DeepSTORM3D: dense three dimensional localization microscopy and point spread function design by deep learning

PLUTO / PLUTO-2 Spatial Light Modulators
Deep Learning / Neuronal Network Microscopy
Published on:
Authors: Nehme, Elias; Freedman, Daniel; Gordon, Racheli; Ferdman, Boris; Weiss, Lucien E.; Alalouf, Onit; Orange, Reut; Michaeli, Tomer & Shechtman, Yoav
Abstract:

“Localization microscopy is an imaging technique in which the positions of individual nanoscale point emitters (e.g. fluorescent molecules) are determined at high precision from their images. This is the key ingredient in single/multiple-particle-tracking and several super-resolution microscopy approaches. Localization in three-dimensions (3D) can be performed by modifying the image that a point-source creates on the camera, namely, the point-spread function (PSF). The PSF is engineered using additional optical elements to vary distinctively with the depth of the point-source. However, localizing multiple adjacent emitters in 3D poses a significant algorithmic challenge, due to the lateral overlap of their PSFs. Here, we train a neural network to receive an image containing densely overlapping PSFs of multiple emitters over a large axial range and output a list of their 3D positions. Furthermore, we then use the network to design the optimal PSF for the multi-emitter case. We demonstrate our approach numerically as well as experimentally by 3D STORM imaging of mitochondria, and volumetric imaging of dozens of fluorescently-labeled telomeres occupying a mammalian nucleus in a single snapshot.”

Open Access

Publication: Nature Methods 17
Issue/Year: Nature Methods 17 (2020) 734-740, Volume 17; Number 7; Pages 734–740; 2019
DOI: 10.1038/s41592-020-0853-5
Link: https://doi.org/10.1038/s41592-020-0853-5

Related Papers

PLUTO / PLUTO-2 Spatial Light Modulators
Authors:Marshall B. Lindsay, Scott D. Kovaleski, Andy G. Varner, Charlie T. Veal, Derek T. Anderson, Stanton R. Price, and Steven R. Price

End-to-end physics-informed deep neural network optimization of sub-Nyquist lenses

Applications: Deep Learning / Neuronal Network
PLUTO / PLUTO-2 Spatial Light Modulators
Authors:Jianan Feng, Hang Chen, Dahai Yang, Junbo Hao, Jie Lin, and Peng Jin

Multi-wavelength diffractive neural network with the weighting method

Applications: Deep Learning / Neuronal Network
PLUTO / PLUTO-2 Spatial Light Modulators
Authors:Zhang, Yantong, Huiling Huang, Feibin Wu, Jun Han, Yi Yang, and Ruyi Li

Imaging through a Scattering Medium under Different Intensities of Ambient Light Interference

Applications: Deep Learning / Neuronal Network,Imaging/ Image Processing,Turbid-/ Opaque Media /Multi Scattering
PLUTO / PLUTO-2 Spatial Light Modulators
Authors:Jiangcheng Zhao, Chuanbiao Bai, Zhiguo Zhang and Qingchuan Zhang

Deep learning-based method for analyzing the optically trapped sperm rotation

Applications: Deep Learning / Neuronal Network,Optical Trapping /-Tweezers
PLUTO / PLUTO-2 Spatial Light Modulators
Authors:Ruibo Shang, Mikaela A. O’Brien, Fei Wang, Guohai Situ & Geoffrey P. Luke

Approximating the uncertainty of deep learning reconstruction predictions in single-pixel imaging

Applications: Deep Learning / Neuronal Network
GAEA Spatial Light Modulators
Authors:Ninghe Liu, Zhengzhong Huang, Zehao He, and Liangcai Cao

DGE-CNN: 2D-to-3D holographic display based on a depth gradient extracting module and ZCNN network

Applications: AR/VR/MR / Holographic Display,Deep Learning / Neuronal Network
PLUTO / PLUTO-2 Spatial Light Modulators
Authors:Zhu, Jialong / Zou, Fucheng / Wang, Le / Lu, Xiaodong / Zhao, Shengmei

Multiplexing Perfect Optical Vortex for Holographic Data Storage

Applications: Deep Learning / Neuronal Network,Digital-/ Computer Holography/ CGH,Higher Order Modes / Optical Vortex / OAM
LC 2012 Spatial Light Modulators
Authors:Weng, Xiaojing / Feng, Ji / Perry, Altai / Vuong, Luat T.

Non-Line-of-Sight Full-Stokes Polarimetric Imaging with Solution-Processed Metagratings and Shallow Neural Networks

Applications: Deep Learning / Neuronal Network,Imaging/ Image Processing,Polarization Generation
PLUTO / PLUTO-2 Spatial Light Modulators
Authors:Yun, Xue / Liang, Yansheng / He, Minru / Zhao, Tianyu / Wang, Shaowei / Lei, Ming / Lei, Ming

Optical trapping of gold microparticles using linearly polarized, centrally obstructed Gaussian beams

Applications: Microscopy,Optical Trapping /-Tweezers
PLUTO / PLUTO-2 Spatial Light Modulators
Authors:Zheng, Huadong / Peng, Junchang / Wang, Zhen / xinghua Shui / yu yingjie / Xia, Xinxing

Diffraction model-driven neural network trained using hybrid domain loss for real-time and high-quality computer-generated holography

Applications: AR/VR/MR / Holographic Display,Deep Learning / Neuronal Network,Digital-/ Computer Holography/ CGH
LC 2012 Spatial Light Modulators
Authors:Tsukada, Takumi / Watanabe, Wataru

Central wavelength estimation in spectral imaging behind a diffuser via deep learning

Applications: Deep Learning / Neuronal Network,Imaging/ Image Processing,Misc.: Speckle / Characterization / Metrology
PLUTO / PLUTO-2 Spatial Light Modulators
Authors:Xu, Ning; Bohndiek, Sarah E.; Li, Zexing; Zhang, Cilong & Tan, Qiaofeng

Mechanical-scan-free and multi-color super-resolutionimaging with diffractive spot array illumination

Applications: Microscopy
PLUTO / PLUTO-2 Spatial Light Modulators
Authors:Wang, Jian; Zhong, Guangchao; Wu, Daixuan; Huang, Sitong; Luo, Zhi-Chao & Shen, Yuecheng

Multimode fiber-based greyscale image projector enabled by neural networks with high generalization ability

Applications: Deep Learning / Neuronal Network
PLUTO / PLUTO-2 Spatial Light Modulators
Authors:Luo, Tianpeng; Yuan, Jing; Chang, Jin; Dai, Yanfeng; Gong, Hui; Luo, Qingming & Yang, Xiaoquan

Resolution and uniformity improvement of parallel confocal microscopy based on microlens arrays and a spatial light modulator

Applications: Microscopy
GAEA Spatial Light Modulators
Authors:Dong, Zhenxing; Xu, Chao; Ling, Yuye; Li, Yan & Su, Yikai

Fourier-inspired neural module for real-time and high-fidelity computer-generated holography

Applications: Deep Learning / Neuronal Network,Digital-/ Computer Holography/ CGH
PLUTO / PLUTO-2 Spatial Light Modulators
Authors:Jia, Qi; Zhang, Yanxia; Shi, Bojian; Li, Hang; Li, Xiaoxin; Feng, Rui; Sun, Fangkui; Cao, Yongyin; Wang, Jian; Qiu, Cheng-Wei & Ding, Weiqiang

Simultaneously sorting vector vortex beams of 120 modes

Applications: Deep Learning / Neuronal Network,Higher Order Modes / Optical Vortex / OAM,Optical Communication
PLUTO / PLUTO-2 Spatial Light Modulators
Authors:Navarro, Gilberto

Modulation of Non-diffracting Hermite Gaussian Beams and Nonlinear Optical Microscopy for Nanoscale Sulfur Imaging

Applications: Beam Shaping / Beam Steering,Higher Order Modes / Optical Vortex / OAM,Microscopy
PLUTO / PLUTO-2 Spatial Light Modulators
Authors:Kürekci, Şahin

Optical classification and reconstruction through multimode fibers

Applications: Deep Learning / Neuronal Network,Optical Communication,Turbid-/ Opaque Media /Multi Scattering
PLUTO / PLUTO-2 Spatial Light Modulators
Authors:Guo, Yong; Wang, Luwei; Zhu, Yinru; Gao, Xinwei; Weng, Xiaoyu; Liu, Jinyuan; Yan, Wei & Qu, Junle

Improving longitudinal resolution of Airy beams two-photon volume imaging with fluorescence lifetime imaging

Applications: Bessel-/ Airy Beam Generation,Microscopy
PLUTO / PLUTO-2 Spatial Light Modulators
Authors:Shui, Xinghua; Zheng, Huadong; Xia, Xinxing; Yang, Furong; Wang, Weisen & Yu, Yingjie

Diffraction model-informed neural network for unsupervised layer-based computer-generated holography

Applications: Deep Learning / Neuronal Network,Digital-/ Computer Holography/ CGH
PLUTO / PLUTO-2 Spatial Light Modulators
Authors:Madan, Ivan; Leccese, Veronica; Mazur, Adam; Barantani, Francesco; LaGrange, Thomas; Sapozhnik, Alexey; Tengdin, Phoebe M.; Gargiulo, Simone; Rotunno, Enzo; Olaya, Jean-Christophe; Kaminer, Ido; Grillo, Vincenzo; de Abajo, F. Javier Garcia; Carbone, Fabrizio & Vanacore, Giovanni Maria

Ultrafast Transverse Modulation of Free Electrons by Interaction with Shaped Optical Fields

Applications: Beam Shaping / Beam Steering,Higher Order Modes / Optical Vortex / OAM,Microscopy
PLUTO / PLUTO-2 Spatial Light Modulators
Authors:Dou, Jiazhen; Dong, Chen; Dai, Siqing; Mi, Jingyu; Luo, Xiangyuan; Di, Jianglei; Zhang, Jiwei & Zhao, Jianlin

High-resolution surface plasmon resonance holographic microscopy based on symmetrical excitation

Applications: Digital-/ Computer Holography/ CGH,Microscopy
GAEA Spatial Light Modulators
Authors:Wu, Peng; Zhang, Dejie; Yuan, Jing; Zeng, Shaoqun; Gong, Hui; Luo, Qingming & Yang, Xiaoquan

Large depth-of-field fluorescence microscopy based on deep learning supported by Fresnel incoherent correlation holography

Applications: Digital-/ Computer Holography/ CGH,Microscopy,Misc.: Speckle / Characterization / Metrology
LC-R 720 Spatial Light Modulators
Authors:Liu, Xuan; Wan, Zhaoxiong; Zhang, Yuanwei & Liu, Yuwei

Optically computed phase microscopy for quantitative dynamic imaging of label-free cells and nanoparticles

Applications: Imaging/ Image Processing,Microscopy