Towards real-time photorealistic 3D holography with deep neural networks

PLUTO / PLUTO-2 Spatial Light Modulators
AR/VR/MR / Holographic Display Digital-/ Computer Holography/ CGH Imaging/ Image Processing
Published on:
Authors: Shi, Liang; Li, Beichen; Kim, Changil; Kellnhofer, Petr & Matusik, Wojciech
Abstract:
“The ability to present three-dimensional (3D) scenes with continuous depth sensation has a profound impact on virtual and augmented reality, human–computer interaction, education and training. Computer-generated holography (CGH) enables high-spatio-angular-resolution 3D projection via numerical simulation of diffraction and interference1. Yet, existing physically based methods fail to produce holograms with both per-pixel focal control and accurate occlusion. The computationally taxing Fresnel diffraction simulation further places an explicit trade-off between image quality and runtime, making dynamic holography impractical. Here we demonstrate a deep-learning-based CGH pipeline capable of synthesizing a photorealistic colour 3D hologram from a single RGB-depth image in real time. Our convolutional neural network (CNN) is extremely memory efficient (below 620 kilobytes) and runs at 60 hertz for a resolution of 1,920 × 1,080 pixels on a single consumer-grade graphics processing unit. Leveraging low-power on-device artificial intelligence acceleration chips, our CNN also runs interactively on mobile (iPhone 11 Pro at 1.1 hertz) and edge (Google Edge TPU at 2.0 hertz) devices, promising real-time performance in future-generation virtual and augmented-reality mobile headsets. We enable this pipeline by introducing a large-scale CGH dataset (MIT-CGH-4K) with 4,000 pairs of RGB-depth images and corresponding 3D holograms. Our CNN is trained with differentiable wave-based loss functions and physically approximates Fresnel diffraction. With an anti-aliasing phase-only encoding method, we experimentally demonstrate speckle-free, natural-looking, high-resolution 3D holograms. Our learning-based approach and the Fresnel hologram dataset will help to unlock the full potential of holography and enable applications in metasurface design, optical and acoustic tweezer-based microscopic manipulation, holographic microscopy and single-exposure volumetric 3D printing.”

Restricted Access

Publication: Nature
Issue/Year: Nature, Volume 591; Number 7849; Pages 234–239; 2021
DOI: 10.1038/s41586-020-03152-0
Link: https://doi.org/10.1038/s41586-020-03152-0

Related Papers

GAEA / GAEA-2 Spatial Light Modulators
Authors:Ekaterina A. Savchenkova, Andrey S. Ovchinnikov, Vladislav G. Rodin, Rostislav S. Starikov, Nikolay N. Evtikhiev, Pavel A. Cheremkhin

Adaptive non-iterative histogram-based hologram quantization

Applications: Deep Learning / Neuronal Network,Digital-/ Computer Holography/ CGH
PLUTO / PLUTO-2 Spatial Light Modulators
Authors:Jixiang Wang, Xin Xu, Hong Ye, Xin Zhang, and Guohua Shi

Efficient generation of high-contrast Bessel light sheets: a flexible method for improved imaging

Applications: Beam Shaping / Beam Steering,Bessel-/ Airy Beam Generation,Imaging/ Image Processing,Microscopy
LC 2012 PLUTO / PLUTO-2 Spatial Light Modulators
Authors:Jiaosheng Li, Yifei Chen, Tianyun Liu, Bo Wu & Qinnan Zhang

Single-pixel Fresnel incoherent correlation holography compressed imaging using a Trumpet network

Applications: Deep Learning / Neuronal Network,Digital-/ Computer Holography/ CGH,Imaging/ Image Processing
LETO / LETO-3 Spatial Light Modulators
Authors: Lukas T. Hiller, Jörg Wallaschek

Lensless projection of automotive light distributions using an advanced printer for volume holographic optical elements

Applications: Adaptive Optics / Wavefront Control,AR/VR/MR / Holographic Display
PLUTO / PLUTO-2 Spatial Light Modulators
Authors: I Komang Januariyasa, Francesco Reda, Fabio Borbone, Marcella Salvatore and Stefano L. Oscurato

Molding three-dimensional azopolymer microstructures with holographically structured light

Applications: Digital-/ Computer Holography/ CGH,Materials Processing / Optical Fabrication
ERIS Spatial Light Modulators
Authors:Sumit Kumar Singh, Kenji Kinashi, Naoto Tsutsumi, Wataru Sakai, and Boaz Jessie Jackin

High-density polarization multiplexed holograms using azo-carbazole films for diverse applications

Applications: Digital-/ Computer Holography/ CGH,Polarization Generation
PLUTO / PLUTO-2 Spatial Light Modulators
Authors:Kyriakos Skarsoulis, Konstantinos Makris, Christophe Moser, and Demetri Psaltis

Ptychographic imaging with a fiber endoscope via wavelength scanning

Applications: Deep Learning / Neuronal Network,Imaging/ Image Processing
LETO / LETO-3 Spatial Light Modulators
Authors:Manu Gopakumar, Gun-Yeal Lee, Suyeon Choi, Brian Chao, Yifan Peng, Jonghyun Kim & Gordon Wetzstein

Full-colour 3D holographic augmented-reality displays with metasurface waveguides

Applications: AR/VR/MR / Holographic Display,Deep Learning / Neuronal Network
PLUTO / PLUTO-2 Spatial Light Modulators
Authors:Ethan Tseng, Grace Kuo, Seung-Hwan Baek, Nathan Matsuda, Andrew Maimone, Florian Schiffers, Praneeth Chakravarthula, Qiang Fu, Wolfgang Heidrich, Douglas Lanman & Felix Heide

Neural étendue expander for ultra-wide-angle high-fidelity holographic display

Applications: AR/VR/MR / Holographic Display,Imaging/ Image Processing
PLUTO / PLUTO-2 Spatial Light Modulators
Authors:Siyuan Wang, Zonglin Guo, Yu Zhao, Yanbo Pei, Hongyan Shi, Jingbo Zhang, Jian Wang

Deep field-of-view and passive tomography based on Airy light field

Applications: Bessel-/ Airy Beam Generation,Imaging/ Image Processing
GAEA / GAEA-2 Spatial Light Modulators
Authors:Xiaofei Li, Xin Liu, Quanying Wu, Jun Zeng, Yangjian Cai, Sergey A. Ponomarenko, Chunhao Liang

Prime number factorization with light beams carrying orbital angular momentum

Applications: Deep Learning / Neuronal Network,Digital-/ Computer Holography/ CGH,Higher Order Modes / Optical Vortex / OAM
GAEA / GAEA-2 Spatial Light Modulators
Authors:Rafal Kukolowicz, Tomasz Kozacki, Maksymilian Chlipala, Moncy Sajeev Idicula, Juan Martinez-Carranza, Weronika Finke, and Izabela Gerej

Digital holographic content manipulation for wide-angle holographic near-eye displays

Applications: AR/VR/MR / Holographic Display
LETO / LETO-3 Spatial Light Modulators
Authors:Chenliang Chang, Xian Ding, Di Wang, Zhizhou Ren, Bo Dai, Qi Wang, Songlin Zhuang, Dawei Zhang

Split Lohmann computer holography: fast generation of 3D hologram in single-step diffraction calculation

Applications: Digital-/ Computer Holography/ CGH
PLUTO / PLUTO-2 Spatial Light Modulators
Authors:Rui Xiong, Xiangchao Zhang, Xinyang Ma, Leheng Li, Zhanghao Ni, and Xiangqian Jiang

Adaptive coded phase mask design and high-quality image reconstruction for interference-less coded aperture correlation holography

Applications: Deep Learning / Neuronal Network,Digital-/ Computer Holography/ CGH,Imaging/ Image Processing
PLUTO / PLUTO-2 Spatial Light Modulators
Authors:Ming Liu, Bing Dong

Principle and application of holographic modal wavefront sensor for extended objects and general aberration modes

Applications: Adaptive Optics / Wavefront Control,Digital-/ Computer Holography/ CGH
LETO / LETO-3 PLUTO / PLUTO-2 Spatial Light Modulators
Authors:Myeong-Ho Choi, Kwang-Soo Shin, Jae-Hyeung Park

Foveated holographic near-eye display with geometric phase lens and camera feedback

Applications: AR/VR/MR / Holographic Display
PLUTO / PLUTO-2 Spatial Light Modulators
Authors:Patrick Cameron, Baptiste Courme, Daniele Faccio, Hugo Defienne

Tutorial: Shaping the Spatial Correlations of Entangled Photon Pairs

Applications: Beam Shaping / Beam Steering,Imaging/ Image Processing,Optical Computing / Quantum Optics
PLUTO / PLUTO-2 Spatial Light Modulators
Authors:Claudio Conci, Laura Sironi, Emanuela Jacchetti, Davide Panzeri, Donato Inverso, Rebeca Martínez Vázquez, Roberto Osellame, Maddalena Collini, Giulio Cerullo, Giuseppe Chirico, Manuela Teresa Raimondi

In vivo label-free tissue histology through a microstructured imaging window

Applications: Beam Shaping / Beam Steering,Digital-/ Computer Holography/ CGH
PLUTO / PLUTO-2 Spatial Light Modulators
Authors:Jiuru He, Pengwei Zhang, Jianpo Su, Junqiao Wang, Yongzhi Tian, Yongsheng Hu, and Fengying Ma

Edge enhancement in three-dimensional vortex imaging based on FINCH by Bessel-like spiral phase modulation

Applications: Imaging/ Image Processing
LETO / LETO-3 Spatial Light Modulators
Authors:Wenxue Zhang, Yuhong Wan, Tianlong Man, Hongqiang Zhou

Computational incoherent edge enhancement imaging based on self-interference digital holography

Applications: Imaging/ Image Processing
PLUTO / PLUTO-2 Spatial Light Modulators
Authors:Dejin Zhang, Wei Sheng, Yaoyao Shi, and Youwen Liu

Imaging objects hidden inside the strongly scattering media based on bidirectional ghost imaging

Applications: Imaging/ Image Processing,Turbid-/ Opaque Media /Multi Scattering
PLUTO / PLUTO-2 Spatial Light Modulators
Authors: Koray Kavaklı , Liang Shi , Hakan Urey , Wojciech Matusik , Kaan Akşit

Multi-color Holograms Improve Brightness in Holographic Displays

Applications: AR/VR/MR / Holographic Display,Digital-/ Computer Holography/ CGH
PLUTO / PLUTO-2 Spatial Light Modulators
Authors: Eric Markley , Nathan Matsuda , Florian Schiffers , Oliver Cossairt , Grace Kuo

Simultaneous Color Computer Generated Holography

Applications: AR/VR/MR / Holographic Display,Digital-/ Computer Holography/ CGH
PLUTO / PLUTO-2 Spatial Light Modulators
Authors: Grace Kuo , Florian Schiffers , Douglas Lanman , Oliver Cossairt , Nathan Matsuda

Multisource Holography

Applications: AR/VR/MR / Holographic Display,Deep Learning / Neuronal Network,Digital-/ Computer Holography/ CGH