Towards real-time photorealistic 3D holography with deep neural networks

Author(s):

Shi, Liang; Li, Beichen; Kim, Changil; Kellnhofer, Petr & Matusik, Wojciech

Abstract:

“The ability to present three-dimensional (3D) scenes with continuous depth sensation has a profound impact on virtual and augmented reality, human–computer interaction, education and training. Computer-generated holography (CGH) enables high-spatio-angular-resolution 3D projection via numerical simulation of diffraction and interference1. Yet, existing physically based methods fail to produce holograms with both per-pixel focal control and accurate occlusion. The computationally taxing Fresnel diffraction simulation further places an explicit trade-off between image quality and runtime, making dynamic holography impractical. Here we demonstrate a deep-learning-based CGH pipeline capable of synthesizing a photorealistic colour 3D hologram from a single RGB-depth image in real time. Our convolutional neural network (CNN) is extremely memory efficient (below 620 kilobytes) and runs at 60 hertz for a resolution of 1,920 × 1,080 pixels on a single consumer-grade graphics processing unit. Leveraging low-power on-device artificial intelligence acceleration chips, our CNN also runs interactively on mobile (iPhone 11 Pro at 1.1 hertz) and edge (Google Edge TPU at 2.0 hertz) devices, promising real-time performance in future-generation virtual and augmented-reality mobile headsets. We enable this pipeline by introducing a large-scale CGH dataset (MIT-CGH-4K) with 4,000 pairs of RGB-depth images and corresponding 3D holograms. Our CNN is trained with differentiable wave-based loss functions and physically approximates Fresnel diffraction. With an anti-aliasing phase-only encoding method, we experimentally demonstrate speckle-free, natural-looking, high-resolution 3D holograms. Our learning-based approach and the Fresnel hologram dataset will help to unlock the full potential of holography and enable applications in metasurface design, optical and acoustic tweezer-based microscopic manipulation, holographic microscopy and single-exposure volumetric 3D printing.”

Link to Publications Page

Publication: Nature
Issue/Year: Nature, Volume 591; Number 7849; Pages 234–239; 2021
DOI: 10.1038/s41586-020-03152-0

A generalized iterative projection framework for pixel-super-resolved holographic imaging

Author(s):

Gao, Yunhui & Cao, Liangcai

Abstract:

“Lensless holographic imaging is challenged by the twin-image artifact due to the missing phase and the aliasing effect due to the undersampled measurement. Therefore, phase retrieval and pixel super-resolution (PSR) techniques serve as the essential ingredients for high-fidelity holographic imaging. In this work, we combine the two in a unified framework by formulating the PSR phase retrieval as a non-convex feasibility problem. An adaptive smoothing strategy for escaping local minima is introduced. Numerical and experimental results are presented and discussed. The proposed framework can be generalized to various physical settings, and is compatible with the state-of-the-art iterative projection algorithms.”

Link to Publications Page

Publication:
Issue/Year: , 2021
DOI: 10.1117/12.2577198

3D reconstruction of weakly scattering objects from 2D intensity-only measurements using the Wolf transform

Author(s):

Ayoub, Ahmed B.; Lim, Joowon; Antoine, Elizabeth E. & Psaltis, Demetri

Abstract:

“A new approach to optical diffraction tomography (ODT) based on intensity measurements is presented. By applying the Wolf transform directly to intensity measurements, we observed unexpected behavior in the 3D reconstruction of the sample. Such a reconstruction does not explicitly represent a quantitative measure of the refractive index of the sample; however, it contains interesting qualitative information. This 3D reconstruction exhibits edge enhancement and contrast enhancement for nanostructures compared with the conventional 3D refractive index reconstruction and thus could be used to localize nanoparticles such as lipids inside a biological sample.”

Link to Publications Page

Publication: Optics Express
Issue/Year: Optics Express, Volume 29; Number 3; Pages 3976; 2021
DOI: 10.1364/oe.414543

Depth-of-field engineering in coded aperture imaging

Author(s):

Rai, Mani Ratnam & Rosen, Joseph

Abstract:

“Extending the depth-of-field (DOF) of an optical imaging system without effecting the other imaging properties has been an important topic of research for a long time. In this work, we propose a new general technique of engineering the DOF of an imaging system beyond just a simple extension of the DOF. Engineering the DOF means in this study that the inherent DOF can be extended to one, or to several, separated different intervals of DOF, with controlled start and end points. Practically, because of the DOF engineering, entire objects in certain separated different input subvolumes are imaged with the same sharpness as if these objects are all in focus. Furthermore, the images from different subvolumes can be laterally shifted, each subvolume in a different shift, relative to their positions in the object space. By doing so, mutual hiding of images can be avoided. The proposed technique is introduced into a system of coded aperture imaging. In other words, the light from the object space is modulated by a coded aperture and recorded into the computer in which the desired image is reconstructed from the recorded pattern. The DOF engineering is done by designing the coded aperture composed of three diffractive elements. One element is a quadratic phase function dictating the start point of the in-focus axial interval and the second element is a quartic phase function which dictates the end point of this interval. Quasi-random coded phase mask is the third element, which enables the digital reconstruction. Multiplexing several sets of diffractive elements, each with different set of phase coefficients, can yield various axial reconstruction curves. The entire diffractive elements are displayed on a spatial light modulator such that real-time DOF engineering is enabled according to the user needs in the course of the observation. Experimental verifications of the proposed system with several examples of DOF engineering are presented, where the entire imaging of the observed scene is done by single camera shot.”

Link to Publications Page

Publication: Optics Express
Issue/Year: Optics Express, Volume 29; Number 2; Pages 1634; 2021
DOI: 10.1364/oe.412744

Partial aperture imaging system based on sparse point spread holograms and nonlinear cross-correlations

Author(s):

Bulbul, Angika & Rosen, Joseph

Abstract:

“Partial aperture imaging system (PAIS) is a recently developed concept in which the traditional disc-shaped aperture is replaced by an aperture with a much smaller area and yet its imaging capabilities are comparable to the full aperture systems. Recently PAIS was demonstrated as an indirect incoherent digital three-dimensional imaging technique. Later it was successfully implemented in the study of the synthetic marginal aperture with revolving telescopes (SMART) to provide superresolution with subaperture area that was less than one percent of the area of the full synthetic disc-shaped aperture. In the study of SMART, the concept of PAIS was tested by placing eight coded phase reflectors along the boundary of the full synthetic aperture. In the current study, various improvements of PAIS are tested and its performance is compared with the other equivalent systems. Among the structural changes, we test ring-shaped eight coded phase subapertures with the same area as of the previous circular subapertures, distributed along the boundary of the full disc-shaped aperture. Another change in the current system is the use of coded phase mask with a point response of a sparse dot pattern. The third change is in the reconstruction process in which a nonlinear correlation with optimal parameters is implemented. With the improved image quality, the modified-PAIS can save weight and cost of imaging devices in general and of space telescopes in particular. Experimental results with reflective objects show that the concept of coded aperture extends the limits of classical imaging”

Link to Publications Page

Publication: Scientific Reports
Issue/Year: Scientific Reports, Volume 10; Number 1; 2020
DOI: 10.1038/s41598-020-77912-3

Maskless lithography for versatile and low cost fabrication of polymer based micro optical structures

Author(s):

Muhammad Shaukat Khan, Roland lachmayer, and Bernhard Roth

Abstract:

“For applications in optical communication, sensing or information projection in automotive lighting, polymer based optical devices are of keen interest. Optical structures such as waveguides and gratings are basic blocks for these devices. We report on a simple, versatile, and yet low-cost fabrication method suited for both binary and multilevel diffractive microstructures as well as multimode optical waveguides in polymers. The fabrication of the diffractive structures, i.e. gratings, with two and multiple levels, is achieved by using a maskless optical lithography system employing a spatial light modulator. With the same system, waveguide cladding structures are realized by stitching of multiple single exposure patterns. For replication of these structures on polymer, e.g. polymethyl methacrylate (PMMA), a lab-made hot embossing machine is used. We then employ UV curable material and doctor blading to realize the waveguide cores. The created diffractive and waveguide structures are characterized in terms of diffraction efficiency and optical propagation loss, respectively, showing good optical quality and performance. With our fabrication system we have demonstrated a diffraction efficiency of 71% for multilevel grating structure and a propagation loss for stitched waveguides of 2.07 dB/cm at a wavelength of 638 nm. These basic elements will be employed to realize entire optical measurement systems for applications in sensing and integrated photonics in the next step.”

Link to Publications Page

Publication: OSA Continuum
Issue/Year: OSA Continuum, Volume 3; Number 10; Pages 2808; 2020
DOI: 10.1364/osac.400056

High-resolution imaging system with an annular aperture of coded phase masks for endoscopic applications

Author(s):

Nitin Dubey, Joseph Rosen, and Israel Gannot

Abstract:

“Partial aperture imaging is a combination of two different techniques; coded aperture imaging and imaging through an aperture that is only a part of the complete disk, commonly used as the aperture of most imaging systems. In the present study, the partial aperture is a ring where the imaging through this aperture resolves small details of the observed scene similarly to the full disk aperture with the same diameter. However, unlike the full aperture, the annular aperture enables using the inner area of the ring for other applications. In this study, we consider the implementation of this special aperture in medical imaging instruments, such as endoscopes, for imaging internal cavities in general and of the human body in particular. By using this annular aperture, it is possible to transfer through the internal open circle of the ring other elements such as surgical tools, fibers and illumination devices. In the proposed configuration, light originated from a source point passes through an annular coded aperture and creates a sparse, randomly distributed, intensity dot pattern on the camera plane. A combination of the dot patterns, each one recorded only once, is used as the point spread hologram of the imaging system. The image is reconstructed digitally by cross correlation between the object intensity response and the point spread hologram.”

Link to Publications Page

Publication: Optics Express
Issue/Year: Vol. 28, Issue 10, pp. 15122-15137
DOI: 10.1364/OE.391713

Fast calculation of computer-generated hologram of line-drawn objects without FFT

Author(s):

Nishitsuji, Takashi; Shimobaba, Tomoyoshi; Kakue, Takashi & Ito, Tomoyoshi

Abstract:

“Although holographic display technology is one of the most promising three-dimensional (3D) display technologies for virtual and augmented reality, the enormous computational effort required to produce computer-generated holograms (CGHs) to digitally record and display 3D images presents a significant roadblock to the implementation of this technology. One of the most effective methods to implement fast CGH calculations is a diffraction calculation (e.g., angular spectrum diffraction) based on the fast-Fourier transform (FFT). Unfortunately, the computational complexity increases with increasing CGH resolution, which is what determines the size of a 3D image. Therefore, enormous calculations are still required to display a reasonably sized 3D image, even for a simple 3D image. To address this issue, we propose herein a fast CGH algorithm for 3D objects comprised of line-drawn objects at layers of different depths. An aperture formed from a continuous line at a single depth can be regarded as a series of aligned point sources of light, and the wavefront converges for a sufficiently long line. Thus, a CGH of a line-drawn object can be calculated by synthesizing converged wavefronts along the line. Numerical experiments indicate that, compared with the FFT-based method, the proposed method offers a factor-56 gain in speed for calculating 16-k-resolution CGHs from 3D objects composed of twelve line-drawn objects at different depths.”

Link to Publications Page

Publication: Optics Express
Issue/Year: Optics Express, Volume 28; Number 11; Pages 15907; 2020
DOI: 10.1364/oe.389778

Non-iterative phase hologram generation with optimized phase modulation

Author(s):

Lizhi Chen, Hao Zhang, Liangcai Cao and Guofan Jin

Abstract:

“A non-iterative algorithm is proposed to generate phase holograms with optimized phase modulation. A quadratic initial phase with continuous distributed spectrum is utilized to iteratively optimize the phase modulation in the reconstruction plane, which can be used as an optimized phase distribution for arbitrary target images. The phase hologram can be calculated directly according to the modulated wave field distribution in the reconstruction plane. Fast generation of the phase holograms can be achieved by this non-iterative implementation, and the avoidance of the random phase modulation helps to suppress the speckle noise. Numerical and
optical experiments have demonstrated that the proposed method can efficiently generate phase holograms with quality reconstructions.”

Link to Publications Page

Publication: Optics Express
Issue/Year: Vol. 28, Issue 8, pp. 11380-11392
DOI: 10.1364/OE.391518

Three-Dimensional Holographic Reconstruction of Brain Tissue Based on Convolution Propagation

Author(s):

Rania M. Abdelazeem and Doaa Youssef and Jala El-Azab and Salah Hassab-Elnaby and Mostafa Agour

Abstract:

” In this study, a dynamic holographic projection system for brain tissue and its anatomical structures extracted from Magnetic Resonance (MR) plane slice is reported. Computer holograms are calculated using a modied Gerchberg-Saxton (GS) iterative algorithm where the projection is based on the plane wave decomposition. First, brain anatomy includes white matter (WM), grey matter (GM) and brain tissue are extracted. Then, phase holograms using the proposed method are generated. Finally, single phase hologram for the whole brain anatomy is generated and is optically reconstructed by a phase-only spatial light modulator (SLM) at dierent depths. The obtained results revealed that the three-dimensional holographic projection of MR brain tissue can aid to provide better interpretation of brain anatomical
structure to achieve better diagnostic results.”

Link to Publications Page

Publication: Journal of Physics: Conference Series
Issue/Year: Vol. 1472
DOI: 10.1088/1742-6596/1472/1/012008