Neural 3D holography

Author(s):

Choi, Suyeon; Gopakumar, Manu; Peng, Yifan; Kim, Jonghyun & Wetzstein, Gordon

Abstract:

“Holographic near-eye displays promise unprecedented capabilities for virtual and augmented reality (VR/AR) systems. The image quality achieved by current holographic displays, however, is limited by the wave propagation models used to simulate the physical optics. We propose a neural network-parameterized plane-to-multiplane wave propagation model that closes the gap between physics and simulation. Our model is automatically trained using camera feedback and it outperforms related techniques in 2D plane-to-plane settings by a large margin. Moreover, it is the first network-parameterized model to naturally extend to 3D settings, enabling high-quality 3D computer-generated holography using a novel phase regularization strategy of the complex-valued wave field. The efficacy of our approach is demonstrated through extensive experimental evaluation with both VR and optical see-through AR display prototypes.”

Link to Publications Page

Publication: ACM Transactions on Graphics
Issue/Year: ACM Transactions on Graphics, Volume 40; Number 6; Pages 1–12; 2021
DOI: 10.1145/3478513.3480542

Speckle-free holography with partially coherent light sources and camera-in-the-loop calibration

Author(s):

Peng, Yifan; Choi, Suyeon; Kim, Jonghyun & Wetzstein, Gordon

Abstract:

“Computer-generated holography (CGH) holds transformative potential for a wide range of applications, including direct-view, virtual and augmented reality, and automotive display systems. While research on holographic displays has recently made impressive progress, image quality and eye safety of holographic displays are fundamentally limited by the speckle introduced by coherent light sources. Here, we develop an approach to CGH using partially coherent sources. For this purpose, we devise a wave propagation model for partially coherent light that is demonstrated in conjunction with a camera-in-the-loop calibration strategy. We evaluate this algorithm using light-emitting diodes (LEDs) and superluminescent LEDs (SLEDs) and demonstrate improved speckle characteristics of the resulting holograms compared with coherent lasers. SLEDs in particular are demonstrated to be promising light sources for holographic display applications, because of their potential to generate sharp and high-contrast two-dimensional (2D) and 3D images that are bright, eye safe, and almost free of speckle.”

Link to Publications Page

Publication: Science Advances
Issue/Year: Science Advances, Volume 7; Number 46; 2021
DOI: 10.1126/sciadv.abg5040

Hologram computation using the radial point spread function

Author(s):

Yasuki, Daiki; Shimobaba, Tomoyoshi; Makowski, Michal; Suszek, Jaroslaw; Kakue, Takashi & Ito, Tomoyoshi

Abstract:

“Holograms are computed by superimposing point spread functions (PSFs), which represent the distribution of light on the hologram plane. The computational cost and the spatial bandwidth product required to generate holograms are significant; therefore, it is challenging to compute high-resolution holograms at the rates required for videos. Among the possible displays, fixed-eye-position holographic displays, such as holographic head-mounted displays, reduce the spatial bandwidth product by fixing eye positions while satisfying almost all human depth cues. In eye-fixed holograms, by calculating a part distribution of the entire PSF, we observe reconstructed images that maintain the image quality and the depth of focus almost as high as those generated by the entire PSF. In this study, we accelerate the calculation of eye-fixed holograms by engineering the PSFs. We propose cross and radial PSFs, and we determine that, out of the two, the radial PSFs have a better image quality. By combining the look-up table method and the wavefront-recording plane method with radial PSFs, we show that the proposed method can rapidly compute holograms.”

Link to Publications Page

Publication: Applied Optics
Issue/Year: Applied Optics, Volume 60; Number 28; Pages 8829; 2021
DOI: 10.1364/ao.437777

Holographic near-eye display based on complex amplitude modulation with band-limited zone plates

Author(s):

Chen, Yun; Hua, Minjie; Zhang, Tianshun; Zhou, Mingxin; Wu, Jianhong & Zou, Wenlong

Abstract:

“A holographic near-eye display (NED) system based on complex amplitude modulation (CAM) with band-limited zone plates is proposed. The whole system mainly consists of a phase-only spatial light modulator (SLM), an Abbe-Porter filter system, an eyepiece, and an image combiner. The point source method based on band limited zone plates is used to accurately control the bandwidth of the target complex amplitude. The effects of intensity modulation coefficient γ in the frequency-filtering method on the intensity and the quality of reconstructed images are analyzed, which provide a judgment basis for selecting the appropriate value of γ. We also derive the expressions of the field of view (FOV) and exit pupil of the NED system. Since the holographic image is magnified in two steps in this system, the large FOV can be obtained. The optical experimental results show that the proposed system can provide a dynamic holographic three-dimensional (3D) augmented reality (AR) display with a 23.5°horizontal FOV.”

Link to Publications Page

Publication: Optics Express
Issue/Year: Optics Express, Volume 29; Number 14; Pages 22749; 2021
DOI: 10.1364/oe.431032

Vision-correcting Holographic Display: Evaluation of Aberration Correcting Hologram

Author(s):

Kim, Dongyeon; Nam, Seung-Woo; Bang, Kiseung; Lee, Byounghyo; Lee, Seungjae; Jeong, Youngmo; Seo, Jongmo & Lee, Byoungho

Abstract:

“Vision-correcting displays are key to achieving physical and physiological comforts to the users with refractive errors. Among such displays are holographic displays, which can provide a high-resolution vision-adaptive solution with complex wavefront modulation. However, none of the existing hologram rendering techniques have considered the optical properties of the human eye nor evaluated the significance of vision correction. Here, we introduce vision-correcting holographic display and hologram acquisition that integrates user-dependent prescriptions and a physical model of the optics, enabling the correction of on-axis and off-axis aberrations. Experimental and empirical evaluations of the vision-correcting holographic displays show the competence of holographic corrections over the conventional vision correction solutions.”

Link to Publications Page

Publication: Biomedical Optics Express
Issue/Year: Biomedical Optics Express,Vol. 12, Issue 8, pp. 5179-5195, 2021
DOI: 10.1364/boe.433919

Holographic contact lens display that provides focusable images for eyes

Author(s):

Sano, Junpei & Takaki, Yasuhiro

Abstract:

“In this paper, we propose a holographic image generation technique for contact lens displays. The proposed technique employs a phase-only spatial light modulator (SLM), a holographic optical element (HOE) backlight, and a polarizer. The proposed holographic technique can generate 3D images apart from the contact lens displays. Therefore, the eyes can focus on the 3D images while simultaneously observing the real scene through the phase-only SLM and the HOE backlight, which provides see-through capability. A bench-top experimental system was constructed to verify the far-distance image generation capability and see-through function.”

Link to Publications Page

Publication: Optics Express
Issue/Year: Optics Express, Volume 29; Number 7; Pages 10568; 2021
DOI: 10.1364/oe.419604

Foveated near-eye display using computational holography

Author(s):

Cem, Ali; Hedili, M. Kivanc; Ulusoy, Erdem & Urey, Hakan

Abstract:

“Holographic display is the only technology that can offer true 3D with all the required depth cues. Holographic head-worn displays (HWD) can provide continuous depth planes with the correct stereoscopic disparity for a comfortable 3D experience. Existing HWD approaches have small field-of-view (FOV) and small exit pupil size, which are limited by the spatial light modulator (SLM). Conventional holographic HWDs are limited to about 20° × 11° FOV using a 4 K SLM panel and have fixed FOV. We present a new optical architecture that can overcome those limitations and substantially extend the FOV supported by the SLM. Our architecture, which does not contain any moving parts, automatically follows the gaze of the viewer’s pupil. Moreover, it mimics human vision by providing varying resolution across the FOV resulting in better utilization of the available space-bandwidth product of the SLM. We propose a system that can provide 28° × 28° instantaneous FOV within an extended FOV (the field of view that is covered by steering the instantaneous FOV in space) of 60° × 40° using a 4 K SLM, effectively providing a total enhancement of > 3 × in instantaneous FOV area, > 10 × in extended FOV area and the space-bandwidth product. We demonstrated 20° × 20° instantaneous FOV and 40° × 20° extended FOV in the experiments.”

Link to Publications Page

Publication: Scientific Reports
Issue/Year: Scientific Reports, Volume 10; Number 1; 2020
DOI: 10.1038/s41598-020-71986-9

Fast calculation of computer-generated hologram of line-drawn objects without FFT

Author(s):

Nishitsuji, Takashi; Shimobaba, Tomoyoshi; Kakue, Takashi & Ito, Tomoyoshi

Abstract:

“Although holographic display technology is one of the most promising three-dimensional (3D) display technologies for virtual and augmented reality, the enormous computational effort required to produce computer-generated holograms (CGHs) to digitally record and display 3D images presents a significant roadblock to the implementation of this technology. One of the most effective methods to implement fast CGH calculations is a diffraction calculation (e.g., angular spectrum diffraction) based on the fast-Fourier transform (FFT). Unfortunately, the computational complexity increases with increasing CGH resolution, which is what determines the size of a 3D image. Therefore, enormous calculations are still required to display a reasonably sized 3D image, even for a simple 3D image. To address this issue, we propose herein a fast CGH algorithm for 3D objects comprised of line-drawn objects at layers of different depths. An aperture formed from a continuous line at a single depth can be regarded as a series of aligned point sources of light, and the wavefront converges for a sufficiently long line. Thus, a CGH of a line-drawn object can be calculated by synthesizing converged wavefronts along the line. Numerical experiments indicate that, compared with the FFT-based method, the proposed method offers a factor-56 gain in speed for calculating 16-k-resolution CGHs from 3D objects composed of twelve line-drawn objects at different depths.”

Link to Publications Page

Publication: Optics Express
Issue/Year: Optics Express, Volume 28; Number 11; Pages 15907; 2020
DOI: 10.1364/oe.389778

Holographic near-eye display with continuously expanded eyebox using two-dimensional replication and angular spectrum wrapping

Author(s):

Myeong-Ho Choi, Yeon-Gyeong Ju and Jae-Hyeung Park

Abstract:

“Holographic near-eye displays present true three-dimensional images with full
monocular depth cues. In this paper, we propose a technique to expand the eyebox of the
holographic near-eye displays. The base eyebox of the holographic near-eye displays is determined
by the space bandwidth product of a spatial light modulator. The proposed technique replicates
and stitches the base eyebox by the combined use of a holographic optical element and high order
diffractions of the spatial light modulator, achieving horizontally and vertically expanded eyebox.
An angular spectrum wrapping technique is also applied to alleviate image distortions observed
at the boundaries between the replicated base eyeboxes.”

Link to Publications Page

Publication: Optics Express
Issue/Year: Vol. 28, Issue 1, pp. 533-547
DOI: 10.1364/OE.381277

Wirtinger holography for near-eye displays

Author(s):

Chakravarthula, Praneeth; Peng, Yifan; Kollin, Joel; Fuchs, Henry & Heide, Felix

Abstract:

“Near-eye displays using holographic projection are emerging as an exciting display approach for virtual and augmented reality at high-resolution without complex optical setups — shifting optical complexity to computation. While precise phase modulation hardware is becoming available, phase retrieval algorithms are still in their infancy, and holographic display approaches resort to heuristic encoding methods or iterative methods relying on various relaxations.
In this work, we depart from such existing approximations and solve the phase retrieval problem for a hologram of a scene at a single depth at a given time by revisiting complex Wirtinger derivatives, also extending our framework to render 3D volumetric scenes. Using Wirtinger derivatives allows us to pose the phase retrieval problem as a quadratic problem which can be minimized with first-order optimization methods. The proposed Wirtinger Holography is flexible and facilitates the use of different loss functions, including learned perceptual losses parametrized by deep neural networks, as well as stochastic optimization methods. We validate this framework by demonstrating holographic reconstructions with an order of magnitude lower error, both in simulation and on an experimental hardware prototype.”

Link to Publications Page

Publication: ACM Transactions on Graphics (TOG)
Issue/Year: ACM Transactions on Graphics (TOG), Volume 38; Number 6; Pages 213; 2019
DOI: 10.1145/3355089.3356539