Learning to Predict Indoor Illumination from a Single Image

We propose an automatic method to infer high dynamic range illumination from a single, limited field-of-view, low dynamic range photograph of an indoor scene. In contrast to previous work that relies on specialized image capture, user input, and/or simple scene models, we train an end-to-end deep neural network that directly regresses a limited field-of-view photo to HDR illumination, without strong assumptions on scene geometry, material properties, or lighting. We show that this can be accomplished in a three step process: 1) we train a robust lighting classifier to automatically annotate the location of light sources in a large dataset of LDR environment maps, 2) we use these annotations to train a deep neural network that predicts the location of lights in a scene from a single limited field-of-view photo, and 3) we fine-tune this network using a small dataset of HDR environment maps to predict light intensities. This allows us to automatically recover high-quality HDR illumination estimates that significantly outperform previous state-of-the-art methods. Consequently, using our illumination estimates for applications like 3D object insertion, we can achieve results that are photo-realistic, which is validated via a perceptual user study.

Paper

Marc-André Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Christian Gagné, and Jean-François Lalonde
Learning to Predict Indoor Illumination from a Single Image
ACM Transactions on Graphics (SIGGRAPH Asia), 9(4), 2017
[arXiv pre-print] [BibTeX]

Supplementary material

We provide additional results in this supplementary page.

Dataset

We provide the HDR dataset we used to train the network. It is free to use for non commercial purposes. The complete dataset (EXR latlong format) can be downloaded here.

Hand-labeled panoramas

We also provide around 400 hand-labeled panoramas from SUN360 that we use to train our light classifier (see Sec.4 in the paper). The light sources have been labeled, with position and type provided for each one. The labels are provided in the form of JSON data (one file per panorama), with a "type" field that can take the following values:
  1. Spotlight, e.g. strong directed light
  2. Lamp, e.g. area light, usually diffuse
  3. Window, e.g. area light with natural light
  4. Bounces, only for the important ones, usually from sunlight
This dataset can be freely used for non commercial purposes only (see SUN360 licence for more details). It can be downloaded here.

Talk

You can download the slides in PDF and PPTX formats. To go along with the PDF, we also provide a zipped folder containing all the videos used in the presentation (MP4 container, H.264, no sound). Please cite the source if you use these slides or videos in a presentation.

Video

Acknowledgements

The authors gratefully acknowledge the following funding sources:

  • A NSERC Alexander Graham Bell Canada Doctoral Scholarship to Marc-André Gardner
  • A generous donation from Adobe to Jean-Francois Lalonde
  • NVIDIA Corporation with the donation of the Tesla K40 and Titan X GPUs used for this research.
  • FRQ-NT New Researcher Grant 2016‐NC‐189939
  • REPARTI Strategic Network

Ulaval logo
Adobe logo