Understanding and Recreating Visual Appearance Under Natural Illumination

Overview of the main results presented in this thesis.



The appearance of an outdoor scene is determined to a great extent by the prevailing illumination conditions. However, most practical computer vision applications treat illumination more as a nuisance rather than a source of signal. In this dissertation, we suggest that we should instead embrace illumination, even in the challenging, uncontrolled world of consumer photographs.

Illumination estimate
Illumination estimate

Our first main contribution is an understanding of natural illumination from images. This is, in general, a hard problem given the wide appearance variation in scenes. Fortunately, natural illumination, while complex, is far from being completely arbitrary. It has a structure that is well understood in atmospheric optics, but which has hardly been exploited in vision and graphics. We introduce methods for automatically estimating the illumination conditions from two types of uncontrolled outdoor image datasets: webcams and single images. The variation in sun position and sky appearance over time can be exploited to obtain viewing and illumination geometry in webcam sequences. For single images, the sky is combined in a probabilistic way with other scene features such as cast shadows and shading on vertical surfaces and convex objects, as well as with illumination priors from large image collections.

Lighting-consistent image composite
Image composite

Our second main contribution is to exploit the knowledge of illumination in order to synthesize novel, realistic visual content. Instead of creating appearance using the traditional computer graphics pipeline, we propose to borrow the appearance of the world that is contained in existing photo collections and webcam datasets. We also demonstrate realistic 3-D object insertion by creating plausible high-dynamic range environment maps. This can be done in image sequences, and even in single images, completely automatically. Addressing such questions has implications in a broad range of applications including intelligent transportation, surveillance, human-robot interaction, and digital entertainment.


Thesis thumbnail Jean-François Lalonde. Understanding and Recreating Visual Appearance Under Natural Illumination, Ph.D. thesis, Carnegie Mellon University, January 2011. [PDF] [BibTeX]

Order the book online!

Lulu thumbnail You can order a hardcover copy of this thesis via lulu.com.


Download the slides from my defense in the following formats:


Download data used in various parts of this thesis. Please cite the relevant papers if you use the data in a publication.


Download code that implements several of the ideas presented in this thesis. Please cite the relevant papers if you use the code in a publication.


This research is supported by:

Copyright notice

Valid XHTML 1.0 Transitional