Illumination and Geometry Inference Using Graphical Models

Abstract

Image formation is a function of three components: scene geometry, surface reflectance and illumination. Estimation of one or more of these components from an image gives rise to inverse rendering problems, such as shape reconstruction or illumination estimation, which are the two major problems of interest in this thesis. We formulate such problems in a way that attempts to bridge the gap between low-level approaches based on the physical laws governing image formation and higher-level models that examine images in a statistical way. We take advantage of the powerful formalism offered by graphical models, which lead to modular frameworks and offer powerful discrete optimization techniques. We first focus on the problem of illumination estimation from a single image, utilizing the information in cast shadows. We start by describing a method to extract cast shadows from an image. We then present three approaches to illumination estimation from shadows: The first models illumination as a mixture of distributions to robustly estimate illumination. The second associates illumination not with pixel intensities but with the existence of shadow edges. The third approach unifies the previous ideas in a Markov Random Field (MRF) framework. Such a model is robust to coarse or incomplete knowledge of geometry, while it can also incorporate geometric parameters, allowing us to jointly infer three major components of the problem: the cast shadows, illumination and geometry. Geometry inference from the information contained in cast shadows can only be coarse, however. We subsequently focus on the problem of inferring geometry from the shading variations in an image. We take a data-driven approach, constructing a dictionary of geometric primitives. To reconstruct an image, we combine local hypotheses from this dictionary in an MRF model. We demonstrate that this approach can effectively reconstruct 3D shapes from real photographs, while removing several important assumptions of previous approaches.