Photometric Stereo has been explored extensively in laboratory conditions since its inception. Recently, attempts have been made at applying this technique under natural outdoor lighting. Outdoor photometric stereo presents additional challenges as one does not have control over illumination anymore. In this paper, we explore the stability of surface normals reconstructed outdoors. We present a data-driven analysis based on a large database of outdoor HDR environment maps. Given a sequence of object images and corresponding sky maps captured in a single day, we investigate natural factors that impact the uncertainty in the estimated surface normals. Quantitative evidence reveals strong dependencies between expected accuracy and the normal orientation, cloud coverage, and sun elevation. In particular, we show that partially cloudy days yield greater accuracy than sunny days with clear skies; furthermore, high sun elevation—recommended in previous work—is in fact not necessarily optimal when taking more elaborate illumination models into account.