Eyes for Relighting

Extracting environment maps for use in integrating and

relighting scenes(Noshino and Nayar)

Previous Work Global illumination

– image based lighting

Mirror Ball environment maps

Eye Shape studies

Idea Find Eye in image by hand Find best fit ellipse Construct ellipsoid in 3 space from

ellipse, focal length, distance to camera and previous knowledge of eyes

Perform inverse ray tracing from camera to ellipsoid

Use law of reflection to project ray onto unit sphere and mark color.

Idea

Camera and Picture

Cornea

Environment Map

Implementation Mark points around

eye, find best fit ellipse.

Paper discusses using gradients to refine initial guess for ellipse parameters, though in most cases the results are satisfactory without.

Implementation Assume limited

perspective, so that the ratio between major and minor axis represents the angle of the plane it lies in with respect to the camera.

Assumes basically level camera.

Implementation Use a (t,theta)

parameterization for the ellipsoid, where t is the height, and theta the radial angle.

Send rays to focal point on camera and sample image (focal point in pixels)

Find reflection along normal

Implementation Issues (t,theta) is not uniformly

distributed, need far more rays for small t (where the ellipsoid is shallow) than large t

No guarantee that whole of sphere is covered, there may be holes.

Implementation Issues To solve these ideally: compute from

sphere to camera and sample image. However this is not trivial since rays are

dependent upon camera position and one would need to try several camera to eye rays to reach a certain point on the sphere.

The solution used just keeps adding rays until there are no gaps in the sphere image

Implementation Once sphere map is calculated, use in

global illumination as light source Assumes the map approximates higher

light level HDR since the reflection reduced small light to negligible levels. Still need to be careful to pick low levels for illumination

Additionally no attempt is made to remove iris texture. Need to pick dark eyes.

Implementation Rendered as a sphere map in

Cinema 4d 7.0 with new objects added.

Blade Runner Suppose we want to add a car to

the background of this image:

Blade Runner First we find the

best fit ellipse

Blade Runner Next we extract

the correct environment map.

(here rendered on a sphere)

Blade Runner Finally we render

the object using the computed environment map

(note that it is untextured)

Blade Runner Finally we composite (fiddle by

hand!)

Blade Runner

Sections not implemented The map can also be

used to unlight a model. Wave a light using the

light distribution found on the eyes, and assumption that skin is lambertian (dot product of normal and light direction) find a normal map.

Sections not implemented Since the lighting is

known and normals can be found, computing the base color is also possible.

All computations take place over set of images with different lighting so best fit matches of both normals and base color can be found.

Sections not implemented

Integration of the normals can be done as in previous work to find a 3d model which fits the lighting description.

This can also be texture mapped with the base texture, and lit as a lambertian surface with shadows.

Sections not implemented The surface can be inserted into a

new scene and lit according to the global illumination of that scene.

This illumination can of course come from eyes!

Sections not implemented