Creating and Exploring a Large Photorealistic Virtual Space

INRIA / CSAIL / Adobe

First IEEE Workshop on Internet Vision, associated with CVPR 2008.

Outline• Introduction• Constructing the image space• Navigating the virtual 3D space• Limitations and Conclusion

Introduction

• We present a system for exploring large collections of photos in a virtual 3D space.

• Let users navigate within each theme using intuitive 3D controls that include move left/right, zoom and rotate.

• In a similar fashion we can create infinite zoom effects that resemble the ”Droste effect”.

Constructing the image space

•The image database– We have collected ~6 million images from Flickr

based on keyword and group searches• typical image size is 500x375 pixels• 720GB of disk space (jpeg compressed)

Image representation

Color layout

GIST [Oliva and Torralba’01]

Original image

Obtaining semantically coherent themesWe further break-up the collection into themes of semantically coherent scenes:

Train SVM-based classifiers from 1-2k training images [Oliva and Torralba, 2001]

Basic camera motions

Forward motion Camera rotation Camera pan

Starting from a single image, find a sequence of images to simulate a camera motion:

3. Find a match to fill the missing pixels

Scene matching with camera view transformations: Translation

1. Move camera

2. View from the virtual camera

4. Locally align images

5. Find a seam

6. Blend in the gradient domain

4. Stitched rotation

Scene matching with camera view transformations: Camera rotation

1. Rotate camera

2. View from the virtual camera

3. Find a match to fill-in the missing pixels

5. Display on a cylinder

Steps

• Collect images• Classify images into topic themes• Calculate the descriptors:

– GIST– RGB

• Build the graph• Find the path for given query image(s)

– Dijkstra algorithm• Alignment

– Gradient decent alignment• Composition

– Poisson blending

More “infinite” images – camera translation

Forward

Rotate (left/right)

Pan (left/right)

• Nodes represent Images

• Edges represent particular motions:

• Edge cost is given by the cost of the image match under the particular transformation

Image graph

Navigating the virtual 3D space

• Virtual space as an image graph

•Virtual image space laid out in 3D

Limitations and Conclusion

• The larger the database, the better the results.• Compositing two distinct images is always a challenge

and at times, the seam is quite visible.• This system can be used to create photorealistic visual

content for large online virtual 3D worlds like Second Life.

• Create infinite panoramas or use the image taxi to generate tailor-made tours in the virtual 3D space. These applications can find use in games, movies and other media creation processes.

Thank you！