6.891 Vision Why study Computer Vision? Why study Computer ...
Where computer vision needs help from computer science (and machine learning)
description
Transcript of Where computer vision needs help from computer science (and machine learning)
![Page 1: Where computer vision needs help from computer science (and machine learning)](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816724550346895ddbae90/html5/thumbnails/1.jpg)
Where computer vision needs help from computer science (and machine learning)
Bill FreemanElectrical Engineering and Computer Science Dept.
Massachusetts Institute of TechnologySeptember 9, 2009
![Page 2: Where computer vision needs help from computer science (and machine learning)](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816724550346895ddbae90/html5/thumbnails/2.jpg)
Outline
• My background• Computer vision applications• Computer vision techniques and problems:
– Low-level vision: underdetermined problems– High-level vision: combinatorial problems– Miscellaneous problems
![Page 3: Where computer vision needs help from computer science (and machine learning)](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816724550346895ddbae90/html5/thumbnails/3.jpg)
At Photokina, in Cologne, Germany
![Page 4: Where computer vision needs help from computer science (and machine learning)](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816724550346895ddbae90/html5/thumbnails/4.jpg)
Me (Foreign Expert) and my wife (English teacher), riding from the Foreigners’ Cafeteria at the Taiyuan University of Technology,
Shanxi, China
![Page 5: Where computer vision needs help from computer science (and machine learning)](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816724550346895ddbae90/html5/thumbnails/5.jpg)
While in China, I read this book (to be re-issued by MIT Press this year), and got very excited about computer vision. Studied for PhD at MIT.
![Page 6: Where computer vision needs help from computer science (and machine learning)](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816724550346895ddbae90/html5/thumbnails/6.jpg)
Worked for 9 years at Mitsubishi Electric Research Labs, an
industrial research lab doing fundamental research across the
street from MIT.
![Page 7: Where computer vision needs help from computer science (and machine learning)](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816724550346895ddbae90/html5/thumbnails/7.jpg)
2001 – present, MIT
![Page 8: Where computer vision needs help from computer science (and machine learning)](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816724550346895ddbae90/html5/thumbnails/8.jpg)
Infinite images
Joint work with:Biliana KanevaJosef SivicShai AvidanAntonio Torralba
![Page 9: Where computer vision needs help from computer science (and machine learning)](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816724550346895ddbae90/html5/thumbnails/9.jpg)
A computer graphics application of belief propagation for optimal seam finding
![Page 10: Where computer vision needs help from computer science (and machine learning)](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816724550346895ddbae90/html5/thumbnails/10.jpg)
The image database
•We have collected ~6 million images from Flickr based on keyword and group searches
– typical image size is 500x375 pixels– 720GB of disk space (jpeg compressed)
![Page 11: Where computer vision needs help from computer science (and machine learning)](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816724550346895ddbae90/html5/thumbnails/11.jpg)
Image representation
Color layout
GIST [Oliva and Torralba’01]
Original image
![Page 12: Where computer vision needs help from computer science (and machine learning)](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816724550346895ddbae90/html5/thumbnails/12.jpg)
Obtaining semantically coherent themesWe further break-up the collection into themes of semantically coherent scenes:
Train SVM-based classifiers from 1-2k training images [Oliva and Torralba, 2001]
![Page 13: Where computer vision needs help from computer science (and machine learning)](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816724550346895ddbae90/html5/thumbnails/13.jpg)
Basic camera motions
Forward motion Camera rotation Camera pan
Starting from a single image, find a sequence of images to simulate a camera motion:
![Page 14: Where computer vision needs help from computer science (and machine learning)](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816724550346895ddbae90/html5/thumbnails/14.jpg)
3. Find a match to fill the missing pixels
Scene matching with camera view transformations: Translation
1. Move camera
2. View from the virtual camera
4. Locally align images
5. Find a seam
6. Blend in the gradient domain
![Page 15: Where computer vision needs help from computer science (and machine learning)](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816724550346895ddbae90/html5/thumbnails/15.jpg)
4. Stitched rotation
Scene matching with camera view transformations: Camera rotation
1. Rotate camera
2. View from the virtual camera
3. Find a match to fill-in the missing pixels
5. Display on a cylinder
![Page 16: Where computer vision needs help from computer science (and machine learning)](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816724550346895ddbae90/html5/thumbnails/16.jpg)
More “infinite” images – camera translation
![Page 17: Where computer vision needs help from computer science (and machine learning)](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816724550346895ddbae90/html5/thumbnails/17.jpg)
![Page 18: Where computer vision needs help from computer science (and machine learning)](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816724550346895ddbae90/html5/thumbnails/18.jpg)
![Page 19: Where computer vision needs help from computer science (and machine learning)](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816724550346895ddbae90/html5/thumbnails/19.jpg)
![Page 20: Where computer vision needs help from computer science (and machine learning)](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816724550346895ddbae90/html5/thumbnails/20.jpg)
Virtual space as an image graph
ForwardRotate (left/right)
Pan (left/right)
• Nodes represent Images
• Edges represent particular motions:
• Edge cost is given by the cost of the image match under the particular transformation
Image graph
Kaneva, Sivic, Torralba, Avidan, and Freeman, Infinite Images, to appear in Proceedings of IEEE.
![Page 21: Where computer vision needs help from computer science (and machine learning)](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816724550346895ddbae90/html5/thumbnails/21.jpg)
Virtual image space laid out in 3D
Kaneva, Sivic, Torralba, Avidan, and Freeman, Infinite Images, to appear in Proceedings of IEEE.