Chapter 4: Tyranny Is Tyranny By: Kriston Costello CHS 245 O: 17981.
On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On...
Transcript of On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On...
![Page 1: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/1.jpg)
On the Tyranny of “Elegant” ideas
or why I like Nearest Neighbors
Alyosha Efros
UC Berkeley
“For every complex problem there is an answer that is clear, simple, and wrong.” -- H. L. Mencken
![Page 2: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/2.jpg)
Point of observation
Figures © Stephen E. Palmer, 2002
3D world 2D image
How do humans see 3D?
But there is a problem…
![Page 3: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/3.jpg)
Emission Theory of Vision
Supported by: • Empedocles • Plato • Euclid (kinda) • Ptolemy • … • 50% of US college students*
*http://www.ncbi.nlm.nih.gov/pubmed/12094435?dopt=Abstract
Eyes send out “feeling rays” into the world
![Page 4: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/4.jpg)
Point of observation
Figures © Stephen E. Palmer, 2002
3D world 2D image
Truth is often ugly
![Page 5: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/5.jpg)
But we still prefer elegant answers… • As CVPR reviewer, which paper would you
rather accept? • Emission theory:
• “mathematically sound formulation” • “Sophisticated approach” • “principled”
• What actually happens: • “inelegant” • “not principled” • “just a dataset”
![Page 6: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/6.jpg)
Our Scientific Narcissism
All things being equal, we prefer to credit our own cleverness
![Page 7: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/7.jpg)
We prefer algorithms to data
Algorithm Features
Data
![Page 8: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/8.jpg)
Face Detection: Big Success Story
• Rowley, Baluja, and Kanade, 1998 • features: pixels, classifier: neural network
• Schniderman & Kanade, 1999 • features: pairs of wavelet coeff., classifier: naïve Bayes
• Viola & Jones, 2001 • features: haar, classifier: boosted cascade
![Page 9: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/9.jpg)
Word embeddings • word2vec
• Matrix factorization
• Normalized Nearest Neighbors
![Page 10: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/10.jpg)
Image captioning • LSTMs
• Feed-forward CNNs
• Language models
• …
• Nearest neighbors
![Page 11: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/11.jpg)
Learning Spectrum
Number of training samples
1 10 102 103 104 105
Extrapolation problem Generalization
Interpolation problem Correspondence
∞ 106
Slide by Antonio Torralba
![Page 12: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/12.jpg)
Part 1: Nearest Neighbors as a negative result
![Page 13: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/13.jpg)
Deformable Part Models
![Page 14: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/14.jpg)
![Page 15: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/15.jpg)
Typical HOG car detector
Felzenszwalb et al, PASCAL 2007
![Page 16: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/16.jpg)
Exemplar-SVMs
Malisiewicz et al, ICCV’11
![Page 17: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/17.jpg)
Showing off correspondences
Malisiewicz et al, ICCV’11
![Page 18: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/18.jpg)
![Page 19: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/19.jpg)
Brute Force Scene Understanding
Label Transfer
Tags: Sky, Water, Beach, Sunny, … Time: 1pm, August, 2006, …
Location: Italy, Greece, Hawaii … Photographer: Flickrbug21, Traveller2
![Page 20: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/20.jpg)
im2GPS (using 6 million GPS-tagged Flickr images)
Im2gps [Hays & Efros, CVPR’08]
![Page 21: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/21.jpg)
2006 to 2016
![Page 22: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/22.jpg)
Deep Features vs. Data
![Page 23: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/23.jpg)
“Unreasonable Effectiveness of Data”
• Parts of our world can be explained by elegant mathematics: – physics, chemistry, astronomy, etc.
• But much cannot: – psychology, genetics, economics, etc.
• Enter: The The Data – Great advances in several fields:
• e.g. speech recognition, machine translation, vision
[Halevy, Norvig, Pereira 2009]
![Page 24: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/24.jpg)
Everything else being equal… … the natural world is just much richer! • MNIST Digits
– 10 digits *
– ~1,000 variations = 10,000
• English words
– ~100,000 words *
– ~5 variations = 500,000
• Natural world
– ~100,000 objects *
– ~10,000 variations (pose, scale, lighting, intra-category)
– = 1,000,000,000 (1 billion!)
MNIST: 60,000 examples
Topic Models (Blei): 11M examples
![Page 25: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/25.jpg)
Yet, we train on 15 examples?!
Caltech 101
![Page 27: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/27.jpg)
The Good News
Really stupid algorithms + Lots of Data = “Unreasonable Effectiveness”
![Page 28: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/28.jpg)
Lessons learned so far…
![Page 29: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/29.jpg)
1. “Please don’t explain – Show me!”
Classifier trained on MSRC cars
MSRC
Caltech101
ImageNet
PASCAL
LabelMe
SUN
[Torralba&Efros,’11]
![Page 30: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/30.jpg)
• Correspondence! • Correspondence!! • Correspondence!!!*
* actually, back in the day, it was “registration”, but we argue that correspondence, alignment, and registration are all talking about the same issue
Takeo Kanade’s top 3 vision problems:
![Page 31: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/31.jpg)
Exemplar-SVMs
Malisiewicz et al, ICCV’11
![Page 32: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/32.jpg)
Showing off correspondences
Malisiewicz et al, ICCV’11
![Page 33: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/33.jpg)
Scene Classification, demystified
Doersh et al, NIPS 2013
![Page 34: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/34.jpg)
Detector w
Appearance
Exemplar
Person Meta-data
Person
Can make predictions
Malisiewicz et al, ICCV’11
![Page 35: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/35.jpg)
2. “Size Does Matter”
Given enough data, most things will be close-by even with the dumb distance
metrics!
![Page 36: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/36.jpg)
![Page 37: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/37.jpg)
2 Million Flickr Images
![Page 38: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/38.jpg)
![Page 39: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/39.jpg)
![Page 40: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/40.jpg)
![Page 41: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/41.jpg)
Nearest neighbors from a collection of 20 thousand images
![Page 42: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/42.jpg)
Nearest neighbors from a collection of 2 million images
![Page 43: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/43.jpg)
3. “Want a bigger party? Shrink the venue!”
Making Closed Worlds
![Page 44: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/44.jpg)
![Page 45: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/45.jpg)
![Page 46: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/46.jpg)
![Page 47: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/47.jpg)
… 200 scene matches
![Page 48: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/48.jpg)
![Page 49: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/49.jpg)
Flickr Paris
![Page 50: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/50.jpg)
Google StreetView Paris
![Page 51: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/51.jpg)
Other obvious examples
• Webcams • First-person (robot) data • Memes
– E.g. http://thehairpin.com/2011/01/women-laughing-alone-with-salad
![Page 52: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/52.jpg)
4. “Don’t give a man a fish – teach him how to fish”
Don’t give the computer the answer (label) right away, may it suffer first…
![Page 53: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/53.jpg)
Paris vs. Not Paris
![Page 54: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/54.jpg)
![Page 55: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/55.jpg)
Step 1: Nearest Neighbors for Every Patch Using normalized correlation of HOG features as a distance metric
patch nearest neighbors
Paris Not
Paris
![Page 56: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/56.jpg)
Step 2: Find the Parisian Clusters by Sorting
patch nearest neighbors patch nearest neighbors
Sor
t by
# P
aris
Nei
ghbo
rs
![Page 57: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/57.jpg)
5. Data is the glue trying everything together
Features / Data / Algorithms
![Page 58: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/58.jpg)
Feature + Data = better feature
![Page 59: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/59.jpg)
Good Patches may have Bad Neighbors!
patch matches
• The naïve distance metric gives equal weight to the vertical bar and the sign.
Paris Not
Paris
![Page 60: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/60.jpg)
Iterate using the new matches matches patch
Orig.
Iteration 1
Iteration 3
Iteration 2
![Page 61: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/61.jpg)
7. “Keep the computer sufficiently stressed”
![Page 62: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/62.jpg)
Latent SVMs getting too cozy…
Initial
Final
Initial
Final
![Page 63: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/63.jpg)
Cross-validation Training
KMeans
Iter 4
Iter 3
Iter 2
Iter 1
![Page 64: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/64.jpg)
Cross-validation Training
KMeans
Iter 4
Iter 3
Iter 2
Iter 1
![Page 65: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/65.jpg)
8. Trust in Data – don’t categorize
![Page 66: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/66.jpg)
The Myth of Semantic Category Generalization
• In non-linear SVMs: – In textbooks, ~10% of data
are support vectors – In recognition, up to 100%
of data are support vectors!!!
• In linear SVMs: – Typical setup: 10,000D
feature. but only 300 “chair” examples
Classic (but wrong!) Figure
![Page 67: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/67.jpg)
Do we really need this linguistic “helper”
“train” category (PASCAL VOC)
![Page 68: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/68.jpg)
Understanding an Image
slide by Fei Fei, Fergus & Torralba
![Page 69: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/69.jpg)
Object naming -> Object categorization
sky
building
flag
wall banner
bus
cars
bus
face
street lamp
slide by Fei Fei, Fergus & Torralba
![Page 70: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/70.jpg)
Object categorization
sky
building
flag
wall banner
bus
cars
bus
face
street lamp
![Page 71: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/71.jpg)
Why Categorize?
1. Knowledge Transfer 2. Communication
Tiger cat
dog
Leopard
![Page 72: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/72.jpg)
Classical View of Categories
•Dates back to Plato & Aristotle 1. Categories are defined by a
list of properties shared by all elements in a category
2. Category membership is binary
3. Every member in the category is equal
![Page 73: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/73.jpg)
Problems with Classical View • Humans don’t do this!
– People don’t rely on abstract definitions / lists of shared properties (Wittgenstein 1953, Rosch 1973) • e.g. define the properties shared by all “games” • e.g. are curtains furniture? Are olives fruit?
– Typicality • e.g. Chicken -> bird, but bird -> eagle, pigeon, etc.
– Language-dependent • e.g. “Women, Fire, and Dangerous Things” category is
Australian aboriginal language (Lakoff 1987) – Doesn’t work even in human-defined domains
• e.g. Is Pluto a planet?
![Page 74: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/74.jpg)
Problems with Visual Categories • A lot of categories are
functional
• World is too varied
Chair
![Page 75: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/75.jpg)
Typical HOG car detector
Felzenszwalb et al, PASCAL 2007
![Page 76: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/76.jpg)
Why not?
+
![Page 77: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/77.jpg)
Solution: hierarchy?
Ontologies, hierarchies, levels of categories (Rosch), etc.
WordNet, ImageNet, etc etc
Tiger cat
dog
Leopard
![Page 78: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/78.jpg)
Still Problematic! – Intransitivity
• e.g. car seat is chair, chair is furniture, but … – Multiple category membership
• it’s not a tree, it’s a forest!
Clay Shirky, “Ontologies are Overrated”
![Page 79: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/79.jpg)
Fundamental Problem with Categorization
Making decisions too early! Why not only categorize at run-time, once we
know the task!
![Page 80: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/80.jpg)
The Dictatorship of Librarians
80
Philosophy and Psychology
Language Arts and recreation
Literature
Technology Religion
![Page 81: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/81.jpg)
categories are losing…
vs.
![Page 82: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/82.jpg)
On-the-fly Categorization?
1. Knowledge Transfer 2. Communication
![Page 83: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/83.jpg)
Association instead of categorization
Ask not “what is this?”, ask “what is this like” – Moshe Bar
• Exemplar Theory (Medin & Schaffer 1978, Nosofsky 1986, Krushke 1992) –categories represented in terms of remembered objects
(exemplars) –Similarity is measured between input and all exemplars –think non-parametric density estimation
• Vanevar Bush (1945), Memex (MEMory EXtender) –Inspired hypertext, WWW, Google…
![Page 84: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/84.jpg)
Bush’s Memex (1945) • Store publications, correspondence, personal work, on
microfilm • Items retrieved rapidly using index codes
– Builds on “rapid selector” • Can annotate text with margin notes, comments • Can construct a trail through the material and save it
– Roots of hypertext • Acts as an external memory
![Page 85: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/85.jpg)
Visual Memex, a proposal [Malisiewicz & Efros]
Nodes = instances Edges = associations
types of edges: • visual similarity • spatial, temporal co-occurrence • geometric structure • language • geography •..
New object
![Page 86: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/86.jpg)
Image Understanding via Memex
![Page 87: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/87.jpg)
Torralba’s Context Challenge
![Page 88: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/88.jpg)
2 1
Torralba’s Context Challenge
Slide by Antonio Torralba
![Page 89: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/89.jpg)
Torralba’s Context Challenge
Chance ~ 1/30000 Slide by Antonio Torralba
![Page 90: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/90.jpg)
Our Challenge Setup
Malisiewicz & Efros, NIPS’09
![Page 91: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/91.jpg)
3 models Visual Memex: exemplars, non-parametric
object-object relationships • Recurse through the graph
Baseline: CoLA: categories, parametric object-
object relationships Reduced Memex: categories, non-parametric
relationships
![Page 92: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/92.jpg)
Qual. results
![Page 93: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/93.jpg)
![Page 94: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/94.jpg)
9. Use other modalities as labels
![Page 95: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/95.jpg)
9. Use other modalities as weak labels
• GPS • Date • Artist • …
![Page 96: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/96.jpg)
Visual + haptic
Active Data-gathering machine
Passive Data-gathering machine
![Page 97: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/97.jpg)
24/7 Poke-bot Farm
![Page 98: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/98.jpg)
Conclusions
• “If you torture data long enough, it might
confess” -- Ronald Coase
![Page 99: On the Tyranny of “Elegant” ideasnegative.vision/slides/Efros_Alexei_NegativeCVPR17.pdf · On the Tyranny of “Elegant” ideas or why I like Nearest Neighbors Alyosha Efros](https://reader034.fdocuments.in/reader034/viewer/2022042302/5ecd6259a8babe7a9b550b87/html5/thumbnails/99.jpg)
Thank You
© Quint Buchholz