Basics of Representations - web.stanford.edu · Basics of Representations (and traditional...
Transcript of Basics of Representations - web.stanford.edu · Basics of Representations (and traditional...
![Page 1: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/1.jpg)
Basics of Representations(and traditional low-level representations)
CS331B: Representation Learning in Computer VisionAmir R. Zamir
Silvio Savarese
![Page 2: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/2.jpg)
(class logistics)● Student paper presentations for 10/12
○ Discriminative learning of deep convolutional feature point descriptors, Simo-Serra, E., Trulls, E., Ferraz, L., Kokkinos, I., Fua, P., & Moreno-Noguer, F., ICCV15
○ Data-Driven 3D Voxel Patterns for Object Category Recognition, Yu Xiang, Wongun Choi, Yuanqing Lin & Silvio Savarese., CVPR15.
○ Convolutional-recursive deep learning for 3d object classification, Socher, R., Huval, B., Bath, B., Manning, C. D., & Ng, A. Y., NIPS12.
2
![Page 3: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/3.jpg)
(class logistics)● A few conceptual and ML oriented papers towards the end of the quarter:
○ Representation learning: A review and new perspectivesY Bengio, A Courville, P Vincent, 2013 PAMI
○ Intelligence without representationRA Brooks - Artificial intelligence, 1991 Elsevier
● Additional ideas for student presentations (extensive papers, talks, etc.) -- prior approval needed.
3
![Page 4: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/4.jpg)
What we talked about so far...
4
![Page 5: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/5.jpg)
Things... Our Knowledge...
5
![Page 6: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/6.jpg)
“Transcript”
Cat
Macbeth was guilty.
6
![Page 7: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/7.jpg)
“Transcript”
Cat
Macbeth was guilty.
[ 81 20 84 64 58 39 17 54 72 15]
Representation Mathematical Model (e.g., classifier)
7
![Page 8: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/8.jpg)
~12 lbs
~8 lbs
-5 0 +207 1511
X XXX XXX XXX XX X XXX XXX XXX XX
w
Weight (w)
Representation Mathematical Model (Classifier)
w>11
X X
Type B
Type A
8
![Page 9: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/9.jpg)
Represent these cats for a cat detector!
9
![Page 10: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/10.jpg)
Represent these cats for a cat detector! (II)
10
![Page 11: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/11.jpg)
Represent these cats for a cat detector! (III)
11
![Page 12: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/12.jpg)
Represent these cats for a cat detector! (IV)
12
![Page 13: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/13.jpg)
Not always as easy (Happy vs Sad)
13
![Page 14: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/14.jpg)
Not always as easy (Sad)
14
![Page 15: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/15.jpg)
Color Histograms
Deformable Part based Models
(DPM)
Histogram of Gradients
(HOG)
Models based Shapes
15Felzenszwalb et al., 2010. Dalal and Triggs, 2005.Beis and Lowe, 1997.
![Page 16: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/16.jpg)
This lecture...
16
![Page 17: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/17.jpg)
Some basics concepts related to representations
17
![Page 18: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/18.jpg)
Concepts● Ill-posedness● Readout Linearity ● Dimensionality● Computational Complexity ● Encoding power (i.e., performance)● Narrowness of application domain (vertical vs horizontal representations)
18
![Page 19: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/19.jpg)
Ill-posedness
19C. F. Bohren, D. R. Huffman, 1983.
![Page 21: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/21.jpg)
Ill-posedness
21
![Page 22: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/22.jpg)
Ill-posedness● 3D pose estimation from 2D gradients is an ill-posed problem.
○ 2D gradient representation is ill-posed wrt 3D pose. ○ 2D gradient representation+full semantics is NOT ill-posed wrt 3D pose.
22
![Page 23: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/23.jpg)
Linearity
23
![Page 24: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/24.jpg)
Linearity
24
● Readout linearity → concerns modeling parameters → Linear classifier, FC● Representation non-linearity → concerns independent variables → ReLU, Neurons, etc.
Linear/Non-linear? Linear/Non-linear?
![Page 25: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/25.jpg)
Linearity
25Linear/Non-linear Linear/Non-linear
● Readout linearity → concerns modeling parameters → Linear classifier, FC● Representation non-linearity → concerns independent variables → ReLU, Neurons, etc.
![Page 26: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/26.jpg)
26
With respect to: {modeling parameters (decision) , independent variables (representation)}
Linear or Non-linear?
Independent var. (x,y)
Modeling Param. (a,b,c,r)
Linear non-Linear
Linear Linear
Decision boundary
Not discussing kernels, reparametrization, etc
![Page 27: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/27.jpg)
Concepts● Ill-posedness● Readout Non-linearity ● Dimensionality● Computational Complexity ● Encoding power (i.e., single-task performance)● Narrowness of application domain (i.e., multi-task performance)
27
More discussions in Lectures 3 & 8
More discussions in Lecture 12
![Page 28: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/28.jpg)
Classical low-level 2D Representations
28
![Page 29: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/29.jpg)
Pixel Gradient based Features
29
![Page 30: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/30.jpg)
Histogram of Gradients (and its descendants)
30Dalal and Triggs, 2005.
![Page 31: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/31.jpg)
HOGgles!Representation ⇄ Data
31Vondrick et al. 2013..
![Page 33: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/33.jpg)
HOGgles -- How: sparse coding
33
![Page 34: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/34.jpg)
HOGgles!
34Vondrick et al. 2013..
![Page 35: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/35.jpg)
HOGgles!
35Vondrick et al. 2013..
![Page 36: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/36.jpg)
HOGgles!
36Vondrick et al. 2013..
![Page 37: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/37.jpg)
HOGgles & ill-posedness
37Vondrick et al. 2013..
Hadamard well-posedness terms:1. A solution exists2. The solution is unique3. Solution's behavior is smooth
![Page 38: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/38.jpg)
Affine-SIFT● Original SIFT: 4-DOF of affine
invariant (translation, scale, rotation)
● ASIFT -- basic idea: exhaustively transform images (w/ sampling and efficiency mechanisms) → then use original SIFT.
38Morel & Yu. 2009.
![Page 39: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/39.jpg)
Self-Similarity See the board!
39Junejo et al. 2008.
![Page 40: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/40.jpg)
(spatial) Self-Similarity
40Shechtman & Irani, 2007.
![Page 41: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/41.jpg)
41Shechtman & Irani, 2007.
(spatial) Self-Similarity
![Page 42: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/42.jpg)
42Shechtman & Irani, 2007.
(spatial) Self-Similarity
![Page 43: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/43.jpg)
Classical Video features
43
![Page 44: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/44.jpg)
3D-SIFTA descriptor for volumetric data (temporal or 3D)
44Scovanner et al. 2007.
2D SIFT Multi-2D SIFT 3D SIFT
![Page 45: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/45.jpg)
3D-SIFT
45Scovanner et al. 2007.
Spatio-temporal cubes Bag-of-words (~cubes) -- based on 3D SIFT similarity
![Page 46: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/46.jpg)
Dense Trajectory Features
46Wang et al. 2011.
Lucas & Kanade. 1981.
![Page 47: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/47.jpg)
Dense Trajectory Features
47Wang et al. 2011.
![Page 48: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/48.jpg)
Dense Trajectory Features
48Wang et al. 2011.
![Page 49: Basics of Representations - web.stanford.edu · Basics of Representations (and traditional low-level representations) CS331B: Representation Learning in Computer Vision Amir R. Zamir](https://reader031.fdocuments.in/reader031/viewer/2022021809/5c5bc97909d3f25e368c6e6f/html5/thumbnails/49.jpg)
Course webpage:http://web.stanford.edu/class/cs331b/
http://www.cs.stanford.edu/~amirz/http://cvgl.stanford.edu/silvio/