Department of Computer Science, University of …castrejon/content/xmodal...Unit 31 (Fountain) Unit...
24
Cross-Modal Scene Representations Lluís Castrejón University of Toronto
Transcript of Department of Computer Science, University of …castrejon/content/xmodal...Unit 31 (Fountain) Unit...
Cross-Modal Scene Representations
Lluís CastrejónUniversity of Toronto
Motivation
Motivation
Motivation
Motivation
Motivation
??
Motivation
?
Cross-Modal Scene Understanding
CMPlaces
Dataset of 205 scene categories
CMPlaces
Dataset of 205 scene categories
CMPlaces
Dataset of 205 scene categories
Strong vs weak alignment
Strong vs weak alignment
Not scalable!
Cross-modal Networks
Cross-modal Networks
Problem: Parts of the network specialize to certain domains
Cross-modal Networks
Solution: Use regularization to enforce alignments
Cross-modal Networks
Chair
BedBedroom
Cross-modal Networks
T-SNE
Visualizing Activations
Cross-Modal Retrieval
Cross-Modal Retrieval
Inverting the representation
Thanks!
http://cmplaces.csail.mit.edu/
