Introduction to Deep Learning - TUM · Introduction to Deep Learning Optimization CNN Introduction...

58
Introduction to Deep Learning Prof. Leal-Taixé and Prof. Niessner 1

Transcript of Introduction to Deep Learning - TUM · Introduction to Deep Learning Optimization CNN Introduction...

Introduction to Deep Learning

Prof. Leal-Taixé and Prof. Niessner 1

LecturersProf. Dr. Laura

Leal-TaixéProf. Dr. Matthias

Niessner

Tim Meinhardt

Tutors

The Team

JiHou

AndreasRössler

Prof. Leal-Taixé and Prof. Niessner 2

What is Computer Vision?

• First defined in the 60s in artificial intelligence groups

• “Mimic the human visual system”

• Center block of robotic intelligence

Prof. Leal-Taixé and Prof. Niessner 3

Prof. Leal-Taixé and Prof. Niessner 4

Computer Vision

Some decades later…

Prof. Leal-Taixé and Prof. Niessner 5

Computer Vision

Physics PsychologyBiology

MathematicsEngineering Computer

scienceArtificial Intelligence

ML

Neuroscience

AlgorithmsOptimization

NLPSpeech

Robotics

OpticsImage

processing

Prof. Leal-Taixé and Prof. Niessner 6

Computer Vision

Physics PsychologyBiology

Engineering ComputerscienceArtificial

IntelligenceML

AlgorithmsOptimization

NLPSpeech

Robotics

OpticsImage

processing

Mathematics

Neuroscience

Prof. Leal-Taixé and Prof. Niessner 7

Computer Vision

Physics PsychologyBiology

Engineering ComputerscienceArtificial

IntelligenceML

AlgorithmsOptimization

NLPSpeech

Robotics

OpticsImage

processing

Mathematics

Neuroscience

Prof. Leal-Taixé and Prof. Niessner 8

Computer Vision

Physics PsychologyBiology

Engineering ComputerscienceArtificial

IntelligenceML

AlgorithmsOptimization

NLPSpeech

Robotics

OpticsImage

processing

Mathematics

Neuroscience

Prof. Leal-Taixé and Prof. Niessner 9

Pre 2012

Image classification

Prof. Leal-Taixé and Prof. Niessner 10

AAwesome magic box

Become magicians Post 2012Open the box

Image classification

Prof. Leal-Taixé and Prof. Niessner 11

Why Deep Learning?

Prof. Leal-Taixé and Prof. Niessner 12

Deep Learning History

Prof. Leal-Taixé and Prof. Niessner 13

The empire strikes back

Prof. Leal-Taixé and Prof. Niessner 14

• MNIST digit recognition dataset

• 107 pixels used in training

• ImageNet image recognition dataset

• 1014 pixels used in training

1988LeCunet al.

2012Krizhevskyet al.

What has changed?

Prof. Leal-Taixé and Prof. Niessner 15

Big Data

Models know where to learn from

Hardware

Models are trainable

Deep

Models are complex

What made this possible?

Prof. Leal-Taixé and Prof. Niessner 16

AlphaGo

Machine translation

Emoticon suggestion

Deep Learning nowadays

Prof. Leal-Taixé and Prof. Niessner 18

Self-driving cars

Deep Learning nowadays

Prof. Leal-Taixé and Prof. Niessner 19

Healthcare, cancer detection

Deep Learning nowadays

Prof. Leal-Taixé and Prof. Niessner 20

Deep Learning market

• […]market research report Deep Learning Market […] Global Forecasts to 2022", the deep learning market is expected to be worth USD 1,722.9 Million by 2022.

Prof. Leal-Taixé and Prof. Niessner 26

S. Caelles, K.K. Maninis, J. Pont-Tuset, L. Leal-Taixé, D. Cremers, and L. Van Gool.One-Shot Video Object Segmentation, CVPR 2017.

Deep Learning at TUM

Prof. Leal-Taixé and Prof. Niessner 27

Deep Learning at TUM

Prof. Leal-Taixé and Prof. Niessner 28

Deep Learning at TUM

CC3

CC2

CC1

Reshape Conv+BN+ReLU Pooling Upsample Concat Score

DDFF

Prof. Leal-Taixé and Prof. Niessner 29

Computer Vision at TUM

ScanNet: Dai, Chang, Savva, Halber, Funkhouser, Niessner., CVPR 2017.

ScanNet Stats:-Kinect-style RGB-D sensors-1513 scans of 3D environments-2.5 Mio RGB-D frames-Dense 3D, crowd-source MTurk labels-Annotations projected to 2D frames

Prof. Leal-Taixé and Prof. Niessner 30

Deep Learning at TUM

Map

Photo

Prof. Leal-Taixé and Prof. Niessner 31

Introduction to Deep Learning

Prof. Leal-Taixé and Prof. Niessner 32

About the lecture

• Theory: 12 lectures

• Every Monday 14-16h (MI HS 1)

• Practice: 3 exercises, practical sessions

• Every Thursday 8-10h (Interim HS1)

• July 2nd: guest lecture by tba

https://dvl.in.tum.de/lectures/dl4cv-ss18.htmlProf. Leal-Taixé and Prof. Niessner 33

Grading system

• Exam: July 16th

• Review: allow until end of July for exam reviews

• Important: no retake exam

• Practice: 4 exercises (Thursdays)

• Bonus 0.3 + questions in the final exam

https://dvl.in.tum.de/lectures/dl4cv-ss18.htmlProf. Leal-Taixé and Prof. Niessner 34

Exercise lectures

• Exercise 1: starting May 3rd

• Thursday lecture 1: DL math background

• Thursday lecture 2: DL math background

• Thursday lecture 3: Python introduction

Prof. Leal-Taixé and Prof. Niessner 35

Introduction to Deep Learning

Optimization

CNN

Introduction to NN

Machine Learning

basics

Back-propagation RNN

Prof. Leal-Taixé and Prof. Niessner 36

Slides• All material will be uploaded on Moodle• Questions regarding the syllabus, exercises or contents

of the lecture, use Moodle!• Questions regarding organization of the course:

• Emails to our individual addresses will not be answered.

[email protected]

Prof. Leal-Taixé and Prof. Niessner 37

Deep Learning at TUM

Intro to Deep

Learning

DL for Physics(Thuerey)

DL for Vision (Niessner,

Leal-Taixe)

DL for Medical Applicat.

(Menze)

DL in Robotics

(Bäuml)

Machine Learning(Günnemann)

Prof. Leal-Taixé and Prof. Niessner 38

Machine Learning

Prof. Leal-Taixé and Prof. Niessner 39

Machine learning

Task

Prof. Leal-Taixé and Prof. Niessner 40

Image classification

Prof. Leal-Taixé and Prof. Niessner 41

Pose Appearance IlluminationProf. Leal-Taixé and Prof. Niessner 42

Image classification

Occlusions

Prof. Leal-Taixé and Prof. Niessner 43

Image classification

Background clutter

Prof. Leal-Taixé and Prof. Niessner 44

Representation

Image classification

Prof. Leal-Taixé and Prof. Niessner 45

Task

Image classification

Experience

Data

Machine learning• How can we learn to perform image classification?

Prof. Leal-Taixé and Prof. Niessner 46

Unsupervised learning Supervised learning

Machine learning

• No label or target class

• Find out properties of the structure of the data

• Clustering (k-means, PCA)

Prof. Leal-Taixé and Prof. Niessner 47

Machine learning

Unsupervised learning Supervised learning

Prof. Leal-Taixé and Prof. Niessner 48

Machine learning

• Labels or target classes

Unsupervised learning Supervised learning

Prof. Leal-Taixé and Prof. Niessner 49

DOG DOG

DOG

CAT

CAT

CAT

Machine learning

Unsupervised learning Supervised learning

Prof. Leal-Taixé and Prof. Niessner 50

Experience

DataTraining dataTest data

Underlying assumption that train and test data come from the same distribution

Machine learning• How can we learn to perform image classification?

Prof. Leal-Taixé and Prof. Niessner 51

Reinforcement learning

Agents Environmentinteraction

Machine learning

Unsupervised learning Supervised learning

Prof. Leal-Taixé and Prof. Niessner 52

Reinforcement learning

Agents Environmentreward

Machine learning

Unsupervised learning Supervised learning

Prof. Leal-Taixé and Prof. Niessner 53

• How can we learn to perform image classification?

Task

Image classification

Experience

DataPerformance

measure

Accuracy

Machine learning

Prof. Leal-Taixé and Prof. Niessner 54

A simple classifier

Prof. Leal-Taixé and Prof. Niessner 55

Nearest Neighbor

?Prof. Leal-Taixé and Prof. Niessner 56

Nearest Neighbor

distance

NN classifier = dog

Prof. Leal-Taixé and Prof. Niessner 57

Nearest Neighbor

distance

k-NN classifier = cat

Prof. Leal-Taixé and Prof. Niessner 58

Nearest Neighbor

Courtesy of Stanford course cs231n

What is the performance on training data for NN classifier?

What classifier is more likely to perform best on test data?

Prof. Leal-Taixé and Prof. Niessner 59

Nearest Neighbor

• Hyperparameters

• These parameters are problem dependent.

• How do we choose these hyperparameters?

Distance (L1, L2)

k (number of neighbors)

Prof. Leal-Taixé and Prof. Niessner 60

Cross validationtrain

validationRun 1

Run 2

Run 3

Run 4

Run 5

Split the training data into N foldsProf. Leal-Taixé and Prof. Niessner 61

Cross validation

train test

train testvalidation

20%

Find your hyperparameters

Prof. Leal-Taixé and Prof. Niessner 62

This lecture: improving our classifier

• Beyond linear classification

• How to train complex models deep networks

• What is happening behind the scenes: optimization, CNN, regularization.

Prof. Leal-Taixé and Prof. Niessner 63

Upcoming lecture• Next Monday: Lecture 2: Machine Learning basics

• Next Thursday: 1st practical lecture (DL math background)

Prof. Leal-Taixé and Prof. Niessner 64