Introduction to Deep Learning (I2DL)

I2DL: Prof. Niessner

Introduction to Deep Learning (I2DL)

1

Exercise 4: Simple Classifier


Today’s Outline• The Pillars of Deep Learning• Exercise 4: Simple Classifier

– Housing Dataset– Submission 2

• Backpropagation• Outlook: Lecture 5 + Exercise 5

2


The Pillars of Deep Learning

3



4

Data Model Solver

Dataset

Dataloader

Network

Loss/Objective

Optimizer

Training Loop

Validation



5

Data

Dataset

Dataloader

Exercise 3: Dataset and Dataloader



6

Model Solver

Network

Loss/Objective

Optimizer

Training Loop

Validation


Exercise 5: Simple Network

Exercise 6: Hyperparameter Tuning


Goal: Exercise 4

7

Solver

Optimizer

Training Loop

Validation

• Goal: Trainings process• Skip: Model Pillar• Simplified Model: Classifier which is a 1-

Layer Neural Network


Goals: Exercises 5++

8

Model

Network

Loss/Objective

• Ex 3 + 4: Dataloading and Trainings process

• Ex 5++: Expand the exercises to more interesting model architectures



9

Data Model Solver

Dataset

Dataloader

Network

Loss/Objective

Optimizer

Training Loop

Validation

Can be implemented once and used in multiple projects



10


Overview Exercise 4• One Notebook

– Logistic regression model

• Submission 2– Several implementation tasks in the notebook– Submission file creation in Notebook

11

Fixed Deadline:Nov 17, 2022 15:59


Housing Dataset

12

• Housing Dataset: Data of ~1400 houses including 81 features like Neighborhood, GrLivArea, YearBuilt, etc.

• Simplified model: 1 input feature to predict the house price


Submission 4 - Classifying House Prices

13

ML Model 𝑴𝑴 𝐱 = 𝐲

Expensive 𝑦 = 1

Low-priced 𝑦 = 0ML Model 𝑴𝑴 𝐱 = 𝐲


Solver

3rd Pillar of Deep Learning

14

Training Data

Validation Data

Loss

Optimizer

Model


Backpropagation

15

𝐿L(𝑦, %𝑦)M! x = $𝑦

d$𝑦/d𝜃 d𝐿/d𝜃d𝐿/d(𝑦

x

"𝑦

Forward pass

Backward pass

Binary Cross Entropy Loss:


Backpropagation

16

𝐿L(𝑦, %𝑦)M! x = $𝑦

d$𝑦/d𝜃 d𝐿/d𝜃d𝐿/d(𝑦

x

"𝑦

Forward pass

Backward pass

𝜃'() = 𝜃' − 𝜆 ⋅ 𝛻*LOptimization with gradient descent:


Backpropagation

17


Model

18

• Input: representing our data with N samples and

D+1 feature dimensions

• Output: Binary labels given by

• Model: Classifier of the form

• Sigmoid function: with

• Weights of the Classifier:


Forward Pass

19

Sample


Input Data X

20


Forward Pass

21Forward Pass

Sample


Backward Pass

22Backward Pass

Sample


Backward Pass

23

• Backward Pass: Derivative of function with respect to weights

of our Classifier

• Attention: Make sure you understand the dimensions here

• Step 1: Forward + Backward Pass for one sample

• Step 2: Forward + Backward Pass for N samples


Outlook

24


Upcoming Lectures

• Next lecture:Lecture 5: Stochastic Gradient Descent

• Next Thursday: Exercise 5: Two-layer Neural Network

25

Model

Network

Loss/Objective


Summary• Monday 15.11 : Watch Lecture 5

– Scaling Optimization to Large Data, Stochastic Gradient Descent

• Wednesday 17.11 15:59: Submit exercise 4– Simple Classifier

• Thursday 18.11: Tutorial 5– Neural Networks

26


See you next week J

27

Introduction to Deep Learning (I2DL)

Documents

Transcript of Introduction to Deep Learning (I2DL)