One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf ·...

35
deeplearning.ai One hidden layer Neural Network Neural Networks Overview

Transcript of One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf ·...

Page 1: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

deeplearning.ai

One hidden layerNeural Network

Neural NetworksOverview

Page 2: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

Andrew Ng

What is a Neural Network?

!"!#!$

%&

x

w

b

' = )*! + , - = .(') ℒ(-, %)

x

4["]

,["]'["] = 4["]! + ,["] -["] = .('["]) '[#] = 4[#]-["] + ,[#] -[#] = .('[#]) ℒ(-[#], %)

!"!#!$

%&

4[#]

,[#]

Page 3: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

deeplearning.ai

One hidden layerNeural Network

Neural NetworkRepresentation

Page 4: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

Andrew Ng

Neural Network Representation

!"

!#

!$

%&

Page 5: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

deeplearning.ai

One hidden layerNeural Network

Computing aNeural Network’s

Output

Page 6: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

Andrew Ng

Neural Network Representation

!"!#!$

%&

' = )!! + +

)!! + +,

!"!#!$

-(') , = %&'

, = -(')

Page 7: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

Andrew Ng

! = #!$ + &

#!$ + &'

$($)$*

+(!) ' = ./!

' = +(!)

Neural Network Representation

$($)$*

./

$($)$*

./

Page 8: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

Andrew Ng

Neural Network Representation

!"!#!$

%&

'""

'#"

'$"

'("

)"" = +"" ,! + /"["], '"["] = 3()"" )

)#" = +#" ,! + /#

["], '#["] = 3()#" )

)$" = +$" ,! + /$["], '$["] = 3()$" )

)(" = +(" ,! + /(

["], '(["] = 3()(" )

Page 9: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

Andrew Ng

Neural Network Representation learningGiven input x:

! " = $ " % + ' "

( " = )(! " )

! , = $ , ( " + ' ,

( , = )(! , )

%"%,%-

./

(""

(,"

(-"

(0"

Page 10: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

deeplearning.ai

One hidden layerNeural Network

Vectorizing across multiple examples

Page 11: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

Andrew Ng

Vectorizing across multiple examples

!"!#!$

%&

' " = ) " ! + + "

, " = -(' " )' # = ) # , " + + #

, # = -(' # )

Page 12: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

Andrew Ng

! " ($) = ' " (($) + * "

+ " ($) = ,(! " $ )! - ($) = ' - + " ($) + * -

+ - ($) = ,(! - $ )

Vectorizing across multiple examplesfor i = 1 to m:

Page 13: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

deeplearning.ai

One hidden layerNeural Network

Explanation for vectorized

implementation

Page 14: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

Andrew Ng

Justification for vectorized implementation

Page 15: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

Andrew Ng

!"!#!$

%&

Recap of vectorizing across multiple examplesfor i = 1 to m

' " ()) = , " !()) + . "

/ " ()) = 0(' " ) )' # ()) = , # / " ()) + . #

/ # ()) = 0(' # ) )…1 = !(") !(#) !(2)

/["](#)A["] = /["](") /["](2)…

6 " = , " 1 + . "

7 " = 0(6 " )6 # = , # 7 " + . #

7 # = 0(6 # )

Page 16: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

deeplearning.ai

One hidden layerNeural Network

Activation functions

Page 17: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

Andrew Ng

Activation functions

!"!#!$

%&

' " = ) " ! + + "

, " = -(' " )' # = ) # , " + + #

, # = -(' # )

Given x:

Page 18: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

Andrew Ng

Pros and cons of activation functionsa

zsigmoid: ! = 1

1 + &'(

z

a

x

a

z

a

Page 19: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

deeplearning.ai

One hidden layerNeural Network

Why do you need non-linear

activation functions?

Page 20: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

Andrew Ng

Activation function

! " = $ " % + ' "

( " = )["](! " )! . = $ . ( " + ' .

( . = )[.](! . )

Given x:

%"%.%/

01

Page 21: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

deeplearning.ai

One hidden layerNeural Network

Derivatives of activation functions

Page 22: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

Andrew Ng

Sigmoid activation function

a

z

!(#) = 11 + )*+

Page 23: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

Andrew Ng

!(#) = tanh(#)

Tanh activation functiona

z

Page 24: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

Andrew Ng

zReLU

a

zLeaky ReLU

a

ReLU and Leaky ReLU

Page 25: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

deeplearning.ai

One hidden layerNeural Network

Gradient descent forneural networks

Page 26: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

Andrew Ng

Gradient descent for neural networks

Page 27: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

Andrew Ng

Formulas for computing derivatives

Page 28: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

deeplearning.ai

One hidden layerNeural Network

Backpropagationintuition (Optional)

Page 29: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

Andrew Ng

Computing gradients

Logistic regression

! = #$% + '%#'

) = *(!) ℒ(), /)

Page 30: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

Andrew Ng

![#] = &[#]' + )[#]'

&["]

)["]

+[#] = ,(![#]) ℒ(+[0], y)![0] = &[0]' + )[0] +[0] = ,(![0])

Neural network gradients&[$]

)[$]

Page 31: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

Andrew Ng

!"[$] = !'[$]( ) *

!+[$] = !'[$]

!'[)] = " $ ,!'[$] ∗ .[)]′(z ) )

!"[)] = !'[)]3,

!+[)] = !'[)]

Summary of gradient descent!'[$] = ([$] − 5

Page 32: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

Andrew Ng

!"[$] = '[$] − )

!*[$] = !"[$]' + ,

!-[$] = !"[$]

!"[+] = * $ .!"[$] ∗ 0[+]′(z + )

!*[+] = !"[+]5.

!-[+] = !"[+]

!6["] = 7["] − 8

!*["] = 1:!6["]7 $ ,

!-["] = 1:;<. >?:(!6 " , '5A> = 1, BCC<!A:> = DE?C)

!6[$] = * " %!6["] ∗ 0[$]′(Z $ )

!*[$] = 1:!6[$]G%

!-[$] = 1:;<. >?:(!6 $ , '5A> = 1, BCC<!A:> = DE?C)

Summary of gradient descent

Page 33: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

deeplearning.ai

One hidden layerNeural Network

Random Initialization

Page 34: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

Andrew Ng

What happens if you initialize weights to zero?

!![!]"#

!$[!]"$%&!!

[$]

Page 35: One hidden layer Neural Network Neural Networks …cs230.stanford.edu/files/C1M3.pdf · 2019-01-07 · Neural Network Gradient descent for neural networks. Andrew Ng Gradient descent

Andrew Ng

Random initialization

!![!]"#

!$[!]"$%&!!

[$]


Learning long-term dependencies with gradient …ai.dinfo.unifi.it/paolo/ps/tnn-94-gradient.pdfTitle Learning long-term dependencies with gradient descent is difficult - Neural Networks,

Learning long-term dependencies with gradient …ai.dinfo.unifi.it/paolo/ps/tnn-94-gradient.pdfTitle Learning long-term dependencies with gradient descent is difficult - Neural Networks,

Neural Proximal Gradient Descent for Compressive Imaging · 2019-02-19 · Neural Proximal Gradient Descent for Compressive Imaging Morteza Mardani1, Qingyun Sun4, Shreyas Vasawanala2,

Neural Proximal Gradient Descent for Compressive Imaging · 2019-02-19 · Neural Proximal Gradient Descent for Compressive Imaging Morteza Mardani1, Qingyun Sun4, Shreyas Vasawanala2,

Chapter 4 (Part 2): Artificial Neural Networksmlittman/courses/ml03/ch4b.pdf · 6 Stochastic Gradient Descent Batch mode Gradient Descent: Do until satisfied 1. Compute the gradient

Chapter 4 (Part 2): Artificial Neural Networksmlittman/courses/ml03/ch4b.pdf · 6 Stochastic Gradient Descent Batch mode Gradient Descent: Do until satisfied 1. Compute the gradient

Learning to learn by gradient descent by gradient descent · Learning to learn by gradient descent by gradient descent Marcin Andrychowicz 1, Misha Denil , Sergio Gómez Colmenarejo

Learning to learn by gradient descent by gradient descent · Learning to learn by gradient descent by gradient descent Marcin Andrychowicz 1, Misha Denil , Sergio Gómez Colmenarejo

Analysis of Gradient Descent on Wide Two-Layer ReLU Neural ...

Analysis of Gradient Descent on Wide Two-Layer ReLU Neural ...

Analysis of Gradient Descent on Wide Two-Layer ReLU Neural ...Analysis of Gradient Descent on Wide Two-Layer ReLU Neural Networks L ena c Chizat*, joint work with Francis Bach+ August

Analysis of Gradient Descent on Wide Two-Layer ReLU Neural ...Analysis of Gradient Descent on Wide Two-Layer ReLU Neural Networks L ena c Chizat*, joint work with Francis Bach+ August

Optimization Algorithm Inspired Deep Neural Network ...some function using the gradient descent algorithm. Based on this observation, we replace the gradient descent algorithm with

Optimization Algorithm Inspired Deep Neural Network ...some function using the gradient descent algorithm. Based on this observation, we replace the gradient descent algorithm with

Exponentiated Gradient versus Gradient Descent for Linear ...manfred/pubs/J36.pdf · Exponentiated Gradient versus Gradient Descent for Linear Predictors* Jyrki Kivinen-Department

Exponentiated Gradient versus Gradient Descent for Linear ...manfred/pubs/J36.pdf · Exponentiated Gradient versus Gradient Descent for Linear Predictors* Jyrki Kivinen-Department

Artificial Neural Networks · 2015-11-24 · Artificial Neural Networks Revised September 2010 ... lPerceptron and Gradient Descent Algorithm lMultilayer Neural Networks lDesigning

Artificial Neural Networks · 2015-11-24 · Artificial Neural Networks Revised September 2010 ... lPerceptron and Gradient Descent Algorithm lMultilayer Neural Networks lDesigning

Gradient Descent Optimization

Gradient Descent Optimization

Gradient Descent: Second Order Momentum and Saturating Errorpapers.nips.cc/paper/454-gradient-descent-second-order... · 2014-04-14 · 1.1 SIMPLE GRADIENT DESCENT First, let us review

Gradient Descent: Second Order Momentum and Saturating Errorpapers.nips.cc/paper/454-gradient-descent-second-order... · 2014-04-14 · 1.1 SIMPLE GRADIENT DESCENT First, let us review

Optimization/Gradient Descent

Optimization/Gradient Descent

Gradient Descent for Spiking Neural Networkspapers.nips.cc/paper/7417-gradient-descent-for-spiking-neural-netwo… · spikes. Research in spike-based computation has been impeded

Gradient Descent for Spiking Neural Networkspapers.nips.cc/paper/7417-gradient-descent-for-spiking-neural-netwo… · spikes. Research in spike-based computation has been impeded

Gradient Descent for Spiking Neural Networks Descent for... · 2019-06-29 · Gradient Descent for Spiking Neural Networks Dongsung Huh Salk Institute La Jolla, CA 92037 huh@salk.edu

Gradient Descent for Spiking Neural Networks Descent for... · 2019-06-29 · Gradient Descent for Spiking Neural Networks Dongsung Huh Salk Institute La Jolla, CA 92037 [email protected]

Dynamics of On-Line Gradient Descent Learning for Multilayer … · 2014. 4. 14. · Dynamics of On-Line Gradient Descent Learning for Multilayer Neural Networks David Saad" Dept.

Dynamics of On-Line Gradient Descent Learning for Multilayer … · 2014. 4. 14. · Dynamics of On-Line Gradient Descent Learning for Multilayer Neural Networks David Saad" Dept.

Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

An Overview of Gradient Descent Optimization Algorithms ... · Outline 1 Introduction Basics 2 Gradient Descent Variants Basic Gradient Descent Algorithms Limitations 3 Gradient Descent

An Overview of Gradient Descent Optimization Algorithms ... · Outline 1 Introduction Basics 2 Gradient Descent Variants Basic Gradient Descent Algorithms Limitations 3 Gradient Descent

DLFS - Training Neural Networks - Databricks...Stochastic Gradient Descent • (Batch) Gradient Descent is computed on the full dataset (not efficient for large scale models and datasets).

DLFS - Training Neural Networks - Databricks...Stochastic Gradient Descent • (Batch) Gradient Descent is computed on the full dataset (not efficient for large scale models and datasets).

CS 5751 Machine Learning Chapter 4 Artificial Neural Networks1 Artificial Neural Networks Threshold units Gradient descent Multilayer networks Backpropagation.

CS 5751 Machine Learning Chapter 4 Artificial Neural Networks1 Artificial Neural Networks Threshold units Gradient descent Multilayer networks Backpropagation.

Gradient Descent Easy version

Gradient Descent Easy version