One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z...
Transcript of One hidden layer Neural Network Neural Networks ...Tanh activation function a z Andrew Ng z ReLU a z...
deeplearning.ai
One hidden layerNeural Network
Neural NetworksOverview
Andrew Ng
What is a Neural Network?
!"!#!$
%&
x
w
b
' = )*! + , - = .(') ℒ(-, %)
x
4["]
,["]'["] = 4["]! + ,["] -["] = .('["]) '[#] = 4[#]-["] + ,[#] -[#] = .('[#]) ℒ(-[#], %)
!"!#!$
%&
4[#]
,[#]
deeplearning.ai
One hidden layerNeural Network
Neural NetworkRepresentation
Andrew Ng
Neural Network Representation
!"
!#
!$
%&
deeplearning.ai
One hidden layerNeural Network
Computing aNeural Network’s
Output
Andrew Ng
Neural Network Representation
!"!#!$
%&
' = )!! + +
)!! + +,
!"!#!$
-(') , = %&'
, = -(')
Andrew Ng
! = #!$ + &
#!$ + &'
$($)$*
+(!) ' = ./!
' = +(!)
Neural Network Representation
$($)$*
./
$($)$*
./
Andrew Ng
Neural Network Representation
!"!#!$
%&
'""
'#"
'$"
'("
)"" = +"" ,! + /"["], '"["] = 3()"" )
)#" = +#" ,! + /#
["], '#["] = 3()#" )
)$" = +$" ,! + /$["], '$["] = 3()$" )
)(" = +(" ,! + /(
["], '(["] = 3()(" )
Andrew Ng
Neural Network Representation learningGiven input x:
! " = $ " % + ' "
( " = )(! " )
! , = $ , ( " + ' ,
( , = )(! , )
%"%,%-
./
(""
(,"
(-"
(0"
deeplearning.ai
One hidden layerNeural Network
Vectorizing across multiple examples
Andrew Ng
Vectorizing across multiple examples
!"!#!$
%&
' " = ) " ! + + "
, " = -(' " )' # = ) # , " + + #
, # = -(' # )
Andrew Ng
! " ($) = ' " (($) + * "
+ " ($) = ,(! " $ )! - ($) = ' - + " ($) + * -
+ - ($) = ,(! - $ )
Vectorizing across multiple examplesfor i = 1 to m:
deeplearning.ai
One hidden layerNeural Network
Explanation for vectorized
implementation
Andrew Ng
Justification for vectorized implementation
Andrew Ng
!"!#!$
%&
Recap of vectorizing across multiple examplesfor i = 1 to m
' " ()) = , " !()) + . "
/ " ()) = 0(' " ) )' # ()) = , # / " ()) + . #
/ # ()) = 0(' # ) )…1 = !(") !(#) !(2)
/["](#)A["] = /["](") /["](2)…
6 " = , " 1 + . "
7 " = 0(6 " )6 # = , # 7 " + . #
7 # = 0(6 # )
deeplearning.ai
One hidden layerNeural Network
Activation functions
Andrew Ng
Activation functions
!"!#!$
%&
' " = ) " ! + + "
, " = -(' " )' # = ) # , " + + #
, # = -(' # )
Given x:
Andrew Ng
Pros and cons of activation functionsa
zsigmoid: ! = 1
1 + &'(
z
a
x
a
z
a
deeplearning.ai
One hidden layerNeural Network
Why do you need non-linear
activation functions?
Andrew Ng
Activation function
! " = $ " % + ' "
( " = )["](! " )! . = $ . ( " + ' .
( . = )[.](! . )
Given x:
%"%.%/
01
deeplearning.ai
One hidden layerNeural Network
Derivatives of activation functions
Andrew Ng
Sigmoid activation function
a
z
!(#) = 11 + )*+
Andrew Ng
!(#) = tanh(#)
Tanh activation functiona
z
Andrew Ng
zReLU
a
zLeaky ReLU
a
ReLU and Leaky ReLU
deeplearning.ai
One hidden layerNeural Network
Gradient descent forneural networks
Andrew Ng
Gradient descent for neural networks
Andrew Ng
Formulas for computing derivatives
deeplearning.ai
One hidden layerNeural Network
Backpropagationintuition (Optional)
Andrew Ng
Computing gradients
Logistic regression
! = #$% + '%#'
) = *(!) ℒ(), /)
Andrew Ng
![#] = &[#]' + )[#]'
&["]
)["]
+[#] = ,(![#]) ℒ(+[0], y)![0] = &[0]' + )[0] +[0] = ,(![0])
Neural network gradients&[$]
)[$]
Andrew Ng
!"[$] = !'[$]( ) *
!+[$] = !'[$]
!'[)] = " $ ,!'[$] ∗ .[)]′(z ) )
!"[)] = !'[)]3,
!+[)] = !'[)]
Summary of gradient descent!'[$] = ([$] − 5
Andrew Ng
!"[$] = '[$] − )
!*[$] = !"[$]' + ,
!-[$] = !"[$]
!"[+] = * $ .!"[$] ∗ 0[+]′(z + )
!*[+] = !"[+]5.
!-[+] = !"[+]
!6["] = 7["] − 8
!*["] = 1:!6["]7 $ ,
!-["] = 1:;<. >?:(!6 " , '5A> = 1, BCC<!A:> = DE?C)
!6[$] = * " %!6["] ∗ 0[$]′(Z $ )
!*[$] = 1:!6[$]G%
!-[$] = 1:;<. >?:(!6 $ , '5A> = 1, BCC<!A:> = DE?C)
Summary of gradient descent
deeplearning.ai
One hidden layerNeural Network
Random Initialization
Andrew Ng
What happens if you initialize weights to zero?
!![!]"#
!$[!]"$%&!!
[$]
Andrew Ng
Random initialization
!![!]"#
!$[!]"$%&!!
[$]