Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization...

31
Optimization Algorithms Mini-batch gradient descent deeplearning.ai

Transcript of Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization...

Page 1: Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Optimization Algorithms

Mini-batchgradient descentdeeplearning.ai

Page 2: Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Andrew Ng

Batch vs. mini-batch gradient descentVectorization allows you to efficiently compute on m examples.

Page 3: Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Andrew Ng

Mini-batch gradient descent

Page 4: Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Optimization Algorithms

Understanding mini-batch

gradient descentdeeplearning.ai

Page 5: Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Andrew Ng

Training with mini batch gradient descent

# iterations

cost

Batch gradient descent

mini batch # (t)

cost

Mini-batch gradient descent

Page 6: Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Andrew Ng

Choosing your mini-batch size

Page 7: Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Andrew Ng

Choosing your mini-batch size

Page 8: Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.
Page 9: Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Andrew Ng

Page 10: Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Andrew Ng

Page 11: Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Optimization Algorithms

Understanding exponentially

weighted averagesdeeplearning.ai

Page 12: Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Andrew Ng

Exponentially weighted averages

days

temperature

!" = $!"%& + (1 − $),"

Page 13: Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Andrew Ng

Exponentially weighted averages

!"## = 0.9!(( + 0.1+"##!(( = 0.9!(, + 0.1+((!(, = 0.9!(- + 0.1+(,

…

!/ = 0!/1" + (1 − 0)+/

Page 14: Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Andrew Ng

Implementing exponentially weighted averages!" = 0!% = &!" + (1 − &)-%

…

!/ = &!% + (1 − &)-/!0 = &!/ + (1 − &)-0

Page 15: Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Optimization Algorithms

Bias correctionin exponentially

weighted averagedeeplearning.ai

Page 16: Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Andrew Ng

Bias correction

days

temperature

!" = $!"%& + (1 − $),"

Page 17: Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Optimization Algorithms

Gradient descent with momentumdeeplearning.ai

Page 18: Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Andrew Ng

Gradient descent example

Page 19: Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Andrew Ng

Implementation details

!"# = %!"# + 1 − % )*!"+ = %!"+ + 1 − % ),* = * − -!"#,

Hyperparameters: -, %

Oniteration8:Compute )*, ),on the current mini-batch

, = , − -!"+

% = 0.9

Page 20: Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Optimization Algorithms

RMSpropdeeplearning.ai

Page 21: Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Andrew Ng

RMSprop

Page 22: Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Optimization Algorithms

Adam optimizationalgorithmdeeplearning.ai

Page 23: Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Andrew Ng

Adam optimization algorithm

yhat = np.array([.9, 0.2, 0.1, .4, .9])

Page 24: Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Andrew Ng

Hyperparameters choice:

Adam Coates

Page 25: Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Optimization Algorithms

Learning rate decaydeeplearning.ai

Page 26: Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Andrew Ng

Learning rate decay

Page 27: Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Andrew Ng

Learning rate decay

Page 28: Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Andrew Ng

Other learning rate decay methods

Page 29: Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Optimization Algorithms

The problem oflocal optimadeeplearning.ai

Page 30: Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Andrew Ng

Local optima in neural networks

Page 31: Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Andrew Ng

Problem of plateaus

• Unlikely to get stuck in a bad local optima• Plateaus can make learning slow


Gradient Descent: Second Order Momentum and Saturating Errorpapers.nips.cc/paper/454-gradient-descent-second-order... · 2014-04-14 · 1.1 SIMPLE GRADIENT DESCENT First, let us review

Gradient Descent: Second Order Momentum and Saturating Errorpapers.nips.cc/paper/454-gradient-descent-second-order... · 2014-04-14 · 1.1 SIMPLE GRADIENT DESCENT First, let us review

The Gradient Descent Algorithm

The Gradient Descent Algorithm

Learning to Learn by Gradient Descent with Rebalancing€¦ · that, for instance, are capable of learning to learn without gradient descent by gradient descent. It should be expected

Learning to Learn by Gradient Descent with Rebalancing€¦ · that, for instance, are capable of learning to learn without gradient descent by gradient descent. It should be expected

Gradient Descent - University of Washington

Gradient Descent - University of Washington

Stochastic Gradient Descent - CMU Statistics

Stochastic Gradient Descent - CMU Statistics

Optimization, Gradient Descent, and Backpropagation

Optimization, Gradient Descent, and Backpropagation

10-315 Recitation Review of Gradient Descent & Kernelsninamf/courses/315sp19/recitations/2_21-… · Review of Gradient Descent & Kernels Misha 21 February 2019. Gradient Descent:

10-315 Recitation Review of Gradient Descent & Kernelsninamf/courses/315sp19/recitations/2_21-… · Review of Gradient Descent & Kernels Misha 21 February 2019. Gradient Descent:

Deep Learning I: Gradient Descentgd.pdf · Roadmap Intro,model,cost Gradient descent Gradient descent (GD) Gradient descent is a learning algorithm. Given: a hypothesis space (or

Deep Learning I: Gradient Descentgd.pdf · Roadmap Intro,model,cost Gradient descent Gradient descent (GD) Gradient descent is a learning algorithm. Given: a hypothesis space (or

Chapter 4 (Part 2): Artificial Neural Networksmlittman/courses/ml03/ch4b.pdf · 6 Stochastic Gradient Descent Batch mode Gradient Descent: Do until satisfied 1. Compute the gradient

Chapter 4 (Part 2): Artificial Neural Networksmlittman/courses/ml03/ch4b.pdf · 6 Stochastic Gradient Descent Batch mode Gradient Descent: Do until satisfied 1. Compute the gradient

Learning Unitaries with gradient descent optimizationqist2019/slides/3rd/...Gradient Descent Numerics A `transition’ occurs when gradient descent is performed in the overparameterized

Learning Unitaries with gradient descent optimizationqist2019/slides/3rd/...Gradient Descent Numerics A `transition’ occurs when gradient descent is performed in the overparameterized

GRADIENT DESCENT - Pomona College

GRADIENT DESCENT - Pomona College

Gradient descent method

Gradient descent method

Linear Regression & Gradient Descent

Linear Regression & Gradient Descent

The General Inefficiency of Batch Training for Gradient ...axon.cs.byu.edu/~martinez/classes/678/Papers/Wilson_batch orig.pdfKey words: Batch training, on-line training, gradient descent,

The General Inefficiency of Batch Training for Gradient ...axon.cs.byu.edu/~martinez/classes/678/Papers/Wilson_batch orig.pdfKey words: Batch training, on-line training, gradient descent,

Exponentiated Gradient versus Gradient Descent for Linear ...manfred/pubs/J36.pdf · Exponentiated Gradient versus Gradient Descent for Linear Predictors* Jyrki Kivinen-Department

Exponentiated Gradient versus Gradient Descent for Linear ...manfred/pubs/J36.pdf · Exponentiated Gradient versus Gradient Descent for Linear Predictors* Jyrki Kivinen-Department

Optimization/Gradient Descent

Optimization/Gradient Descent

When Big Data and Machine Learning meet Partial ... · Learning Phase Gradient back-propagation aka Stochastic Gradient Descent Present the examples 1 by 1 or mini-batch by mini-batch

When Big Data and Machine Learning meet Partial ... · Learning Phase Gradient back-propagation aka Stochastic Gradient Descent Present the examples 1 by 1 or mini-batch by mini-batch

Proximal Gradient Descent › ~aarti › Class › 10725_Fall17 › Lecture_Slides › ...Proximal gradient descent has convergence rate O(1=k), or O(1= ) Same as gradient descent!

Proximal Gradient Descent › ~aarti › Class › 10725_Fall17 › Lecture_Slides › ...Proximal gradient descent has convergence rate O(1=k), or O(1= ) Same as gradient descent!

Midterm Review - CS230 Deep Learning · Optimization Methods Gradient Descent - update parameters in the opposite direction of their gradient. Stochastic - batch size of 1. Mini Batch

Midterm Review - CS230 Deep Learning · Optimization Methods Gradient Descent - update parameters in the opposite direction of their gradient. Stochastic - batch size of 1. Mini Batch

Linear Regression and Gradient Descent

Linear Regression and Gradient Descent