Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization...

Optimization Algorithms

Mini-batchgradient descentdeeplearning.ai

Andrew Ng

Batch vs. mini-batch gradient descentVectorization allows you to efficiently compute on m examples.

Andrew Ng

Mini-batch gradient descent

Understanding mini-batch

gradient descentdeeplearning.ai

Andrew Ng

Training with mini batch gradient descent

# iterations

Batch gradient descent

mini batch # (t)

Mini-batch gradient descent

Andrew Ng

Choosing your mini-batch size

Andrew Ng

Choosing your mini-batch size

Andrew Ng

Understanding exponentially

weighted averagesdeeplearning.ai

Andrew Ng

Exponentially weighted averages

temperature

!" = $!"%& + (1 − $),"

Andrew Ng

Exponentially weighted averages

!"## = 0.9!(( + 0.1+"##!(( = 0.9!(, + 0.1+((!(, = 0.9!(- + 0.1+(,

!/ = 0!/1" + (1 − 0)+/

Andrew Ng

Implementing exponentially weighted averages!" = 0!% = &!" + (1 − &)-%

!/ = &!% + (1 − &)-/!0 = &!/ + (1 − &)-0

Bias correctionin exponentially

weighted averagedeeplearning.ai

Andrew Ng

Bias correction

temperature

!" = $!"%& + (1 − $),"

Gradient descent with momentumdeeplearning.ai

Andrew Ng

Gradient descent example

Andrew Ng

Implementation details

!"# = %!"# + 1 − % )*!"+ = %!"+ + 1 − % ),* = * − -!"#,

Hyperparameters: -, %

Oniteration8:Compute )*, ),on the current mini-batch

, = , − -!"+

% = 0.9

RMSpropdeeplearning.ai

Andrew Ng

RMSprop

Adam optimizationalgorithmdeeplearning.ai

Andrew Ng

Adam optimization algorithm

yhat = np.array([.9, 0.2, 0.1, .4, .9])

Andrew Ng

Hyperparameters choice:

Adam Coates

Learning rate decaydeeplearning.ai

Andrew Ng

Learning rate decay

Andrew Ng

Learning rate decay

Andrew Ng

Other learning rate decay methods

The problem oflocal optimadeeplearning.ai

Andrew Ng

Local optima in neural networks

Andrew Ng

Problem of plateaus

• Unlikely to get stuck in a bad local optima• Plateaus can make learning slow

Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization...

Documents

Transcript of Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization...

Gradient Descent - cs.cmu.edu · Gradient Descent • Now that we have seen how horrible gradient descent is, and how there are so many methods with better guarantees, let’s now

The general inefﬁciency of batch training for gradient …axon.cs.byu.edu/papers/Wilson.nn03.batch.pdfThe general inefﬁciency of batch training for gradient descent learning D.

10-315 Recitation Review of Gradient Descent & Kernelsninamf/courses/315sp19/recitations/2_21-… · Review of Gradient Descent & Kernels Misha 21 February 2019. Gradient Descent:

Gradient descent

Proximal Gradient Descent › ~aarti › Class › 10725_Fall17 › Lecture_Slides › ...Proximal gradient descent has convergence rate O(1=k), or O(1= ) Same as gradient descent!

Learning to learn by gradient descent by gradient descentpapers.nips.cc/...to...descent-by-gradient-descent.pdf · Learning to learn by gradient descent by gradient descent Marcin

Semi-Stochastic Gradient Descent Methods

An Overview of Gradient Descent Optimization Algorithms ... · Outline 1 Introduction Basics 2 Gradient Descent Variants Basic Gradient Descent Algorithms Limitations 3 Gradient Descent

Gradient descent method

Learning to Learn by Gradient Descent with Rebalancing€¦ · that, for instance, are capable of learning to learn without gradient descent by gradient descent. It should be expected

Gradient Descent Rule Tuning

Optimization based on Gradient Descent

Linear Regression & Gradient Descent

When Big Data and Machine Learning meet Partial ... · Learning Phase Gradient back-propagation aka Stochastic Gradient Descent Present the examples 1 by 1 or mini-batch by mini-batch

Overview of Stochastic Gradient Descent Algorithmscs6320/cv_files/GradientDescentOverview.… · •Mini-batch gradient descent is typically the algorithm of choice when training

Regression and Gradient Descent - GitHub Pages · 2020. 12. 20. · Linear regression with Batch Gradient Descent Repeat { (for every ) } Learning rate is typically held constant.

Stochastic Gradient Descent Methods

Learning to learn by gradient descent by gradient descent · Learning to learn by gradient descent by gradient descent Marcin Andrychowicz 1, Misha Denil , Sergio Gómez Colmenarejo

by gradient descent · Learning to learn by gradient descent by gradient descent Marcin Andrychowicz 1, Misha Denil , Sergio Gómez Colmenarejo , Matthew W. Hoffman , David Pfau 1,

Optimization, Gradient Descent, and Backpropagation