1 Artificial Neural Networks Sanun Srisuk 42973003 [email protected] EECP0720 Expert Systems –...
-
Upload
clyde-stokes -
Category
Documents
-
view
224 -
download
0
Transcript of 1 Artificial Neural Networks Sanun Srisuk 42973003 [email protected] EECP0720 Expert Systems –...
1
Artificial Neural Networks
Sanun Srisuk 42973003
EECP0720 Expert Systems – Artificial Neural Networks
2
Introduction
Artificial neural networks (ANNs) provide a general, practical method for learning real-valued, discrete-valued, and vector-valued functions from examples. Algorithms such as BACKPROPAGATION use gradient descent to tune network parameters to best fit a training set of input-output pairs. ANN learning is robust to errors in the training data and has been successfully applied to problems such as face recognition/detection, speech recognition, and learning robot control strategies.
EECP0720 Expert Systems – Artificial Neural Networks
3
Autonomous Vehicle SteeringEECP0720 Expert Systems – Artificial Neural Networks
4
Characteristics of ANNs
Instances are represented by many attribute-value pairs.
The target function output may be discrete-valued, real-valued, or a vector of several real- or discrete-valued attributes.
The training examples may contain errors.
Long training times are acceptable.
Fast evaluation of the learned target function may be required.
The ability of humans to understand the learned target function is not important.
EECP0720 Expert Systems – Artificial Neural Networks
5
One type of ANN system is based on a unit called a perceptron.
The perceptron function can sometimes be written as
The space H of candidate hypotheses considered in perceptron learning is the set of all possible real-valued weight vectors.
PerceptronsEECP0720 Expert Systems – Artificial Neural Networks
6
Representational Power of Perceptrons
EECP0720 Expert Systems – Artificial Neural Networks
7
Decision surfaceEECP0720 Expert Systems – Artificial Neural Networks
linear decision surface nonlinear decision surface
Programming Example of Decision Surface
8
The Perceptron Training RuleOne way to learn an acceptable weight vector is to begin with random weights, then iteratively apply the perceptron to each training example, modifying the perceptron weights whenever it misclassifies an example. This process is repeated, iterating through the training examples as many times as needed until the perceptron classifies all training examples correctly. Weights are modified at each step according to the perceptron training rule, which revises the weight associated with input according to the rule
EECP0720 Expert Systems – Artificial Neural Networks
9
Gradient Descent and Delta Rule
The delta training rule is best understood by considering the task of training an unthresholded perceptron; that is, a linear unit for which the output o is given by
In order to derive a weight learning rule for linear units, let us begin by specifying a measure for the training error of a hypothesis (weight vector), relative to the training examples.
EECP0720 Expert Systems – Artificial Neural Networks
10
Visualizing the Hypothesis SpaceEECP0720 Expert Systems – Artificial Neural Networks
minimum error
initial weight vector by random
11
Derivation of the Gradient Descent Rule
The vector derivative is called the gradient of E with respect to , written
The gradient specifies the direction that produces the steepest increase in E. The negative of this vector therefore gives the direction of steepest decrease. The training rule for gradient descent is
EECP0720 Expert Systems – Artificial Neural Networks
12
Derivation of the Gradient Descent Rule (cont.)
The negative sign is presented because we want to move the weight vector in the direction that decreases E. This training rule can also written in its component form
which makes it clear that steepest descent is achieved by altering each component of in proportion to .
EECP0720 Expert Systems – Artificial Neural Networks
13
The vector of derivatives that form the gradient can be obtained by differentiating E
EECP0720 Expert Systems – Artificial Neural Networks
Derivation of the Gradient Descent Rule (cont.)
The weight update rule for standard gradient descent can be summarized as
14
Stochastic Approximation to Gradient DescentEECP0720 Expert Systems – Artificial Neural Networks
15
Summary of Perceptron
Perceptron training rule guaranteed to succeed if
training examples are linearly separable
sufficiently small learning rate
Linear unit training rule uses gradient descent
guaranteed to converge to hypothesis with minimum squared error
given sufficiently small learning rate
even when training data contains noise
EECP0720 Expert Systems – Artificial Neural Networks
16
BACKPROPAGATION Algorithm
EECP0720 Expert Systems – Artificial Neural Networks
17
Error Function
The Backpropagation algorithm learns the weights for a multilayer network, given a network with a fixed set of units and interconnections. It employs gradient descent to attempt to minimize the squared error between the network output values and the target values for those outputs. We begin by redefining E to sum the errors over all of the network output units
where outputs is the set of output units in the network, and tkd and okd are the target and output values associated with the kth output unit and training example d.
EECP0720 Expert Systems – Artificial Neural Networks
18
Architecture of Backpropagation EECP0720 Expert Systems – Artificial Neural Networks
19
Backpropagation Learning Algorithm
EECP0720 Expert Systems – Artificial Neural Networks
20
Backpropagation Learning Algorithm (cont.)
EECP0720 Expert Systems – Artificial Neural Networks
21
Backpropagation Learning Algorithm (cont.)
EECP0720 Expert Systems – Artificial Neural Networks
22
Backpropagation Learning Algorithm (cont.)
EECP0720 Expert Systems – Artificial Neural Networks
23
Backpropagation Learning Algorithm (cont.)
EECP0720 Expert Systems – Artificial Neural Networks
24
Face Detection using Neural NetworksEECP0720 Expert Systems – Artificial Neural Networks
Neural
Network
Face Database
Non-Face Database
Training ProcessOutput=1, for face database
Output=0, for non-face database
Face
orNon-
Face?
Test
ing P
roc e
ss
25
End of Presentation
EECP0720 Expert Systems – Artificial Neural Networks
26
Derivation of Backpropagation
EECP0720 Expert Systems – Artificial Neural Networks
27
Derivation of Backpropagation (cont.)
EECP0720 Expert Systems – Artificial Neural Networks