# Artificial Neural Network

date post

24-Feb-2016Category

## Documents

view

68download

0

Embed Size (px)

description

### Transcript of Artificial Neural Network

Neural Network

Artificial Neural NetworkYalong Li

Some slides are from http://www.cs.cmu.edu/~tom/10701_sp11/slides/NNets-701-3_24_2011_ann.pdfStructureMotivationArtificial neural networksLearning: Backpropagation AlgorithmOverfittingExpressive Capabilities of ANNsSummarySome facts about our brainPerformance tends to degrade gracefully under partial damageLearn (reorganize itself) from experienceRecovery from damageis possiblePerforms massively parallel computations extremely efficientlyFor example, complex visual perception occurs within less than 100 ms, that is, 10 processing steps!(processing speed of synapses about 100hz)

Supports our intelligence and self-awarenessNeural Networks in the BrainCortex, midbrain, brainstem and cerebellum

Visual System10 or 11 processing stages have been identifiedfeedforwardearlier processing stages (near the sensory input) to later ones (near the motor output)feedback

Neurons and Synapses Basic computational unit in the nervous system is the nerve cell, orneuron.

Synaptic LearningOne way brain learn is by altering the strengths of connections between neurons, and by adding or deleting connections between neurons LTP(long-term potentiation)Long-Term Potentiation:An enduring (>1 hour) increase in synaptic efficacy that results from high-frequency stimulation of an afferent (input) pathway

The efficacy of a synapse can change as a result of experience, providing both memory and learning throughlong-term potentiation. One way this happens is through release of more neurotransmitter.

Hebbs Postulate:"When an axon of cell A... excites[s] cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells so that A's efficiency as one of the cells firing B is increased.

Points to note about LTP:Synapses become more or less important over time (plasticity)LTP is based on experienceLTP is based only onlocalinformation (Hebb's postulate)

??StructureMotivationArtificial neural networksBackpropagation AlgorithmOverfittingExpressive Capabilities of ANNsSummary

Multilayer Networks of Sigmoid Units

Multilayer Networks of Sigmoid Units

Connectionist ModelsConsider humans:Neuron switching time ~.001 secondNumber of neurons ~1010fsConnections per neuron ~104-5Scene recognition time ~.1 second100 inference steps doesnt seem like enough Much parallel compution

Properties of artificial neural nets(ANNs):Many neuron-like threshold switching unitsMany weighted interconnections among unitsHighly parallel, distributed process

StructureMotivationArtificial neural networksLearning: Backpropagation AlgorithmOverfittingExpressive Capabilities of ANNsSummaryBackpropagation AlgorithmLooks for the minium of the error function in weight space using the method of gradient descent.The combination of weights which minimizes the error functionis considered to be a solution of the learning problem.

Sigmoid unit

Error Gradient for a Sigmoid Unit

Gradient Descent

Incremental(Stochastic) Gradient Descent

Backpropagation Algorithm(MLE)

Backpropagation Algorithm(MLE)Derivation of the BP rule:

Goal:Error:Notation:

Backpropagation Algorithm(MLE)For ouput unit j:

Backpropagation Algorithm(MLE)For hidden unit j:

More on Backpropagation

StructureMotivationArtificial neural networksLearning: Backpropagation AlgorithmOverfittingExpressive Capabilities of ANNsSummaryOverfitting in ANNs

Dealing with Overfitting

Dealing with Overfitting

K-Fold Cross Validation

Leave-Out-One Cross Validation

StructureMotivationArtificial neural networksBackpropagation AlgorithmOverfittingExpressive Capabilities of ANNsSummaryExpressive Capabilities of ANNsSingle Layer: Preceptron

XOR problem

8-3-8 problemSingle Layer: Perceptron

Single Layer: PerceptronRepresentational Power of Perceptrons hyperplane decision surface in the n-dimensional space of instances wx = 0

Linear separable setsLogical: and, or,

How to learn w ?

Single Layer: PerceptronNonliear sets of examples?

Multi-layer perceptron, XOR

Multi-layer perceptron

Expressive Capabilities of ANNs

Leaning Hidden Layer Representations8-3-8 problem

Leaning Hidden Layer Representations8-3-8 problem

Leaning Hidden Layer Representations8-3-8 problem

Auto Encoder?Training

Training

Training

Neural Nets for Face Recognition

Leaning Hidden Layer Representations

Leaning Hidden Layer Representations

StructureMotivationArtificial neural networksBackpropagation AlgorithmOverfittingExpressive Capabilities of ANNsSummarySummary

Thank you!