Introduction to Neural Net
description
Transcript of Introduction to Neural Net
![Page 1: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/1.jpg)
Introduction to Neural Net
CS480 Computer Science Seminar
Fall 2002
![Page 2: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/2.jpg)
Characteristics not present in von Neumann or modern parallel computers
![Page 3: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/3.jpg)
Typical tasks a neural net is capable of performing
• Pattern classification
• Clustering/categorization
• Function approximation
• prediction/forecasting
• Optimization
• Content-addressable memory
• control
![Page 4: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/4.jpg)
![Page 5: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/5.jpg)
![Page 6: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/6.jpg)
![Page 7: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/7.jpg)
![Page 8: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/8.jpg)
![Page 9: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/9.jpg)
![Page 10: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/10.jpg)
![Page 11: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/11.jpg)
![Page 12: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/12.jpg)
![Page 13: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/13.jpg)
![Page 14: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/14.jpg)
![Page 15: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/15.jpg)
![Page 16: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/16.jpg)
![Page 17: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/17.jpg)
![Page 18: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/18.jpg)
![Page 19: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/19.jpg)
![Page 20: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/20.jpg)
![Page 21: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/21.jpg)
![Page 22: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/22.jpg)
Kohonen’s Self-Organizing Map
![Page 23: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/23.jpg)
![Page 24: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/24.jpg)
![Page 25: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/25.jpg)
Neural network definition
• A machine that is designed to model the way in which the brain performs a particular task or function of interest.
• It is a massively parallel distributed processor that has a natural propensity for storing experiential knowledge and making it available for use. It resembles brain in two respects: 1. Knowledge is acquired through learning process, 2. Interneuron connection strengths known as synaptic weights are used to stored the knowledge.
![Page 26: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/26.jpg)
Some interesting facts
• Silicon gate speed: 10-9 sec• Neural event: 10-3 sec• # gates on Pentium III: < 107
• # neurons in human brain: 1010 with 60x1012
synapses or connections• Power consumption: for human brain---10-16 joules
per operations per second; for today's best computer---10-6 joules per operations per second
![Page 27: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/27.jpg)
Some interesting facts (cont’d)
• The brain is a highly complex, nonlinear, and parallel processor.
• Brain is superior in performing pattern recognition, perception, and motor control), e.g., it takes a brain 100-200 msec to recognize a familiar face embedded in an unfamiliar scene (will take days for the computer to do the similar tasks)
![Page 28: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/28.jpg)
Some interesting facts (cont’d)
• Brain has the ability to build up its own rules through “experience”over the years with most dramatic development in the first two years from birth (about 106 synapses formed per second)
![Page 29: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/29.jpg)
Neural net architecture
• Feed-forward neural net– Single layer– Multi-layer
• Recurrent neural net– There are closed loops (feedback) paths.
![Page 30: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/30.jpg)
Training the neural net
• Training: setting the weights– Supervised– Unsupervised
![Page 31: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/31.jpg)
Supervised learning• Supervised training: presenting a sequence of training
vectors or patterns, each is associated with an associated target output vector. The weight is then adjusted according to a learning algorithm.
• Examples– Pattern classification: the output is either 1 or –1 (belong or not
belong to the category)– Patten association: to associate a set of input vectors with a
corresponding set of output vectors (the neural net is called associative memory). After training, the the neural net can identify input vectors that are sufficiently close to (but not necessary the same as the input vectors).
![Page 32: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/32.jpg)
History of neural net• 1940s: McCulloch-Pitts neurons (perform
logic/combinational functions); introduced the idea of “threshold”.
• 1949: Hebb learning algorithm---if two neurons are active, the connection strength (weight) should be increased.
• 1950s and 1960s: the first golden age of neural net– Perceptrons (Rosenblatt) for pattern recognition– Adaline (adaptive linear neurons).
• 1970s: the quiet years (research funding stopped)– Kohonen, Anderson, Grossberg, and Carpenter
![Page 33: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/33.jpg)
History of neural net (cont’d)
• 1980s:– Backpropagation (of errors) learning algorithm– Hopfield nets: capable of solving constraint satisfaction
problems such as traveling salesman problem– Neocognitron: a self-organizing neural for pattern
recognition (position or rotation-distorted characters)– Boltzmann machine: non-deterministic neural net
(weights or activations are changed on the basis of a probabilistic density function)
– Hardware implementation: digital and analog implementations using using VLSI technology
![Page 34: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/34.jpg)
Neural net architecture
• Feed-forward neural net– Single layer– Multi-layer
• Recurrent neural net– There are closed loops (feedback) paths.
![Page 35: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/35.jpg)
Training the neural net
• Training: setting the weights– Supervised– Unsupervised
![Page 36: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/36.jpg)
Supervised learning• Supervised training: presenting a sequence of training
vectors or patterns, each is associated with an associated target output vector. The weight is then adjusted according to a learning algorithm.
• Examples– Pattern classification: the output is either 1 or –1 (belong or not
belong to the category)– Patten association: to associate a set of input vectors with a
corresponding set of output vectors (the neural net is called associative memory). After training, the net is the neural net can identify input vectors that are sufficiently close to (but not necessary the same as the input vecotrs)
![Page 37: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/37.jpg)
A simple example: McCulloch-Pitts neurons that perform logic operations
• Presumption:– Binary activation: neuron either fires or does not fire– Neurons are connected by directed, weighted paths– A connection path is excitatory if the weight on the path is
positive; otherwise, its inhibitory– Each neuron has a fixed threshold such that if the net input
into a neuron is greater than the threshold, the neuron fires.– The threshold is set so that inhibition is absolute, i.e., any
nonzero inhibitory input will prevent the neuron from firing– It takes one time step for a signal to pass over one
connection link
![Page 38: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/38.jpg)
McCulloch-Pitts Neuron: AND function
The threshold on y is 2
![Page 39: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/39.jpg)
McCulloch-Pitts Neuron: OR function
![Page 40: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/40.jpg)
McCulloch-Pitts Neuron: OR function
The threshold on y is 2
![Page 41: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/41.jpg)
McCulloch-Pitts Neuron: AND-NOT function
![Page 42: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/42.jpg)
McCulloch-Pitts Neuron: AND-NOT function
The threshold on y is 2
![Page 43: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/43.jpg)
McCulloch-Pitts Neural net: XOR function
The threshold on y is 2
![Page 44: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/44.jpg)
McCulloch-Pitts Neural net: XOR function
The threshold on y is 2
![Page 45: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/45.jpg)
McCulloch-Pitts Neural net application: modeling the perception of heat and cold
• Interesting physiological phenomenon: if cold stimulus is applied to a person’s skin for a very short period of time, the person will perceive heat. If the same stimulus is applied for a longer period, the person will perceive cold.
![Page 46: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/46.jpg)
McCulloch-Pitts Neural net: modeling the perception of heat and cold (time step functions are included in the modeling)
![Page 47: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/47.jpg)
A cold stimulus applied for one time step
![Page 48: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/48.jpg)
A cold stimulus applied for two time steps
![Page 49: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/49.jpg)
Another example: character recognition
![Page 50: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/50.jpg)
Noisy input
![Page 51: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/51.jpg)
Possible topics for further investigation
• Current development and applications
• Architectures
• Training algorithms
• Fuzzy logic and fuzzy-neural systems
![Page 52: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/52.jpg)
Supervised learning• Supervised training: presenting a sequence of training
vectors or patterns, each is associated with an associated target output vector. The weight is then adjusted according to a learning algorithm.
• Examples– Pattern classification: the output is either 1 or –1 (belong or not
belong to the category)– Patten association: to associate a set of input vectors with a
corresponding set of output vectors (the neural net is called associative memory). After training, the net is the neural net can identify input vectors that are sufficiently close to (but not necessary the same as the input vecotrs)
![Page 53: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/53.jpg)
McCulloch-Pitts Neuron: AND function
![Page 54: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/54.jpg)
McCulloch-Pitts Neuron: AND function
The threshold on y is 2
![Page 55: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/55.jpg)
McCulloch-Pitts Neuron: OR function
![Page 56: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/56.jpg)
McCulloch-Pitts Neuron: OR function
The threshold on y is 2
![Page 57: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/57.jpg)
McCulloch-Pitts Neuron: AND-NOT function
![Page 58: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/58.jpg)
McCulloch-Pitts Neuron: AND-NOT function
The threshold on y is 2
![Page 59: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/59.jpg)
McCulloch-Pitts Neural net: XOR function
The threshold on y is 2
![Page 60: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/60.jpg)
McCulloch-Pitts Neural net: XOR function
The threshold on y is 2
![Page 61: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/61.jpg)
McCulloch-Pitts Neural net application: modeling the perception of heat and cold
• Interesting physiological phenomenon: if cold stimulus is applied to a person’s skin for a very short period of time, the person will perceive heat. If the same stimulus is applied for a longer period, the person will perceive cold.
![Page 62: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/62.jpg)
McCulloch-Pitts Neural net: modeling the perception of heat and cold (time step functions are included in the modeling)
![Page 63: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/63.jpg)
A cold stimulus applied for one time step
![Page 64: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/64.jpg)
A cold stimulus applied for two time steps
![Page 65: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/65.jpg)
Another example: character recognition
![Page 66: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/66.jpg)
Noisy input
![Page 67: Introduction to Neural Net](https://reader033.fdocuments.in/reader033/viewer/2022061509/568148f5550346895db6159e/html5/thumbnails/67.jpg)
Possible topics for further investigation
• Current development and applications
• Architectures
• Training algorithms
• Fuzzy logic and fuzzy-neural systems