Neural Networks in Electrical Engineering Prof. Howard Silver School of Computer Sciences and...
-
date post
21-Dec-2015 -
Category
Documents
-
view
216 -
download
0
Transcript of Neural Networks in Electrical Engineering Prof. Howard Silver School of Computer Sciences and...
Neural Networks in Electrical Engineering
Prof. Howard SilverSchool of Computer Sciences and Engineering
Fairleigh Dickinson UniversityTeaneck, New Jersey
Axon fromAnother Neuron
Axon fromAnother Neuron
Synaptic Gap
Synaptic Gap
Soma
Axon
Dendrite
Dendrite ofAnother Neuron
Dendrite ofAnother Neuron
Biological Neuron
Steps in applying Perceptron:
! Initialize the weights and bias to 0.! Set the learning rate alpha (0 < α <= 1) and threshold θ.! For each input pattern,!__Compute for each output ! yin = b + x1 * w1 + x2 * w2 + x3 * w3 + ...... ! and set !__y = ‑1 for yin < ‑θ !__y = 0 for -θ<= yin <= θ!__y = 1 for yin > θ!__If the jth output yj is not equal to tj, set!____wij(new) = wij(old) + α * xi * tj
!____bj(new) = bj(old) + α * tj
!__(else no change in wij and bj)
Example of Supervised Learning Algorithm - Perceptron
Perceptron Applied to Character Recognition
Character Inputs ("A") (Binary)..#.. 00100.#.#. 01010#...# 10001##### 11111#...# 10001
'0010001010100011111110001'
'10000000000000000000000000'
Neural net inputs x1 to x25
Binary target output string (t1 to t26)
EDU» a_zlearnCharacter training set:..#.. ##### ##### ###.. ##### ##### ##### #...# ##### ....# #...# #.... #...# .#.#. #...# #.... #..#. #.... #.... #.... #...# ..#.. ....# #..#. #.... ##.## #...# ####. #.... #...# ##### ####. #..## ##### ..#.. ....# ###.. #.... #.#.# ##### #...# #.... #..#. #.... #.... #...# #...# ..#.. #...# #..#. #.... #...# #...# ##### ##### ###.. ##### #.... ##### #...# ##### ##### #...# ##### #...# #...# ##### ##### ##### ##### ##### ##### #...# #...# #...# #...# #...# ##### ##..# #...# #...# #...# #...# #.... ..#.. #...# #...# #...# .#.#. .#.#. ...#. #.#.# #...# ##### #.#.# ##### ##### ..#.. #...# #...# #...# ..#.. ..#.. ..#.. #..## #...# #.... #..## #..#. ....# ..#.. #...# .#.#. #.#.# .#.#. ..#.. .#... #...# ##### #.... ##### #...# ##### ..#.. ##### ..#.. .#.#. #...# ..#.. #####
Enter number of training epochs (m) 5Enter learning rate (alpha) 1Enter threshold value (theta) 0.1
Character training set:..#.. ##### ##### ###.. ##### ##### ##### #...# ##### ....# #...# #.... #...# .#.#. #...# #.... #..#. #.... #.... #.... #...# ..#.. ....# #..#. #.... ##.## #...# ####. #.... #...# ##### ####. #..## ##### ..#.. ....# ###.. #.... #.#.# ##### #...# #.... #..#. #.... #.... #...# #...# ..#.. #...# #..#. #.... #...# #...# ##### ##### ###.. ##### #.... ##### #...# ##### ##### #...# ##### #...# #...# ##### ##### ##### ##### ##### ##### #...# #...# #...# #...# #...# ##### ##..# #...# #...# #...# #...# #.... ..#.. #...# #...# #...# .#.#. .#.#. ...#. #.#.# #...# ##### #.#.# ##### ##### ..#.. #...# #...# #...# ..#.. ..#.. ..#.. #..## #...# #.... #..## #..#. ....# ..#.. #...# .#.#. #.#.# .#.#. ..#.. .#... #...# ##### #.... ##### #...# ##### ..#.. ##### ..#.. .#.#. #...# ..#.. #####
Final outputs after training:ABCDEFGHIJKLMNOPQRSTUVWXYZ10000000000000000000000000 A01000000000000000000100000 B00000000000000000000000000 C00010000000000000000000000 D00001000000000000000000000 E00100100000000000000000000 F000000?0000000100000000000 G00000001000000000000000000 H00000000100000000000000000 I00000000010000000000000000 J00000000001000000000000000 K00000010000100000000000000 L00000000000010000000000000 M00000000000001000000000000 N00?00000000000000000000000 O00000000000000010000000000 P00000000000000001000000000 Q00000000000000000100000000 R00000000000000000010000000 S00000000000000000001000000 T00000000000000000000000000 U00000000000000000000010000 V00000000000000100000001000 W00000000000000000000000100 X00000000000000000000000010 Y00000000000000?00000000001 Z
Enter a test pattern (1s and 0s in quotes)'1010001010100011111110001'Character entered:#.#...#.#.#...#######...#Resulting outputs:ABCDEFGHIJKLMNOPQRSTUVWXYZ10000000000000000000000000Sorted outputs before activation (strongest first):AWRODVYUSLKIHJGFECZXTMNPQB
Enter a test pattern (1s and 0s in quotes)'1010101010100011111110001'Character entered:#.#.#.#.#.#...#######...#Resulting outputs:ABCDEFGHIJKLMNOPQRSTUVWXYZ?0000000000000000000000000Sorted outputs before activation (strongest first):AWRVSODYUKGIHXJECMLFZTNQBP
Enter a test pattern (1s and 0s in quotes)'0010001010111111000110001'Character entered:..#...#.#.######...##...#Resulting outputs:ABCDEFGHIJKLMNOPQRSTUVWXYZ00000?0100101?010010000010Sorted outputs before activation (strongest first):YHSPMKNFEWUARODVLJZXBGTQIC
SN = 5
Signal Classification with Perceptron
SN = 5
SN=5
NNET11H.M
EDU» nnet11hEnter number of training epochs (m) 20Enter learning rate (alpha) 1Enter threshold value (theta) 0.2Final outputs after training:100011000:EDU» nnet11hEnter number of training epochs (m) 30Enter learning rate (alpha) 1Enter threshold value (theta) 0.2Final outputs after training:101010000:
EDU» nnet11hEnter number of training epochs (m) 40Enter learning rate (alpha) 1Enter threshold value (theta) 0.2Final outputs after training:100010001Enter signal to noise ratio 100Classification of signals embedded in noise100010001:Enter signal to noise ratio 10Classification of signals embedded in noise100010001:
Enter signal to noise ratio 5Classification of signals embedded in noise100010000:Enter signal to noise ratio 2Classification of signals embedded in noise100010000:Enter signal to noise ratio 1Classification of signals embedded in noise100010010:Enter signal to noise ratio 1Classification of signals embedded in noise100011001
>> nnet11iEnter frequency separation in pct. (del) 100Number of samples per cycle (xtot) 64Enter number of training epochs (m) 100 Final outputs after training: 100010001
Enter signal to noise ratio (SN) - zero to exit 1 Classification of signals embedded in noise 100010001
Classification of Three Sinusoids of Different Frequency
Signals Noisy Signals
>> nnet11iEnter frequency separation in pct. (del) 10Number of samples per cycle (xtot) 64Enter number of training epochs (m) 100 Final outputs after training: 100010001
Enter signal to noise ratio (SN) - zero to exit 10 Classification of signals embedded in noise 100010001
Signals Noisy Signals
Signals Noisy Signals
Enter signal to noise ratio (SN) - zero to exit 10 Classification of signals embedded in noise 0?0010001
>> nnet11iEnter frequency separation in pct. (del) 5Number of samples per cycle (xtot) 64Enter number of training epochs (m) 500 Final outputs after training: 100010001
Signals Noisy Signals
>> nnet11iEnter frequency separation in pct. (del) 1Number of samples per cycle (xtot) 1000Enter number of training epochs (m) 10000 Final outputs after training: 100010001
Enter signal to noise ratio (SN) - zero to exit 100 Classification of signals embedded in noise 1000?0001
K o ho n e n L e arn in g a nd Se lf O rg a n iz in g M a psL ine a r a rra y : # (R=0) * # * (R=1) * * # * * (R=2)* * * # * * * (R=3)R e c ta n g u lar g rid : # (R=0)
* * * * # * (R=1) * * *
* * * * * * * * * * * * # * * (R=2) * * * * * * * * * *
Unsupervised Learning
Initialize the weights (e.g. random values).
Set the neighborhood radius (R) and a learning rate (α).
Repeat the steps below until convergence or a maximum number of epochs is reached.
For each input pattern X = [x1 x2 x3 ......]
Compute a "distance"
D(j) = (w1j ‑ x1)2 + (w2j ‑ x2)2 + (w3j ‑ x3)2 + ...... for each cluster (i.e. all j), and find jmin, the value of j corresponding to the minimum D(j).
If j is "in the neighborhood of" jmin, wij(new) = wij(old) + α [xi ‑ wij(old)] for all i.
Decrease α (linearly or geometrically) and reduce R (at a specified rate) if R > 0.
Kohonen Learning Steps
S e lf O rg a n iz ing M ap s fo r A lp ha b e tic C ha rac te r S e t
E xa m p le 4.5 in Fa u se tt
C ha ra c te r tra in in g se t :..##... ######. ..####. #####.. ####### ...#### ###..## ...#... .#....# .#....# .#...#. .#....# .....#. .#..#.. ...#... .#....# #...... .#....# .#..... .....#. .#.#... ..#.#.. .#....# #...... .#....# .#.#... .....#. .##.... ..#.#.. .#####. #...... .#....# .###... .....#. .##.... .#####. .#....# #...... .#....# .#.#... .....#. .#.#... .#...#. .#....# #...... .#....# .#..... .#...#. .#..#.. .#...#. .#....# .#....# .#...#. .#....# .#...#. .#...#. ###.### ######. ..####. #####.. ####### ..###.. ###..## A1 B1 C1 D1 E1 J1 K1
...#... ######. ..###.. #####.. ####### .....#. #....#.
...#... #.....# .#...#. #....#. #...... .....#. #...#..
...#... #.....# #.....# #.....# #...... .....#. #..#...
..#.#.. #.....# #...... #.....# #...... .....#. #.#....
..#.#.. ######. #...... #.....# #####.. .....#. ##.....
.#...#. #.....# #...... #.....# #...... .....#. #.#....
.#####. #.....# #.....# #.....# #...... .#...#. #..#...
.#...#. #.....# .#...#. #....#. #...... .#...#. #...#..
.#...#. ######. ..###.. #####.. ####### ..###.. #....#. A2 B2 C2 D2 E2 J2 K2
...#... ######. ..###.# #####.. ####### ....### ###..##
...#... .#....# .#...## .#...#. .#....# .....#. .#...#.
..#.#.. .#....# #.....# .#....# .#..#.. .....#. .#..#..
..#.#.. .#####. #...... .#....# .####.. .....#. .#.#...
.#...#. .#....# #...... .#....# .#..#.. .....#. .##....
.#####. .#....# #...... .#....# .#..... .....#. .#.#... #.....# .#....# #.....# .#....# .#..... .....#. .#..#.. #.....# .#....# .#...#. .#...#. .#....# .#...#. .#...#. ##...## ######. ..###.. #####.. ####### ..###.. ###..## A3 B3 C3 D3 E3 J3 K3
###.### ######. ..####. #####.. ####### ..###.. ###..## A1 B1 C1 D1 E1 J1 K1
.#...#. ######. ..###.. #####.. ####### ..###.. #....#. A2 B2 C2 D2 E2 J2 K2
##...## ######. ..###.. #####.. ####### ..###.. ###..## A3 B3 C3 D3 E3 J3 K3
NNET19.M
T he u n its a ssoc ia te d w ith e a c h p a tte rn
(a s sho w n in Fa u se tt fo r E x a m p les 4 .5 a n d 4 .6 ).(F o r in it ia l R = 0 )Unit Patterns2 B1, B3, D1, D3, E1, E3, K1, K311 A1, A2, A314 C1, C2, C3, J1, J2, J325 B2, D2, E2, K2(F o r in it ia l R = 1 )Unit Patterns2 C14 C2, C36 J1, J2, J38 D1, D39 B1, B310 E111 E312 K1, K314 K216 D217 B2, E219 A120 A221 A3
N N E T 1 9 A .M
S a m e c h a rac te r p a tte rn s are o rga n iz e d into a tw o -d im e ns io na l re c ta ng u la r a rra y o f c lus t e r u n its .
A sa m p le o utp u t o f t h is pro g ra m a fte r 1 0 0 e p oc h sR o w : 3 3 4 1 5 1 5 5 5 1 5 1 1 5 1 3 3 3 1 4 1C o lum n : 2 2 1 4 3 4 1 5 5 5 4 5 3 2 3 4 4 5 2 2 2
Column 1 2 3 4 5 1 K1,K3 E1,E3 B1,B3 D1,D3 2Row 3 A1,A2 J1,J2 J3 4 A3 K2 5 C1 E2 B2 D2 C2,C3
Two input neurons ‑ one each for the x and y coordinate numbers for the cities
The Traveling Salesman Problem
Distance function:
D(j) = (w1j ‑ x1)2 + (w2j ‑ x2)2
Example - 4 cities on the vertices of a square
! C ity A : -1 , -1! C ity B : -1 , 1! C ity C : 1 , -1! C ity D : 1 , 1
N N E T 1 9C .M
The coordinates of the cities:
Weight matrices from three different runs:
W = [ 1 1 ‑1 ‑1 1 ‑1 ‑1 1]
W = [ 1 ‑1 ‑1 1 1 1 ‑1 ‑1]
W = [‑1 1 1 ‑1 ‑1 ‑1 1 1]
100 “Randomly” Located Cities
0.1 < α < 0.25, 100 epochs, Initial R = 2