Reactive Behavior Modeling Neural Networks (GATE-561)
description
Transcript of Reactive Behavior Modeling Neural Networks (GATE-561)
1GATE-561
Reactive Behavior ModelingNeural Networks
(GATE-561)
Dr.Çağatay ÜNDEĞER
InstructorMiddle East Technical University, GameTechnologies
Bilkent University, Computer Engineering
&
General ManagerSimBT Inc.
e-mail : [email protected]
Game Technologies Program – Middle East Technical University – Fall 2009
2GATE-561
Outline• Neural Networks
3GATE-561
Artificial Neural Networks (NNets)
• Inspired by human brain.• Fundamental functional units of brain are:
– Neurons / soma / nerve cells and– Axons / dendrites connecting nerve cells.
4GATE-561
Artificial Neural Networks (NNets)
• Signals can be thought as simple electrical impulses from axons to dendrites.
• Dendrites collect the signals.• Soma performs a kind of summation.• Axon fire and transmit the signal.
5GATE-561
Artificial Neural Networks (NNets)
• Neural networks use a similar approach.• Consist of:
– Input arcs collect the signals,– Neuron sums the signals and– Output arcs transmit the summed signal based on
a transmit function.
• Neural networks can be learned.
6GATE-561
Neural Network Architecture
i
i
i
h
h
h
h
h
h
o
o
h h
input layer hidden layers output layer
input 1
input 2
input 3
output 1
output 2
Arcs (storing weights) Neurons
7GATE-561
Usage of NNets
• Environmental Scanning and Classification:– Interpret vision and auditory information
• Memory:– Can learn through experience
• Behavior Control:– Outputs can be used as actions to be
performed.• Response Mapping:
– Any input output mapping and association.
8GATE-561
Usage of NNets
• The usage of NNets is a little bit fuzzy.• No exact cases and rules.• Assume it as a tool and use your creativity to
get whatever you like.
• What NNets do is– To seperate a solution space into partitions
9GATE-561
A Neuron Architecture
Y
X1
X2
Xi
Xn
w1
w2
wi
wn
Xi = input i of neuron
wi = weight of input i
Y = summing junction
y = output of neuron
yto input of other
neurons
y is based on a transfer or activation function
B=1
b
B = a constant bias node, which may represent past history
10GATE-561
Neuron Architecture
Y
X1
X2
Xi
Xn
w1
w2
wi
wn
y
B=1
b
Ya = B*b + Σ Xi*wii=1
n
Summing is referred as the input activation Ya
Input activation is fed into the activation function fa(x)
and output y is acquired
y = fa(x) Inputs and weights can be in
range (-∞, +∞)
11GATE-561
Some Activation Functions
1
0 x
y
θ
Step function
fs(x) = 1, if x≥θ0, if x<θ
1
0 x
y
Linear function
fs(x) = x
12GATE-561
Common Activation Function
1
0 x
y
Exponential / Sigmoid function
fs(x) = 1 / ( 1 + e-θx )
θ = 1
You can also scale and shift the output if required
13GATE-561
A Group Of Neurons
• A single neuron will not do a lot for us• We need a group of them
A single layer NNet A two layer NNet
14GATE-561
Simple Logic Functions
• Created by McCulloch and Pitts in 1943.• AND, OR, XOR functions could be done with
simple neural network structures.
15GATE-561
Simple Logic Function
AND
OR
XOR
16GATE-561
Learning NNets
• Weights of NNets can be learned by various algorithms.
• The most common algorşthm is the Backpropagation.
17GATE-561
Backpropagation
• Can learn weights of a multi-layer network – With fixed set of units and interconnections
• Employs a gradient descent search– To minimize the squared errors between
target output and network output• It is sub-optimal and may fail to reach true
optimal point.
18GATE-561
Backpropagation
• Can learn weights of a multi-layer network – With fixed set of units and interconnections
• Employs a gradient descent search– To minimize the squared errors between
target output and network output
19GATE-561
Backpropagation
• Error surface of the network may have multiple local minima.
• Therefore backprogation guarantees only to converge a local-optimum and may fail to reach the true-optimum.
20GATE-561
Backpropagation
01
02
03
11
12
13
21
22
14
input 1
input 2
input 3
output 1
output 2
Initialize the network weights with small random numbers (e.g. -0.5 ... 0.5) Train the network with m training examples repeatedly for multiple iterations.Stop training when a convergence criteria is met.
21GATE-561
Backpropagation
01
02
03
11
12
13
21
22
14
o1
o2
For one training example, assume that:Inputs are x01, x02 and x03 andExpected outputs are t1 and t2
We propogate the inputs through the network and Compute the actual outputs o1, o2 and o3
t1
t2
expectedactualx01
x02
x03
22GATE-561
Backpropagation11
12
13
21
22
14
o1
o2
Errors will occur between actual and expected outputs.We propagate errors backward through the network.
Compute error on each output unit k ( δk ) using:
δk = ok (1 - ok) (tk - ok)
t1
t2
expectedactual
δ21
δ22
output errors
e.g. δ21 = o21 (1 - o21) (t21 - o21)
x01
x02
x03
01
02
03
23GATE-561
Backpropagation11
12
13
21
22
14
o1
o2
Compute error on each hidden unit h (δh) using:
δh = oh (1 - oh) Σ wh-k δk
t1
t2
expectedactual
δ21
δ22
output errors
w11-21
w11-22
e.g. δ11 = o11 (1 - o11) (w11-21 δ21 + w11-22 δ22 )
δ11
δ12
δ13
δ14
hidden unit errors
x01
x02
x03
01
02
03
24GATE-561
Backpropagation11
12
13
21
22
14
o1
o2
Update each weight w (wf-t) using:
wf-t = wf-t + η δt xf-t
t1
t2
expectedactual
δ21
δ22
output errors
w11-21
w11-22
δ11
δ12
δ13
δ14
hidden unit errors
w01-11
w01-12x01
x02
x03
η = learning rate (e.g. 0.05)e.g. w01-11 = w01-11 + η δ11 x01-11
x01
01
02
03
25GATE-561
Backpropagation
• The main loop of the algorithm should repeatedly iterate over the training examples
• May iterate thousands of times using the same training examples.
• Until the network performs acceptably well:– Test with the entire training set or– A better way: test with a sparate test set.
26GATE-561
An Action Selector Neural Network
• The inputs are state variables that equal to either 0 (no) or 1 (yes).
• Outputs are action variables that are between 0 and 1.
• The action with the greatest output is selected.
Enemy
Sound
Dead
Low Health
Attack
Retreat
Wander
Search
Spawn
27GATE-561
A Number Recognizer Neural Network
• Problem : To recognize normalized bitmaps of handwritten digits from “0” to “9”.
• Data Set : Collected from 43 people– 30 people to training set– 13 people to test set– 32x32 bitmaps are divided into 4x4 blocks and
black pixels are counted– So we have 8x8 matrices and each member
stores a number between 0 and 16
28GATE-561
A Number Recognizer Neural Network
• 8x8 pixels = 64 inputs• No Missing Input Value• Target Value : class number
from 0 to 9
• Training Set : 3823 Instances• Test Set : 1797 Instances
29GATE-561
A Number Recognizer Neural Network
x011 = 1x111 1
x0641 = 1x1641 64
x012 = 1
x112 1
n
x212
x6412
x013 = 1
x113 1
10
x213
xn13
“0”
“9”
.
.
.
.
.
Max
Bias Control Unit
64 Pixel values (0..16) of 8x8 image
.
.
.
.
.
.
.
.
.
.
.
.
Result
30GATE-561
A Number Recognizer Neural Network
• For each optic character (0..9), an output unit is used
• The class of the output unit having the highest value is chosen as result.
• Sample target values for training:
‘0’ 0.9 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1
‘1’ 0.1 0.9 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1
‘2’ 0.1 0.1 0.9 0.1 0.1 0.1 0.1 0.1 0.1 0.1
31GATE-561
A Number Recognizer Neural Network
• Random weights are given (–0.5 ... 0.5)• A fixed number of iterations are performed on
Training Set– Meanwhile weights of iterations are saved
• A test phase is performed using saved weights on Test Set to avoid overfitting
• The iteration weights that maximizes the test set accuracy is chosen
32GATE-561
Test Platform
General InformationNew InstanceReal Value
Result of NNet
Output layer of NNet
Loaded Instance Set
Current instance
33GATE-561
First Attempt
• 4 hidden units • 0.1 as learning rate• 5000 iterations are performed (>12 hours)• Best on Test Set = 86.9% on 450th iteration
– Meanwhile 92.2% on Training Set– 82.5% accuracy on 5000th iteration
• Best on Training Set = 93.0% on 1110th iteration– Meanwhile 86.1% on Test Set
34GATE-561
Graph of First Attempt
35GATE-561
Second Attempt
• 4 hidden units to 8• 0.1 as learning rate (same)• 5000 iterations to 2500• Best on Test Set = 94.0% on 100th iteration
– Meanwhile 97.7% on Training Set– 88.9% accuracy on 2500th iteration
• Best on Training Set = 98.0% on 140th iteration– Meanwhile 93.6% on Test Set
36GATE-561
Graph of Second Attempt
37GATE-561
Third Attempt
• 8 hidden units to 16• 0.1 as learning rate (same)• 2500 iterations to 200• Best on Test Set = 95.2% on 176th iteration
– Meanwhile 98.7% on Training Set– 95.2% accuracy on 200th iteration
• Best on Training Set = 98.7% on 176th iteration– Meanwhile 95.2% on Test Set
38GATE-561
Graph of Third Attempt
39GATE-561
Fourth Attempt
• 16 hidden units (same)• 0.1 as learning rate to 0.05• 200 iterations to 400• Best on Test Set = 95.0% on 122th iteration
– Meanwhile 98.4% on Training Set– 94.4% accuracy on 400th iteration
• Best on Training Set = 98.7% on 394th iteration– Meanwhile 94.4% on Test Set
40GATE-561
Graph of Fourth Attempt
41GATE-561
Last Attempt
• 16 hidden units to 32• 0.05 as learning rate (same)• 400 iterations to 1000 (4 hours and 38 minutes)• Best on Test Set = 96.4% on 315th iteration
– Meanwhile 99.5% on Training Set– 96.0% accuracy on 1000th iteration
• Best on Training Set = 99.9% on 945th iteration– Meanwhile 96.1% on Test Set
42GATE-561
Graph of Last Attempt
43GATE-561
Advantages
• Handle errors well.
• Graceful degradation.
• Can learn novel solutions.
44GATE-561
Disadvantages
• Can’t understand how the learned network works, therefore they are the second best way to do something.
• Need as many examples as possible.
• Learning takes too much time / processing.
• Sometimes the network may not converge.