EE749Neural Networks

How to design a good performance NN?

Kasin PrakobwaitayakitDepartment of Electrical Engineering

Chiangmai University

Glossary• Pattern: a complete set of data inputs and

outputs that provide a snapshot of the system being modelled

– also called examples, cases

• Feature: an identifying characteristic in the data that the model will ideally capture

• Domain: a set of boundaries that define the range of expected / observed data for a particular model or problem

Backpropagation Modelling Heuristics• Selection of model inputs and outputs

• Defining the model domain

• Data pre-processing

• Selection of training/testing cases

• Data scaling

• Number of hidden layers

• Number of neurons

• Activation function selection

• Initial weight values

• Learning rate and momentum

• Presentation of patterns

• Stopping criteria for training

• Improving model performance

Selection of Model Inputs and Outputs• Start with a set of inputs that are KNOWN

to affect the process – then add other inputs that are suspected of

having a relationship in the process one at a time

• Eliminate input variables that are redundant (high covariance)

• Eliminate data patterns that do not contribute to training (no new information)

• Identify / eliminate data patterns with the same inputs that have different outputs

Defining the Model Domain• ANNs should be confined to a limited

domain– develop separate models for contradictory areas

of the domain

• Build models that predict a single output– link multiple models together, if required

Data Pre-processing• Any time the dynamic range of an input is

over a few orders of magnitude, a logarithmic transformation should be applied

• Transformations may also be useful in “reconditioning” input data when an ANN has trouble converging

• Non-numerical inputs need to be codified

Selection of Training/Testing Cases• Training cases should be representative of

the problem domain

• For good generalization capacity, the training set must be complete

– important variable must be measured

• Data set is randomly divided into training and testing sets

– in a 70:30 ratio, for example

• Rules of thumb– The number of training patterns should be at

least 5 times the number of nodes in the network – The number of training cases should be roughly

= the number of weights*the inverse of the accuracy parameter (e, e=0.9 means 90% accuracy of prediction is required)

Data Scaling• Output variables should be scaled in the 0.1

to 0.9 range – avoids operating in the saturation range of the

sigmoid function

Number of Hidden Layers• Increasing the number of hidden layers

increases both the time and number of patterns (examples) required for training

• For most problems, one hidden layer will suffice

– problems can always be solved with two

• Multiple slabs (Ward Nets) supposedly increase processing power

– each slab (group of neurons) acts as a detector for one or more input features

Number of Neurons• One neuron in the input layer for each input

• One neuron in the output layer for each output

• Proper number of hidden neurons is often determined experimentally

– too few = poor ability to capture features– too many = poor ability to generalize (ANN

simply memorizes training data)

• Various rules of thumb have been reported– 0.75*N– 2*N +1

N = number of inputs

• In general, more weights introduce more local minima to the error surface

• Flat regions in the error surface can mislead gradient-based error minimization methods (backpropagation)

• Start with a small network and then add connections as needed

– avoids convergence problems as the networks get too large

• The optimum ratio of hidden neurons in the first to second hidden layer is 3:1

Activation Function Selection• Sigmoidal (logistic) activation functions are

the most widely used

• Thresholding functions are only useful for categorical outputs

Initial Weight Values• Weights need to be randomized initially

– if all weights are set to the same number, the GDR would never be able to leave the starting point

• The backpropagation algorithm may also have difficulty if the connection weights are prematurely saturated (>0.9)

Learning Rate and Momentum• Learning rate () affects the speed of

convergence. – If it is large (>0.5), the weights will be changed

more drastically, but this may cause the optimum to be overshot

– If it is small (<0.2) the weights will be changed in smaller increments, thus causing the system to converge more slowly with little oscillation

• The best training occurs when the learning rate is as large as possible without leading to oscillation

• The learning rate can be increased as learning progresses, or a momentum term added to improve network learning speed

• The momentum factor () has the ability to dampen out oscillations caused by increasing the learning rate

– a momentum of 0.9 allows higher learning rates

Presenting Patterns to the ANN• Present patterns in a random fashion

• If input patterns can be easily classified, do not train the ANN on all patterns in a class in succession

– the ANN will forget information as it moves from class to class

• Shaping can be used to improve network training

– involves starting off on a very small cohesive data set and then adding more patterns that have greater deviations from variable means as training progresses

Stopping Criteria for Training• Training should be stopped when one of the

following conditions is met– the testing error is sufficiently small– the testing error begins to increase– a set number of iterations has passed

Improving Model Performance• Re-initialize network weights to a new set of

random values– re-train the model

• Adjust learning rate and momentum

• Modify stopping criteria

• Prune network weights

• Use genetic algorithms to adjust network topology

• Add noise to training cases to decrease the chance of memorization

ANN Modelling ApproachASS ES S NEE DS AND

SUITABIL ITY

CO LLEC T AND ANALY ZE DATA

APP LY M O DEL DEV ELO PM ENT

PRO TO CO L

EVALUATE PE RFO RM ANCE

SE LE CT AN D O RG ANIZE DATA

PATTE RNS

SE T ARCHITECTURE

CHARA CTE RISTIC S

EVALUATE M O DEL STABIL ITY

FIN E-TUNE M O DE L

SE LE CT INP UTS AND O UTPUTS

Needs and Suitability Assessment

• What are the modelling needs ?

• Is the ANN technique suitable to meet these needs ?

• Can the following be met ?– data requirements– software requirements– hardware requirements– personnel requirements

Data Collection and Analysis

• Successful models require careful attention to data details

• Recall : relevant historical data is a key requirement

• For data collection, investigate:– data availability

• parameters, time-frame, frequency, format

– QA/QC protocols • data reliability

– process changes

• Data requirements and guidelines:– data for each of the parameters must be

available– at least one full cycle of data must be available– appropriate QA/QC protocols must be in place – data collected prior to major process changes

should generally not be used

• Data analysis involves:– data characterization – a complete statistical analysis

• Data characterization for each parameter– qualitative assessment of hourly, daily, seasonal

trends (graphical examination of data)– time-series analyses may be warranted

• Statistical analysis for each parameter– measures of central tendency

• mean, median, mode

– measures of variability• standard deviation, variance

– percentile analyses– identification of outliers, erroneous entries, non-

entries

Application of a Model-Building Protocol

• There is no accepted “best method” of developing ANN models

• An infinite number of distinct architectures are possible

• A protocol is to reduce the number of architectures that are evaluated

• A sample five-step protocol:

ASS ES S NEE DS AND SUITABIL ITY

CO LLEC T AND ANALY ZE DATA

APP LY M O DEL DEV ELO PM ENT

PRO TO CO L

EVALUATE PE RFO RM ANCE

SE LE CT AN D O RG ANIZE DATA

PATTE RNS

SE T ARCHITECTURE

CHARA CTE RISTIC S

EVALUATE M O DEL STABIL ITY

FIN E-TUNE M O DE L

SE LE CT INP UTS AND O UTPUTS

• Selection of model inputs and outputsWhy it’s important: – ANN models are based on process inputs and

outputs

How it’s done:– first, select the model output

• best models only have one output parameter

– next, select model inputs from available input parameters

– selection is based on data availability, literature, expert knowledge

• Selection and organization of data patternsWhy it’s important: – ANN models are only as good as the data used– separate independent data sets are required to test

and validate the model

How it’s done: – examine each data pattern for erroneous entries,

outliers, blank entries– delete questionable data patterns– sort and divide data into training, testing, and

production sets– perform a statistical analysis on each of the three

data sets

• Determination of architecture characteristicsWhy it’s important: – each modeling scenario will have an optimal

architecture

How it’s done: – initially, hold many factors at software defaults

or at pre-determined values• use modelling heuristics from literature along with

expert knowledge

– determine the number of hidden layer neurons• compare results for different runs

• Evaluation of model stabilityWhy it’s important: – ensure that model results are independent of the

method of data sorting

How it’s done: – build new training, testing, and production sets

from the original database – re-train models on the “new” data sets – compare results with initial runs

• Model fine tuningWhy it’s important:– some models require minor improvements in

order to meet process operating criteria

How it’s done: – modeling parameters previously held constant

can be varied to improve model results– the model fine-tuning methodology is typically

researcher-specific

• In many situations, more than one good model can be developed

• The best model is the one that both– meets the modelling needs initially identified– offers the smallest prediction errors

• Therefore need to be able to evaluate model performance

Evaluating Model Performance

• Prediction errors can be assessed– graphically

• visual representation of missed predictions

0

1

2

3

4

5

6

71 11 21 31 41 51 61 71 81 91 101

111

121

131

141

151

161

171

181

Pattern #

Cla

rifi

er E

fflu

ent

Col

our

(TC

U)

Actual Network

– using statistics• absolute measures of error

– mean absolute error (MAE)

– maximum absolute error

• relative measures of error

– mean absolute percent error (MAPE)

• coefficients of correlation

– coefficient of correlation (r)

– coefficient of multiple correlation (R)

• coefficients of determination

– coefficient of determination (r2)

– coefficient of multiple determination (R2)

• Model residuals should also be studied– residual = prediction error– residuals should be

• Normally distributed

– plot a histogram of the residuals

• have a mean = 0

• independent

– plot the residuals (in time order if applicable)

– plot should be free of obvious trends

• have constant variance

– plot the residuals against the predicted values

– plot should not show spreading, converging, or other trends

Model Evaluation Using Real-time Data

• Need to consider:– changes in the frequency of data collection– changes in the methodology of data collection– existence of QA/QC protocols to detect

erroneous data

• The evaluation can take the form of– simulated real-time testing on a stand-alone PC– online testing in real-time

• Methodology– select the time frame of the test– the data is ported to the developed models– process each data pattern and record the results– the model predicted values are compared to the

actual values– prediction errors are determined