9. Lecture Neural Networks · Preliminaries • Neural networks can model any non-linear relations...
Transcript of 9. Lecture Neural Networks · Preliminaries • Neural networks can model any non-linear relations...
9. Lecture
Neural Networks
Application in Automation
Engineering
Soft Control
(AT 3, RMA)
SC
234WS 18/19 Georg Frey
Outline of the lecture
1. Introduction to Soft Control: definition and limitations, basics of
"smart" systems
2. Knowledge representation and knowledge processing (Symbolic AI)
Application: expert systems
3. Fuzzy systems: Dealing with fuzzy knowledge
Application: Fuzzy Control
4. Connective systems: Neural Networks
Application: Identification and neural control
1. Basics
2. Learning method
3. Application in Automation Engineering
5. Genetic algorithms: Stochastic Optimization
Application: Optimization
6. Summary & Literature
SC
235WS 18/19 Georg Frey
Contents of 9th Lecture
• Modelling of Systems by NN
Preliminaries
Direct Model
Inverse Model
• Application
Control
“Virtual” Sensors
• Assessment of NN
• Comparison of NN und Fuzzy
• Possible combinations
• Application examples: Load forecasting
SC
236WS 18/19 Georg Frey
Preliminaries
• Neural networks can model any non-linear relations among multiple
input and output variables of a system
• Pure feed-forward networks can only model static relationships
Solution 1: Recurrent Networks
- Training is difficult
Solution 2: External feedback i.e., processing of past values
+ Simple learning algorithm like backpropagation can be used
- The number of past values must be fixed
• Identification with past values: discrete model
SC
237WS 18/19 Georg Frey
Generating the model of a process
• Objective:
Modeling of a process
Networks models the function yk = f(uk-1, yk-1)
For systems of higher order: yk = f(uk-1,uk-2,... ,yk-1,yk-2,...)
• Input:
Current and past values of the process input u
past values of the process output y
• Output:
Current process output yk
• Example
SC
238WS 18/19 Georg Frey
Generating inverse process model
• Objective:
Modeling of inverse process model
Network models the function uk-1 = f(yk, yk-1) or uk-1 = f( yk ,yk-1,yk-2,... uk-2,uk-3,... )
• Inputs:
Current and past values of the process output y
Previous process inputs u
• output:
Current process input uk-1
• Example
SC
239WS 18/19 Georg Frey
Application of the direct model
• Estimation of state variables which are not measurable online to use
in closed-loop controllers (virtual sensor, observers)
Controller Route
NN Model
w u ym
logical
interconnection
yNN
y
-
SC
240WS 18/19 Georg Frey
Application of the inverse model (ideal)
• If the model is ideal it is possible to achieve open-loop control using
inverse model
But:
• Model is not ideal
• There are noises
Routeinverse NN Modellw u y
SC
241WS 18/19 Georg Frey
Application of the inverse model (real)
• Use of a controller to remove the noises and to compensate for the
errors in the model
Routeinverse NN Modely
Lin. Controllerw u
-
SC
242WS 18/19 Georg Frey
Summary of applications of NN in AT (Automation)
• Besides the "classical" tasks such as pattern recognition,
classification, etc. NN can also be used for performing core
functionalities of AT (Automation)
Observer or virtual Sensor
Closed-loop control (in combination with conventional control)
Combinations of the above are also possible
• In addition to the basic structures discussed, there could be many
other structures
SC
243WS 18/19 Georg Frey
Evaluation of NN
• Neural networks can be trained on the basis of data
no modelling of the processes necessary
• Successful applications show the potential of the method
• Knowledge is encoded in the structure of the NN
A verification, interpretation of the calculated values is virtually
not possible (raises acceptance problems!)
• NN training is extensive
• Acquisition of "good" data can be problematic
• To fix the structural parameters, e.g.,
Number of hidden layers
Number of neurons in the hidden layers
Type of network
Type of activation functions
Learning parameters and criteria for stopping training
use of heuristics is preferable in most cases.
SC
244WS 18/19 Georg Frey
Comparison NN vs. Fuzzy
- Often very long computing
times
- Convergence is not ensured
- Long computing
times during
training,
- Many competing
network structures
to choose from
- Extrapolation not
possible, i.e., good
results are achieved
only in the range of
training data
- Knowledge in the
network hardly
interpretable
- Difficult knowledge
acquisition phase
- Optimization phase
often slow
- Unusual way of
thinking
- Application to complex
processes very
cumbersome and
expensive
- Control specialists are
needed to write and
amend the algorithms
- There are scarce
standard tools for
implementing the
algorithms on standard
hardware (e.g., PLC)
+ Like NN but
+ Better interpretation of
knowledge,
+ Knowledge through learning
can gradually be
complemented
+ Adaptive and
adaptable to very
complex dynamic
processes,
+ Possible to retrain
when the process
undergoes changes
+ Simple and
comprehensive form of
algorithms,
+ Easily extensible rule-
base
+ Integration of
knowledge from more
than one source is
possible
+ In-depth process
understanding based on
process analysis
+ Generally the outcome
is very good and optimal
solutions can be
achieved
+ Stability proves are
possible
Neuro-FuzzyNeuronal NetworksFuzzy ControlClassic Method
SC
245WS 18/19 Georg Frey
Approaches for the combination of NN and fuzzy
(A) Cooperative neuro-fuzzy systems: Fuzzy systems which can be trained by neural networks. A neural network connected serially with the fuzzy system can, for example, be used to learn the suitability of a rule in certain situations.
(B) Rule-based training of a simple neural network
(C) Hybrid Neuro-Fuzzy-systems: simple neural networks that uses "fuzzy neurons" (e.g., min-/max-Neurons) and "fuzzy weights". The structure of the fuzzy system can be recognized from the network topology.
(D) Neural networks that can be trained by fuzzy-learning method. The changes of the weights between the neurons is calculated by a fuzzy system at each step.
(E) topological configuration of a neural network, with more or less complex fuzzy systems as neurons.
(F) A mix of classic expert systems and one of the above approaches.
• Important approaches are A, B and C.
• Other approaches are not as widespread the previous ones.
SC
246WS 18/19 Georg Frey
Cooperative neuro-fuzzy systems (2 approaches)
Fine-tuning a fuzzy controller by NN
• A fuzzy controller will be followed by a neural network
• The output of the fuzzy system will be immediately processed by the NN
• Thus based upon a basic knowledge (of the fuzzy system) a non-linear system can be built, which additionally renders adaptability to certain special situations which are not defined by the basic knowledge.
• Thus NN performs the "fine tuning" of output of the fuzzy system. The NN can learn which tuning is necessary for which input.
• The fuzzy system must not deliver defuzzyfied output this task can also be performed by the NN.
Preprocessing the input values of a fuzzy controller by NN
• fuzzy controller is preceded by a NN
• The output of the NN is fed to the fuzzy controller for processing.
• Thus, changes in the input data, which cannot be processed by the fuzzy system can be compensated by NN.
SC
247WS 18/19 Georg Frey
Rule based training of NN
• NN can only be trained by numerical data
• Often a rough knowledge of the process is available in the form of
fuzzy rules
• Solution: mapping of linguistic rules (qualitative) to the training data
(quantitative)
The linguistic terms are mapped to values (according to the membership
functions)
The rules are then defined by the corresponding values
• During training NN interpolates among the values
• Example:
Three variables X1, X2 and Y with values of Small, Medium and Large within the
range of [0, 1] have to mapped to numeric values. It is given that small = 0;
resources = 0.5; Large = 1.
The rule IF X1=small AND X2 = large THEN Y = large
Results in the data set X = (0, 1); Y = 1
SC
248WS 18/19 Georg Frey
Hybrid neuro-fuzzy systems
• Mapping of a fuzzy controller to a neural network
• Example:
1st Layer: input Fuzzy Sets
2nd Layer: evaluate the degree of fulfilment of the rules
3rd Layer: output fuzzy sets
4th Layer: De-fuzzyfication
• Other variants define the fuzzy sets in the weights
• Training with data
• Interpretation of the rules learned as weights (weights between
Layer 1 and 2 or 2 to 3)
SC
250WS 18/19 Georg Frey
Example: Load forecast in electrical energy supply networks
• Motivation
• Last curve analysis
• Forecast with Artificial Neural Networks (ANN)
• Wavelet transformation
• Assessment of results
• Summary
SC
251WS 18/19 Georg Frey
Motivation
Structure of an electric power supply network
Power PlantNetwork
(Low storing capacity)
Consumer
Logic
on/off
deterministic, knownnot deterministic,
only past behaviour known
PI PO
SC
252WS 18/19 Georg Frey
Motivation
• Last curves forecast plays a major role in the operation of power
networks
Power is cost-effective
Electrical energy is difficult to save
• It should be possible to only produce as much electrical energy as
needed
PI=PO
• Therefore one needs to recorded consumption profiles based
Forecast
Under forecast leads to inadequate provision of spare capacity
Over forecast caused unnecessary spare capacity
SC
253WS 18/19 Georg Frey
Last curve-Analysis
• Network load from 09.06.2003 to 29.06.2003 (individual than three weeks) in the control zone RWE's electricity transportation network
1. From Monday to Sunday, from 0 clock to 24 clock
2. Given are 15-minute averages
3. 4 * 24 = 96 test points per day, 96 * 7 = 672 measuring points per
week
MW
SC
254WS 18/19 Georg Frey
Forecast with artificial neural networks (ANN)
• Forecast runoff
Last curve normalization
Forecast basic idea
KNN-Definition
• Structure, vector input, output vector, activation function
KNN training (with a whole week (this week 1))
• Back propagation-Algorithms
Learning rate
KNN-Application (with Week 2 oder 3)
Results Denormalization
KNN
Modell
( 1, 2,... 8)Lk k k ( 4)L k
Fig 3 : Drei-Schichten-Feed-Forward-StrukturFig 2 : Einschicht-Neuron
SC
255WS 18/19 Georg Frey
Forecast with artificial neural networks (ANN)
• Last curve-Normalization
SC
256WS 18/19 Georg Frey
Forecast with artificial neural networks (ANN)
KNN
Modell
( 1, 2,... 8)Lk k k ( 4)L k
Three layers feed forward structure
Monolayer neuron Forecast basic idea
1
2
......
8
L k
L k
L k
p 4a Lk
Last course (distribution) of
the last two hours
Last in an hour
SC
257WS 18/19 Georg Frey
Forecast with artificial neural networks (ANN)
• Four-step forecast results
Training of KNN with Week 1
Target vector (SimT): Last curve Week 3
Output vector(Y): Forecast of Week 3
SC
258WS 18/19 Georg Frey
Forecast with artificial neural networks (KNN)
• In many places, the relative error is greater than 10%
The accuracy must be improved
Idea: Installation of Wavelet transformation
Relativer Fehler
SC
259WS 18/19 Georg Frey
Wavelet transformation
• Development of Wavelet transformation Fourier transformation
Transformation from Time- to Frequency Domain
Short-Time-Fourier transformation
Additional Information which Frequency in occurs which time frame
Continuous Wavelet transformation
Transformation of time in frequency and time domain
Discrete Wavelet transformation (DWT)
Realization in Computer
A Trous algorithm of Wavelet transformation
• Shift invariant
• Same in data length in different frequency domains
• suitable for real-time systems
f t Ff
1 2
10
, ,...tt tt
t tt
ft FfFf
,f t Ff
SC
260WS 18/19 Georg Frey
Discrete Wavelet transformation (DWT) (implementation)
• Analysis of a signal
HP
TP
2
High pass filter
Low pass filter
Down sampling
f<fs/16fs/16<f<fs/8 f<fs/8fs/8<f<fs/4 f<fs/4fs/4<f<fs/2 f<fs/2Frequency response
N/8N/8N/4N/4N/2N/2NSampling points
a3d3a2d2a1d1x
SC
261WS 18/19 Georg Frey
Discrete Wavelet transformation (DWT)
• Example
SC
262WS 18/19 Georg Frey
Discrete Wavelet transformation (DWT)
• Synthesis of a signal
Upsampling
SC
263WS 18/19 Georg Frey
Discrete wavelet transformation (DWT)
• Requirement of DWT in the analysis of real-time system
Localization time points in different scales
Shift invariance of the system
Move original
curve
Wavelet-
transformation
Wavelet-
Coefficient
Move
Coefficient
Wavelet-
transformation
Wavelet-
Coefficient
SC
264WS 18/19 Georg Frey
Á-Trous algorithm of Wavelet transformation
d1
d2
d3
a3
• Properties of the A-Trous algorithm
Shift invariance
Same data length of all the different scales Wavelet coefficient
g[n] : Tiefpassfilter
h[n] : Hochpassfilter
SC
265WS 18/19 Georg Frey
Wavelet transformation
• Example A-Trous algorithm
Week 1 load curve is split into 4 layers
a4: Approximations signal; d4, d3, d2, d1: detail signals
a4 has the largest amplitude and the lowest frequency
d1 is the smallest and the largest amplitude frequency
SC
266WS 18/19 Georg Frey
Forecast: ANN + A Trous
• Forecast runoff with KNN and A-Trous
For each split signal, a ANN model
The more layers, the higher the accuracy of the load curve synthesis
d1 is the prognosis regarded as noise and neglected.
Recorded
load curves
a4
d4
d3
d2
d1
netA4
netD4
netD3
netD2
netD1
Predicted
Last curve
Wavelet
Re-
transformation
Ã-Trous
Wavelet
SC
267WS 18/19 Georg Frey
Forecast: ANN + A Trous
• Four-step forecast results
Training with Week1
Target vector(SimT): Last curve Week3
Output vector(Y): Forecast of Week3
SC
268WS 18/19 Georg Frey
Forecast: ANN + A Trous
• At the most points the relative error less than 2%
• The error is never greater than 6%
In comparison to ANN without A-Trous, the accuracy improved significantly
Relativer Fehler
SC
269WS 18/19 Georg Frey
Summary and learning from the 9th Lecture
Know basic applications of NN in AT
Model shapes in the identification and their target
directly
Inverse
Neural networks with other approaches to (especially fuzzy) compare
Deduce reasons for neuro-fuzzy
Know possible ways of combining NN with fuzzy and can explain the basic idea
Use of neural networks has been shown to predict
Neural networks applied to isolated not bring satisfactory results in the load curve forecasting
In combination with wavelet transform results could be significantly improved