Post on 07-Jun-2019
Radial Basis Function Networks Notes © Bill Wilson, 2008 1
Radial Basis Function Networks
Aim • to study radial basis function networks
Reference Haykin, Chapter 7 provides a theoretical treatment.
Another view is found in section 5.3 of Karray & de Silva: Soft Computing and
Intelligent Systems Design, Addison-Wesley, 2004. ISBN 0 321 11617 8
Unfortunately this is not currently in UNSW Library (but is on order).
Keywords radial basis function, receptive field, hybrid learning
Plan • define radial basis function (RBF)
• introduce RBF network
• training (estimating parameters for) RBFNs
• example
• applications
Radial Basis Function Networks Notes © Bill Wilson, 2008 2
Introduction
• Radial basis function networks feedforward networks, with some nodes using a different
activation function from the regular sigmoid node in a backprop net – a radial activation function:
... ... ...
inputs hiddens
(RBF nodes)outputs
1wij
x1
x2
xn
o1
o2
om
• The connections from the input units to the hidden units are fixed at 1.
• Each RBF/hidden node has two parameters, termed the centre
!
v (a vector) and the scalar width
!
" . The function computed by the i-th node will be of the form
!
gi x( ) = rix" vi
# i
$
% &
'
( )
Radial Basis Function Networks Notes © Bill Wilson, 2008 3
Radial Activation Functions
• Clearly the value of
!
gi x( ) depends on the distance of the input vector
!
x from the centre
!
vi, scaled
by the width
!
" i . All points
!
x that are equally close to
!
vi will be treated in the same way by the
function
!
gi x( ). Hence the name radial activation function.
• The most widely used radial basis function is the Gaussian function:
!
gi x( ) = exp" x" vi
2
2# i2
$
%
& &
'
(
) )
• There is no squashing function in the output neurons: the value from the j-th output unit of our
RBF network, assuming n hidden layer units and r output units, is:
!
oj (x) = wijgi(x)i=1
n
" j =1,...,r
• Another possible radial activation function is
!
gi x( ) =1
1+ exp( x" vi2/# i2).
• Note that in both cases,
!
gi(x)" 0, and so
!
oj (x)" 0, as
!
x" vi #$
Radial Basis Function Networks Notes © Bill Wilson, 2008 4
“Training” Radial Activation Functions
• Training the RBF involves estimating the best values for the
!
vi, the
!
" i , and the
!
wij.
• The standard technique is the hybrid approach, which is a two-stage learning strategy:
1) Unsupervised clustering algorithm used to find the
!
vi and
!
" i for the RBFs.
There are a number of possible clustering algorithms to try, the random input vector method,
the k-means-based method, maximum likelihood estimate-based method, standard deviations-
based method, and self-organising map method.
2) Supervised algorithm used to find values for the
!
wij.
There is only one layer with weights, so this is not as complicated as with MLPs/backprop.
We’ll describe one way to do this.
Radial Basis Function Networks Notes © Bill Wilson, 2008 5
Finding Centres and Widths
• The random input vector method involves choosing the locations of centres randomly from the
training data set. All RBFs get the same width
!
" =dmax
2m1
, where
!
dmax is the maximum
distance between the chosen centres, and
!
m1 is the the number of centres.
• An alternative is to give centres differing widths depending on the local data density (broader
width if density is low).
• The k-means-based method is a clustering algorithm:
1 Initialise: choose small random values for the initial (but distinct) centres
!
tk 0( ).
2 Sample: choose a vector x from the input space – we’ll call this
!
x(n).
3 Match: Let
!
k(x) be the index of the centre
!
tk 0( ) that minimises
!
x(n)" tk (n) .
4 Update: Set
!
tk n+1( ) = tk n( ) +" x n( )# tk n( )[ ] for
!
k = k x( );
!
tk n+1( ) = tk n( ) otherwise.
5 Continue: add 1 to n and go to step 2. Continue until no further significant changes.
• The width(s) can be done in the same way as that used for the random input vector method.
Radial Basis Function Networks Notes © Bill Wilson, 2008 6
Finding Weights
• A range of supervised learning methods can be used to find the weight values. Here is one that is
appropriate for function interpolation, given a set of input vectors
!
xk and corresponding output
values
!
dk. In this case, it makes sense to use one RBF node for each pair (
!
xk,
!
dk), and the centres
for the RBF nodes are just the
!
xk, i.e.
!
vk = xk .
• Let
!
D be the matrix of desired outputs – i.e. if the function to be interpolated is
!
f :
!
"n #"m, and
!
dk is the desired output for the input
!
xk, so that
!
f xk( ) = dk , then
!
D = dk[ ],
!
k =1,...,p, where p is
the number of hidden/RBF nodes. We can also write
!
dkj = f j xk( ) = wijgii=1
p
" xk( ) .
• Let
!
gki be
!
gi(xk ), i.e.
!
exp" xi " xk
2
2# i2
$
%
& &
'
(
) ) if we are using Gaussian RBFs. We can write
!
G = gki[ ].
Then
!
dkj = f j xk( ) = wijgii=1
p
" xk( ) = wijgkii=1
p
" = gkiwiji=1
p
" , or
!
dkj = GW[ ]kj . That is,
!
D =GW.
Radial Basis Function Networks Notes © Bill Wilson, 2008 7
Finding Weights 2
• Since
!
D =GW, if G is invertible, then
!
G"1D =W
• If G is not invertible (because
!
G = 0, or G is non-square, or G is ill-conditioned, i.e.
!
G is close to
0), then we replace
!
G"1
by
!
G+, the Moore-Penrose pseudo-inverse of G, (ako generalised
inverse), which is the unique matrix satisfying:
(1)
!
G =GG+G (2)
!
G+
=G+GG
+
(3)1
!
GG+( )*
=GG+ (4)
!
G+G( )*
=G+G
• If the columns of G are linearly independent, then
!
GTG is invertible, and then
!
G+ = G
TG( )
"1GT
.
Note that then
!
G+G = G
TG( )
"1GTG = I. So
!
W =G+D.
1
!
A* is the conjugate transpose of
!
A . This is the same as
!
AT
if
!
A is all real (not complex).
Radial Basis Function Networks Notes © Bill Wilson, 2008 8
Effect of Width on Approximation Performance
• The diagrams on the next slide use the same centres, but three different values of
!
" - but the
same
!
" for all RBFs in each graph. The top pair of graphs show
!
" = 0.5, the middle pair have
!
" = 2.1, and the bottom pair have
!
" = 8.5
• Inspection shows us that with the small value of , the approximation is “too wiggly”, for the large value of the approximation is too smooth, and the middle value is “just right”.
• In practice, one would use an unsupervised algorithm to estimate the best value for
!
" , as described above.
• The diagram comes from Karray & de Silva, Figure 5.8 a, b & c:
Radial Basis Function Networks Notes © Bill Wilson, 2008 9
Radial Basis Function Networks Notes © Bill Wilson, 2008 10
Applications of RBF Networks
• RBF networks have universal approximation capabilities (on compact subsets of
!
"n), and they
have efficient training algorithms.
• They have been used for
– control systems
– audio and video signal processing
– pattern recognition
– chaotic time series prediction (e.g. weather forecasting, power load forecasting).
Radial Basis Function Networks Notes © Bill Wilson, 2008 11
David Broomhead
Professor of Applied Mathematics at UMIST and
works in applied nonlinear dynamical systems
and mathematical biology. Since the early 1980's
he has developed methods for time series
analysis and nonlinear signal processing using
techniques from nonlinear dynamics, including
modelling oculomotor control. In 1989 he was
awarded the John Benjamin Memorial Prize for
his work on radial basis function neural
networks.
D.S. Broomhead and D. Lowe. Multivariable
functional interpolation and adaptive
networks. Complex Systems, 2:321-355. 1988.
Radial Basis Function Networks Notes © Bill Wilson, 2008 12