ISNN 2008

A gradient-based sequential radial basis function neural networkmodeling method

Wen Yao Æ Xiaoqian Chen Æ Wencai Luo

Received: 20 July 2008 / Accepted: 12 February 2009 / Published online: 5 March 2009

� Springer-Verlag London Limited 2009

Abstract Radial basis function neural network (RBFNN)

is widely used in nonlinear function approximation. One of

the key issues in RBFNN modeling is to improve the

approximation ability with samples as few as possible, so

as to limit the network’s complexity. To solve this prob-

lem, a gradient-based sequential RBFNN modeling method

is proposed. This method can utilize the gradient infor-

mation of the present model to expand the sample set and

refine the model sequentially, so as to improve the

approximation accuracy effectively. Two mathematical

examples and one practical problem are tested to verify the

efficiency of this method.

Keywords RBFNN � Sequential modeling � Gradient

1 Introduction

In multidisciplinary design optimization (MDO) of com-

plex system, it is very time-consuming to run a simulation,

or very expensive to conduct an experiment. Especially in

reliability or robust design considering the uncertainties in

realistic world, it even needs much more simulation time to

estimate the uncertain characteristics of the system. An

effective method to solve this problem is to use an

approximate model as a simple and inexpensive replace-

ment of the high fidelity model in the design and

optimization process. The approximate model is also called

metamodel, which means model of the model. There are

various types of metamodels, including response surface

model (regression polynomial), adaptive regression spline,

kriging model and radial basis function neural network

(RBFNN) model, etc. RBFNN is an interpolation approx-

imation method, which can fit all the sample data exactly,

and has been shown to be suitable in approximating high

nonlinear functions [1]. So in recent research, RBFNN has

been widely studied and used in approximation application.

The accuracy of an RBFNN model depends on the

selected basis function as well as the sample data [2]. The

most commonly used basis functions are linear, thin-plate

spline, Gaussian, and multiquadric functions. With appro-

priate basis function and sufficient sample data, RBFNN

can infinitely approximate a highly nonlinear response to

any degree of accuracy. Researchers have developed new

basis functions such as compactly supported basis func-

tions to produce positive definite interpolation matrix [3,

4], and augment the classical radial basis functions with

polynomial terms to improve the capability in modeling the

constant and higher order nonlinear functions. Another

efficient method to improve the approximation quality is to

increase the number of samples, which is directly relative

to the number of neurons in hidden layer of the network.

But in traditional approximation methods, the approxima-

tion modeling is carried out in one step, which samples all

the data at one time and constructs the approximation

model. It is very difficult to define the sample number

reasonably. If a large sample set is chosen for accuracy, the

network would be extremely complicated and the calcu-

lation efficiency would be influenced with a big number of

This article was originally presented in the fifth International

Symposium on Neural Networks.

W. Yao (&) � X. Chen � W. Luo

Multidisciplinary Aerospace Design Optimization Research

Center, College of Aerospace and Material Engineering,

National University of Defense Technology,

410073 Changsha, China

e-mail: wendy0782@126.com

123

Neural Comput & Applic (2009) 18:477–484

DOI 10.1007/s00521-009-0249-z

hidden layer neurons. Orthogonal least-square (OLS)

method has been developed to select the appropriate subset

of the whole samples to constitute the hidden layer neurons

so as to reduce the size of the network [5]. But it is very

hard to design the first whole set of experiment sample

data, which directly affects approximation accuracy. And

for a large set of initial sample data, it needs very long time

to complete this process. Some researchers proposed using

support vector machine (SVM) learning method to obtain a

good initial structure and parameters of RBFNN model,

and then using other methods, such as BP algorithm, to

improve the net [6]. But the initial sample set is difficult to

select and the SVM learning process is very complex with

many variables and large sample set. In recent years,

researchers proposed using sequential design method to

model highly nonlinear responses with limited number of

samples, which has been widely studied and used in

engineering design and optimization [7, 8]. There are

several sequential sampling strategies, such as progressive

lattice sampling, maximum entropy design strategy, cross

validation strategy, maximum scaled distance approach,

etc. [9, 10]. These methods can add data which have the

maximum expected information or can fill the space uni-

formly. But in some cases, part regions of the space are

highly nonlinear, while other parts are quite flat, so it is

much more feasible to allocate more points in those highly

nonlinear regions to obtain system characteristics rather

than distribute the samples all over the design space uni-

formly. Based on this point of view, a gradient-based

sequential RBFNN modeling method is proposed in this

paper. In the sequential modeling process, it selects the

future samples which have the maximum expected gradient

in present approximation model and refresh the model with

these data, so as to improve the accuracy of the model,

especially in the highly nonlinear areas. To still keep a

comprehensive understanding of the whole design space, a

sequential grouping design method based on optimum

Latin hypercube design (LHD) of experiment [11, 12] is

adopted, which can add data uniformly distributed in the

space to the sample set sequentially along with the points

having maximum expected gradient. By this sequential

modeling method, samples can be added step by step until

the metamodel meets the accuracy requirement, so as to

construct the approximation model with minimum sample

number, and control the complexity of the RBFNN to an

acceptable degree.

2 RBFNN model

In classical RBFNN model, the interpolation surface is

defined as a linear combination of radial functions.

yi ¼ siðxÞ ¼XN

k¼1

wik/ðx; ckÞ ¼XN

k¼1

wik/ x� ckk k2

� �;

i ¼ 1; 2; . . .;m

ð1Þ

/ is radial basis function of the distance x� ckk k2 from

the kth basis function center ck, and wik is output layer

weight. m is the number of system outputs. The center ck is

the sample point xk. N is the number of hidden layer

neurons, and also is the number of sample data points with

known function values fi(xk) such that

siðxkÞ ¼ fiðxkÞ ð2Þ

The Euclidean norm x� ckk k2 represents the radial

distance r between the point x and the center ck. It is

defined as

rðx; ckÞ ¼ x� ckk k2¼ ðXndv

i¼1

ðxi � cikÞ

2Þ12 ð3Þ

ndv is the number of design variables. For the sake of

convenience in discussion, assume that m = 1, implying

that the output weights wik constitute a vector

w ¼ ½w1;w2; . . .;wN �T , and the accurate responses of the

samples constitute a vector F ¼ ½f ðx1Þ; f ðx2Þ; . . .; f ðxNÞ�T ,

and the network outputs of the samples constitute a vector

S ¼ ½sðx1Þ; sðx2Þ; . . .; sðxNÞ�T . The unknown interpolation

coefficients w can be defined by minimizing the function

JðwÞ ¼ 1

2

XN

k¼1

ðf ðxkÞ � sðxkÞÞ2 ð4Þ

Considering (2), the minimization equation in matrix

form can be written as

Uw ¼ F ð5Þ

U ¼

/ðx1; c1Þ /ðx1; c2Þ � � � /ðx1; cNÞ/ðx2; c1Þ /ðx2; c2Þ � � � /ðx2; cNÞ

..

. ... ..

. ...

/ðxN ; c1Þ /ðxN ; c2Þ � � � /ðxN ; cNÞ

2

6664

3

7775 ð6Þ

From (5) and (6) we can see that the RBF centers are

fixed and the weight vector can be calculated directly from

the system of linear equations, so there is no need to

conduct complex iterative procedure to select the RBF

centers and calculate the weight vector, which may result

in complex convergence problem.

For an unknown point x in the space, the expected

response is

yðxÞ ¼ sðxÞ ¼XN

k¼1

wk/ðx; ckÞ ð7Þ

In this paper, the Gaussian basis function is discussed,

which is defined as

478 Neural Comput & Applic (2009) 18:477–484

123

/ðrÞ ¼ expð� r2

r2Þ ð8Þ

So the expected first order partial derivative of the

unknown point can be obtained by differentiating (7) as

y0xiðxÞ ¼ s0xi

ðxÞ ¼ os

oxi¼XN

k¼1

wko/ðrÞ

or� or

oxið9Þ

The gradient at x is

grad s ¼ rs ¼ ½ os

ox1

;os

ox2

; � � � ; os

oxndv

�T ð10Þ

y0ðxÞk k2¼ rsk k2¼Xndv

i¼1

os

oxi

� �2 !1

2

ð11Þ

3 The gradient-based sequential RBFNN modeling

method

Based on the RBFNN model introduced above, we can get

estimated system response and gradient information at any

point of the space through the approximation model.

Although the metamodel can hardly completely fit the

accurate model, especially when the sample data set is not

large and the information about the target system is not

sufficient, it can still represent the trend in the design space of

the true model to some extent. So utilizing information from

the present model to add data to the sample set and to rebuild

the model purposively is a more effective way to improve the

model quality than just expanding the samples randomly and

blindly. In this paper, we consider the gradient distribution as

the information we care for, as it can identify the high non-

linear region and offer information for optimization.

To begin the sequential modeling process, an initial data

set should be firstly constructed and the model be initial-

ized. To ensure good approximation qualities of the model

all over the space, a comprehensive understanding of the

space should be obtained through a space filling and uni-

formly distributed experiment design. There are many such

kind experiment design methods, such as optimum LHD,

full-factorial design method, orthogonal array sampling,

etc. The method should be selected according to the spe-

cific approximation problem.

During the sequential modeling process, every time a

model is built, a set of nval validation points are selected to

verify the accuracy of the model. The root mean square

error (RMSE) is chosen as the criteria to judge the accuracy

of the approximation model. It is defined as

RMSE ¼

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiPnval

i¼1

f ðxiÞ � sðxiÞð Þ2

nval

vuuutð12Þ

If the RMSE does not meet the required accuracy, the

gradient of the validation points are calculated. The points

with maximum gradient are selected to add to the sample

set and the model is rebuilt. In this way, more information

of the regions around the points with expected sharp

gradient can be obtained and the accuracy in these regions

can be improved.

Along with the process of refining the high nonlinear

regions, the accuracy of the model all over the space should

be maintained. So samples should also be augmented with

data filling the space. This part of adding data is accom-

plished by the method of sequential grouping design strategy

based on optimum LHD of experiment. In this method, an

optimum LHD is conducted firstly to construct the initial

uniformly distributed sample data filling the design space.

Then these points are divided uniformly and randomly into

groups with equal number of samples. Each group can be

added to the sample set according to certain strategy. In this

gradient-based sequential RBFNN modeling method, we

define that when there is no improvement of accuracy by

adding points with maximum expected gradient, a group of

these space filling data is added to the sample set.

Based on these sequential design strategies, the

sequential RBFNN modeling can be carried out in accor-

dance with the flowchart depicted in Fig. 1.

4 Tests

This section describes three tests of the gradient-based

sequential RBFNN modeling method, including one 1-

dimensional example, one 2-dimensional example, and one

5-dimensional practical approximation problem.

4.1 Test 1: Quasi_sinusoidal function

The n-dimensional quasi_sinusoidal [13] function is

defined as

f ðxÞ ¼Xm

i¼1

½0:3þ sinð16xi=15� eÞ þ sin2ð16xi=15� eÞ� e

¼ 0:7; xi 2 �1; 2½ � ð13Þ

For one dimension problem we set m = 1. For this

simple example, we set the required RMSE value to be

0.01. The initial sample set is constructed by four

uniformly distributed points and the initial model is built.

Then a point with maximum gradient is added into the

sample set during each iteration. The validation data are 15

uniformly distributed points within the design space. After

three iterations, the RMSE of the model meet the preset

requirement. The sequential modeling process is shown in

Fig. 2. In the figure, the circle represents the sample data.

Neural Comput & Applic (2009) 18:477–484 479

123

To illustrate the efficiency of this gradient-based

sequential modeling method, a sequential modeling method

without adding gradient information during the iteration

process is used to compare the results. In this no gradient

sequential modeling method, the model is built and the

accuracy of the model is checked. If it doesn’t meet the

requirement, the next group of data designed by optimum

LHD is added into the sampling set, and the model is rebuilt.

The only difference between this method and the gradient-

based sequential modeling method is that there is no gradient

information analysis in this one, and sample set is augmented

randomly and uniformly only according to the optimum

LHD design. The modeling result is shown in Fig. 3.

In Fig. 2, it takes three iterations and 6 points to get the

required accuracy, while in Fig. 3 it takes five iterations

and 12 points to meet the accuracy requirement. It also can

be seen clearly in Fig. 2 that the sample set augmented

with points that lie in region of great gradient, which can

improve the metamodel quality in these areas purposefully,

while in Fig. 3 the sample set is expanded randomly, which

influences the modeling efficiency greatly.

To further demonstrate the performance of this proposed

sequential sampling method, it is also compared with other

two most widely used sequential sampling strategies in RBF

modeling, including maximum entropy strategy and cross

validation strategy. Approximation accuracy (RMSE),

sample set number and modeling time are chosen to be

comparison indexes. The results are listed in Table 1. From

the table we can see that all these three indexes are quite

similar, only that the proposed method has a very narrow

advantage margin in modeling time, as the gradient pre-

diction equation is quite simple, while entropy calculation is

complex and cross validation needs building leave-one-out

models and the procedure is lengthy and time-consuming.

4.2 Test 2: Akley function

The 2-dimensional Akley function is defined as

f ðx1; x2Þ ¼ 20� 20 expð�0:2ffiffiffiffiffiffiffiffiffiffiffiffiffiffix2

1 þ x22

qÞ

� expcosð2px1Þ þ cosð2px2Þ

2

� �

x1; x2 2 ½�5; 30�

ð14Þ

Fig. 1 Gradient-based sequential RBFNN modeling flowchart

Fig. 2 Gradient-based

sequential RBFNN modeling of

test one. a Accurate function

curve. b Iteration one.

c Iteration two. d Iteration three.

e RMSE iteration history

480 Neural Comput & Applic (2009) 18:477–484

123

We set the required RMSE value to be 0.5. The initial

sample set is constructed by 16 uniformly distributed

points designed by full-factorial design method and the

initial model is built. Then five points with maximum

gradient are added into the sample set during each iteration.

The validation data are 100 uniformly distributed points

within the design space designed by full-factorial design

method. After two iterations, the RMSE of the model meet

the preset requirement. The sequential modeling process is

shown in Fig. 4. In the figure, the square represents the

sample data. The no gradient sequential modeling method

is also used to approximate the function. The initial sample

set, LHD sample data and grouping are the same with the

data used in the gradient-based method. In Fig. 4 it takes

two iteration and 19 points to meet the accuracy

requirement, while in Fig. 5 it takes nine iterations and

96 points to obtain a model without meeting the accuracy

requirement.

Because of the uncertain and random characteristic of

the LHD, we run several times with the no gradient

method. The results show that it can achieve good effect in

some cases, but the probability is small. So the modeling

capability is not robust merely by adding uniformly dis-

tributed space filling data randomly.

The proposed sequential sampling method is also com-

pared with maximum entropy strategy and cross validation

strategy in this test. The results are listed in Table 2. It is

clearly shown in the table that the gradient-based method

has the best performance in all of the three indexes. From

Fig. 6 we can see that the distribution of Akley function

is quite irregular. It has a very sharp region in

½�10; 0� � ½�10; 0�, while other parts are relatively flat. So

with little more samples added in this sharp part, the

accuracy of the model can be greatly improved. So adding

samples with great gradient sequentially can obtain good

approximation quality. The maximum entropy method

tends to add samples which are far away from the existing

samples, so this method can’t obtain sufficient information

in the sharp region and the approximation accuracy is not

good. The cross validation method utilizes the leave-one-

out model to estimate the model prediction error. The

maximum predicted error data points it identifies not only

distribute in the sharp region, but also in other flat regions,

so the modeling efficiency is affected. This is why gradi-

ent-based method can achieve good performance with

much less samples and modeling time in this test.

4.3 Test 3: Satellite system design approximation

problem

To demonstrate the validity of the gradient-based sequen-

tial RBFNN modeling method in practical engineering

applications, the proposed method is tested in a satellite

system design approximation problem.

In satellite conceptual system design, satellite weight is

an important index in scheme design decision-making. The

calculation of weight consists of several coupled disci-

plines, such as structure, propulsion and power, etc., and

the calculation is very complex and time-consuming. So in

system sensitive analysis and design optimization, it is

more preferable to use approximation model instead of

accurate weight calculation model. We choose a virtual

earth observation satellite as the discussion object. The

disciplinary models of the satellite to calculate weight can

Fig. 3 No gradient sequential

RBFNN modeling of test one.

a Iteration one. b Iteration two.

c Iteration three. d Iteration

four. e Iteration five. f RMSE

iteration history

Table 1 Comparison of different adding data strategies in test one

Strategy Sample

number

RMSE Modeling

time (s)

Gradient 6 0.006 1

Entropy 6 0.007 2

Cross-validation 7 0.006 2

Neural Comput & Applic (2009) 18:477–484 481

123

be referenced in [11]. In the model there are five design

variables, including orbit altitude, camera focus, satellite

body length, width and height. The satellite weight is the

output of the model. We utilize the proposed method to

construct the approximation model of weight in respect to

the five design variables. In the design space, the weight

output is mostly distributed between 1,000 and 2,000 kg,

so we set the required RMSE value to be 50 kg. The initial

sample set is constructed by 60 uniformly distributed

points designed by optimum LHD and the initial model is

built. Then ten points with maximum gradient are added

into the sample set during each iteration. The validation

data are 100 uniformly distributed points within the design

space designed by optimum LHD. After four iterations, the

RMSE of the model meet the preset requirement. The

sequential modeling process is shown in Fig. 7 depicted by

the line with triangle marks. To demonstrate the efficiency

of the proposed method, we compare it with approximation

models constructed with uniformly and randomly space

filling data obtained from optimum LHD. The sample data

number ranges from 60 to 150, with interval of 10. The

result is shown in Fig. 7 depicted by the line with square

marks. From the figure we can see that using the proposed

method the approximation model RMSE falls down to

39.83 only with four iterations, and the final sample data

number is 100. By uniformly sampling data in the design

space, it needs 120 samples to reduce RMSE to 48.49 and

140 samples to reduce RMSE to 34.07. It is obvious that

Fig. 4 Gradient-based

sequential RBFNN modeling

of test two. a Iteration one.

b Iteration two. c RMSE

iteration history

Fig. 5 No gradient sequential

RBFNN modeling of test two.

a Iteration one. b Iteration three.

c Iteration five.d Iteration seven.

e Iteration nine. f RMSE

iteration history

Table 2 Comparison of different adding data strategies in test two

Strategy Sample

number

RMSE Modeling

time (s)

Gradient 19 0.283 21

Entropy 48 1.245 62

Cross-validation 23 0.365 31

Fig. 6 Akley function

482 Neural Comput & Applic (2009) 18:477–484

123

the proposed method is efficient in improving the approx-

imation model accuracy with fewer samples, which can

directly reduce the calculation burden.

5 Conclusion

In this paper the RBFNN modeling method is discussed. To

construct the model with required accuracy and limited

samples, a gradient-based sequential RBFNN modeling

method is proposed. This method has following merits:

1. It builds the metamodel sequentially. During each

iteration process, several new points are added to the

sample set to rebuild the model. So we can use a small

set to initiate the RBFNN, and then refine the model

step by step until it meets the accuracy requirement. In

this way, we can avoid the difficulty in defining the

initial sample size and data in one-step modeling

method, and meanwhile minimize the sample number.

2. In the process of rebuilding the model, the gradient

information of the present model is analyzed firstly. By

adding the points with maximum expected gradient to

the sample set, the information in the highly nonlinear

regions can be integrated into the new model, and the

approximation accuracy in these areas can be

improved purposefully.

3. By adding groups of uniformly and randomly space

filling data obtained from optimum LHD according to

certain strategy (when it can’t improve the accuracy of

the model by merely adding expected gradient infor-

mation, add these groups of data), a comprehensive

understanding of the whole design space can be

maintained, and the accuracy of the model all over

the space can be improved.

4. In combination of adding data with maximum gradient

information and data uniformly and randomly filling

the design space, it can reduce the impact of random

and uncertain characteristics of LHD on metamodel-

ing, so as to make the sequential modeling process

more stable and robust. With growing number of

sample data, the RBFNN model can continuously

improve its accuracy until it meets the preset require-

ment, and the convergence rate is fast based on the

sample set augmentation strategy. The calculation of

predicted gradient and operation of adding data

sequentially are very simple, so it is easy to implement

and apply this method.

In the test section, two mathematical problems are used to

examine the proposed method. In every test problem, the

models achieved by this method are compared with those

achieved by the sequential modeling method without adding

gradient information during the modeling process. The

results show that this gradient-based method is more effi-

cient than the other one. Especially under the uncertainty

affection of LHD, it can achieve good approximation model

with much higher probability than the other one. The pro-

posed method is also compared with two widely used

sequential sampling strategies in the tests. The results

demonstrate that the proposed method has comparable

approximation ability with the classical ones. Especially in

test two, it is obvious that the new method has great

advantages in modeling irregular distributed functions. In

test three, the proposed method is applied in a practical

problem, which is to approximate the satellite total weight

function in respect to a design vector of five variables in

satellite conceptual system design. The model achieved by

the proposed method is compared with the model con-

structed with data by optimum LHD. The result shows that

the proposed method can meet the preset accuracy require-

ment with much less samples, which can greatly reduce the

modeling time and calculation burden. So the test results

prove that this gradient-based sequential RBFNN modeling

method is efficient and feasible. Future research of this

method should be conducted in high dimension problems.

The gradient analysis method and gradient-based sample set

augmentation strategy should be further studied.

Because of the good capability of this approximation

method in modeling high nonlinear functions, it can be

applied in engineering design and optimization, especially

in the uncertainty MDO of complex system, such as satellite

system design and other spacecraft, etc., so as to decrease the

calculation burden and enhance optimization efficiency.

References

1. Park J, Sandberg W (1991) Universal approximation using radial

basis function networks. Neural Comput 3(2):246–257. doi:

10.1162/neco.1991.3.2.246

Fig. 7 Comparison between proposed method and uniformly sam-

pling method in test three

Neural Comput & Applic (2009) 18:477–484 483

123

2. Fang H, Horstemeyer (2005) Metamodeling with radial basis

functions. In: The 46th AIAA/ASME/ASCE/AHS/ASC struc-

tures, structural dynamics and materials conferrence, paper no.

AIAA 2005-2059, Austin

3. Wu Z (1995) Compactly supported positive definite radial func-

tion. Adv Comput Math 4:283–292

4. Wendland H (1999) On the smoothness of positive definite and

radial functions. J Comput Appl Math 101:177–188. doi:

10.1016/S0377-0427(98)00218-0

5. Chen S, Cowan CF, Grant PM (1991) Orthogonal least square

algorithm for radial basis function networks. IEEE Trans Neural

Netw 2:302–309. doi:10.1109/72.80341

6. Xiaofang Y, Yaonan W, Wei S, Huiqian Y (2005) A hybrid

learning algorithm for RBF neural networks based on support

vector machines and BP algorithms. J Hunan Univ 32(3):88–92

in Chinese. Natural Sciences

7. Lin Y, Mistree F, Allen JK, Tsui KL, Chen VCP (2004)

Sequential metamodeling in engineering design. In: The 10th

AIAA/ISSMO multidisciplinary analysis and optimization con-

ference, paper no. AIAA 2004-4304, Albany

8. Lin Y (2004) An efficient robust concept exploration method and

sequential exploratory experimental design. PhD dissertation,

Philosophy in Mechanical Engineering, Georgia Institute of

Technology

9. Krishnamurthy T (2003) Response surface approximation with

augmented and compactly supported radial basis functions. In:

The 44th AIAA/ASME/ASCE/AHS/ASC structures, structural

dynamics, and materials conference, paper no. AIAA 2003-1748.

Norfolk

10. Jin R, Chen W, Sudjianto A (2002) On sequential sampling for

global metamodeling in engineering design. In: ASME 2002

design engineering technical conferences and computer and

information in engineering conference, paper no. DETC2002/

DAC-34092, Montreal

11. Yao W (2007) Research on uncertainty multidisciplinary design

optimization theory and application to satellite system design (in

Chinese). Master of engineering dissertation, National University

of Defense Technology

12. Giunta AA, Wojtkiewicz SF Jr, Eldred MS (2003) Overview of

modern design of experiments methods for computational simu-

lations. In: 41st aerospace sciences meeting and exhibit, paper no.

AIAA 2003-649, Reno

13. Giunta AA, Watson LT (1998) A comparison of approximation

modeling techniques: polynomial versus interpolating models. In:

The 7th AIAA/USAF/NASA/ISSMO symposium on multidisci-

plinary analysis and optimization, paper no. AIAA 98-4758,

Louis

484 Neural Comput & Applic (2009) 18:477–484

123