Abstract - The study develops a software reliability growth model (SRGM) with confidence intervals that provides software developers useful information to decide the optimal software release time and to refine the quality of software testing tasks. The developed SRGM of this study is based on stochastic calculus to deduce the confidence intervals of the mean value function at different confidence level. Owing to less clear explanation of the variance in the mean value function of cumulative software errors in most software reliability growth models, it might not be effective in deducing the confidence interval concerning the mean value function. Therefore, software developers cannot estimate the possible risk variation in software reliability, and it might diminish the value of practical applications. In this study, we utilize the method of stochastic differential equations and five classic models (Goel and Okumoto’s model (1979), Yamada’s Delayed S-shaped model (1983), Ohba’s Inflection S-shaped model (1984), Yamada’s exponential model (1992), Chiu and Haung’s learning effect model (2008)) to build the SRGM with confidence intervals that can assist the software developers in determining the optimal release times at different confidence levels. With regard to the software failure phenomena, they were supposed as Non-homogeneous Poisson Process (NHPP) in this study.

Keywords - Software Reliability Growth Model;

software reliability; confidence intervals; stochastic differential equations; Non-homogeneous Poisson process

I. INTRODUCTION Software reliability growth is important in software industry, since it can provide critical information for developers and testing staff during the testing/debugging phase. In order to control the cost in the software testing/debugging phase, the software testing staff has to notice the variation of software reliability and the testing costs at any time in the process. Thus, it should be seriously monitored the issue of reliability, cost, and release time in a software development process. In practice, studies of how to accurately predict software reliability are important to software industry, since they are able to provide developers and testing staff with effective suggestions to enhance the quality of the software system during the testing/debugging phase. Various software reliability growth models (SRGM) have been proposed, some of which described the software failure phenomena based on the Non-Homogeneous Poisson process (NHPP). Therefore, the objective in developing a software reliability model is to accurately estimate software testing costs with the intention of determining an adequate time

to stop debugging and release the software to the users. However, a vast amount of literature deterministically discussed software testing costs without considering the existing randomness among related parameters, and therefore leaded to incomplete decision-making processes. Hence, decision makers can only adopt a neutral strategy instead of more sufficient strategies with the consideration of stochastic parameter evaluations. On the other hand, since a number of researchers might improperly deduce the variance regarding the mean value function, its confidence interval would not conform to the real world situation, and thus lower the effectiveness of the estimated software testing costs. According to the above mentions, the objective of this study is to propose several SRGMs with confidence intervals by using the method of SDE (Stochastic Differential Equations) along with the error detection function. Software developers can estimate the possible risk variation in software reliability by using the randomness of mean value function, and can determine the optimal software release times at different confidence levels, with the assumption that the variance in the mean value function stems from the error detection process.

II. RELATED WORK For the last two decades, various stochastic models have been proposed to assess software reliability. The software failure phenomenon is depicted mostly based on the Non-homogeneous Poisson Process (NHPP). In fact, these models are fairly effective in describing the error-detection process with a time-dependent error-detection rate. However, the assumption is that each time an error occurs the fault that caused it can be immediately removed, leading to no new problems, which is usually called perfect debugging. Imperfect debugging has also been discussed to loosen such a restricted assumption (Ohba, 1984b). Gokhale and Trivedi (1999) stated that the assumption of statistical independence for the numbers of events occurring in disjoint time intervals is constantly violated when the software reliability growth models developed by NHPP, and proposed an enhanced NHPP model to allow the time-dependent failures occurring in the debugging process. However, the enhanced NHPP model has not yet been widely adopted and applied to the field of software reliability growth. Goel and Okumoto (1979) proposed an exponential software reliability growth model that the curve of the cumulative number of faults is often related to an S-

Confidence Interval Estimation of Software Reliability Growth Models Derived from Stochastic Differential Equations

Chih-Chiang Fang 1 and Chun-Wu Yeh 2 * 1 Department of Information Management, Shu-Te University, Taiwan, R.O.C.

2 * Department of Information Management, Kun Shan University, Taiwan, R.O.C. (1 ccfang@stu.edu.tw3 2 davidyeh929@gmail.com )

978-1-4577-0739-1/11/$26.00 ©2011 IEEE 1843

shaped or exponential-shaped mean value function. In addition, a number of generalizations or modifications have been proposed, such as the delayed S-shaped reliability growth model proposed by Yamada (1983) and the inflection S-shaped reliability growth model proposed by Ohba (1984a). Generally, the S-shaped model can explain the learning process by which people become familiar with the software and the test tools. Their debugging skills improve gradually and testing effectiveness also increases significantly. Hence, there is a tendency that the cumulative numbers of errors have an S-shaped form, which implies that the increase of reliability in the beginning is only of modest size due to low testing effectiveness. The reliability then increases more quickly and later the improvement slows down. Hence, discovering the confidence intervals of software reliability can enhance the decision of software releases and control the related expenditures for software testing. Yamada and Osaki (1985) considered that the maximum likelihood estimates (MLE) concerning the confidence interval of the mean value function can be estimated as /2( ) Z ( )CRm T m T± , where CR denotes the critical region, /2ZCR denotes the value providing an area

2CR of the standard normal distribution. Yin and Trivedi (1999) obtained the confidence bounds for the model parameters via the Bayesian approach by quoting the estimation method of Yamada and Osaki (1985). Huang (2005) also adopted the method of Yamada and Osaki (1985) to draw a graph to illustrate the confidence interval of the mean value function.

Some researchers in the field of software reliability considered that the variation in the mean value function stems from the error detection process, and deduce the mean value function by using a method of stochastic differential equations (SDE) (Lee et al., 2004; Tamura and Yamada, 2006). Tamura and Yamada (2006) derived a flexible SDE model to describe the fault-detection process during the system testing phase of the distributed development environment by using an inflection S-shaped SRGM and an Itô type SDE (Karatsas and Shreve, 1997). However, such a model improperly presents the expectation and variance of the mean value function, and lacks a complete process for obtaining the parameters in their model. The possible reason of existing improper variances lies in the misconception that merely the randomness of the mean value function should account for the final estimation error of software reliability. This turns out that the variance of mean value function will increase in time as shown in Fig. 1 (Huang, 2005). However, since the number of software errors is finite, there should be less software errors to be detected with time. Therefore, the variance of cumulative software errors should be decreasing instead of increasing as time goes by. In this regard, it is assumed that the variance in the mean value function mainly stems from the error detection process in this study, and the confidence interval of the mean value

function would thus diminish as remaining software errors decrease, which can be observed in Fig. 2

Cum. errors

Testing time

Confidence Interval

Fig. 1 The confidence interval under the condition that the variance in mean value functions does not stem from the

error detection process

Cum. errors

Testing time

Confidence Interval

Fig. 2 The confidence interval under the condition that the variance in mean value functions mainly stems from the

error detection process

According to the above discussion, several SRGMs with confidence intervals by using the method of SDE will be proposed in this study. The proposed SRGMs with confidence intervals could assess the variance in the mean value function more reasonably, and assist software developers in properly dealing with the risks of software reliability estimation.

III. METHODOLOGY 3.1 Model Development

For the last two decades, various software reliability growth models have been proposed, and some of these models were fairly effective in prediction of the average number of cumulative software errors during the testing task. These models include Goel and Okumoto’s model (1979), Yamada’s delayed S-shaped model (1983), Ohba’s inflection S-shaped model (1984), Yamada’s exponential model (1992), Chiu and Haung’s learning effect model (2008). Table 1 shows the corresponding mean value functions and the error detection rate functions for the five classic models. Table 1 Summary of mean value and error detection rate

functions for the classic models

1844

Model Mean value function and error detection rate function

Goel and Okumoto’s model

(1979)

( ) (1 )btm t a e−= − ( )d t b=

Yamada’s Delayed S-shaped model

(1983)

( ) (1 (1 ) )btm t a bt e−= − + 2

( )(1 )

b td tbt

=+

Ohba’s Inflection S-shaped model

(1984)

1( )1

bt

bt

em t ace

−

−

⎛ ⎞−= ⎜ ⎟+⎝ ⎠

( )(1 )bt

bd tce−

=+

Yamada’s exponential model

(1992)

( )(1 )( ) 1tr em t a e

βα −− −= −

( ) td t re βαβ −=

Chiu and Haung’s learning effect model (2008)

( )

1( ) 1

t

m t ae α η

ηα

ηα

+

⎛ ⎞+⎜ ⎟= −⎜ ⎟

⎜ ⎟+⎜ ⎟⎝ ⎠

( )( ) ( ) 1

td t

e α η

ηα ηα η+

⎛ ⎞= + −⎜ ⎟+⎝ ⎠

The following notations will be used to deduce the

model of this study: Notations:

a : the expected number of all potential errors in the software system before the software testing begins

( )m t : the mean value function of the software error detection process, which is the expected number of errors detected within time (0, )t

( )M t : the function of the residual software errors at time t

( )d t : the error detection rate per error at time t ( )W t : the variable for Wiener process (Wiener and

Masani, 1957) σ : the standard deviation stems from testing process,

and it is a positive constant to denote the magnitude of the irregular fluctuation.

Due to the fact that the above-mentioned models in

Table 1 lack of the discussion about the corresponding confidence intervals, the decision makers cannot evaluate the possible variation in the mean value function itself during the testing task. Accordingly, we use the models in Table 1 as basis to develop their corresponding confidence intervals of the mean value function.

According to the behavior of testing and debugging, we assume the irregular fluctuation is influenced by the error detection rate, and it can be represented by the following stochastic differential equation:

d ( )d ( )+ d ( )

( )

m tt d t W t

a m tσ=

−. (1)

Let ( )M t substitute for ( )a m t− , and Equation (1) can be rewritten as follows:

( )d ( )

d ( )+ d ( )( )

M tt d t W t

M tσ= − . (2)

Besides, Let [ ]( ) ln ( )Z t M t= . According to Itô’s calculus, we can get the following equation:

21d ( ) ( ) ( ) d d ( )2

Z t d t t W tσ σ⎧ ⎫= − − − −⎨ ⎬⎩ ⎭

. (3)

By way of the integrating process from 0 to T, the form of ( )M T can be obtained by following procedure:

2

0 0 0 0

1d ( ) ( )d d d ( )2

T T T T

Z t d t t t W tσ σ= − − −∫ ∫ ∫ ∫ (4)

→ 2

0 0 0 0

1( ) ( )d d d ( )2

T T TTZ t d t t t W tσ σ= − − −∫ ∫ ∫

→ [ ] [ ] 2

0 0 0

1ln ( ) ln (0) ( )d d d ( )2

T T TM T M d t t t W tσ σ− = − − −∫ ∫ ∫

(∵ [ ]( ) ln ( )Z t M t= )

→ [ ] [ ] 2

0 0 0

1ln ( ) ln ( )d d d ( )2

T T T

M T a d t t t W tσ σ− = − − −∫ ∫ ∫

(∵ (0)M a= , the total number of software errors before

the testing work begins must be equal to a)

→ [ ] [ ] 2

0 0 0

1ln ( ) ( )d ln d d ( )2

T T T

M T d t t a t W tσ σ= − + − −∫ ∫ ∫

→2

0

1( )d ( )2( )

T

d t t T W TM T ae

σ σ− − −∫= . (5)

Consider ( )M T as a random variable, and the corresponding expectation can be obtained via the following procedures:

[ ]2

0

2

0

1( )d ( )2

1( )d ( )2

( )T

T

d t t T W T

d t t T W T

E M T E ae

a e e E e

σ σ

σ σ

− − −

− − −

⎡ ⎤∫= ⎢ ⎥⎣ ⎦

⎛ ⎞⎛ ⎞∫ ⎡ ⎤= ⎜ ⎟⎜ ⎟ ⎣ ⎦⎝ ⎠⎝ ⎠

. (6)

Since2

21( ) 2 21 d

2

x TW T x TE e e e x eT

σσ σ

π∞ −− −

−∞⎡ ⎤ = =⎣ ⎦ ∫ ,

Equation (6) can be simplified as follows:

[ ]2 2

0 0

1 1( )d ( )d2 2( )

T T

d t t d t tT TE M T a e e e a e

σ σ− −−⎛ ⎞ ⎛ ⎞⎛ ⎞∫ ∫= =⎜ ⎟ ⎜ ⎟⎜ ⎟⎝ ⎠⎝ ⎠ ⎝ ⎠

. (7)

Accordingly, the expectation of the mean value function can be presented as Equation (7)

1845

[ ] [ ][ ]( )

0

0

( )d

d ln ( ) dd

( ) ( )

( ) ( )

T

T

d t t

M t tt

E m T E a M T a a e

a ae a M t m T

−⎛ ⎞∫= − = − ⎜ ⎟⎝ ⎠

∫= − = − =

. (8)

Likewise, the variance of the mean value function can be deduced as follow:

[ ] [ ] [ ]22( ) ( ) ( ) ( )Var m T Var M T E M T E M T⎡ ⎤= = −⎣ ⎦ . (9)

In order to obtain [ ]( )Var m T , 2( )E M T⎡ ⎤⎣ ⎦ and

[ ]2( )E M T need to be deduced in advance. The deduction

process of 2( )E M T⎡ ⎤⎣ ⎦ and [ ]2( )E M T can be seen as follows:

( )

2

0

20

2 ( )d 2 ( )2 2

2 ( )d2 2 ( )

( )T

T

d t t T W T

d t t T W T

E M T E a e

a e e E e

σ σ

σ σ

− − −

− − −

⎡ ⎤∫⎡ ⎤ = ⎢ ⎥⎣ ⎦⎣ ⎦

⎛ ⎞∫ ⎡ ⎤= ⎜ ⎟ ⎣ ⎦⎝ ⎠

. (10)

Since 2

22 ( ) 2 221 d2

xW T x TTE e e e x e

Tσ σ σ

π∞ −− −

−∞⎡ ⎤ = =⎣ ⎦ ∫ ,

Equation (10) can be simplified as follows:

( )20

2 ( )d2 2( )T

d t t TE M T a e eσ−⎛ ⎞∫⎡ ⎤ = ⎜ ⎟⎣ ⎦⎝ ⎠

. (11)

Likewise, referring to Equation (7), we have known

[ ] 0( )d

( )T

d t tE M T a e

−⎛ ⎞∫= ⎜ ⎟⎝ ⎠

so [ ]2( )E M T can be obtained as

follows by intuition:

[ ] 02 ( )d2 2( )

T

d t tE M T a e

−⎛ ⎞∫= ⎜ ⎟⎝ ⎠

. (12)

Therefore, according to Equations (11) and (12), the variance of the mean value function can be presented as follows:

[ ] ( )[ ]( ) ( ) ( ) ( )

20

2 2

2 ( )d2

2 2

( ) 1

( ) 1 ( ) 1

T

d t t T

T T

Var m T a e e

a E m T e a m T e

σ

σ σ

−⎛ ⎞∫= −⎜ ⎟⎝ ⎠

= − − = − −. (13)

According to Equations (8) and (13), we have got the general forms of the expectation [ ]( )E m T and the

variance [ ]( )Var m T of the mean value function. However, the application in practice will be needed to specify which model you choose. Therefore, we use the detection rate function in Table 1 to calculate the definite integral in [0, T]. The real form of expectation and the variance of the mean value function are presented in Table 2.

Table 2 Expectation and variance of the mean value function for the five classic models

Model Expectation and Variance of Mean Value Function

Goel and Okumoto’s

model (1979)

[ ]( ) (1 )bTE m T a e−= −

[ ] ( )22 2( ) 1bT TVar m T a e eσ−= −

Yamada’s Delayed S-

shaped model (1983)

[ ]( ) (1 (1 ) )bTE m T a bT e−= − +

[ ] ( )22 2 2( ) (1 ) 1bt TVar m T a bT e eσ−= + −

Ohba’s Inflection S-shaped model

(1984)

[ ] 1( )1

bT

bT

eE m t ace

−

−

⎛ ⎞−= ⎜ ⎟+⎝ ⎠

[ ] ( )2

2

2 (1 )( ) 11

bTT

bT

c eVar m T a ece

σ−

−

⎛ ⎞+= −⎜ ⎟+⎝ ⎠

Yamada’s exponential

model (1992)

[ ] ( )(1 )( ) 1Tr eE m T a e

βα −− −= −

[ ] ( )22 2 (1 )( ) 1Tr e TVar m T a e e

βα σ−− −= −

Chiu and Haung’s

learning effect model (2008)

[ ]( )

1( ) 1

T

E m T ae α η

ηα

ηα

+

⎛ ⎞+⎜ ⎟= −⎜ ⎟

⎜ ⎟+⎜ ⎟⎝ ⎠

[ ] ( )2

2

2

( )

1( ) 1T

T

Var m T a ee

σ

α η

ηα

ηα

+

⎛ ⎞+⎜ ⎟= −⎜ ⎟

⎜ ⎟+⎜ ⎟⎝ ⎠

3.2 Estimation of Parameters

Since these models in Table 2 involve unknown parameters which can be estimated from the observed data, the method of maximum likelihood is utilized to estimate the values of unknown parameters. The set of paired data ( , )i iT m can be collected in practice, where im is the total number of errors detected until iT . Also, suppose that the unknown parameters of the specified SRGM are determined by the 1n + observed paired data:

0 0 1 1 2 2( , ), ( , ), ( , ),..., ( , ),n nT m T m T m T m then the likelihood function for the SRGMs can be expressed as

( ) ( )1 1

1 1 2 2 3 3

( ) ( ( ) ( ))1

1 1

Pr{ ( ) , ( ) , ( ) ,..., ( ) }

( ) ( )( )!

i i i i

n n

m m m T m Tni i

i i i

L N T m N T m N T m N T m

m T m T em m

− −− − −

−

= −

= = = = =

−=

−∏. (14)

Taking a logarithm of the likelihood function, the MLE of the unknown parameters can be obtained by numerically solving the simultaneous equation

ln( )(unknown parameter 1)

L∂∂

= ln( )(unknown parameter 2)

L∂∂

=

… = ln( )(unknown parameter k)

L∂∂

=0.

3.3 Estimation of Confidence Intervals

In this section, the relevant confidence intervals of the SRGMs in Table 2 are developed; the confidence interval of the mean value function can be developed using Equations (8) and (13), which is given by

1846

[ ] ( )( )

2

/2,2

1

1( ) [ ( )] 1CR n k n

ii

T tE m T t m T

n t tσ−

=

−± + +

−∑, (15)

where CR denotes the critical region, /2,CR n kt − denotes the value providing an area 2CR of the Student-t distribution with n k− degrees of freedom.

However, in the classical model, owing to the different expression of 2σ̂ , the confidence intervals that are correspondent with Equation (15) would be somewhat dissimilar. For instance, in the classical model, the confidence interval of the mean value function is expressed as

[ ]( ) ( )

( )

22

1/2,

2

1

( )1( ) 1

n

i ii

CR n k n

ii

m m t T tE m T t

n k n t t

=−

=

⎛ ⎞⎛ ⎞−⎜ ⎟⎜ ⎟−⎜ ⎟⎜ ⎟± + +

−⎜ ⎟⎜ ⎟−⎜ ⎟⎜ ⎟⎝ ⎠⎝ ⎠

∑

∑. (16)

Since the classical model assumes that the variance in mean value functions stems from ( )m T , whereas the proposed model assumes that the variance in mean value functions stems from ( )d T , confidence intervals in the classical model would be divergent in the late testing phase, but convergent for the proposed model. This can be explained by the fact that the possibility of finding new software errors becomes lower due to less software errors remaining in the late testing phase. In such a case, the variance in mean value functions turns out to be fewer in the proposed model, which is opposite to the case in the classical model.

IV. CONCLUSION In this paper, a software reliability model is presented

using the SDE method to construct confidence intervals regarding the mean value function whose variance is assumed to stem from the error detection rate. It can assist software developers in determining optimal release times at different confidence levels. From the inference procedure,, the effects of time on the error detection rate and the variance in the mean value function would be crucial to the shapes and confidence intervals regarding the mean value function.

Furthermore, after comparing the proposed model with the classical model, it is discovered that confidence intervals regarding the mean value function would be convergent in the proposed model but divergent in the classical model as testing time elapsed, since the possibility of finding new software errors becomes lower due to the fewer remaining software errors in the late testing phase, and the variance in mean value functions then turns out to be smaller in the proposed model, which is opposite to the case of the classical model.

Future research may estimate the unknown parameters in the proposed model through expert opinions due to the lack of historical data, and this can be

conducted by using a Bayesian decision approach coupled with the proposed model.

REFERENCES

[1] Goel, A.L. and Okumoto, K. (1979) “Time-dependent fault detection rate model for software and other performance measures”, IEEE Transactions on Reliability, 28: 206-211.

[2] Gokhale, S. S. and Trivedi, K. S. (1999) “A time/structure based software reliability model”, Annals of Software Engineering, 8: 85-121.

[3] Huang, C.-Y. (2005) “Performance analysis of software reliability growth models with testing-effort and change-point”, Journal of Systems and Software, 76: 181-194.

[4] Karatsas, I., Shreve, S. (1997) Brownian Motion and Stochastic Calculus, 2nd ed. Springer-Verlag: New York.

[5] Lee, C.H., Kim, Y.T., Park, D.H. (2004) “S-shaped software reliability growth models derived from stochastic differential equations”, IIE Transactions, 36: 1193-1199.

[6] Ohba, M. (1984a) “Inflexion S-shaped software reliability growth models”, in Stochastic Models in Reliability Theory, Osaki, S. and Hatoyama, Y., Eds. Berlin, Germany: Springer-Verlag, 144-162.

[7] Ohba, M. (1984b) “Software reliability analysis models”, IBM Journal of Research and Development, 28: 428-443.

[8] Tamura, Y., Yamada, S. (2006) “A flexible stochastic differential equation model in distributed development environment”, European Journal of Operational Research, 168: 143-152.

[9] Yamada, S., Ohba, M., and Osaki, S. (1983) “S-Shaped software reliability modeling for software error detection”, IEEE Transactions on Reliability, 32: 475-484.

[10] Yamada S., Osaki. S. (1985) “Software reliability growth modeling: Models and applications”, IEEE Transactions on Software Engineering, 11: 1431-1437.

[11] Yin, L., Trivedi, K.S. (1999) “Confidence Interval Estimation of NHPP-Based Software Reliability Models”, Proceedings of the 10th International Symposium on Software Reliability Engineering, November: 6-11.

1847