IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. SE-6, NO. 3, MAY 1980

Error Sensitive Test Cases Analysis (ESTCA)KENNETH A. FOSTER, MEMBER, IEEE

Abstract-A hardware failure analysis technique adapted to softwareyielded three rules for generating test cases sensitive to code errors.These rules, and a procedure for generating these cases, are given withexamples. Areas for further study are recommended.

Index Terms-Program correctness, progran testing, software errors,software reliability, test data generation.

-----:> oTEST CASES l.

I

E;SPECIFICATION:COVER?-:..--!

(MarnualiExercise)

II lJlJTCOMPUT'ER COrLE :, ---: C()VER?

, :~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~1

II ( Ex.ecute).-

I. INTRODUCTIONCOMPUTER program testing is done to show compliance

of a program with its specification or to detect noncom-pliance errors. We require a test to have three elements, a unitunder test (UUT), a specification to test against, and test casesto apply to both the unit and its specification. The test casesmust "cover" all specification requirements and source/objectcode statements to be minimally thorough. When the testcases are applied to both the specification and the unit, the ac-tual results agree with specified results, the test is said to be"passed." Note that neither passing tests nor demonstratingcompliance imply correctness. Fig. 1 illustrates these concepts.

Code ErrorsCode errors are mistakes, "typos," or "bugs" which cannot

be detected in compilation or assembly such as:1) omitted or out of sequence operations or conditions;2) references to the wrong variables;3) substitution of relations or conjunctions in simple or

compound conditions (LT for LE; OR for AND, etc.);4) missing or incorrectly placed parentheses, incorrect

grouping of variables in an arithmetic expression;5) substitution of arithmetic operations (* for **, etc.);6) incorrect constant values as factors in arithmetic compu-

tations or an limit counters;7) arithmetic operations on variables in the wrong sign form

(load or store positive, negative, with complement sign,signed, etc.).

Programs with these code errors will usually compile, assem-ble, and execute. Though program testing rules should at leastbe adequate to detect such code errors, Goodenough and Ger-hart [11 and Howden [2] show that current structural or to-pological testing rules such as

1) execute all logical paths through the program, or2) execute all instructions at least once, or3) exercise all individual predicates

will not necessarily detect these errors.

Manuscript received February 1, 1979; revised July 16, 1979.The author is with GTE Sylvania, Inc., Needham Heights, MA 02194.

-------

SPECIFIED: RESULTS 5

AC TTLALRESULTS 5_ _ _ _ _ _ _

AGREE? ---:>Noy Faily Reject

Yesy Pass; Accept

Fig. 1. Test concepts.

II. ANALYSIS APPROACHTo overcome current test inadequacies, the approach taken

was substantially the same as the early investigations into hard-ware fault pattern coverage and minimization. In the hard-ware studies, faults were simulated at every gate input-outputand test patterns were determined that would detect each fault.Then the test patterns which "covered" less faults than otherswere systematically eliminated. Eventually, algorithms that donot require identifying or enumerating the faults in order tocover them were developed. Hardware faults are usually lim-ited to "stuck-at-one, stuck-at-zero" nodes, but software hasrelation, conjunction, operation, and symbol errors in manymore modes than hardware.In this software study, combinations of compilable state-

ments in various source languages were written in the generalforms:

1) IF ARG-1 RELATION ARG-2 CONJUNCTION ARG-3 RE-LATION ARG-4 THEN DISPLAY "1" ELSE DISPLAY "0I"

2) MOVE ARG-1 TO ARG-X, MOVE ARG-2 TO ARG-Y,MOVE ARG-3 TO ARG-Z, DISPLAY ARG-X, ARG-Y,ARG-Z

3) COMPUTE ARG-1 = ARG-2 OPERATION ARG-3 OPERA.TION ARG-4, DISPLAY ARG-1

All ARG-n were defmed as signed integers or Boolean values.Actual relations (LT, LE, EQ, NE, GE, and GT) conjunctions(AND, OR), operations (+, -, /, *, and ** with parentheses) andsymbols (ARG-2, -3, -4, for ARG-1, etc.) were substituted inthese statements one at a time so that any one could be con-sidered correct and all others in error.The statements were compiled and executed using test cases

0098-5589/80/0500-0258$00.75 © 1980 IEEE

258

FOSTER: ERROR SENSITIVE TEST CASES ANALYSIS

TABLE ICONDITION CODES, VARIABLE VALUES, AND BRANCHES

Condition Values/Branch

ARG-1 Code ARG"2 A<B AxB A>B

B,1 0,B 1h0

A LT B

A LE

A EQ

A NE B

GE B

A GT 8

T F F

T T r

F r F

T T

F T T

f F T

TABLE IICONDITION CODES, CONSTANTS, AND BRANCHES

Condltlon ARGwI Values/Branch

ARG-1 Code CnSt

ARG-1 l,T 2'

ARG-1 bE 2

ARGwl EQ 2

ARGI1 NE 2

ARG-1 GE 2

ARGw 6T 2

ARG-1 L#r 3

with various values. The results and cases were then analyzedto determine the minimum number of differentiating cases,

and deduce rules for an algorithm to generate the error sensi-tive ones. Short routines such as sort three variables, find the

biggest number, etc., were also written to verify that the rulesapplied to combinations of code statements in meaningful con-

texts. The limitation of this study to signed integer arithmetic(though fractional results were allowed) precludes any com-

ment on errors in significance or precision of real numbers dueto incorrect normalization, truncation, scaling, or floating-point operations.

Error Sensitive Test Case Analysis (ESTCA) RulesThe foregoing analysis resulted in three surprisingly simple,

almost obvious, common-sense rules for generating error sensi-tive test cases:

1) Assign values to variables ARG-1 and ARG-2 at every sim-ple condition such that all strict ordering relations (<, , and>) exist between them where logically possible.To verify this rule, consider Table I. From Table I we can

see that if we meant to code NE and used test case valuesA,B =0, 1 and 0,0 (thus taking both branches) the cases

would take the same branches with the incorrect code LT. Or,if we use A, B = 0, 0 and 1, 0 again taking both branches, thecode could have been incorrectly written GT without detectingthe error. Since the same reasoning holds for any coded condi-tion, the test data must include values with all three strict or-

dering relations between arguments at every condition irre-spective of the actual code.2) When a variable is required to be less than (greater than)

a constant, assign the variable value so it differs from the con-

stant by the smallest decrement (increment) allowed.This rule makes the "off by 1" type errors in increments,

decrements, and limits detectable. For example, Table II

shows the branches taken for ARG-2 = 2, 3, and 4 with valuesfor ARG-1 ranging from 1 thru 5. Assume ARG-2 is supposedto be 3, as shown in the middle section of Table II. The upper

and lower sections show the branches when ARG-1 iS com-

pared with 2 and 4, respectively (off-by-one). Incorrectbranches are marked "*" for detectable erroneous compari-sons against 2 and 4. Neither test values 1 nor 5 for ARG-1

ARG-l

ARG

ARG-i

ARG-1

ARG.l

ILE 3

EQ 3

NE 3

GE 3

Gr 3

ARG-1 LT 4

ANG-1 LE 4

ARG.1 EQ 4

APG-1

ARG-1

ARC-1

N'E 4

GE 4

GT 4

T T F F F

T T Tr r

r T r r

T T T T

F T T T

FF rF T T

I T T*r r

T T T T* F

r r r* T* F

T T T* F* T

r r F T T

r F F T

take any incorrect branches with the "off-by-one" error,

therefore the < and > values used for the variable in a variablewith constant comparison should differ by the smallest valuefrom the constant.When the constant is a nonnumeric literal and the condition

is coded as EQ or NE, only the two test cases of equality andinequality are required because the manual specification exer-

cise should reveal "misspelling" errors.

3) Assign values to external input variables so that they donot have the same value and sign in every test case and also donot have the same value and sign as any other variable in alltest cases. Also, ensure that the variables in arithmetic opera-tions have a measurable effect on the sign and magnitude ofthe result.

If an external input variable always has the same value or

sign in every test case then the code could be in error by ref-erencing a constant rather than the variable. And if two or

more variables always have equal values then any error refer-ence to either could be addressing the wrong one. Note thatthe application of rules 1 and 2 will often ensure meeting rule3 except when there is straight line code without conditionalbranches or the variables referenced in conditional compari-sons are not the same as those referenced in other operations.

(< a I )

1 2 3 4 5

T r* r r r

T T r* F

F T* * 7 F

T F* T*T T

F T* T T T

r r T*T r

259

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. SE-6, NO. 3, MAY 1980

These rules are clearly necessary (though not necessarily suf-ficient) to make code errors detectable. The rules imply threecases for every simple condition and two or more cases forevery decision to decision branch involving any operations(other than GOTO). However, there is usually a need to mini-mize the number of cases so that they may be manually veri-fied against the specification in a reasonable time. Some casesmay be eliminated because there would be no difference in re-sults if the incorrect path were taken. For example, the codein an ascending bubble sort IF B < A THEN INTERCHANGEB, A ELSE GET NEXT PAIR wiul requlre values A < B and A >B, but the pair A = B can be omitted because the results willbe the same for both the THEN and ELSE branches; there is noway to tell from the results whether A and B were interchangedor not. (Though LE instead of LT is inefficient, it is not logi-cally incorrect in this example.) In applying rules 1 or 2 toeach simple condition, it seems only necessary to use test caseswhich meet prior and subsequent conditions on a "criticalpath" in the code.Definition: A critical path is a set of logically connected

branches from entrance to exit which traverses the highestnumber of simple conditions and operations and exits a termi-nal condition on the branch having external results differentfrom other terminal branches.The critical path concept is an adaptation of hardware path

sensitization. The idea is that external input which passesthrough the most conditions and operations will usually pro-duce extemal results different from results on other paths.Hence if a condition or operation on that path has a code errorthe input will produce a different result than specified. Onecan frequently determine a critical path by taking Falsebranches from OR and inequality conditions, and Truebranches from AND and equality conditions. The applicationof rules 1 or 2 to conditions on a critical path provides execu-tion of all branches from that path.

III. ESTCA GENERATING PROCEDUREGenerating test cases to meet rules 1 and 2 is comparable to

solving the "traveling salesman" problem where the code is anetwork of nodes (conditions) and branches (operations andGOTO's), but the values at each condition are dependent onoperations along the precedent branches. This "predicate de-termination" problem does not have an automatic solution atthe present time, so the procedure described here is manualand heuristic. The procedure has been developed only for top-down, structured code modules with single entry and exitpoints.The procedure is as follows.1) Obtain a flowchart or indented listing of the source code

with all compound conditions expressed as simple conditions.A flowchart is not a strict necessity but is a convenient way offinding paths and branches.2) Find the most critical path. If there are unvalued inter-

nal variables along a critical path, first select some other path(s)which will value these variables.3) At the last simple condition on a critical path, assign vari-

able values to meet rules 1 or 2. If there are no branching con-ditions, assign variable values per rule 3, or use the proceduregiven in the next section for arithmetic expressions.

4) Work assigned values backwards from the exit to the en-try along this path by "playing computer" with the flowchartor indented listing. At each reference to an undefined extemalvariable assign a value (preferably not equal to zero, one, orpreviously assigned to another variable) such that the criticalpath will be taken. Use rule 3 to assign variable values to op-erations on a branch.5) At each call or invocation of a subordinate module or

subroutine assign a case with extreme values to pass to the sub-ordinate, and as many cases as necessary to return extreme val-ues from the subordinate. If the values are signed numerics,also provide the same cases with complemented signs. Providestub code which will return the appropriate values for thesecases.This step is necessary to catch interface errors between mod-

ules such as source of inputs, destination of outputs, and possi-ble truncation or re-signing. The stub code would be deletedwhen the subordinate module has been fully tested and inte-grated. Thus, limits or boundary tests become the responsibil-ity of the using, rather than the subordinate module. Also,since each subordinate module should be tested before integra-tion, the superior modules are not required to fully exercisethe subordinate.6) Select the next prior simple condition on this path and

establish the relations between its arguments. Provide valuesfor any missing relations unless the results would be the sameeven if the incorrect branch of that condition were taken.7) Repeat steps 2 thru 6 for a path, then select the next crit-

ical path, until all conditions meet rules 1 or 2, and all opera-tions on every branch meet rule 3.

Arithmetic ExpressionsSince the primary concern was with nonnumeric applications,

the arithmetic expressions analyzed had insufficient variety foruniversal application. However, the following procedure wasfound to be sufficient for generating cases to meet rule 3 ininteger arithmetic computations.

1) Take the set of integers from 2 to v + 1 where v is equalto the number of variables. Delete any integer used as a con-stant in the expression, and add 1 to the highest member foreach constant deleted. Note that zero and one are not used inthis set because they usually will not produce different resultswhen there are either symbol or operation errors.2) To generate test case 1, assign lowest odd values from the

set to exponent variables, lowest even values to exponentiatedvariables, and next higher values to divisors. Assign the re-maining values to the other variables.3) If the variables are constrained as absolute (signs cannot

be changed) then add 1 to test case 1 values to generate testcase 2 and omit the next step. Else, complement the sign ofall variable exponents and an odd number of variables in everysum and difference term of test case 1 to generate test case 2.4) Complement the sign of all variables in test case 2. If the

set is identical to test case 1 delete it, else use the set as testcase 3.5) If any term of the expression is not within an order of

magnitude of the result of the expression, rescale the setformed in step 1 and repeat steps 2 thru 4.For example (assuming divisor F cannot equal zero):

260

FOSTER: ERROR SENSITIVE TEST CASES ANALYSIS

COMPUTE X = A + B ** C + (D * E) / F

Case 1: 5 2 3 6 7 4Case 2: -5 -2 3 -6 -7 -4Case 3: 5 2 -3 6 7 4

The results--for x, to two decimal places, are 23.50, -23.50, and15.63. If the variables had been constrained as absolute, testcases 2 and 3 would have been replaced by A, B, C, D, E, F =6, 3, 4, 7, 8, 5 with result 98.20. Since each term is within anorder of magnitude of the result, the rescaling of step 5 is notnecessary.When the result of an arithmetic expression must be con-

strained to some value to meet rules 1 or 2, it may be neces-sary to use some data set other than the one proposed in step1. When this occurs, equal values may be used for some vari-ables providing rule 3 is still met. Note that this proceduredoes not verify signifilcance and precision requirements, norprove that the code is a ''correct" implementation of the spec-ified function.

IV. EXAMPLESExample 1Consider the specification "if the Boolean variables A, B and

c are all set (=1), output 'TRUE' otherwise output 'FALSE' toillustrate a software equivalent of a hardware three-input ANDgate." One possible software implementation is

1020304050

60708090100110

VAR

A, B, C, RESULT: BOOLEAN

BEGINREAD (A, B, C)IF (A=B)AND (B = C)AND (C = 1)

THEN RESULT:= TRUEELSE RESULT: = FALSE: END

WRITE(RESULT)END.

The critical path is through the True branches of lines 50,60, and 70 for RESULT = TRUE. Using 1 for TRUE and 0 forFALSE, test case I iS A, B, C = 1, 1, 1.The relationship of c to 1 in the first test case is equal; fol-

lowing rule 2 we would like to get a value c > 1 (impossible)and a value c < 1 (possible) at 70. Therefore our next testcase following most of the critical path is A, B, C = 0, 0. 0.The condition at line 70 has been fully exercised, so we

move backwards along the critical path to the preceding condi-tion at line 60. Both previous cases had B = C, so we wouldnow like B > c and B < C, also with A = B. The only way toget B > c and A = B is with A, B, C = 1, 1, 0. But this test case

would have the same result if there were no error in 60 be-cause 70 would take the false branch anyway. We are left withA, B, C = 0, 0, 1 and only the B <C test case relation.Line 50 is the next condition. We have no cases there for

A > B and A < B. To follow the critical path (in the event ofoniission or error in 50) both B, C must equal 1, 1, leaving us

with the fmal case A, B, C = 0, 1, 1.The test cases are therefore A, B, C = 1, 1, 1; 0, 0,0;0,0, 1

and 0, 1, 1 which should result in TRUE, FALSE, FALSE, and

FALSE, respectively. The reader can verify that these cases willdetect all single errors in this source code. The other casespossible (0, 1, 0; 1, 0, 0; 1, 0, 1 and 1, 1, 0) detect no singleerrors.

It is interesting to note that the test cases for a hardwarethree-input AND gate in any circuit technology and non-redundant realization are always A, B, C = 1, 1, 1; 0, 1, 1; 1, 0,1 and 1, 1, 0, but software tests depend on the actual codeimplementation.

Example 2Because ESTCA is sensitive to code errors it is also sensitive

to what the code actually does. This example, from Good-enough and Gerhart [1 ] , illustrates its effectiveness on a designor logical error, rather than a single code error.Specification:

1) x, Y, and z are positive integers including zero;2) for each x, Y, z output "YES" if they are all equal, else

output "NO."Code:

1) COMPUTE W =( IXI+ IYI+ IZI)/ 32) IF W = X PRINT "YES"3) ELSE PRINT "NO."

Applying the ESTCA procedure to the code, we would workbackward from the condition and assign the values as follows.

1) Assign x = 2, then assign Y = 3, z = 4, yielding w = 3. Wenow have w > x and result "NO."

2) Assign x = 3, then assign Y = 2, z = 4, yielding w = 3.Now we have w = x and result "YES."

3) Assign x = -3, Y = -2, z = -4 again yielding w = 3, show-ing that the signs do not affect the result. w = x, and theresult is "YES."

4) Assign x = 4, then assign Y = 2, z = 3, yielding w = 3. Fi-nally we have w < x and result "NO."

The test cases meet the applicable rules and cover the code(= 100 percent). But when we apply the test cases to the speci-fication we fmd that there is no case x = y = z (<100 percentcoverage) and cases 2 and 3 specified results do not agree withactual results. If we had derived the test cases randomly andexecuted both branches we would have had less than a 3.2 per-cent chance of detecting the error. (From x, Y, z = 0, 0, 0 to9, 9, 9 only 32 combinations give wrong results.) Similarly, ifwe had used a rule like "select cases at lower and upperbounds and in the mid-range" we would generate cases likex, Y, z = 000, 001, 010, 100, 999, 998, 989, 899, 555, 554,545,455-none of which can detect the error.

Example 3The following example ilustrates the procedure using a pro-

gram originally due to Naur [3]. The history of "successful"proofs of incorrect versions of this program is extensively dis-cussed in Goodenough and Gerhart [11 together with their"corrected" version containing two errors [4]. The programalso has several loops which show how ESTCA's essentiallycombinatorial rules are adapted to looping programs.Specification:1) The input is a series of characters, classified as break

and nonbreak characters. A break character is a blank,"-" (newline indicator), or "." (end of text).

261

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. SE-6, NO. 3, MAY 1980

10 CONST20 MAXPOS = 630 VAR40 CW: CHAR;50 BUFFER: ARRAYEO. .MAXPOS3 OF CHAR;60 ALARM: BOOLEAN;70 BUFPOS, FILLY K: INTEGER;

B 1 80 BEGIN90 ALARM F=FALSE;

100 BUFPOS :- 0;110 FILL :° 0;

IR 2 120 REPEAT130 READ(TTYPCW);

I 3 140 IF (CW = '150 OR (CW = '-)160 OR (CW = '.')

T 3 170 THENB 4 180 BEGINI 5 190 IF BUFPOS # 0

T 5 200 THENB 6 210 BEGINI 7 220 IF (FILL + BUFPOS < MAXPOS)

230 AND (FILL # 0)

T 7 240 THENB 8 250 BEGIN

260 WRITE(TTY.' ';270 FILL := FILL + 1;

E 8 280 ENDS 7 290 ELSE

B 9 300 BEGIN310 WRITE(TTY,'-');320 FILL :- 0;

E 9 330 END;340 FOR K : 1 TO BUFPOS DiO350 WRITE(TTY.BUFFEREK3);360 FILL FILL + BUFPOS;370 BUFPOS := 0;

E 6 380 ENDE 4 390 ENDS 3 400 ELSE

I 10 410 IF BUFPOS - MAXPOST 10 420 THEN ALARM :- TRUES 10 430 ELSE

B 11 440 BEGIN450 BUFPOS : BUFPCS + 1;460 BUFFEREBUFPUS3 :I= CW;

E 11 470 ENDU 2 480 UNTIL (ALARM)

490 OR (CW = 1.')f500 WRITE(TTY,ALARM);

E 1 510 END.

Fig. 2. Naur program.

2) The final input character is a ".".

3) A word is a nonempty sequence of nonbreak characters.4) A break is a sequence of one or more contiguous break

characters.(Thus the input can be viewed as a sequence of words

separated by breaks with possible leading and trailingbreaks, and ending with ".".)

5) The program output shall be words with characters inthe same sequence as the input, and newline indicators.

6) A new output line shall start only between words and atthe beginning of the output text, if any.

7) A break in the input shall be reduced to a single breakin output.

8) As many words as possible shall be placed on each line[i.e., between successive newline indicators ("-")] .

9) No line may contain more than MAXPOS characters(words and blanks).

10) An oversize word (i.e., a word containing more thanMAXPOS characters) shall cause the variable ALARM tobe set true and an immediate program exit.

11) The program shall be written to permit MAXPOS to beeasily changed for various applications.

An indented listing of an implementation (Version 1 Pascalfor the DEC 20) to meet this specification is shown in Fig. 2,

and its flowchart in Fig. 3. From the indented code or theflowchart we find that the critical path is through lines 140False, 150 True (150 False and 160 True cause a prematureexit at 490 True) 190 True, 220 True, 230 True, 340 loop,480 False and 490 False, then back to 120 REPEAT. On thispath we encounter the variable MAXPOS which has an unspeci-fied value.We wish to minimize MAXPOS for testing purposes subject to

the path constraints that it can be <, =, and > the sum FILL +BUFPOS, both of which must at least be equal to 1 to traversethis path. We would like, per rule 3, to have FILL, BUFPOS =3,2 and 2,3 for MAXPOS=5, but since FILL will be incre-mented in line 270 before the critical path can be reentered,we assign MAXPOS = 6 as shown in statement 20. For testingpurposes we also assign the READ'S and WRITE'S to TTY, andadd line 500 to WRITE the ALARM state at exit.Note that in exercising line 220 FILL + BUFPOS < MAXPOS

conditions we must satisfy both rules 2 and 3 because we haveset MAXPOS to a constant and because the condition is basedon the sum of two variables. The test cases therefore mustshow that FILL iS altered independent of BUFPOS and viceversa, and that the sum is off-by-one and equal for eachvariable.We cannot take the critical path initially in applying the

ESTCA procedure because 190 BUFPOS = 0. Since BUFPOS iSonly incremented in 450, we must first take the path 140False, 150 False, 160 False, 410 False, 480 False, and 490False which requires a nonbreak character to be input first.We therefore assign "A" to cw and follow that path.Having set BUFPOS = 1 in 450 we can now revert towards the

critical path assigning "-" to cw. The "-" will follow the path140 False, 150 True, 190 True, 220 True, 230 False, 480 False,490 False, and back to the REPEAT. During the exercise of310 through 370 newline A would be output, BUFPOS reset to0, and FILL set to 1.We next input the characters B-C- and D- which would be

output as blank B blank c and newline D. This series incre-ments FILL from 1 to 3 and 5 and sets BUFPOS = 1 when en-tering the FILL + BUFPOS < MAXPOS condition. We have onlysatisfied the equality relation of rule 2 for FILL because theless than relations were not off-by-one.We then input EF- followed by G- and H-. This series will be

output as blank EF blank G and newline H. The series bringsFILL from 1 to 4 and 6 with BUFPOS = 1 thus satisfying theoff-by-one rule for both the less than and greater than rela-tions at the line 220 condition. At this point we have com-pleted the cases necessary to vary FILL.Now with FILL = 1, we input IJKL- which sets BUFPOS = 4,

meets the less than by-one relation, and would output blankIJKL. Next, inputting M- followed by NOPQR- brings FILL = 1and BUFPOS = 5 to meet the equality relation, and would out-put M on one line and NOPQR on the next. Finally we inputs- followed by TUVWXY, again setting FILL = 1 but settingBUFPOS = 6, to complete the greater than by-one relation. Theoutput should be newline s then newline TUVWXY. This com-pletes the exercise of varying BUFPOS to meet rules 2 and 3.We now input two newlines, a blank, z, and a period to exer-

cise the true branches of the OR'S at lines 140 and 160, the

262

FOSTER: ERROR SENSITIVE TEST CASES ANALYSIS

:ALARM 5 FALSE 90::BUFPOS : = O 100:FILL 0=O 110

!READ[TTYPCW3 130:

:ECW = ' ') OR 140:.-_T_: [CW = '-') OR 150:_F .--------------::CW = '.'3 160:

.-____________T_:BUFPOS 0 190:_F_. .-T- BUFPOS = MAXPOS 410X _F-.

---------: --------- ALARM : = TRUE 420' BUFPOS t = BUFPOS + 1[EFILL + BUFPOS < :--------------- -: 450-460:

._T_tMAXPOS3 AND 2201-F- .---------------------:CFILL * 03 230: :BUFFEREBUFPOS3 := CW

uWRITEETTYP' '3 260: :WRITE[TTYr'-'3 310:FILL :- FILL + 1 2701 FILL := 0 320:

.F-!K'= 1 TO BUFPOS 340:

uWRITEETTYPBUFFER[K33 350:.

*FILL:.=FILL+BUFPOS 360::BUFPOS := 0 370:.

:.ALARM OR 480:[ECu = '.'] 490L-F.--

!uRITEETTYPALARM3 5oo:

Fig. 3. Naur flowchart.

false branch of the line 190 BUFPOS NE 0 condition, and exitthe program. The output should be newline z and ALARM

"false." Test case 1 is now completed, and consists of the fol-lowing character string: "A-B-C-D-EF-G-H-IJKL-M-NOPQR-S-TUVWXY--Z ."

We now simply have to exercise the line 410 True branch ofBUFPOS = MAXPOS by a string of seven characters: "ABCDEFG"which will invoke line 480 True branch to exit with ALARM"true."An evaluation of the cases shows that all code lines are cov-

ered and the test case rules are met with the exception ofmagnitude relations at 140 through 160. Since these charactercodes are nonnumeric literals and the comparisons are equali-ties, no additional test case values are necessary. Given thetest cases and the value of MAXPOS used, we would arrive atspecified results:

1. A B CD EF G

H IJKLM

NOPQRS

TUVWXYz

2.

falsetrue

which agree with actual results from executing the program.

Specification items 5 through 10 are met. Item 11 can be veri-fied by inspection of line 20 and other lines referencingMAXPOS.

Error AnalysisWe now show that if the test cases were derived from erro-

neous code then the test cases generated would not cover thespecification or specified results would not equal actual results.An ANSI Fortran version of the Naur program was coded

and subjected to program mutation as described by Lipton andSayward [5]. Fourteen hundred and ninety single error mu-tants had different results than specified, thus detecting thesingle errors. Twenty-four mutants produced the same resultsas the original code, but are logically equivalent programs (e.g.,NE zero is equivalent to GT zero for positive numbers) or pe-culiar to the Fortran implementation. Only one nonequivalentmutant, line 270 FILL := FILL + FILL instead of FILLFILL + 1, produced the specified results. Inspection of thecode verifies that the correct increment was used and thereforethe nonequivalent mutant may be disregarded. However, if wehad generated the test cases from this erroneous version, wewould set MAXPOS = 2(3) + 2 = 8 since FILL would be dou-bled in line 270 before the critical path could be reentered.The test cases would start as in the correct versions, but theinitial results (literally, without translating "-" to new-line)are: "-A B C-D" which has only five characters on the first line,thus violating specification statements 8 and 9. In an earlier

263

264 IEEE TRANSACT

version of this paper the author failed to follow rule 3 andused MAXPOS = 4. Program mutation showed that some errorswere not detectable when all the rules were not followed, eventhough both the code and the specification were "covered"and the results agreed.Goodenough and Gerhart identified several logical or multi-

ple code errors in the Naur program to illustrate the ineffec-tiveness of basing tests solely on knowledge of a program's in-temal structure (not using ESTCA rules). These errors were:

1) incorrect predicate 220: FILL + BUFPOS <= MAXPOS2) missing action 270: FILL := FILL + 13) missing predicate 190: BUFPOS #04) missing predicate 230: AND (FILL #0)5) missing predicate 160: OR (cw ='.')

Error 1 is a logical or single code error where <= has beenused instead of <. The test generation procedure and resultingcases would be the same, but the actual result:

-A B C D-EF G H-IJKL M-NOPQR S-TUVWXY-Z'

has seven characters on the first and fourth lines violating spec-ification statements 9 and 10.Error 2 is a design or multiple code error. MAXPOS would be

set to 5, and the beginning test cases A-B-C-D-E would be gen-erated. However, the actual result:

-A B C D-E'

has seven characters on the first line again violating thespecification.Error 3 would be caught at compilation if associated 200

THEN, 210 BEGIN, and 380 or 390 END statements are not re-moved. When they are removed, the critical path would beexercised first with a "-" (newline, blank) which would resultin the initial line being indented one character, violating speci-fication statements 6 and 7.Error 4 would not affect test case generation, but the output

would start with a blank, violating specification statement 6.Error 5 also would not affect test case generation. However,

the output would not include newline z, thus violating specifi-cation statements 5, 6, and 7.

V. CONCLUSIONHowden [6] states that actual data testing is not reliable in

detecting path-domain errors because there is no reason that aprogrammer will be lucky enough to choose data from an in-correct part of the domain. The ESTCA rules and procedureremove the "luck" part, and are necessary, though possiblynot sufficient.

IONS ON SOFTWARE ENGINEERING, VOL. SE-6, NO. 3, MAY 1980

Because the study was based on pragmatic, engineering con-siderations, the rules and procedure cannot be claimed to bethe complete answer to test data generation. The procedurefor arithmetic expressions lacks proof, and should be extendedto reals. Formalization ofthe rules,as initiated by Goodenoughand Gerhart, is clearly desirable.Much work remains to be done in test case generation;

ESTCA is a step in this direction which overcomes several de-ficiencies in current testing practice.

ACKNOWLEDGMENTThe author is grateful to R. McDowell of GTE Sylvania for

many helpful discussions. Special thanks are due to Prof. F.Sayward of Yale University for the Fortran coding and pro-gram mutation of the Naur example.

REFERENCES

[1] J. B. Goodenough and S. L. Gerhart, "Toward a theory of testdata selection," IEEE Trans. Software Engineering, vol. SE-1, June1975; also, R. T. Yeh, Ed., Current Trends in Programming Meth-odology, Vol. II, Program Validation. Englewood Cliffs, NJ:Prentice-Hall, 1977, pp. 44-78.

[2] W. E. Howden, "Reliability of the path analysis testing strategy,"IEEE Trans. Software Eng., vol. SE-2, Sept. 1976.

[3] P. Naur, "Programming by action clusters," BIT, vol. 9, pp. 250-258, 1969.

[4] J. B. Goodenough and S. L. Gerhart, "Correction to 'Toward atheory of test data selection,"' IEEE Trans. Software Eng., vol.SE-1, Dec. 1975.

[5] R. J. Lipton and F. G. Sayward, "The status of research on pro-gram mutation," in Dig. Workshop on Software Testing and TestDocumentation, Ft. Lauderdale, FL, Dec. 18-20, 1978, IEEEComputer Society Tech. Committee on Software Engineering.

[6] W. E. Howden, "Symbolic testing and the dissect symbolic evalua-tion system," IEEE Trans. Software Eng., vol. SE-3, July 1977.

Kenneth A. Foster (A'68) received the B.B.A.degree from Boston University, Boston, MA, in1949, and the C.D.P. in 1962.From 1953 to 1956 he was a Programmer/

Systems Analyst with Sperry Rand Corpora-tion. He joined GTE Sylvania in 1956, initially

_at their Data Processing Center in Camillus, NY.He has been with GTE Sylvania Eastern Divi-

- sion, Electronics Systems Group, NeedhamHeights, MA, since 1960, where he is currentlyManager, Software Quality Assurance.

Mr. Foster is a member of the Association for Computing Machineryand the IEEE Computer Society.