8/3/2019 Logic of Testing

1/422 CROSSTALK The Journal of Defense Software Engineering March 2006

Test case is a fundamental term usedby all software testers. Numerouspublished sources, including the Instituteof Electrical and Electronics Engineers(IEEE) Standard 610 (IEEE Std. 610),define what a test case is, techniques todesign test cases, and templates to docu-ment them. However, testers in the fieldstill find these definitions confusing, andthey frequently mean different things

when referring to test cases.

A common misunderstanding of testcases can be a symptom of a larger issue

a misunderstanding of the logic of soft-ware testing. The main purpose of soft-ware testing can be defined as exploringthe software product to derive and report

valid conclusions about its quality andsuitability for use. In this regard, softwaretesting is similar to mathematics theyboth need proofs for their conclusions.However, mathematicians surpass soft-

ware testers in deriving and proving theirconclusions thanks to their skill in using a

powerful tool called deductive reasoning. Toconstruct valid arguments, logicians havedeveloped proof strategies and tech-niques based on the concepts of symbol-ic logic and proof theory [1, 2, 3].

On critical software projects, testershave always been required to present validevidence supporting their conclusionsabout the products quality. The recentSarbanes-Oxley Act1 makes this require-ment much more important going for-

ward. In this article, I discuss the logic ofone of the conventional levels of testing

system test [4] and propose a formal

approach to constructing valid argumentssupporting testers conclusions. Finally,understanding the system test logic canhelp testers better understand the mean-ing of test cases.

Proofs and Software TestingSoftware testers have always dealt withproofs on their projects. One example canbe concluding that a system passed testing.As testers can never prove the absence ofbugs in a software product, their claimthat a system passed testing is conditional

upon the evidence and arguments sup-porting such a claim. On critical projects,either the projects manager, end-users, ora compliance department commonlyrequire documented test cases and testexecution logs to be used as grounds forsupporting testers conclusion that a soft-ware product passed testing.

Another example is reporting a systemfailure. Regardless whether it is formaltesting or unscripted exploratory testing,

testers are required to document andreport the defects they find. By reportinga defect, a tester first claims that a certainsystem feature failed testing and, second,presents an argument in the form of adefect description to support the claim.Such an argument should be logically

valid to be sufficiently convincing fordevelopers.

Deriving conclusions and presenting valid proofs, also known in mathematicsas logical arguments, is frequently not atrivial matter. That is why mathematiciansuse deductive reasoning as a foundation fortheir strategies and techniques to deriveconclusions and present valid arguments.Deductive reasoning is the type of rea-soning used when deriving a conclusionfrom other sentences that are accepted astrue [3]. As I discuss in this article, soft-

ware testers can also benefit from usingdeductive reasoning. First, they can betterunderstand the logic of software testingand, second, they can construct validproofs better supporting their conclu-sions about product quality.

Applying DeductiveReasoning to Software TestingIn mathematics, the process of deriving aconclusion results in presenting a deduc-tive argument or proof that is defined asa convincing argument that starts fromthe premises and logically deduces thedesired conclusion. The proof theory dis-cusses various logical patterns for deriv-ing conclusions, called rules of inference,that are used as a basis for constructing

valid arguments [2, 3]. An argument issaid to be valid if the conclusion neces-

sarily follows from its premise. Hence, anargument consists of two parts a con-clusion, and premises offered in its sup-port. Premises, in turn, are composed ofan implication and evidence. Implicationsare usually presented as conditionalpropositions, for example, (if A, then B)

They serve as a bridge between the con-clusion and the evidence from which theconclusion is derived [1]. Thus, the impli-cation is very important for constructing

a logical argument as it sets the argumentsstructure and meaning. In the following, I

will apply this concept to software testingand identify the argument componentsthat can be used in testing to construct

valid proofs.In software testing, we derive and

report conclusions about the quality of aproduct under test. In particular, in sys-tem testing a common unit of a testers

work is testing a software feature (alsoknown in Rational Unified Process as tesrequirement ); the objective is to derive conclusion about the features testing sta-tus. Hence, the feature status, commonlycaptured aspassorfail, can be considereda conclusion of the logical argument. Toderive such a conclusion, test cases aredesigned and executed. By executing testcases, information is gained, i.e., evidenceis acquired that will support the conclu-sion. To derive a valid conclusion, alsoneeded are implications that in systemtesting are known as a features pass/failcriteria. Finally, both the featurespass/fail criteria and the test case execu-tion results are the premises from which a

tester derives a conclusion. The lack ofunderstanding of how system testinglogic works can lead to various issues some of the most common of which I

will discuss next.

Common Issues With

Testing LogicIssue 1: Disagreeing About the

Meaning of Test CasesSoftware testers frequently disagree aboutthe meaning of test cases. Many testers

would define a test case as the whole set

Understanding the Logic of System TestingDr. Yuri Chernak

Valley Forge Consulting, Inc

What do system testing and mathematics have in common? They both deal with proofs. This article discusses the logic of sys-

tem testing, and the steps to construct valid proofs that testers need to perform for their conclusions about the quality of a soft-

ware product.

Software Engineering Technology

8/3/2019 Logic of Testing

2/4

Understanding the Logic of System Testing

March 2006 www.stsc.hill.af.mil 23

of information designed for testing thesame software feature and presented as atest-case specification. Their argument isthat all test inputs and expected results aredesigned for the same objective, i.e., test-ing the same feature, and they all are usedas supporting evidence that the featurepassed testing.

For other testers, a test caseconsists ofeach pair input and its expected result

in the same test-case specification. Intheir view, such a test-case specificationpresents a set of test cases. To supporttheir point, they refer to various text-books on test design, for example [4, 5],that teach how to design test cases forboundary conditions, valid and invaliddomains, and so on. Commonly, thesetextbooks focus their discussion ondesigning test cases that can be effectivein finding bugs. Therefore, they call eachpair of test input and its expected outputa test case because, assuming a bug is inthe code, such a pair provides sufficientinformation to find the bug and concludethat the feature failed testing.

Despite these different views, bothgroups actually imply the same meaningof the term test case: information that pro-

vides grounds for deriving a conclusionabout the features testing status.However, there is an important differ-ence: The first group calls test case theinformation that supports the featurespass status, while the second group callstest casethe information that supports thefeatures fail status. Such confusion appar-

ently stems from the fact that all knowndefinitions of the term test case do notrelate it to a features pass/fail criteria.However, as the discussion in the sidebarshows, these criteria are key to under-standing the meaning of test cases.

Issue 2: Presenting an ArgumentWithout a Conclusion

This issue is also very common. As dis-cussed earlier, an important part of a log-ical argument is its conclusion. However,a lack of understanding of this conceptcan lead to presenting arguments withoutconclusions. On a number of projects, Ihave seen testers produce test case docu-mentation in the form of huge tables orExcel spreadsheets listing their test cases.In such tables, each row shows a test caserepresented by a few columns such as testcase number, test input, expected result,and test case execution (pass/fail) status.

What is missing in this documentation isa description of what features testersintend to evaluate using these test cases.

As a result, it is difficult to judge the valid-ity and verify the completeness of such

test cases as the underlying purpose forwhich they were designed is not known.Such documentation suggests that thetesters who designed it do not completelyunderstand the logic of software testing.

Issue 3: Presenting an ArgumentWithout an Implication

This issue also stems from a lack ofunderstanding of the structure of a logi-cal argument, specifically that having an

implication is necessary for deriving a valid conclusion. In software testing,such implications are a features pass/failcriteria. The issue arises when such crite-ria are either forgotten or not clearlydefined and understood by testers. Thiscan lead to a situation where testers losesight of what kind of conclusions theyneed to report. As a result, instead ofderiving a conclusion about the featureand then reporting its testing status, theyreport the status of each executed testcase. This situation presents an issue

illustrated in the following example.Let us assume a tester needs to test 10

software features, and he or she designed10 test cases for each of the featuresunder test. Thus, the entire testingrequires executing 100 test cases. Now

while executing test cases, the testefound that one test case failed for each ofthe features. In our example, the tester didnot define and did not think about thefeature pass/fail criteria. Instead, the

tester reported to a project manager thetesting status for each executed test case.

Thus, at the end of the testing cycle, theresults show that 90 percent of testing

was successful. When seeing such resultsa manager would be fairly satisfied andcould even make a decision about releas-ing the system.

The project manager would see completely different picture if the fea-tures pass/fail criteria were not forgot-ten. In this case, the testers would reportthe testing status for each feature as

What Do We Call a Test Case?

Most of the published sources defining the term test casefollow the definitions givenin the Institute of Electrical and Electronics Engineers (IEEE) Standard 610 (IEEEStd. 610):

a) Test Case: A set of test inputs, execution conditions, and expected results devel-oped for a particular objective such as to exercise a particular program path orto verify compliance with a specific requirement.

b) Test Case: Documentation specifying inputs, predicted results, and a set of exe-cution conditions for a test item.

Despite the fact that this standard was published many years ago, testers in thefield still do not have a consistent understanding of the meaning of test cases. To bet-ter understand this meaning, we can use the concept of deductive reasoning.Following this concept, system testing can be viewed as a process of deducing validconclusions about the testing status of system features based on evidence acquiredby executing test cases. Hence, the main purpose of executing test cases is to gaininformation about the system implementation. This information can be used togetherwith the features pass/fail criteria to derive and support conclusions about the statusof feature testing. The features pass/fail criteria are important implications in the test-ing argument that determine the meaning of testers conclusions. These criteria are alink between a testers conclusion about the feature status and the test cases used tosupport the conclusion. Hence, the meaning of test cases follows from the definitionof the features pass/fail criteria.

In system testing, the mission is finding and reporting software defects; the featurefail criterion is commonly defined as, If any of the features test cases fails, then the

feature fails testing. What follows from this implication is that a test case is informa-tion that is sufficient to identify a software defect by causing a system feature to fail.The features pass criterion can further explain the meaning of test cases. It is com-monly defined as, The feature passes the test only if all of its test cases pass testing.According to this definition, we imply that the system feature passed testing only if thewhole group of its test cases passed testing. Hence, test cases are used as collectiveevidence to support the features pass status. It should be noted, however, that thisinterpretation of the test-case meaning refers to the system test only. In contrast, inacceptance testing a testing mission and pass/fail criteria can be defined differentlyfrom the system testing. Correspondingly, the meaning of test cases can be differentas well.

If the IEEE definitions of test cases are examined again, we can see that these def-initions are not specific to a particular testing mission, nor are they explicit about whichtesting conclusion, i.e., pass or fail, a test case is intended to support. Instead, theyfocus primarily on the test case structure: test inputs and expected results. As a result,

these definitions alone and without the features pass/fail criteria lack clarity about thetest case purpose and meaning.

8/3/2019 Logic of Testing

3/4

Software Engineering Technology

24 CROSSTALK The Journal of Defense Software Engineering March 2006

opposed to each test case. If the featurefail criterion were defined as, If any ofthe features test cases fails, then the fea-ture fails testing, then the testing end-result in our example would have beenquite the opposite and have shown thatnone of the software features passed test-ing; they all should be re-tested when thebugs are fixed.

An Approach to ConstructingA Valid ProofConstructing a valid proof in system test-ing can be defined as a four-step proce-dure. The following sections discuss eachstep in detail, and explain how to con-struct a valid argument to support a test-ing conclusion.

Step 1: Define a Conclusion of theArgumentIn constructing a proof, always begin bydefining what needs to be proven, i.e., the

conclusion. In system testing, the ultimategoal is to evaluate a software product. This is achieved by decomposing theentire functional domain into a set offunctional features where each feature tobe tested is a unit of a testers work thatresults in one of the two possible conclu-sions about a features testing status passorfail. At any given time, only one of thetwo conclusions is valid.

The term software featureis defined inthe IEEE Std. 610 as follows:a) A distinguished characteristic of a

software item.

b) A software characteristic specified orimplemented by requirements docu-mentation.From a testers perspective, a software

feature means any characteristic of a soft- ware product that the tester believesmight not work as expected and, there-fore, should be tested. Deciding what fea-tures should be in the scope of testing isdone at the test-planning phase and docu-mented in a test plan document. Later, atthe test design phase, the list of features isrefined and enhanced based on a betterunderstanding of the products function-ality and its quality risks. At this phase,each feature and its testing logic aredescribed in more detail. This informa-

tion is presented either in test designspecifications and/or in test case specifi-cations.

The test design specification com-monly covers a set of related features;

whereas, the test case specification com-monly addresses testing of a single fea-ture. At this point, a tester should alreadyknow which quality risks to focus on infeature testing. Understanding the fea-

tures quality risks, i.e., how the featurecan fail, is important for designing effec-tive test cases that a tester executes toevaluate the features implementation andderive a conclusion about its testing sta-tus. Performing Step 1 can help testersavoid Issue No. 2 as discussed earlier.

Step 2: Define an Implication ofthe Argument The next important step is to define animplication of an argument. An implica-tion of a logical argument defines an

important relation between the conclusionand the premises given in its support.Correspondingly, the features pass/fail cri-teria define the relation between the resultsof test-case execution and the conclusionabout the features evaluation status.

According to the IEEE Std. 829, thefeatures pass/fail criteria should bedefined in the test design specification;this standard provides an example ofsuch a specification. However, it does notprovide any guidance on how to definethese criteria, apparently assuming thisbeing an obvious matter that testers know

how to handle. Neither do the textbookson software testing methodology and testdesign techniques. Contrary to this view, Ifeel that defining these criteria is one ofthe critical steps in test design thatdeserves a special consideration. As I dis-cussed earlier and illustrated as Issue No.3, the lack of understanding of the roleand meaning of the features pass/fail cri-teria can lead to logically invalid testingconclusions in system testing. Also, as dis-cussed in the sidebar, from the well-defined implications, i.e., the featurespass/fail criteria, testers can better under-stand the meaning of test cases and avoidthe confusion discussed earlier as IssueNo. 1.

The rationale for defining the featurespass/fail criteria stems from the systemtest mission that can be defined as critically examining the software system under the fullrange of use to expose defects and to report conditions under which the system is not compliantwith its requirements. As such, the systemtest mission is driven by the assumptionthat a system is not yet stable and hasbugs; the testers job is to identify condi-

tions where the system fails. Hence, ourprimary goal in system testing is to provethat a feature fails the test. To do thattesters develop ideas about how the fea-ture can fail. Then, based on these ideastesters design various test cases for a fea-ture and execute them to expose defectsin the feature implementation. If thishappens, each test case failure providessufficient grounds to conclude that thefeature failed testing. Based on this logicthe features fail criterion can be definedas, If any of the features test cases failthen the feature fails testing. In logic, thisis known as a sufficient condition (if A,then B ). The validity of this implicationcan also be formally proved using thetruth-table technique [1]; however, thisgoes beyond the scope of this article.

Defining the features pass criterion isa separate task. In system testing, testerscan never prove that a system has nobugs, nor can they test the system foreverHowever, at some point and under certainconditions they have to make a claim thata feature passed testing. Hence, the sup-porting evidence, i.e., the test case execu-

tion results, can only be a necessary (C,only if D), but not a sufficient condition ofthe features pass status. Based on thislogic, the features pass criterion can bedefined as, The feature passes the testonly if all of its test cases pass testing. Inthis case, the features pass criterionmeans two things:a) The feature pass conclusion is condi-

tional upon the test execution resultspresented in its support.

b) Another condition may exist thatcould cause the feature to fail.

Step 3: Select a Technique to Derivea ConclusionOnce we have defined all components ofa testing argument, the next step is toselect a technique that can be used toderive a valid conclusion from the premis-es. The word validis very important at thispoint as we are concerned with deducingthe conclusion that logically follows fromits premises. In the proof theory, suchtechniques are known as rules of inference[1, 2]. By using these rules, a valid argu-ment can be constructed and its conclu-

Modus Ponens Form Testing Argument Form

1. IfA, then B means 1. Ifany test case fails, then a feature fails (implication).

2. A is true means 2. We know that at least one test case failed (evidence).

3. Then B is true means 3. Then the feature fails the test (conclusion).

Table 1: Deriving a Feature Fail Conclusion

8/3/2019 Logic of Testing

4/4

Understanding the Logic of System Testing

March 2006 www.stsc.hill.af.mil 25

sion deduced through a sequence ofstatements where each of them is knownto be true and valid. In system testing,there are two types of conclusions a fea-ture fail status and a feature pass status.Correspondingly, for each of these con-clusions, a technique to construct a validargument is discussed. On software pro-jects, testers should discuss and define thelogic of constructing valid proofs before

they begin their test design. For example,they can present this logic in the Test

Approachsection of a test plan document.

Deriving a Feature Fail Conclusion

I defined the features fail criterion as aconditional proposition in the form (if A,then B), which means if any of the testcases fail, then testers can conclude thatthe feature fails as well. This also meansthat each failed test case can provide suf-ficient evidence for the conclusion. In thiscase, a valid argument can be presentedbased on the rule of inference, known as

Modus Ponens[1, 2]. This rule is defined asa sequence of three statements (see Table1). The first two statements are premisesknown to be true and lead to the thirdstatement, which is a valid conclusion.

Deriving a Feature Pass Conclusion

The features pass criterion was defined asa conditional proposition in the form (C,only if D), which means a feature passes thetest only if all of its test cases pass testing.This also means that such a conclusion isderived only when all of the features test

cases have been executed. At this point,the feature status can be either pass or fail,but not anything else. Hence, the rule ofinference can be used, known as DisjunctiveSyllogism[1, 2], which is presented as threeconsecutive statements that comprise avalid argument (see Table 2).

Step 4: Present an Argument for aConclusion

At this point, there is a clear plan on howto construct valid arguments in systemtesting. The actual process of deriving atesting conclusion begins with executingtest cases. By executing test cases, thetesters can learn the systems behavior andanalyze the feature implementation bycomparing it to its requirements capturedby expected results of test cases. As aresult, testers can acquire evidence from

which they can derive and report a validtesting conclusion, i.e., a feature pass orfail testing status.

Concluding a Feature Fail Status

The feature fail criterion is defined as, Ifany of the test cases fails, then the feature

fails testing. According to the ModusPonens rule, this means that each failedtest case provides grounds for the validconclusion that the feature has failed test-ing. As a feature can fail on more than oneof its test cases, after finding the firstdefect a tester should continue featuretesting and execute all of its test cases.

After that, the tester should report allinstances of the feature failure by submit-ting defect reports, where each defect

report should be a valid argument thatincludes the evidence supporting the fea-ture fail status.

On the other hand, if a given test casepassed testing, the Modus Ponens ruledoes not apply, and there are no groundsfor any conclusion at this point, i.e., thefeature has neither passed nor failed test-ing. Finally, only when all of the featurestest cases have been executed should it bedecided whether there are grounds for thefeature pass status as discussed in the nextsection.

Concluding a Feature Pass Status

Obviously, if the feature has alreadyfailed, the pass status cannot havegrounds. However, if none of the testcases failed, then the DisjunctiveSyllogism rule can be applied. Accordingto this rule, the fact that none of the testcases failed provides grounds for a validconclusion: the feature passed testing. Tosupport this claim, evidence is provided test case execution results. However, thisconclusion should not be confused withthe claim that the feature implementation

has no bugs, which we know is impossibleto prove. The conclusion means only thatthe feature did not fail on the executedtest cases that were presented as evidencesupporting the conclusion.N

References1. Copi, I., and C. Cohen. Introduction

to Logic. 11th ed. Prentice-Hall, 2002.2. Bloch, E. Proof and Fundamentals.

Boston: Birkhauser, 2000.3. Rodgers, N. Learning to Reason. An

Introduction to Logic, Sets, and

Relations. John Willey & Sons, 2000.4. Meyers, G. The Art of Software

Testing. John Wiley & Sons, 1979.5. Kit, E. Software Testing In the Rea

World. Addison-Wesley, 1995.

Note1. The Sarbanes-Oxley Act .

AcknowledgementsI am grateful to the CrossTalk review-ers, to the distinguished professor Sergei

Artemov at the graduate center of thCity University of New York, and toRobin Goldsmith at GoPro Managementfor their feedback and comments thathelped me improve this article.

Disjunctive Syllogism Form Testing Argument Form

1. EitherP orQ is true means 1. After all test cases have been executed, a feature

status can be eitherfail (P) orpass (Q) (implication).

2. P is not true means 2. We know that the feature did not fail the test for

all of its test cases (evidence).

3. Then Q is true means 3. Then the feature passes the test (conclusion).

Table 2: Deriving a Feature Pass Conclusion

About the Author

Yuri Chernak, Ph.D., is

the president and princi-

pal consultant of ValleyForge Consulting, Inc.

As a consultant, Chernak

has worked for a number

of major financial firms in New York

helping senior management improve the

software testing process. Currently, his

research focuses on aspect-oriented

requirements engineering, use-case-dri-

ven testing, and test process assessment

and improvement. Chernak is a member

of the Institute of Electrical and

Electronics Engineers (IEEE) Computer

Society. He has been a speaker at severalinternational conferences, and has pub-

lished papers on software testing in

IEEE publications and other profes-

sional journals. Chernak has a doctorate

in computer science.

Valley Forge Consulting, Inc.

233 Cambridge Oaks ST

Park Ridge, NJ 07656

Phone: (201) 307-4802

E-mail: ychernak@yahoo.com