Download - Black-Box Testing

Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING

Black-Box Testing

Claus Brabrand[ [email protected] ]

( “FÅP”: First-year Project Course, ITU, Denmark )

[ 2 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING

Reminder: Learning & Exam Goals ”Product”:

”Oral Exam”:


Outline

Introduction: ”Bugs” and ”Black-box testing”

Testing the Specification: [”static black-box testing”]

Testing the Code: [”dynamic black-box testing”]: ”Equivalence partitioning”

Other techniques: Model-checking & Static-analysis

Exercises: Work on your group exercises


Definition: ”bug”Main entry: 2bug

Pronunciation: /’bəg/

Function: noun

Etymology: origin unknown

Date: 16221 a: an insect or other creeping or crawling invertebrate (as a spider or centipede)

b: any of several insects (as the bedbug or cockroach) commonly considered obnoxious

c: any of an order (Hemiptera and especially its suborder Heteroptera) of insects that have sucking mouthparts, forewings thickened at the base, and incomplete metamorphosis and are often economic pests —called also true bug

2: an unexpected defect, fault, flaw, or imperfection <the software was full of bugs>

3 a: a germ or microorganism especially when causing disease b: an unspecified or nonspecific sickness usually presumed due to a bug

4: a sudden enthusiasm

5: enthusiast <a camera bug>

6: a prominent person

7: a crazy person

8: a concealed listening device

9: a weight allowance given apprentice jockeys


Example of a Bug :-)

Hej Claus,

A propos test og datoberegninger:

Fredag morgen mente min PDA kalender (Microsoft Windows CE) at vi skrev lørdag 1. marts. Jeg stillede selvfølgelig tålmodigt uret tilbage til fredag 29. februar 2008 (dette fandt sted på et møde hos Microsoft i Vedbæk ;-) Lørdag formiddag startede den så med at vise min (ret tomme) kalender for august 2049. Det indbyggede ur stod på 1. januar 1601. Ahem.

Peter


Terminology (for ’Bugs’)

”Severe conditions”: Fault Failure Defect

”Unintended operation”: Anomaly Incident Variance Feature (intended)

”Generic terms”: Problem Error Bug ( we’ll use this term)

Typically imply blame; as in:- ”it was his fault”

Famous quote:- ”It’s not a bug, it’s a feature”


”The Harvard Mark II Bug”

Photo of firstactual ”bug”:

“The first documented computer bug was a moth found trapped between points at Relay # 70, Panel F, of the Mark II Aiken Relay Calculator while it was being tested”

Harvard University, Sep. 9, 1947


Definition: ”Bug”

A ”bug” is a relation between: Specification & Implementation

Whether or not specification is: Explicit or Implicit Written or Oral Formal or Informal

(e.g., ”product spec” vs. ”back-of-envellope”)

implementation(aka., ’program’)

specification(aka., ’spec’)

~bug


Specification

A spec states things an impl…: Should do! Shouldn’t do! Unspecified? (’unclear’, ’unmentioned’, or ’left open’)

Should do!

Shouldn’t do!

Unspecified?

S

Specs are often intentionally under-specified. It’s often better to not ”prematurely

commit” to a particular solution (by specifying exactly how a task should be done) –

and instead just state which overall tasks the system should do.


Formal Definition: ”Bug”

A (software) bug occurs when: 1. Impl doesn’t do sth spec says it should:

2. Impl does sth spec says it shouldn’t:

3. Impl does sth spec doesn’t mention:

4. Spec doesn’t mention sth, but should:

5. Impl is hard to understand/use by user(s):

~

~?!

(or does it incorrectly)IS

I S

S I

?!I

S

[cf. ”Software Testing”, R.Patton, p.15]

!~ideal


Consequences…

Consequently, …: ”Additional functionality”?

e.g., a calculator which also does square root (but wasn’t supposed to)

”Easter egg”? e.g., hit [Alt+Shift+2] in Solitaire to win game

…is a bug !

…is a bug !


Question

Question:

A) R.Patton interprets as (”spec := impl”):

trivially no bugs in impl ! (according to [own] definition #1,#2,#3)

B) More sensible to interpret as ”no spec”: entire impl is essentially one big bug !

(…according to definition #3+#4 [p.15])

”Because the software is already complete, you have the perfect specification

– the product itself” [R.Patton, ”Software Testing”, p.31]

”an impl w/o a spec” ?

bug~


Often due to: complexity tight schedule under-specification under-documentation

Number one Cause of Bugs…[cf. ”Software Testing”, R.Patton, p.17]

(NB: no percentages given)

spec

…is the specification:

design

impl

oth

er


Cost of (Fixing) Bugs

Cost of bugs increases exponentially (over time):

log

spec design code test release

$1$100

$10,000

$1,000,000

$10,000,000



Tester’s Goal

The goal of a software tester is:

Testers (therefore) often aren’t the most popular members of a project team

- 1) “to find bugs”; - 2) “find them as early as possible”; and - 3) “make sure they get fixed”

[R.Patton, “Software Testing”, p. 19]



”The Pesticide Paradox”

”The pesticide paradox”: ”The more you test a software,

the more immune it becomes to your tests”

B.Beizer, “Software Testing Techniques”, 1990


Bugs

Bugs often occur in groups If you find one, chances are there are more nearby

Not all bugs will be fixed Too costly? Too risky? Not ”cost-effective”? Bug unknown :-)


Programming vs. Testing

Constructive thinking: ”Test-to-pass”

Often not a good idea to ”test” your own code :-(

Destructive thinking: ”Test-to-fail”

(aka., ”error-forcing”)

Often better to test/break someone else’s code :-)

Advice:

”test-to-pass” before ”test-to-fail”

(i.e., test ”normal usage” first)


Black-Box Testing

The goal of black-box testing is: Make sure impl solves problem it is supposed to:

i.e.,

Point of departure: spec, not impl

not a particular program which ”claims” to solve problem testing w/o insights of code

impl ~ spec

Static(test spec):

Dynamic(test impl):

output

input

bug~


Outline





Exercises: Work with your group exercises


Spec ”Warning Words” (I/III)

Unconditionals (universally): ’Always’, ’for every’, ’for each’, ’for all’, …

Try to violate (i.e., find exemptions from rule)!

Unconditionals (never): ’None’, ’never’, ’not under any circumstances’, …

Try to violate (i.e., find exemptions from rule)!

Tautologies: ’Trivially’, ’certainly’, ’clearly’, ’obviously’, ’evidently’, …

Check assumptions (that nothing’s swept under the rug)!



Spec ”Warning Words” (II/III)

Unspecified conditionals: ’Some(-times)’, ’often’, ’usually’, ’ordinarilly’, ’mostly’, …

Unclear spec (under which circumstances)?

Continuations: ’Etcetera’, ’and so forth’, ’and so on’, …

Check that spec is comprehensively unambiguous?

Examples: ’E.g.’, ’for example’, ’such as’, …

Is example representative (what about other examples)?



Spec ”Warning Words” (III/III)

Positive adjectives: ’Good’, ’fast’, ’efficient’, ’small’, ’reliable’, ’stable’, …

Subjective (needs objectification if to be used for testing)!

Alegedly completed: ’Handled’, ’processed’, ’taken care of’, ’eliminated’, …

Is something hidden?

Incompleted: ’Skipped’, ’unnecessary’, ’superfluous’, ’rejected’, …

Is something forgotten?



Spec ”Warning Words” (if-then)

Finally, watch out for: ”If … Then” (with missing ”Else”):

Check what happens in the ”Else-case”


IF … THEN … ELSE … ?!


Question: which is ”best” (…and why)?

A) white-box testing ; black-box testing ; (i.e., white-box testing first)

B) black-box testing ; white-box testing ; (i.e., black-box testing first)

Answer: ’B’ Settle overall impl ~ spec problems first Before zooming in on the impl [as in white-box testing]

Test Sequencing

…or…

~


Outline







Test ”Boundaries”

Programs are vulnerable ”around the edges”: e.g. testing legal inputs (time, in hours):

e.g. testing legal inputs (dates, in April):

PropertyMinimum-1MinimumMiddle repr.MaximumMaximum+1

Input-1 011 (e.g.)2324

Expected outputinvalidvalidvalidvalidinvalid

Actual output


Input00/401/412/4 (e.g.)30/431/4


Actual output


Test ”Powers-of-Two”

Programs vulnerable ”around powers-of-two”: e.g. years of age (assume held in a byte):

e.g. #game-spectators (assume held in a 16-bit word):


Input-1 027 (e.g.)255256


Actual output


Input-1 012345 (e.g.)6553565536


Actual output


Test ”Empty Input”

Default / empty / blank / null / zero / none / : e.g., ’any program’:

PropertyNo input

Input

Expected outputError message

Actual output


Test ”Invalid Input”

Invalid / illegal / wrong / garbage / bogus data: e.g., calculator:

PropertyInvalid inputBogus data!!!

Input+*31#$+~´?=!

Expected outputError messageError message

Actual output


Equivalence Partitioning

Partitions and

Equivalence Relations


Testing: Infinite process

Recall: ”testing is an incomplete process” (i.e., ”testing can’t prove absence of bugs”)

There are infinitely many possible inputs: (hence, testing will take an infinite amount of time)

sum(x,y)

(0,0)

0

sum(x,y)

(1,0)

1

sum(x,y)

(0,1)

1

sum(x,y)

(-1,0)

-1

sum(x,y)

(0,-1)

-1

sum(x,y)

(1,1)

2

sum(x,y)

(-1,-1)

-2

…


Crash course on Relations

Relations

Equivalence Relations


Relations

Example1: “even” relation: Written as: as a short-hand for: … and as: as a short-hand for:

Example2: “equals” relation: Written as: as a short-hand for: … and as: as a short-hand for:

Example3: “dist-btwn” relation: Written as: as short-hand for:

|_even Z

|_even 4

|_even 5

4 |_even

5 |_even

2 3 (2,3) ‘=’

‘=’ Z Z

(2,2) ‘=’2 = 2

‘’ CITY Z CITY

Aarhus Copenhagen310

(Aarhus, 310, Copenhagen) ‘’


Equivalence Relation

Let ‘~’ be a binary relation over set A: ‘~’ A A

~ is an equivalence relation iff: Reflexive:

Symmetric:

Transitive:

xA: x ~ x

x,yA: x ~ y y ~ x

x,y,zA: x ~ y y ~ z x ~ z


Exercise: Eq.Rel.

Which relations are equivalence relations?: …and which are not (and why not)?:

a) The "less-than-or-equal-to" relation: ' ' { (n,m) | n,m Z, n m }

b) The ”almost-total-relation-on-integers”, (relating all numbers except 42, but relating 42 with 42):

{ (n,m) | n,m ( Z \ {42} ) } { (42,42) } c) The "is-congruent-modulo-three" relation:

{ (n,m) | n,m Z, (n % 3) = (m % 3) } d) The "have-talked-together" relation:

{ (p,q) | n,m PEOPLE, p and q have talked together } e) The "is-in-the-same-group-as" relation:

{ (p,q) | n,m PEOPLE, p and q are in same FÅP group }


Eq. Rel. Partition

Equivalence relation ’~’ = { (A,A), (B,B), (A,B), (B,A),

(P,P), (X,X), (Y,Y), (Z,Z), (X,Y), (Y,X), (X,Z), (Z,X), (Y,Z), (Z,Y) }

E.g.: A ~ B, P ~ P, X ~ X, X ~ Z A ~ P, B ~ Y, P ~ Z

Partition ’~’:

A

B

P

X

Y

Z

”Canonical representatives”: [A] = [B] = { A, B } [P] = { P } [X] = [Y] = [Z] = { X, Y, Z }

Capture the same information;

i.e., notion of ”equivalence”


Testing: Infinite process

Recall: ”testing is an incomplete process” (i.e., ”testing can’t prove absence of bugs”)

There are infinitely many possible inputs: (hence, testing will take an infinite amount of time)

sum(x,y)

(0,0)

0

sum(x,y)

(1,0)

1

sum(x,y)

(0,1)

1

sum(x,y)

(-1,0)

-1

sum(x,y)

(0,-1)

-1

sum(x,y)

(1,1)

2

sum(x,y)

(-1,-1)

-2

…


Equivalence Partitioning

Partition input:

Finitary partition: If finite # categories (aka. ”equivalence classes”)

Here 3: { ”zero”, ”pos”, ”neg” }

We can now test all equivalence classes Using representative elements from each category

neg pos

0

zero

1 2 3 …… -3 -2 -1


Test Sum (cont’d)

We can now test all equivalence classes Using representative input from each category

Sum (testing all equivalence classes):PropertyPos, PosNeg, PosZero,PosPos, NegNeg, NegZero,NegPos, ZeroNeg, ZeroZero,Zero

Input(1,2)(-3,4)(0,5)(6,-7)(-8,-9)(0,-10)(11,0)(-12,0)(0,0)

Expected output315-1-17-1011-120

Actual output


Frequent Partitions for Testing

Numbers: Positive, Negative, Zero Zero, One, Two, Many (aka. ”Greenlandic Numbers”)

Lists: Length-0, Length-1, Length-2, Length-3+ Ascending-elements, Descending-elements, AscDesc-elements

…Advice: - consider how problem might be solved

- partition into qualitatively different

categories such that:

(likely: ”same category same error”)


Outline







Model-checking

Modelling

Verification

Model-checking


Program worldModel world

ConcreteAbstract

P M

Program vs. Model World

abstraction

concretization

(Java)(modeling lang)


Methodology: Model-based Design Design abstract model

Decompose model

Reason/Test/Verify model individual parts and whole

Recompose insights make model safe

Impl. concrete program

concretize

REAL PROBLEM

SAFE MODEL

SAFE PROGRAM

abstract

? ?

?

?

MODEL

testreason

verify


Program worldModel world

ConcreteAbstract

P M

1. P ? 3. M ?

4. verify!!

5. M !7. P !

2. abstract

6. concretize

Model-checking

(Java)(modeling lang)

If models are “finite”, we can have

a computer test all possibilities.

“Never send a human to do a machine’s job”-- A.Smith (’99)


Static Analysis

Static Analysis

Undecidability


Undecidability

Most interesting properties are undecidable: e.g., can program P have a type error (when run):

Compilers use approximations (computed via ”static analyses”):

correct incorrect

?

correct incorrect

safe (over-)approximation

no program can ”decide” this line automatically (in all cases)


Undecidability (self-referentiality)

Consider "The Book-of-all-Books": This book contains the titles of all books

that do not have a self-reference (i.e. don't contain their title inside)

Finitely many books; i.e.: We can sit down & figure out whether to include or not...

Q: What about "The Book-of-all-Books"; Should it be included or not?

"Self-referential paradox" (many guises): e.g. "This sentence is false"

"The Bible"

"War and Peace"

"Programming Languages,

An Interp.-Based Approach"

...

The Book-of-all-Books


Termination Undecidable!

Assume termination is decidable (in Java); i.e. some program, halts: PROG BOOL

Q: Does P0 loop or terminate...? :)

Hence: "Termination is undecidable" ...for Java, C, C++, Pascal, -Calculus, Haskell, ...

bool halts(Program p) { ... }

Program p0 = read_program("P0.java");if (halts(p0) == “halts”) loop();else halt();

-- P0.java --

bool halts(Program p) { ... }


Example: Type Checking

Will this program have type error (when run)?

Type error <EXP> evaluates to true

void f() { var b; if (<EXP>) { b = 42; } else { b = true; } /* some code */ if (b) ...;}

Undecidable

(in all cases)

i.e., what <EXP>

evaluates to (when run)


Example: Type Checking (cont’d)

Hence, languages use static requirements:

All variables must be declared And have constant type (throughout the program)

void f() { bool b;

/* some code */ if (<EXP>) { b = 42; } else { b = true; } /* some more code */}

Static compiler error:

Regardless of what <EXP>

evaluates to when run


Outline




