Model-Based Testing (DIT848 / DAT261)

Spring 2017 Lecture 4

Testing: The Bigger Picture

Gerardo Schneider Department of Computer Science and Engineering

Chalmers | University of Gothenburg

Some slides based on material by Magnus Björk and Thomas Arts 1

The Bigger Picture

Requirements

Specification

Architectural Design

Detailed Design

Coding

Unit Test

Integration Test

System Test

Acceptance Test

The techniques we have seen so far are mainly at this

level

2

Unit tests �  Testthesmallestcomponentsindividually

�  O4endonebytheprogrammerwhowrotethecode

�  Lessstrictrequirementsofdocumenta<on�  Largepartofdocumenta<onreplacedbyexecutabletestsuites(Cunit,Junit

orsimilar),whichmustthereforebeclearlywriCen

�  Nolessimportantthananyothertest!

�  Infact,maybethemostimportanttest:�  Unittestseasiertodothanothertests–wellinvested<me�  Bugsdiscoveredearlyeasiertofix�  Sospendingeffortonunittestsreducesworklater

�  Recommendedeffort:equalamountof<mespentwri<ngcodeandunittests

3

Unit tests – typical flow Programmer:

�  writescode�  runssta<cverifica<ontoolsuchassplint(forC)�  writesandrunsunittestsuitetotestthecode

�  UsingframeworksuchasJUnit,CUnit

�  complementsblackboxtestsuitewithwhiteboxtechniques�  Coveragechecking(iden<fymissingtestcases)–Gcov,Emma�  Valgrind:MonitormemorybehaviourofC/C++programs

Thecolleaguesoftheprogrammerdo:

�  Codereview4

Unit tests Test Driven Development (TDD) Programmer:

�  writesunittestcases�  Runstestsuite,makessureitfails

�  writescodeun<ltestsuitedoesnotfail�  Addsmoretestcasesifneeded

�  runssta<cverifica<ontoolsuchassplint(forC)

�  complementsblackboxtestsuitewithwhiteboxtechniques�  Coveragechecking(iden<fymissingtestcases)–Gcov,Emma�  Valgrind:MonitormemorybehaviourofC/C++programs

Thecolleaguesoftheprogrammerdo:

�  Codereview 5

Benefits of TDD �  Focusonwhatthecodedoesbeforeimplementa<on

�  Helpfulwhenwri<ngthecode

�  Programmergetsveryquickfeedback

�  Easiertomaintain�  resultsinbeCercoverageofunittestsuite

�  Note:TDD”mandatory”inXP6

Integration tests �  Testdifferentcombina<onsofcomponents

�  Differentstrategiesforintegra<on�  Bigbang�  BoCom-up�  Top-Down�  Sandwich

User interface

Function 1 Function 2

Lib 1 Lib 2 Lib 3

Lib 4

Thanks to Bruegge & Dutoit for material (through Magnus) 7

Integration tests: Big Bang approach

�  A4erunittests,integrateallcomponentsatonce

�  Essen<allyasystemtest

�  Badidea!Don’tuseit�  Hardtolocatebugs(havetosearchthewholesystem)�  Cri<calandperipheralcomponentsgetthesame

aCen<on�  Onlypossibleverylateindevelopmentcycle

8

�  Startwiththesubsystemsinthelowestlayerofcallhierarchy

�  Integratesuchcomponentswithcomponentsthatusethem

�  Donerepeatedlyun<lwholesystemisintegrated

�  Specialcodeneeded:Testdriver�  Arou<nethatcallssubsystemsandpassestestcasestoit

�  CanbedoneinframeworkssuchasCUnit

Integration tests: Bottom-up approach

9

Integration tests: Bottom-up approach

User interface

Function 1 Function 2

Lib 1 Lib 2 Lib 3

Lib 4

Lib 4

Driver

Lib 1

Driver

Lib 4 Lib 1

Driver

Lib 4

Lib 2

Driver

Lib 4

Lib 2

Functions 1

Driver

Driver: A routine that calls subsystems and passes test cases to it

10

Integration tests: Top-down approach

�  Testtoplevelcomponentsfirst,itera<velyintegratecomponentsthatarecalledbythecomponentsthatarealreadyincluded.Repeatun<lthewholesystemisintegrated

�  Specialcodeneeded:Stub�  Hasthesameinterfaceasthecomponentitreplaces�  Returnsfakedata(probablydescribedinthetestcase)�  Passesinforma<onofthecalltothetestframework

�  XUnitmaybeuseful

�  Pros:�  Testcasesdefinedintermsofprogramspec.�  Easytoseebehaviourateachstage(userinterface)

�  Cons:�  Wri<ngstubsdifficultandtedious�  Makingautomatedtestsuitemaybeharder(e.g.ifGUI)

11

User interface

Function 1 Function 2

Lib 1 Lib 2 Lib 3

Lib 4

User interface

Stub Stub

User interface

Fun 1 Stub

Stub Stub

User interface

Fun 1 Stub

Stub Lib 2

Stub Stub: a piece of (dummy) code used to stand in for some other programming functionality

Integration tests: Top-down approach

12

Sandwich approach

�  DobothboComupandtopdown,meetinthemiddle

�  Muchparalleliza<on:�  Firstphase:

�  Toplayerwithstubs� Middlelayerwithdriversandstubs�  BoComlayerwithdrivers

�  Secondphase:�  Topandmiddlelayer(toplayerreplacesdrivers)� MiddleandboComlayer(boComlayerreplacesstubs)

13

Integration tests: Sandwich approach

Driver

Function 1 Function 2

Lib 1 Lib 2 Lib 3

Lib 4

User interface

Stub Stub

Lib 1

Lib 4

Driver

Lib 2

Lib 4

Driver

Lib 3

Driver

Driver

Function 1 Function 2

Stub Stub Stub

User interface

Function 1 Function 2

Stub Stub Stub 14

Integration Tests: what to consider when choosing approach

�  Whichpartsofthesystemaremostcri<cal?�  Choosestrategythatrevealserrorincri<calpartsearly,and

includescri<calpartsinmanytests

�  Whichapproachmeanslesswork?�  Topleveltestmaybehardertoautomate(e.g.GUIs)�  Howtominimizeworkspentwri<ngdriversandstubs

�  Availabilityofcomponents�  IfcodingdoneboCom-up,thenboCom-upintegra<ontestscan

bestartedearlier15

To make integration tests easier

� Dothoroughunittests

� Makewelldefinedinterfacesbetweenmodules

16

System tests �  Testthefullsystem

�  Coverfullspecifica<on�  Testautoma<onmaybehardtoachive

�  Systemtestsmaybeexpensiveandtedious

�  Combineblackandwhiteboxtes<ngasbefore

�  Testbothnormalandabnormalusesofthesystem�  Performancetes<ng

�  Pushsystemtoitslimits�  Thegoalistotrytobreakthesystem�  Maybeusedtoiden<fyboClenecks,tobedealtwithinnextitera<on

ofdevelopment17

System tests �  Stresstes'ng:exceedparameters:numberofrequests,…

�  Volumetes'ng:largeamountsofdata

�  Configura'ontes'ng:differentcombina<onsofHW&SW

�  Compa'bilitytes'ng:usewitholdersystems

�  Securitytes'ng:trytobreakin�  Timingtes'ng:<meresponses&func<ons

�  Environmentaltes'ng:effectsoftemperature,movement,…

�  Qualitytes'ng:reliability,maintainability,availability

�  Recoverytes'ng:erroneousormissinginput

�  Humanfactorstes'ng:testuserinterfaceonusers 18

Acceptance tests

�  Customermostlyresponsibleforacceptancetest

�  Alphates<ng�  Donebycustomerundersupervisionofdeveloper�  Usuallydoneincontrolledenvironment(developer’ssystems)�  Developercanquicklyfixbugs

�  Betates<ng�  Productusedbycustomersinrealenvironment�  Developerstypicallynotpresent�  Differencefromrestofproductlife<me:

�  O4enonlyselectedcustomers�  Customercannotrelyonso4ware

19

Fixing bugs �  Ac<ondependsonseverityofbug

�  Low-priorityfailuresmaybeputon”knownbugs”list,includedinreleasenotes

�  Alwaysdoregressiontesta4erfixingbugs!�  Bugfixesarelikelytobreaksomethingelse

�  Bugtrackingtoolso4enuseful(Example:Bugzilla)�  Maintainslistofbugs�  Assignspriori<esandresponsiblepeopleforeachbug�  Keepsremindingpeopleabouttheirhighprioritybugs�  Searchablebugindex(withhistory)

20

Regression tests �  Mustbedonea4ereverychangetosourcecode

�  Regressiontestssignificantlycheaperiftestsuiteisautomated

�  Some<mesnotfeasibletoredoalltests.Ifso,iden<fyasubsetofcasesthatcoverasmuchaspossible

�  Tool:Tinderbox�  Automa<callychecksoutcommiCedcode,compilesitandruns

testsuite(needsothertoolforthat,suchasDejaGnu)�  Iden<fiescompila<onerrorsandfailingtestcases�  Pointsoutwho’sresponsible�  Maintainshistory�  O4enruns100%ofthe<meonabunchofdedicatedmachines

21

Test in General: a test… Determinewhetherthestatementsaretrueorfalse.Ifastatementisfalse,jus'fyyouranswer

1.  Therearetwokindsoftes<ng:dynamicandsta<c.

2.  Ifyouget100%codecoveragethenyoucanguaranteethatyourso4warehasbeenthoroughlytestedandcanstoptes<ng.

3.  XUnitisbeCerthanJUnitsinceyoucangetbeCertests.4.  TheVmodelteachesusthatwecandoacceptancetestsassoonaswehavethe

requirements,evenbeforewestartdeveloping.

5.  Differenttes<ngmethodsandtechniquesapplytoeachtestlevel(aspresentedintheVmodel).

6.  Tes<nganddebuggingarethesame.

7.  Onegoodthingaboutthesandwichapproachforintegra<ontes<ngisthatenhancesparalleliza<on(thatis,developersandtesterscanworkinparallel).

8.  Performancetes<ngisonekindoftest,partoftheso-calledsystemtes<ng.

9.  Thebestwaytodointegra<ontestisthesandwichapproach.Groups 2-5 persons: 15 min 22

Test in General: solution… 1.  F–Tes<ngisbydefini<ondynamic

2.  F–Codecoverageisonlyoneaspect;thereisnoguaranteeingeneraltoget100%confidence

3.  F–XUnitisafamilyoftestunits,includingCUnit

4.  FAcceptancetestisonlydonea4erthereissomethingtotestagainsttherequirements.Donebycustomer

5.  T6.  F–Tes<ng:establishtheexistenceofdefects;debugging:

loca<ng&repairingthoseerrorsfoundduringtes<ng

7.  T8.  T9.  F–dependsonhowthesystemisbuilt 23

Next lecture

24

�  EFSM: Extended Finite State Machines