The Magazine for Agile Developers and Agile Testers
© iStockphoto.com/GarySandyWales
April 2011
issue 6www.agilerecord.com freedigitalversion madeinGermany ISSN2191-1320
26 www.agilerecord.com
Theinformationtechnologyindustryincreasinglyrealizestheim-portanceofconducting,inacarefulandefficientmanner,verifi-cationandvalidationactivities,whichincludessoftwaretesting.Herethetestingindividuals,alongwiththerestoftheteam,worktoassurethatthedevelopedsoftwaremeetsallclients’needsandisofahighqualitystandard.Toachievethisgoal,aneffec-tivetestplanisindispensable.Fromthebeginningoftheproject,atestengineershouldbepresent,becausethisallowsustoplanaheadandtofindandfixdefectsassoonaspossibleinthede-velopmentlifecycle.Afterall,asmentionedintestingliterature,thesoonerdefectsare found, the lowerwillbe thecosts tofixthemandthehigheristheprobabilitythattheircorrectionwon’tcausenewbugs(GlenfordJ.Myers,“Theartofsoftwaretesting”).
Thisarticle,describeshowtestingactivitieswereperformed inamobilegamedevelopmentprojectusingSCRUMastheman-agementprocess.Itwilldescribeindetailthetestingstrategiesused, alongwith the best practices identified and the learnedlessons.Themaingoalof thearticle is toassistother testen-gineerswhoarestartingingamedevelopmentprojects,sothattheycaneasilyand rapidlyadapt to thedifferencescomparedtostandardsoftwaredevelopmentprojects.Thiswillalsoallowthemtocontributetothecreationofnewandevenmoreeffec-tivetestingtechniques.
The projectAllthetechniquesandlearnedlessonsdescribedinthisarticlewereexperiencedduringaprojectdevelopedatC.E.S.A.R. (Re-cife’sCenterofAdvancedStudiesandSystems)fromDecember2007toMarch2008.
Sincetheprojectincluded,amongstothers,elementslikeshortduration, a small team, frequent client involvement, and con-stantrequirementchanges,itwasdecidedtoapplySCRUMandanagiledevelopmentmethodologytohelpmanageallactivitiesduringtheproject.InaccordancewithSCRUM,alltaskstobede-velopedwerelistedasbacklogitems(BLI).Thesewereelectedto
bedevelopedinshortdevelopmentcycles(“sprints”),wherebyattheendofeachsprintanewversionoftheproductwasreleasedtotheclient.Inourproject,eachsprintlasted10workdays,andtheitemstobedevelopedwerechosenbytheteamduringsprintplanningmeetings.
Duringthesprintplanningmeeting,allteammembershadtopri-oritizeallBLIs,inordertohelpdecidewhichtaskshadtobede-velopedduringthefollowingsprint.Fromatestengineer’spointofview,theapproachwastoalwaystrytoanticipatethefeaturesthatappearedtobecriticalforsystembehaviorandformeetingtheclient’sneeds.Prioritizationcouldbeassignedduetocom-plexityorimportance;theintentionwastopreventbugsrelatingtothesefeaturesfrombeingfoundlateintheprojectlifecycle.
Testing strategyPlanning – First sprintTesting activities started before the end of the project’s firstsprintwiththearrivalofatestengineer.Asafirsttask,acom-plete analysis of the Basic GameDesign Specification (BGDS)wasmade.Thisdocumentsummarizesallbasicgamefeatures.AfterevaluatingtheBGDSandallclient’srequestsandneeds,asimpleteststrategywasdefinedwhichinvolveddocumentingmanualtestcasesassoonasthesprintstartedusingasimplepriorityschemebasedonthecomplexityoftheselectedstoryandtheimplementationorder.Atthisinitialstagewealsoidentifiedandsolvedalltestenvironmentneeds,likeavailablehardware,SIMcards,bug trackingsoftware,etc. Finally, a setof generaltestcasesmadeavailablebytheclientwerealsoevaluatedpriortostartingtestexecution.
Test case designOneoftheinitiallydefinedconstraintstotheprojectwasthatalldocumentedBLIsshouldbecoveredbytestcases,andtheBGDSwereconsideredasthetestoracle.Basedonprojectcharacteris-ticslikeshortduration,scarceBLIdocumentation,andfrequentchanges,wedecidedtodesigntestcasesinamoregeneralway
© iStockphoto.com
/uriy2007
Test Planning and Execution in a Mo-bile Game Development Project using SCRUMby José Carréra Alvares Neto
27www.agilerecord.com
withafocusontestingthegame’sbasicfunctionalitiesforeachBLI.Thisresultedinasetoftestcasessimilartothoseusedfor“sanity”tests.Thiswayweexpectedtomaximizethetimespentontestexecutionandtoavoidspendingexcessivetimeondocu-mentation.
Test executionThe testspecificationconsistedofaspreadsheetwithasetoftestcasessentbythecustomerandagroupofspecificscenar-iosdesignedspecifically for thegamebythetestengineers.,AMANTISbugtracker(http://www.mantisbt.org/)wasusedasthedefectmanagementtool.
Duringtheexecutionphase,thefollowingactivitieswereplannedtobeexecutedineachsprint:Themainfocuswastheexecutionoftheexploratorytestsasfeatureswerereleasedthroughoutthesprint,alongwiththeexecutionoftheclientsetoftestcasesandthegame’sspecificgroupoftestcases.Theuseofexploratorytestingisgenerallyencouragedforprojectswithcharacteristicssimilar to ours, and when executed by experienced test engi-neers,such”exploratory testscanbemuchmoreefficient thanthetestsperformedfollowingscriptedtestcases”(JamesBach).
Duringthecourseofeachsprint,an importanttaskperformedby the testengineerwas toeffectivelymonitor theprogressofalldevelopers’activitiesonthecurrentsprint.Thiswasdoneinordertodefinethebesttimetorequestthereleaseofintermedi-ateversionsofthegameforcomponenttestingandalsotoavoiddefectsorchangerequestsbeingraisedforfeaturesthathadnotbeenfullyreleasedbythedevelopmentteam.Thismonitoringofactivitieswasmadeeasierby theSCRUMmethodologywhere,duringthedailymeetings,wecouldeasilyfollowprojectactivitiesthroughtheburndowngraph.
Aspreviouslymentioned, thedecision toprioritize theexplora-tory testswasmadedue to theproject’smain characteristics,suchasalackofaextensivedocumentationatthebeginningoftheproject,andfrequentchangesofclient’sneedsandrequire-ments.
Formal testsThecompletesetoftestcasesconsistedoftheclient’sstandardtestcasestogetherwithgame-specifictestcaseswhichcametoatotalofaround15Otests.However,notallspecifictestswereexecuted in the initial sprints, sincemostof the featureswerenot yetdeveloped.A complete test execution cyclewould takeanaverageofthreedays.Duringtherestofthesprintothertest-ingactivitiesliketestdesignandmaintenancewereperformedalongwithexploratorytestingandchangerequestvalidation.
The client’s set of test casesmainly focused on assuring thatthecompany’sstandardswerebeingfollowedbyourteam.Thisconcernedfeatures likekeymapping,performance, interactionanduserinterface.Takingthisintoconsideration,itwasmanda-torytorunthesetestsforeachsprintinordertoassurethatthedevelopedgamesuitedallclients’demandsand,aboveall,thattheywouldn’tinterferewiththemobilephone’sbasicfeatures.
At theendofeachsprintan intermediateversionof thegamewasreleasedto theclient,whocouldanalyze thedeliveryandprovidefeedbacktoourteam.Thisusuallyinvolvedaspectslikegameplay,gamedesignanddefects.Throughananalysisofthisfeedbackwecouldfigureoutwhichareasweremorerelevanttoourclient,adjustourteststrategyaccordingly,andthenfocusonthemisseddefectsinthenextsprint.
Exploratory testsExploratory tests, which were chosen as ourmain test execu-tionstrategyduetotheprojectprofile,beganearlyintheinitialsprints.Inatraditionalapproach,informaltestcharterswerepre-paredfocusingonaspecificareaorBLItobetested.Duringthecourseoftheproject,astheGameDesignSpecificationbecamemoremature,westartedusingtwonewapproachesforrunningtheexploratorytests,whichwillbedescribedbelow.
Test case basedIn thisapproach theactual testcaseswereusedas the focusareaforeachexploratorytest,wherebythestepsofthetestcasewereruninanunusualway.Thetesterisencouragedtodivergeasmuchaspossiblefromthespecifiedteststeps,andtotryandthinkofalternativepathsthatcouldbetakeninsteadoftheonesuggestedbythetestcase.Themainideaistousetheexistingtestcasesjustasareferenceinordertocoverallthefeaturesof theapplicationandto leave it to the testengineer toevalu-ate relevanceof the features to the system.By doing this,weencouragethetestengineertothinkcreativelytofindnewways,orwaysnotpreviouslyforeseen,tobreakthesoftware.Thisap-proachachievedverygood resultsandcertainly increased thetotalnumberofrelevantdefectsfound.
Ifthisapproachistoachieveahighdegreeofsuccess,itisveryimportant for the testengineer toknowallexisting featuresofthesystem,themarketandtheclient’sexpectations.Heneedstofullyconcentrateonhisworkinordertonoticedetailsthatmayhaveescapedbefore.Itisalsoimportantthattheengineercanworkinacomfortableenvironmentduringtestexecutionwithoutbeingconstantly interrupted,thusallowinganefficientanalysisoftheexistingtestcasesandunexploredpossibilities.
GDS scanningA different approach, which we used in the later sprints, wasbasedontheexecutionoftheexploratoryteststhroughacom-pletescanoftheGDS.Thisapproachcouldn’tbeappliedfullyintheinitialsprints,becauseonlytheBGDSwasavailable,whichdidn’tcontainenough information toallowamoredetailedex-ecution.
Forthistechniquetobeappliedsuccessfully,thetestengineerneedstohavealreadyreadandunderstoodthedocument,,andthere shouldbenoopenquestions. Themain ideaof this ap-proach is tomake sure that every description included in theGDS iscorrectlycoded in thegame.Bysimultaneouslyanalyz-ingthedocumentandexploringthegame,itbecomesvisibleifanyimportantscenarioisn’twelldescribedinthetext.Withthisapproachwecancombinethebenefitsofstatictestingofdocu-
28 www.agilerecord.com
mentationwiththeadvantagesofexploratorytestingoftheap-plication.Anygapsbetweengameanddocumentationcaneasilybefoundandthegamedesignercanclarifythetypeofanybugsfound.
Registered defectsDuring the course of the project, 122 change requests (CRs)wereregistered,whichweresimplyclassifiedforthisarticleintofourcategories,accordingtothetypeofdefect,asfollows:func-tional, user interface, art, and sound effects. Later on in thisarticle isdescribedhoweachoftheseareaspresenteduniqueimportantcharacteristics.Inadditiontothesedescriptions,twographswillindicatetheamountofCRspertypeandtheseverityoftheregistereddefects.
Theseverityofeachbugwasclassifiedasfollows:(i)minor(fordefectsthatdonotblockthegame’scorrectbehavior,e.g.,thephonedoesn’tvibratewhenanewlevelisunlocked);(ii)normal(for defects that affect important elements of the game, butdo not block the game’s execution, e.g., a special effect lastsshorterthanthevaluespecifiedintheGDS);(iii)major(forde-fectsthatdirectlyaffectgameplayorusersatisfaction,thathaveadirectimpactonthegame’sleveldesign,andthatprohibittheplayerfromproceeding,e.g.,scenarioswherethegamefreezes);(iv)critical(fordefectssimilartomajordefects,butwithanevenhigherimpactonthegame’scorrectoperation,e.g.,levelisnotunlockedaftercompletingtasksorphonedoesn’treceivecallswhilegameisstarted).
FunctionalTheCRsfromthisgrouparerelatedtoinconsistenciesregardingthegame’sdesigned rulesand logic.This typeofdefect,eventhosewith lower severity, have a direct impact on the game’ssuccess, because theyusually get in theway of a smoothun-derstandingofthegame’sobjectives.Theymayevenblocktheplayer from overcoming the challenges presented, turning thegameintoanimpossiblemission.
Scenarios like score limits, lousy player and excellent playerapproach, and other possible features that involve testing thegame’sfeaturesandlimits,wereteststhatusuallydetectedthiskindofdefect.ThesescenarioswerenotalwayswelldescribedintheGDS,anddevelopersdidn’ttakethemintoconsiderationorunittestthemproperly.
User interfaceInterface defects were connected to failures during display oftexts, openingof pop-upwindows, screen limits, etc. These is-suescouldbeeasilynoticedbyanyplayer,andwoulddefinitelygivetheideaofapoorlydevelopedgame,withoutcarefordetails.
Thesedefects,althougheasilydetected, frequentlyescape thedevelopingteamandeventheinitialtestcycles.Thiscanhappenbecausedevelopersusuallyruntestsusingasimulatorandnotarealdevice.Itispartofthetestengineer’sjobtoassurethatallgamescreensandtextsarecheckedforthesupporteddevices.
ArtAllchangerequestsofthe“art”categoryarerelatedtofeatureslikeimagerendering,lightening,andanyotheraspectsrelatedtotheelementsproducedbytheartteam(althoughsomeofthemmayalsobeperformedbythedevelopmentteam).
Thistypeofdefectvarieswidelyfromhugeperceptiblefailures,whichcanbeeasilynoticed,tospecificscenariosthatarecausedonlybyadefinedsequenceofactions.Thiskindofdefectcanbefoundnotonlybyfocusingonthisaspectduringtesting,butalsowhilerunninganyothertypeoftest.Allthatisneededisahighlyalerttesterwhopaysattentiontodetails.
Itishighlyimportantforthetestengineer,especiallyifheisnotexperiencedwiththiskindofdefect,tointeractwiththeartteamtoclarifythequestionsrelatedtopossibledefects,andindoingsobeginning tounderstand the features, their limitsand theirsolutions.
Sound effectsWealsoassigneddefects toa soundeffect category,becauseit turnedouttobeakeyareawhere initiallywedidnotexpecttofindarelevantamountofbugs.However,testingshowedthatthis assumptionwaswrong.Several defectswere foundwhichdemandedconsiderableworkfromthedevelopmentteamtogetthemfixed.Oneaspectobserved,theconcurrenteventsexecu-tion,causedcomplicationsinthegame’sgeneralbehavior.Thisconcernsscenariossuchasexecutingasoundwhileanotheroneisalreadyplaying,user-initiatedpausesof thegame,disablingandenablingthesound. Inaddition,severeperformanceprob-lemscouldresultfromsomeofthegame’ssoundevents.
Throughout the course of the project, this kind of the defect,whichinitiallywasunderestimated,gainedhigherpriorityandat-tention.Wefoundthatinthisareawehadahigherprobabilityofcausingotherdefectswhilefixingone.
Atfirst,testexecutionforthisaspectwasimpactedbythequalityoftheavailablehardware,whichpresentedabadsoundquality.Lateron,withthearrivalofnewhardware,testscouldbeeasilyexecutedandshowedbetterresults.Thereforeitisimportantforthetestingteamtoensuretheavailabilityofthecorrecthardwareatthebeginningoftheproject.
Figure 1 – Amount of defects registered by type
29www.agilerecord.com
Best practicesBelowwedescribesomeofthepracticesthatwereappliedinourprojectandthatpresentedgoodresults.Thesecouldthereforeevenbeappliedtoprojectsindifferentareas.
BLI defect trackingEvery time a new change request was submitted during thecourseoftheproject,alongwithitsshortdescription,atagwasaddedtoidentifytowhichBLIitwasrelated.Atthestart,thiswasonlyusedtohelptheConfigurationControlBoard(CCB)withtheCRassignment,butlateronitaidedtheteaminevaluatingwhichBLIspresentedmoredefectsofhigherseverity.
Onthebasisofthisinformationwecouldplantestexecutionfo-cusingontwoaspects:(i)validateifBLIswithfewornodefectsweresufficiently tested; (ii)analyzeBLIs thatshowedagreateramountofdefects,theircharacteristicsandwhichtestscenarioscouldbere-testedoraddedtoallowthediscoveryofnewbugs.Greater emphasis was applied to the second aspect, as de-scribedbyMyers“Theprobabilityoftheexistenceofmoreerrorsinasectionofaprogramisproportionaltothenumberoferrorsalreadyfoundonthatsection”.
Thisapproachprovedtobeefficientasnewerrorswerefound.Someadjustmentsweremadetomakethistaskfaster.Forex-ample,insteadofaddingtheBLIidentificationinthedescriptionheader,weaddedanewtextfieldtothedefectmanagementtoolsowecouldidentifytheBLIandlateroneasily identifythede-fectsfound.
Greater level of detail in defect descriptionOneof themost importantactivitiesof the testengineer is re-cordingthedefects,foundinadefectmanagementtool.Never-theless, somepeopledidn’t perform this taskasexpected. Is-suesweresometimesnotcompletelydescribed,makingitharderformanagersanddeveloperstounderstandtheerror.Lateronthisgeneratedseveralinterruptionsforthetestengineerinordertoclarifytheissuedescriptionor,evenworse,todiscardthede-fect.Therefore,itisveryimportantforthedefectstobereportedinadetailedanddidacticmanner,makingeveryone’sjobeasier,includingothertestersthatmightbe involved inretestingaftertheissuegetsfixed.
Toassistinthistask,onerealbenefitistoaddarecordedvideooftheissueor,ifthatisnotpossible,atleastascreenshot.Iftheissuecan’tbereproducedonthesimulator,usearegularcameratocaptureit.Justrememberthatthisisnotarule.It’suptoyoutoevaluatewhetheraCRshouldbe improvedbyaddingsomeextraresource(afterall,notallissueshaveavisualfeedback).
Defect management toolAnotherhighlyimportantassettoassistthetestengineer’sworkis thedefectmanagement tool. Inourproject theopensourcetoolMANTIS was used. It provides the engineer with effectivecontrol of the registered defects, allows trends analysis and,mostofall,enhancesteamcommunication.
Taking notesKeepinganelectronicnotepadalwaysopenorevenapieceofpaperandpenonyourdeskcanreallyhelpduringtestactivities.Withthetimepressureandtightdeadlinesalwayspresent,itisnotalwayspossibletotestallthescenariosthatweforeseedur-ingtesting.Thismayhappenbecauseoftheneedofkeepingfo-cusedonthecurrenttestcycleorothertasksbeingperformed.Ifnotwrittendownsomewhere,theseotheractivitiesorevenhintsobservedduringteammeetingsmaygetlostandnevertested.
Makingahabitofnote-taking forany relevant information thatmighthelpinfutureactivities(forexample,anewtestscenarioorsystemcharacteristic,someuserfeedback,etc.),willhelpthetest engineer to avoid forgetting interesting investigations thatcouldbeperformedandassistinfindingnewbugs.
LEASSONS LEARNEDThroughouttheproject,manynewexperienceswerefacedandalotwaslearnedbyworkinginaprojectwithverydifferentcharac-teristicstothosefoundinregularsoftwaredevelopmentprojects.
Test case designAfter gaining some experience in test case design for games,weobservedthatteststhatwererelatedtofunctionalitiesdidn’tneed tobe repeatedondifferentgamescenariosbecause theerrorfoundappliedtothewholeapplication(e.g.pressingaspe-cifickeydoesn’tproduce theexpectedbehavior).On theotherhand,whenconsideringtheuserinterfaceandart,thistypeoftestneededtoberepeatedforeachscenariooftheapplication,sincesomeerrorscanbefoundonlyinspecificscenarios.
Using cheat codesAsthegameevolved,acoupleofcheatcodes(codingthatpro-videdspecialgameadvantagesthatwouldnotbeavailableonthe final version, such as “invincibility”),were created tomakesome testseasier toexecute. forbothdevelopersand testers.However,weneedtobeextremelycarefulwhendecidingtousethistypeofassistance.Iftheyaremisused,itcanleadtohiddenbugsor,conversely,bugsthatonlyappearduetotheexistenceofthecheatcode.Anexamplethathappenedtouswasacheatcodethatunlockedagamelevel,whichblockeddevelopersfromreproducingabugreportedbytesters.
Figure 2 – Severity of reported defects
30 www.agilerecord.com
Ifcorrectlyused,cheatcodescanincreasetheteam’sperform-anceandevenhelpanticipatingbugs.
Team communicationIt isalso important for the testing teamtodefinespecific timeintervalsthroughouttheprojectwherereportsfromtestexecu-tioncycleswillbesentfortherestoftheteam.Thishelpsustomakeourworkvisibleandunderstandablefortheentireteam.Although SCRUM already makes tasks communication easieramongteammembersbyapplyingthedailymeetingsandburn-downgraphs,westillneedamoreformaltypeofcommunication.Arecommendedmomentforthesereportsisattheendofeachsprint.
Activities followed up using the burndown graphAstestengineerswecommonlyneedtoassurethatweareus-ing the latest available versions for testing. During the courseofasprint,thetimeforrequestingnewpartialversionscanbedeterminedthroughthedailymeetingsandtheburndowngraph,wherewecanbeawareofthestageofeachBLIandsetupwiththedevelopersthenumberoffeaturesavailablefortesting.Thetestengineerneeds tokeepconstantly in touchwith the teamleader toassurethat these intermediateversionsget releasedforcomponentandexploratorytesting.
BLIs changes closely followedGenerally on Agile projects, but especially on ours, a greatamountofchangehappenedtotheBLIswhichdirectlyimpactsonthepreviouslydesignedtestcases.Therefore,itishighlyrel-evanttostaytunedtochangescausedbyclient’sfeedbacks,us-abilitytestsandreports,meetingsandalsotechnicallimitations.This follow-up ismadeeasierwhen theGameDesignerworkscloselywiththerestoftheteamandkeepseveryoneinformedwhenchangesoccur.Thisway,anyquestionsrelatedtoanyfea-tureofthegamecanbeeasilydiscussedandclarifiedwiththeGameDesigner.
Informally reported defects don’t get fixedJustlikeatesterforgetstotestscenariosthathedoesn’tdocu-ment,byinformallyreportingadefecttoateammember,(devel-oper,GDorartist),thereisnoguaranteeofafix.Nomatterwhattheseverityofthereportis,theinformalcommunicationcreatesahighriskthatitdoesn’tgetfixedor,iffixed,thatitmaynotbevalidated.
Retest with different devicesThroughouttheprojectseveralerrorswerefoundwhichwerere-latedtothegame’sinteractionwithspecificdevicesorexternalapplicationssuchasphonecallsorSMS.Thiskindoferrorcaus-esconsiderableeffort for thedevelopment teamto investigatetheissueandtofindoutthecause.However,thiskindoferrorcan’tbesolvedbyourteamsinceitisnotrelatedtoourgame.Therefore,itisagoodapproachtoretestwithdifferentdevicesordifferentgameswithsimilarcharacteristics.Thisextrainforma-tionwon’teliminatetheCR,butdeveloperscanstartanalyzingwiththisaspectinmind.
Thisapproachcanalsobeappliedtodifferenttypeofscenarios,such as performance, sound effects, rendering and other fea-tureswhichcanbecomparedwithsimilargames.
CONCLUSIONNomatterwhat stagewe are at in the development life cycleortheexperiencelevelofthedevelopingteam,therearemanycauses forsoftwarebugs.Mostof themarenot related to thecodeitself,buttootherproblems,suchasincompleteorunclearrequirements,hardwareissuesandintegration.Therefore,con-sideringthepracticesandlessonslearnedfromthisproject,weareconvincedthatsoftwarequalityisahighpriorityformodernsoftware products, likemobile games, and that achieving thisshallbeacommongoalfortheentireteamwithacleardivisionof responsibilities.Making sure that there are no conflicts be-tween developers, testers and other teammembers regardingquality, everyonemust work together to deliver a high qualityproduct.Onlybyplacingpriorityonqualitywewillbeabletode-liverproductsthatfullymeetourclients’needsandexpectations.
José CarréraMSc, has been test engi-neer at C.E.S.A.R. (Recife’s Center for Advanced Stu-dies and Systems) since 2006 and Professor of Computer Science at FA-TEC (Faculdade de Tec-nologia de Pernambuco), Brazil, since 2010. He ob-tained his master degree
in software engineering (2009), graduated in computer science (2007), and is a Certified Tester - Foundation Level (CTFL) by the ISTQB (2009). His main research in-terests include quality assurance, agile methodologies, software engineering and performance testing.
> About the author
Top Related