Agile Testing for Embedded and IoT Software Development

W16 Agile Testing 10/5/16 15:00

Agile Testing for Embedded and IoT Software Development

Presented by:

Thomas Stiehm

Coveros, Inc.

Brought to you by:

350 Corporate Way, Suite 400, Orange Park, FL 32073 888-‐-‐-‐268-‐-‐-‐8770 ·∙·∙ 904-‐-‐-‐278-‐-‐-‐0524 -‐ [email protected] -‐ http://www.starwest.techwell.com/

Thomas Stiehm Thomas Stiehm has been developing applications and managing software development teams for eighteen years. As CTO of Coveros, he is responsible for the oversight of all technical projects and integrating new technologies and application security practices into software development projects. Most recently, Thomas has been focusing on how to incorporate DevOps best practices into distributed agile development projects using cloud-‐based solutions and how to achieve a balance between team productivity and cost while mitigating project risks. Previously, as a managing architect involved in agile development at Digital Focus, Thomas found that agile is the only development methodology that makes the business reality of constant change central to the development process.

9/21/16

1

©COPYRIGHT2016COVEROS,INC.ALLRIGHTSRESERVED. 1

Agility.Security.Delivered.

Agile Tes)ng for Embedded So3ware Development


About Coveros

•  Coverosbuildssecurity-criIcalapplicaIonsusingagilemethods.

•  CoverosServices•  AgiletransformaIons•  AgiledevelopmentandtesIng•  DevOpsandconInuousintegraIon•  ApplicaIonsecurityanalysis

•  Agile&Securitytraining

•  GovernmentqualificaIons•  DCAAapprovedratesandaccounIng•  TSfacilityclearance

AreasofExper8se

9/21/16

2


What is embedded so3ware

•  EmbeddedsoTwareissoTwarethatcontrolsadevice• AdeviceisacombinaIonofsoTware,firmwareandhardwarethatmakeupasystem•  Thesystemhashardwarecomponentsthatareeithersensorsormanipulators•  ThesoTwareandfirmwarecontrolthehardwarecomponents,usingthesensorstodiscoverandthemanipulatorstocarryoutacIons•  Examplesinclude:•  Medicaldevices•  Selfdrivingcars•  Fitnesswearables•  HVACsystems


What is agile tes)ng

• AgiletesIngisthethesetofpracIcesusedtobuildqualityinandmakesurethatsoTwarecreatedbytheteamisofhighquality•  Thewholeteamisresponsibleforquality•  TestautomaIonisintegratedintotheConInuousIntegraIonandConInuousDeliverypipeline•  TesIngofnewfuncIonalityisconductedduringthesamesprintoriteraIoninwhichthefuncIonalityisdeveloped• UnittesIngisusedaspartofthequalityassuranceprocess• AcceptancetesIngisconductedusingacceptancecriteriacreateddirectlyfromtherequirements

9/21/16

3


User Acceptance Tes)ng (UAT)

• UATistheprocessofge]ngstakeholderstowritedownwhatresultstheywanttoseefromthesoTwareinordertodetermineitworksastherequirementsintended• AcceptancecriteriaarewriênbeforedevelopmentofthesoTwarebegins• AcceptancecriteriaareaformofrequirementsthatareusedtolettheteamknowwhatkeyaspectsoftherequirementsneedtobemetinorderforthestakeholderstofeelthattheycanacceptthesoTwareincrement•  TesIngofacceptancecriteriahappensattheendofanincrementandgivestheteamfeedbackonhowwelltheyhavecreatedsoTwarethatthestakeholdersexpected.


Behavior Driven Development (BDD)

• BDDisatestautomaIonprocessfortesIngsoTwarebasedontherequirements•  Thetestsarewriênbeforethecodeiswriênoratleastindependentlyofthecode•  ForsometeamsBDDisusedtoautomateUATtests• ManyBDDtoolsexistsuchasCucumberandBehave• MosttoolsusetheGherkinlanguage,whichusesthefollowingformat:•  Givenini8alcondi8on• Whensomeac8onortrigger•  Thenresul8ngac8on

9/21/16

4


Test Driven Development (TDD)

•  TDD,alsocalledTestFirstDevelopment,istheprocessofusingunitteststohelpdesignthecodeforanapplicaIon•  Thedeveloperwillwriteatestofwhattheyintendthecodetodoandthenwritethecodetomakethetestpass• Red,Green,Refactorcycle•  ThishasanumberofposiIveresults:•  Makesthecodemoretestable•  Makesthecodemoremodularorselfcontained•  Makesthecodemoremaintainable•  Givesthecodearegressiontestsuite•  Makestheintentofthecodeeasiertounderstand


Con)nuous Integra)on (CI)

• CIistheprocessofintegraIngandtesIngcodeuponcheckin•  InCI,integraIonistheprocessofintegraIngadeveloper’scodewiththecodealreadyintherepository•  ThegoalofCIistomakesurethecodecanbuildandpassaniniIalsetofregressiontestsbeforeothertesIngandqualityassuranceisappliedtoaspecificbuild• CIusesabuildserversuchasJenkinstodoacleanbuildofthecodeandrunasuiteofunittests•  Ifthebuildbreaks,thepeoplethatcheckedinsincethelastsuccessfulbuildareinformedandtheyareexpectedtogetthebuildbacktoaworkingstate

9/21/16

5


Con)nuous Delivery (CD)

• CDistheprocessoftakingabuildthroughabuildpipelinethatconsistsofaCIprocessfollowedbyseveralqualitygatesthatverifythecodethroughvariousqualitychecksincluding:•  AutomatedfuncIonaltests•  StaIccodeanalysis•  Automatednon-funcIonaltestslikeperformanceandsecurity•  Automateddeploymentintoprogressivelyhigherenvironments•  Itcouldincludeondemanddeploymenttoamanualtestenvironment•  OndemanddeploymenttoaUATenvironment•  Ondemanddeploymenttoademoenvironment

• ConInuousDeploymentisautomaIcallyprogressingtoproducIonincludingautomaIcallydeployingintoproducIon


Value Stream Mapping

• ValuestreammappingistheprocessofdetermininghowaproductiscreatedinyourenvironmentfromincepIontodeploymentanduseinproducIon• BycreaIngavaluestreammapyoucanfigureouthowmanystepsyouhaveinyourdeliveryprocessandhowlongeachsteptakes• OnceyouhaveamapofyourprocessyoucanstarttoopImizeandautomatethedifferentstepstostreamlineyourprocess•  ThegoalistoreducetheImeittakestomoverequirementsthroughyourprocessandgetnewfeaturesintoproducIon

9/21/16

6


Build Pipeline

•  Thebuildpipelineisthepartofyourvaluestreamthattakescheckedincodeandmakesitintohighquality,testedsoTwarethatiseitherinproducIonorreadytobedistributedtocustomers• CI/CDareasubsetofyourbuildpipeline• AutomaIon,includingtestautomaIon,istheonlywaytomakeabuildpipelineworkinafastandefficientmanner• ManyofthequalitygatesinabuildpipelinearetesIngfocusedandrequirestrongtesIngacumeninordertohelpdeliverhighqualitysoTware


Version Control

• VersioncontrolistheprocessofmanagingthecodeoverImebyallowingaprojecttokeeptrackofthestateofthecodeasitprogresses• Allcode,includingtes6ngcodeandtestautoma6oncodeneedstobecheckedintoversioncontrol• ApplicaIoncodeandtesIngcodeshouldbeversionedtogethersincethetestcodeactsontheapplicaIonandchangesoverImetokeepinsyncwithchangestotheapplicaIoncode• Versioncontroltoolsinclude:•  Subversion•  Git•  Others

9/21/16

7


Configura)on Management

• ConfiguraIonManagementarethetoolsandprocessusedtodeterminehowsoTwareissetupinanygivenenvironmentandhowtoaccountforallofthedifferentfilesneededtomakethesoTwarework•  Thisincludesbinaries,configuraIonfilesandenvironmentse]ngs• MostbuildpipelineshaveacomponentthatmanagesconfiguraIonelementsforeachdeploymentsothattheteamcanknowexactlywhatisdeployedandtestedforeveryenvironment


Tes)ng So3ware vs. Hardware vs. System

• WithembeddedsystemsyouaretesIngthreethings:•  SoTware–Thecodethatmanagesthesystem•  Hardware–Themechanicaland/orelectricalpartsofthesystem•  System–ThecombinaIonofhardwareandsoTware

•  SomeImesfirmwareistestedlikesoTware,someImesitistestedlikehardware(dependingonhowfirmitis)•  IsolaIngwhatyouaretesIngcanhelpyoucreatetestautomaIonthatcoversamuchoftheproductaspossible•  SomeamountofmanualtesIngmightbeunavoidable

9/21/16

8


So3ware Tes)ng

• DuringsoTwaretesIngyouaretesIngonlythesoTwarepartofyoursystem•  Thismeansthatallinput(sensordata)shouldbesuppliedbyaknownsource,ifpossiblebyrecordingthedataandplayingitback,orsimulaIngit• Outputcanbeignoredorsimulated• Recordandplayback,orsimulaIoncanbelessexpensiveandeasiertosetup•  YoushouldbefocusedontesIngthebusinesslogicofthesoTware,i.e.thecalculaIonsanddeterminaIonsthatsoTwaremakes•  YoucanalsofocusontesIngtheuserinterfaceorinteracIveelementsofthesystem


Hardware Tes)ng

• HardwaretesIngcanbeagileaswell• HardwarecanhaveadifferentcadencethansoTware• AgilehardwaretesIngisaboutautomaIngasmuchofthetesIngprocessaspossible•  SoTwaretools,bothhomegrownandthirdparty,canbeusedtoautomatehardwaretesIng• UsingthesystemsoTwarethecontrolsthathardwaretotestthehardwareisacceptablebutkeepthefocusonthehardware

9/21/16

9


System Tes)ng

•  SystemtesIngisthecombinaIonofthesoTwareandhardwareworkingtogether• Dependingonthenatureofyourembeddedsystem,thiscanbehardtoautomatedorautomatecompletely• Dependingonthenatureofyourembeddedsystemtheremaybeagooddealoftoolingthatcanhelpyoutestthesystemortheremaybenothingatall•  TesIngthesystemiscriIcalbecauseyoudon’tknowhowthesoTwareandhardwarewillinteractunIlyouactuallyhavetheminteract


Test Automa)on vs. Manual Tes)ng

• AgiletesIngfocusesontestautomaIonandinsometypesofapplicaIonsitispossibletoautomatenearlyallofthetesIng• WiththefocusontestautomatonitiseasytothinkthatmanuallytesInghasnoplaceinagilebutthatisn’ttrue• ManualtesIngisoTenaprecursortofuncIonaltestautomaIon• ManualtesIngcanbecheaperormoreeffecIveforsomefeaturesofanembeddedsystem•  ExploratorytesIngcanexercisethesysteminawaythattestautomaIonwillnotandfinddeeperissues

9/21/16

10


How much can you automate?

•  Lookatthisfromanumberofpointsofview:• Whatcanyouafford• WhatdoyouhavetheskillsorabiliIestoautomate• Whatispossible• Whatisthemostbeneficial

•  FindingabalancebetweentestautomaIonandmanualtesIngwillbeachallengethatallprojectsface•  TheanswerwillchangeoverImeasyoulearnandasyourcodeimproves


Coding Standards (for tests too)

• Usingcodingstandardsmeans:•  Makingthecodingstandards•  Enforcingcodingstandardsusingcodereviews

•  Itdoesn’tmaêrwhatspecificcodingstandardsyouusesomuchasyouhavethemandtheteamagreestothem•  Ifpossible,useanexisIngcodingstandardforyourtechnology• Makecodingstandardscompliancepartofyourcodereview• Beêryet,useacodeformaêrtoapplythestandardduringcodecommit• Applycodingstandardstoyourtestsaswell

9/21/16

11


Code Inspec)ons

• CodeinspecIonsincludeanumberofthings:•  Codereviews•  StaIccodeanalysis•  Securecodereviews•  Formatandstylereviews

• Automatewhatyoucan,formatandstyleareeasytoautomate•  StaIccodeanalysiscanbeagoodinputintocodereviews• Codereviewtoolscanhelp,thingslikeCrucibleandReviewBoard• Asecurecodereviewsisdifferentthanacodereviewandisadifferentskillset,youshouldmasterboth


Unit Tes)ng

• Unittestsaredeveloperteststhatareusedtomakesurethatthecodethattheywritebehavesasthedeveloperintended• Unittestsarewri^eninthelanguageandtechnologyplanormoftheapplicaIoncodeandoTenuseaframeworklikeUnity(seeh^p://www.throwtheswitch.org/unity/)• Unittestsmightberunonthedevplanormorthetargetplanorm,thisoTendependsonhowthetoolingworks•  Samplecode:voidtest_FuncIonWhichReturnsLocalVariable_ShouldReturnTheCurrentCounterValueAgain(void)

{ TEST_ASSERT_EQUAL(0,FindFuncIon_WhichIsBroken(78)); TEST_ASSERT_EQUAL_HEX(0x5a5a,FuncIonWhichReturnsLocalVariable()); }

9/21/16

12


BDD Sample Code

Scenario:SubtracttwonumbersGivenIhaveacalculatoronWhenIenter”25"intothecalculatorAndIenter”10"intothecalculatorAndIpresssubtractThentheresultshouldbe"15"onthescreenScenario:AddtwonumbersGivenIhaveacalculatoronWhenIenter”25"intothecalculatorAndIenter”45"intothecalculatorAndIpressaddThentheresultshouldbe”70"onthescreen


Mock Objects

• Mockobjectsarereplacementsforthedependenciesofthecodeundertest•  TheyareusedtomakeiteasiertotestthecodebyrespondingthesamewayalltheIme•  ThereareanumberofgoodmockobjectframeworksincludingCMock(seeh^p://www.throwtheswitch.org/cmock/)

9/21/16

13


Test Fixtures

•  TestfixturesarethecodethatcreatesafixedstatethatisusedtobaselinetestssothatthetestenvironmentisthesameeachImethetestsarerun•  TestframeworksdothisbyexposingmethodsorfuncIonsthatallowfortheconfiguraIonofdependentobjectsorforthecreaIonofmockobjects•  Foradatarelatedtestatestfixturemightloadaspecificsetofdataintothedatabase• MostunittesIngframeworksandmanytestautomaIontoolshavetestfixturesbuildintothem


Working with Simulators

• AsimulatorissoTwareorhardwareusedfortesIngthathasthesameinterfaceastheapplicaIonorsystemitisreplacing•  SimulatorsprovideaconsistentinterfacefortesIng,muchliketestfixturesormockobjects•  Simulatorsaremorefullfeaturedandcanreacttoinput,inmorevariedwaysthatothertesIngtools•  Theideaisthatasimulatorwillreactmoreliketherealcomponentintherealworld•  SimulatorscanbeexpensivetobuildandmaintainbuttheremaybenootheropIon

9/21/16

14


Tes)ng services for embedded so3ware

• ManyInternetofThings(IoT)devicesuseservices•  ServicesareInternetaccessiblebitsoffuncIonalitythatdothingslike:•  Collectdata•  Calculateresults•  Coordinateresources

•  TesIngtheseservicescanbeeasiertoautomatethenthedevicesthemselves•  Scalecanbeanissueforservices• Networkavailabilityorreliabilitycanbeanissueforthedevices• PerformancetesIngisoTenimportantfortesIngservices


What to take away from the presenta)on

•  TestautomaIonisthekeytoagiletesIng• DotestautomaIonplanningwhileyoudotestplanning• CreatetestsuitesthattestyoursoTwareseparatelyfromyourhardwareandanothersuitethatteststhesystemasawhole

9/21/16

15


Ques)ons?

•  TomSIehm•  [email protected]• @thomassIehm

Agile Testing for Embedded and IoT Software Development

Software

Transcript of Agile Testing for Embedded and IoT Software Development