MMIS Testing Without Getting Testy

MESC August, 18, 2016

MMISTesting……WithoutGettingTestyTheNewHampshireExperience

DianeDelisle– NewHampshireDirector,MMIS

LauraHall,PMP– CognosanteProjectManager

MMISTesting…WithoutGettingTesty- TheNewHampshireExperience

Today’sTopics…

1. MMISTesting…

2. …WithoutGettingTesty

3

MMISTesting…WithoutGettingTesty- TheNewHampshireExperience

MMISTesting

DianeDelisle

4

TheState’sResponsibility

• TheMMIS istheState’ssystem

• Thecontractorshandoffatestedsolution

• TheStatemustexecuteitsownteststoconfirmthatthesystemoperatesandperformsaccordingtospecs

5

ADedicatedStateTeam

• FullyimmersedintheDDIeffort• Knowsrequirementsandhowsolutionaddressesthem• Buildsunderstandingofsystemfunctions,processes,anddata

• Developsstandardmethodsandexecutes logically• CoordinatesandcollaborateswithDevandQAcontractors

6

Governance/Procedures/Communication

• StructuredChangeManagement• SystematizedDefectTracking• PlannedandCoordinatedReleaseStrategy• Collaborative IssueResolution• ContingencyPlanning• EffectiveCommunication– BetweenTeams,TradingPartners,andBusinessUsers

7

TestEnvironments

• Statecontrolsitstestenvironment(s)• Useracceptanceenvironment isamirrorofproduction

• Clusteredvs.un-clustered• Allcapabilitiessupported

• Routineexecutionofdailyprocesses,weekly financials,capitationcycles,andreporting

• Differentenvironments fordifferentpurposes• MembereligibilityorEDItesting

8

TestData

• Hasproduction-likequality• Expansivetocovereverydayconditions• Refreshedaccordingtoplan• Integrityisprotected• Canbealignedwithtradingpartnerdataneeds• TheStateneedsaccesstoviewdatainthedatabase

9

InternalTestingTestNewChangesorDefectFixes

• Positiveandnegative• Makesurefixhappenswhenitshould

• Generateandverifyinputsandoutputs• Executeupdates

• Online,database,interfaceupdates• Adds,changes,voids,deletes

• Confirmusersecurityprivileges

10

ExternalTestingWithInterfaceandTradingPartners

• Planapproachanddataneeds• Coordinatedeliveryandtesting• Executetestingandvalidation• Processinboundandoutboundtransactions• Reconcileandshareresultsandidentifieddefects

• Insidertip:Run1stinboundproductionfilesinUAT

11

RegressionTesting

• Executeroutinesystemprocesses• Dailyeligibilityfiles• Allreimbursementmethods• Weeklyfinancialcycles• Monthlycapitationcycles• Letter&reportoutputs• Periodicspecialjobs

• Validatecriticalprocesses• Evenifnotdirectlyimpactedbythefixorcoderelease

12

Follow-Through

• Documentnewdefects(ifany)• Usedetaileddescriptiveinformationandevidence

• Screenshots• Reportoutput• Datafiledetails,etc.

• Close-outdefectsandchangerequests• Educateusers

• New/changedfunctionality13

PrepforNextRelease

• UpdateRegressionTestingscenarios

• Accountforthenewsystemfunctionality

• Updatesystemdocumentation• IncludeallimplementedCRsandDefectfixes

• ApplyLessonsLearned• Modifyanyaffectedproceduresanddocumentation

• UpdateOperationalProcedures• New/changedfunctions

14

MMISTesting…WithoutGettingTesty- TheNewHampshireExperience

Part2

…WithoutGettingTesty

LauraHall15

16

WhyDoWeCare?

17

• Accomplishinganythingsignificantrequiresworkingwithhumanbeings.

• Humansareless effectivewhentheyareirritated.

Avoidingit– WhatSheSaid

18

• Establishprocedures• Establishsupportiveenvironment• Defineroles• Buildcollaborationthemeandcollaborativeteam

Avoidingit,Fixingit– MoreIdeas

• Maintainyourcomposure/balance• EatyourWheaties• Take10deepbreaths• LifeorDeathsituation?

• Stepintotheotherperson’sshoes• Listen• Observe• Imagine

19

Avoidingit,Fixingit

• Communicate• Mindyourtone(teamplayer)• Listen(andothersaremoreopentolistentoyou)

• Email• Lookslikeitiscomingfromamachine,buttheemailisusuallybetweenpeople.

• Ifemailsgetcontentious,confusing,callonthephoneandlisten.• SecretWeapon

20

21

AnIdeaWorthSharing

22

“I'velearnedthatpeoplewillforgetwhatyousaid,

peoplewillforgetwhatyoudid,butpeoplewillneverforget

howyoumadethemfeel.”

- MayaAngelou

MMISTesting…WithoutGettingTesty- TheNewHampshireExperience

Contacts

DianeDelisle– NewHampshireDirector,MMISDiane.Delisle@dhhs.nh.gov (603)223-4744

LauraHall,PMP– CognosanteProjectManagerLaura.Hall@Cognosante.com (505)699-7166

23

Missouri - Joint Testing Strategy for Eligibility and Enrollment

PaulaPeters,MEDESProjectDirectorKimberlyBrandt,IV&VProjectManager

MEDESHistoryOctober 1, 2013 - Implementation of the Citizen Portal

January 6, 2014 – Implementation of the Worker Portal

June 2015 –• Appointed a State Project Director• Contract with IBM to provide analysis of remaining

MEDES work

August 2015 - Aggressive MEDES Roadmap was created

MissouriMEDES

"Insanity:doingthesamething overandoveragainandexpectingdifferentresults."- AlbertEinstein

MovingMEDESForward

• Since August 2015 – Over 60 Production Implementations

• Aggressive schedule required changes in testing methodology

TestPlanning

MEDESTestingMethodology– TheApproach• Functional Design Documents/Test Plans

• Time Travel vs Non-Time Travel

• Test Environments/Coordination with downstream systems

• Execution • Unit Testing• System Integration Testing (SIT)• Joint SIT /User Acceptance Testing• Final UAT

MEDESTestingMethodology– TheToolsandResources

• JIRA

• Organizational Change Management

• “Tiger Team”

• FSD Integration Team (FIT)

• Field staff participate in MEDES testing

MEDESJointSIT/UAT

• Vendor onsite presence

• Creation/execution of Joint SIT/UAT test scripts

• Defect logging and triage

• State completes validation

• Regression testing

MEDESFinalUAT• Creation of Final UAT test scripts

• Focus on “failed” test scenarios from Joint SIT/UAT

• Volume of test scenarios is 1/3 of Joint SIT/UAT

• Selected end-to-end test scenarios

• State executes and validates

MEDESTestingLessonsLearned

• Communication and coordination critical

• Daily tracking against plan thru dashboards

• Vendor onsite presence is key

• Triage room

QuestionsandContactInformation

Contact Information:

PaulaPeterspaula.peters@oa.mo.gov

KimberlyBrandtkimberly.brandt@dss.mo.gov

MMISTesting:TestingWithoutBeingTesty

NewMEDSProjectAugust2016

O f f i c e o f t h e G ov e r n o r | M i s s i s s ip p i D iv i s ion o f Me d i c a id

ProblemStatementQuestion:HowcanIavoiddiscoveringtoomany

defectsandidentifyingnewChangeRequestsduringUAT?

StephenM.Oshinsky,ManagingConsultant, iTECH

MarkA.Joyce,PMP,CSM,Director– Business Development,StateandLocalGovernment

Strategy• AfterdiscussionswiththeDDIvendor,DOMdecidedto

requeststwo“sprints”priortoUATtoassistthevendorindiscoveringdefectsandrequiredchanges.

• TheSprintswerereleasedwithincreasingfunctionalityandrefinedconversiondata.

2016Jan Mar May Jul Sep Nov 2016 Mar May Jul

NEWMedsProjectkickoff1/5/2015

NEWMedsstatewideimplementation8/8/2016

Sprint1 10/5/2015- 10/9/2015

Sprint2 12/7/2015- 12/11/2015

8/31/2015- 4/4/2016DDIvendorsystemtestperiod

4/11/2016- 5/13/2016UAT

SprintTestResultsAsdepictedinthefollowinggraph,eachsprintdetectedasignificantnumberofnewdefects(andCRscreated).Thesedefectswereuncoveredacrossallsubsystems.Subsequenttoeachsprint,defectlevelsremainedstableforseveralweeks.

EstimateofUATDefects

• TheincreasingtrendlineindefectsfoundsoclosetoUATwasadefiniterisk.

• Basedonthepreviousdata,theIV&VteamdevelopedamodeltoestimatethenumberofdefectsthatwouldbeopenattheendofUAT.

• Inputsincluded:Numberofteststoberun(635)Numberoftesters(20)Numberofdaysoftesting(10forinitialtests)Averagedefects/tester(97)

DOMUATAnalysis• Basedonthemodel,DOMrealizedUATwouldnotfinishinthe

timeallotted

Week1 Week2 Week3 Week4 Week5Found 71 71 3 3 3

TotalDef 71 143 146 149 152

ClosedWk1 4 26 16 8 3

ClosedWk2 26 16 8 3ClosedWk3 1 1 0ClosedWk4 1 1ClosedWk5 1

TotalClosed 0 4 51 32 17 8

Rejected 0 0 5 3 2 1NetClosed 0 4 56 36 19 9OpenDef 71 139 96 70 57 50

UATDONE

UATWeeks

50opendefectswerepredictedattheendofUAT

• DOMandtheSLIIV&Vteamdevelopedastrategytomitigatetherisk.

MitigationStrategy

• DOMworkedwiththevendorforthreeextrareleases,thatwerethentested.

• ReleasesaddednewfunctionalityassociatedwithCRsandasmanydefect fixesaspossible

• Aftereachrelease, testcaseswererefinedandsupplementedsothenumberoftestsgrewfrom635to1400

• Duetolargernumberoftestcases,additionaltesterswereaddedtoUAT

ResultsofAllDOMTesting

• Defects foundduringextrareleases:176• Defects foundduringUAT:141• Drivenbythedefectdatacollectedduringthesprintsandtheadditionaltestingassociatedwiththereleases,UATprovedtobemanageableandtheexitcriteriawasmet

NewMEDSisLIVE!

LessonLearned• PlanspecificagilelikeSprintsduringsystemtestwhereUATtestersfullyexerciseadelivered featureset

• HavingactualE&Ecaseworkersdoingtestingwasinstrumental

• Employadatadrivenapproachtodecideonadditionaltesting

• UATtesterstestprevious functionalityandnewfunctionalityoneachsprintusingUATstyletestcases

• EachSprintbuildsonthepreviousSprintwithdefectfixes, changes,andnewfunctionality

• Releasesallowforrefiningandsupplementingtestcases,wherenecessary

BreakingDownOrganizationalBarriers

• Collaborationbetween Stateandvendortestteamsiscriticalforasuccessfulproduct

• DiminishthewallsthattypicallypreventUATtestersfromtestingduringtheSystemTestphase

• TestearlyinacollaborativewaytoensurethevendorandStatehaveaclearandunifiedsoftwaredevelopment vision

ContributiontoSuccess

• SLIasourIV&Vvendorcontributedindustryexperiencetohelpmoldtheprocess.• SLIDefecttrackingprovidedclearvisibilityintoprogressduring

testing.• DefecttrendinganalysishelpedconvinceDOMandDDIVendorto

doadditionalsprints• SLIindustryknowledgeandsubjectmatterexpertscontributedin

testcasedevelopment– PreviousUATtestingatDOMused~700testcasesbutSLIidentifiedandmanagedthedevelopmentof~1400cases

ContactInformationStephenM.OshinskyManagingConsultant,iTECHOfficeoftheGovernor,DivisionofMedicaid601-942-9624Stephen.Oshinsky@medicaid.ms.govwww.medicaid.ms.gov

MarkA.Joyce,PMP,CSMDirector– BusinessDevelopment,StateandLocalGovernmentSolutions307-220-8855mjoyce@SLIGlobalSolutions.comwww.SLIglobalsolutions.com

Questions?