MMIS Testing Without Getting Testy · • Can be aligned with trading partner data needs • The...
Transcript of MMIS Testing Without Getting Testy · • Can be aligned with trading partner data needs • The...
MMIS Testing Without Getting Testy
MESC August, 18, 2016
MMISTesting……WithoutGettingTestyTheNewHampshireExperience
DianeDelisle– NewHampshireDirector,MMIS
LauraHall,PMP– CognosanteProjectManager
MMISTesting…WithoutGettingTesty- TheNewHampshireExperience
Today’sTopics…
1. MMISTesting…
2. …WithoutGettingTesty
3
MMISTesting…WithoutGettingTesty- TheNewHampshireExperience
MMISTesting
DianeDelisle
4
TheState’sResponsibility
• TheMMIS istheState’ssystem
• Thecontractorshandoffatestedsolution
• TheStatemustexecuteitsownteststoconfirmthatthesystemoperatesandperformsaccordingtospecs
5
ADedicatedStateTeam
• FullyimmersedintheDDIeffort• Knowsrequirementsandhowsolutionaddressesthem• Buildsunderstandingofsystemfunctions,processes,anddata
• Developsstandardmethodsandexecutes logically• CoordinatesandcollaborateswithDevandQAcontractors
6
Governance/Procedures/Communication
• StructuredChangeManagement• SystematizedDefectTracking• PlannedandCoordinatedReleaseStrategy• Collaborative IssueResolution• ContingencyPlanning• EffectiveCommunication– BetweenTeams,TradingPartners,andBusinessUsers
7
TestEnvironments
• Statecontrolsitstestenvironment(s)• Useracceptanceenvironment isamirrorofproduction
• Clusteredvs.un-clustered• Allcapabilitiessupported
• Routineexecutionofdailyprocesses,weekly financials,capitationcycles,andreporting
• Differentenvironments fordifferentpurposes• MembereligibilityorEDItesting
8
TestData
• Hasproduction-likequality• Expansivetocovereverydayconditions• Refreshedaccordingtoplan• Integrityisprotected• Canbealignedwithtradingpartnerdataneeds• TheStateneedsaccesstoviewdatainthedatabase
9
InternalTestingTestNewChangesorDefectFixes
• Positiveandnegative• Makesurefixhappenswhenitshould
• Generateandverifyinputsandoutputs• Executeupdates
• Online,database,interfaceupdates• Adds,changes,voids,deletes
• Confirmusersecurityprivileges
10
ExternalTestingWithInterfaceandTradingPartners
• Planapproachanddataneeds• Coordinatedeliveryandtesting• Executetestingandvalidation• Processinboundandoutboundtransactions• Reconcileandshareresultsandidentifieddefects
• Insidertip:Run1stinboundproductionfilesinUAT
11
RegressionTesting
• Executeroutinesystemprocesses• Dailyeligibilityfiles• Allreimbursementmethods• Weeklyfinancialcycles• Monthlycapitationcycles• Letter&reportoutputs• Periodicspecialjobs
• Validatecriticalprocesses• Evenifnotdirectlyimpactedbythefixorcoderelease
12
Follow-Through
• Documentnewdefects(ifany)• Usedetaileddescriptiveinformationandevidence
• Screenshots• Reportoutput• Datafiledetails,etc.
• Close-outdefectsandchangerequests• Educateusers
• New/changedfunctionality13
PrepforNextRelease
• UpdateRegressionTestingscenarios
• Accountforthenewsystemfunctionality
• Updatesystemdocumentation• IncludeallimplementedCRsandDefectfixes
• ApplyLessonsLearned• Modifyanyaffectedproceduresanddocumentation
• UpdateOperationalProcedures• New/changedfunctions
14
MMISTesting…WithoutGettingTesty- TheNewHampshireExperience
Part2
…WithoutGettingTesty
LauraHall15
16
WhyDoWeCare?
17
• Accomplishinganythingsignificantrequiresworkingwithhumanbeings.
• Humansareless effectivewhentheyareirritated.
Avoidingit– WhatSheSaid
18
• Establishprocedures• Establishsupportiveenvironment• Defineroles• Buildcollaborationthemeandcollaborativeteam
Avoidingit,Fixingit– MoreIdeas
• Maintainyourcomposure/balance• EatyourWheaties• Take10deepbreaths• LifeorDeathsituation?
• Stepintotheotherperson’sshoes• Listen• Observe• Imagine
19
Avoidingit,Fixingit
• Communicate• Mindyourtone(teamplayer)• Listen(andothersaremoreopentolistentoyou)
• Email• Lookslikeitiscomingfromamachine,buttheemailisusuallybetweenpeople.
• Ifemailsgetcontentious,confusing,callonthephoneandlisten.• SecretWeapon
20
21
AnIdeaWorthSharing
22
“I'velearnedthatpeoplewillforgetwhatyousaid,
peoplewillforgetwhatyoudid,butpeoplewillneverforget
howyoumadethemfeel.”
- MayaAngelou
MMISTesting…WithoutGettingTesty- TheNewHampshireExperience
Contacts
DianeDelisle– NewHampshireDirector,[email protected] (603)223-4744
LauraHall,PMP– [email protected] (505)699-7166
23
Missouri - Joint Testing Strategy for Eligibility and Enrollment
PaulaPeters,MEDESProjectDirectorKimberlyBrandt,IV&VProjectManager
MEDESHistoryOctober 1, 2013 - Implementation of the Citizen Portal
January 6, 2014 – Implementation of the Worker Portal
June 2015 –• Appointed a State Project Director• Contract with IBM to provide analysis of remaining
MEDES work
August 2015 - Aggressive MEDES Roadmap was created
MissouriMEDES
"Insanity:doingthesamething overandoveragainandexpectingdifferentresults."- AlbertEinstein
MovingMEDESForward
• Since August 2015 – Over 60 Production Implementations
• Aggressive schedule required changes in testing methodology
TestPlanning
MEDESTestingMethodology– TheApproach• Functional Design Documents/Test Plans
• Time Travel vs Non-Time Travel
• Test Environments/Coordination with downstream systems
• Execution • Unit Testing• System Integration Testing (SIT)• Joint SIT /User Acceptance Testing• Final UAT
MEDESTestingMethodology– TheToolsandResources
• JIRA
• Organizational Change Management
• “Tiger Team”
• FSD Integration Team (FIT)
• Field staff participate in MEDES testing
MEDESJointSIT/UAT
• Vendor onsite presence
• Creation/execution of Joint SIT/UAT test scripts
• Defect logging and triage
• State completes validation
• Regression testing
MEDESFinalUAT• Creation of Final UAT test scripts
• Focus on “failed” test scenarios from Joint SIT/UAT
• Volume of test scenarios is 1/3 of Joint SIT/UAT
• Selected end-to-end test scenarios
• State executes and validates
MEDESTestingLessonsLearned
• Communication and coordination critical
• Daily tracking against plan thru dashboards
• Vendor onsite presence is key
• Triage room
MMISTesting:TestingWithoutBeingTesty
NewMEDSProjectAugust2016
O f f i c e o f t h e G ov e r n o r | M i s s i s s ip p i D iv i s ion o f Me d i c a id
ProblemStatementQuestion:HowcanIavoiddiscoveringtoomany
defectsandidentifyingnewChangeRequestsduringUAT?
StephenM.Oshinsky,ManagingConsultant, iTECH
MarkA.Joyce,PMP,CSM,Director– Business Development,StateandLocalGovernment
Strategy• AfterdiscussionswiththeDDIvendor,DOMdecidedto
requeststwo“sprints”priortoUATtoassistthevendorindiscoveringdefectsandrequiredchanges.
• TheSprintswerereleasedwithincreasingfunctionalityandrefinedconversiondata.
2016Jan Mar May Jul Sep Nov 2016 Mar May Jul
NEWMedsProjectkickoff1/5/2015
NEWMedsstatewideimplementation8/8/2016
Sprint1 10/5/2015- 10/9/2015
Sprint2 12/7/2015- 12/11/2015
8/31/2015- 4/4/2016DDIvendorsystemtestperiod
4/11/2016- 5/13/2016UAT
SprintTestResultsAsdepictedinthefollowinggraph,eachsprintdetectedasignificantnumberofnewdefects(andCRscreated).Thesedefectswereuncoveredacrossallsubsystems.Subsequenttoeachsprint,defectlevelsremainedstableforseveralweeks.
EstimateofUATDefects
• TheincreasingtrendlineindefectsfoundsoclosetoUATwasadefiniterisk.
• Basedonthepreviousdata,theIV&VteamdevelopedamodeltoestimatethenumberofdefectsthatwouldbeopenattheendofUAT.
• Inputsincluded:Numberofteststoberun(635)Numberoftesters(20)Numberofdaysoftesting(10forinitialtests)Averagedefects/tester(97)
DOMUATAnalysis• Basedonthemodel,DOMrealizedUATwouldnotfinishinthe
timeallotted
Week1 Week2 Week3 Week4 Week5Found 71 71 3 3 3
TotalDef 71 143 146 149 152
ClosedWk1 4 26 16 8 3
ClosedWk2 26 16 8 3ClosedWk3 1 1 0ClosedWk4 1 1ClosedWk5 1
TotalClosed 0 4 51 32 17 8
Rejected 0 0 5 3 2 1NetClosed 0 4 56 36 19 9OpenDef 71 139 96 70 57 50
UATDONE
UATWeeks
50opendefectswerepredictedattheendofUAT
• DOMandtheSLIIV&Vteamdevelopedastrategytomitigatetherisk.
MitigationStrategy
• DOMworkedwiththevendorforthreeextrareleases,thatwerethentested.
• ReleasesaddednewfunctionalityassociatedwithCRsandasmanydefect fixesaspossible
• Aftereachrelease, testcaseswererefinedandsupplementedsothenumberoftestsgrewfrom635to1400
• Duetolargernumberoftestcases,additionaltesterswereaddedtoUAT
ResultsofAllDOMTesting
• Defects foundduringextrareleases:176• Defects foundduringUAT:141• Drivenbythedefectdatacollectedduringthesprintsandtheadditionaltestingassociatedwiththereleases,UATprovedtobemanageableandtheexitcriteriawasmet
NewMEDSisLIVE!
LessonLearned• PlanspecificagilelikeSprintsduringsystemtestwhereUATtestersfullyexerciseadelivered featureset
• HavingactualE&Ecaseworkersdoingtestingwasinstrumental
• Employadatadrivenapproachtodecideonadditionaltesting
• UATtesterstestprevious functionalityandnewfunctionalityoneachsprintusingUATstyletestcases
• EachSprintbuildsonthepreviousSprintwithdefectfixes, changes,andnewfunctionality
• Releasesallowforrefiningandsupplementingtestcases,wherenecessary
BreakingDownOrganizationalBarriers
• Collaborationbetween Stateandvendortestteamsiscriticalforasuccessfulproduct
• DiminishthewallsthattypicallypreventUATtestersfromtestingduringtheSystemTestphase
• TestearlyinacollaborativewaytoensurethevendorandStatehaveaclearandunifiedsoftwaredevelopment vision
ContributiontoSuccess
• SLIasourIV&Vvendorcontributedindustryexperiencetohelpmoldtheprocess.• SLIDefecttrackingprovidedclearvisibilityintoprogressduring
testing.• DefecttrendinganalysishelpedconvinceDOMandDDIVendorto
doadditionalsprints• SLIindustryknowledgeandsubjectmatterexpertscontributedin
testcasedevelopment– PreviousUATtestingatDOMused~700testcasesbutSLIidentifiedandmanagedthedevelopmentof~1400cases
ContactInformationStephenM.OshinskyManagingConsultant,iTECHOfficeoftheGovernor,DivisionofMedicaid601-942-9624Stephen.Oshinsky@medicaid.ms.govwww.medicaid.ms.gov
MarkA.Joyce,PMP,CSMDirector– BusinessDevelopment,StateandLocalGovernmentSolutions307-220-8855mjoyce@SLIGlobalSolutions.comwww.SLIglobalsolutions.com
Questions?