Post on 14-Apr-2017
World®’16
TechTalk:SometimesLessisMore–VisualizationCanReduceyourTestDatawhileEnhancingQuality!JamesWalker– PrincipalSoftwareEngineer– CATechnologies
DO5T06T
DEVOPS
2 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
©2016CA.Allrightsreserved.Alltrademarksreferencedhereinbelongtotheirrespectivecompanies.
Thecontentprovidedinthis CAWorld2016presentationisintendedforinformationalpurposesonlyanddoesnotformanytypeofwarranty. The informationprovidedbyaCApartnerand/orCAcustomerhasnotbeenreviewedforaccuracybyCA.
ForInformationalPurposesOnlyTermsofthisPresentation
3 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Abstract
Effectivetestingrequireshighqualitytestdata,butmostorganizationsstillrelyonproductiondatawhichprovidesjust10-20%functionalcoverage.Thisdataisdrawnfrom“businessasusual”scenariosthathaveoccurredinthepast,andsorarelyprovidethenegativescenariosandoutliersneededtorigorouslytestsoftware.Thedatathatcomesthickandfastintoproductionisfurthertoolargeandcomplicatedforanyhumanmindtoevaluate,sothatprofilingormodellingtechnologyisneeded.Toensurethattheyhavethequalitydataneededfortesting,organizationsneedtobeabletoevaluatewhichattributesexistinexistingdata,aswellashowtheycombine.Onlythencantheyevaluatewhichmissingattributesareneededtoexecutethetestsneededtodeliverqualitysoftware.
ThisTechTalkwillshowhowdatavisualizationprovidesaquickandreliablemethodtomeasurethetestcoverageprovidedbyexistingtestdata,spottinganymissingorinvaliddataataglance.Presentingdataattributesanddimensionsinpictorialformallowsuserstounderstandwhatdatatheyhave,howitsattributesrelate,andwhatdataismissing.TheaccuratemodelcreatedinCA’sDataVisualizationcanfurtherthenbefedintoCAAgileRequirementsDesignerandCATestDataManager,creatingthesmallestsetofdataneededtosatisfyeverypossibletestautomatically.
JamesWalker
CATechnologiesPrincipalSoftwareEngineer,CAAgileRequirementsDesigner
4 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
AboutMe
§ BSc,MRes,PhD– SwanseaUniversity,Wales
§ ResearchinDataVisualisation/BigDataproblems
§ Grid-Tools– SoftwareEngineer(2012– 2015)
§ CA– LeadSoftwareEngineerARD
5 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
MOTIVATION
INTRODUCTIONTODATAVISUALIZATION
DATAVISUALIZATIONFORTESTDATA
DEMO
1
2
3
4
Agenda
CONCLUDINGTHOUGHTS5
6 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
TestDataChallenges
§ Productionisoftenviewedasthetruesourceofgoodtestdata(defaulttakeacopyofproductiondata-masking)
§ Productiondatahashighvolumesandlowvariance(edgecases)
§ Betterdataindevelopmentshouldbethegoal(subjective–whatisbetter?)
Todothisyouneedtobeabletoanswerawholebunchofquestions
7 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
TestDataChallenges
“WhatdatadoIhave?”
“Whatdatadon’tIhave?”“DoIhave thedataIneed
formytests?”
“WhereamIundertesting?”
“WhereamIovertesting?”
“Howeffectiveismytestdata?”
“Whatismydatacoverage?”
8 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
CustomSQL
Data Views/Cubes Off-the-shelf visualisationtool
OldWorldOrder…1.
2. 3.
9 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
NewWorldOrder…TestingtotheBigDataFieldTechnologicaladvancementsoverthepastdecadehaveincreasedourabilitytocollectdatatopreviouslyunimaginablevolumes
Estimatedthatpeoplewillgenerate4.3exabytes ofdataintheirlifetime(1).
Datacontainshugeamountsofvalueforgaininginsight,understanding,decisionmaking,andprediction.
Virtuallyeveryfieldofscienceandindustryistakingadvantageofanalytics(medicine,sports,weather,finances,etc).
Testingislatetotheparty – Hugeopportunitiesforbigdatatechniquestohelpustestoursoftware,understandtheresults,gaininsightintoquality,makedecisions(shouldwerelease?),andeventuallypredictresultsbeforewe’veevenranasingletestcase...
10 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
IntroductiontoDataVisualization
11 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
“Thepurposeofcomputingisinsight,notnumbers”
Visualization:
§ Atoolthatallowstheusertogaininsightintodata
§ Toformamentalvision,image,orpictureof(somethingnotvisibleorpresenttothesightoranabstraction);tomakevisibletothemindorimagination[OxfordEnglishDictionary,1989]
RichardW.Hamming,1962
14 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
http://www.comm-dev.org/
12 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
VisualizationisVeryOld
§ Oftenanintuitivesteptomakephenomenaclearere.g.agraph
§ Classical(easy)approachesknownfrombusinessgraphics(excel,etc)
§ Onlynowinthepastdecadeisthevaluestartingtobecomeprevalent
https://utah.com/parowan-gap
13 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DataSetsAreEver-increasinginSize– AGraphicalApproachIsNecessaryBefore – Simpletabulardata(verylownumberofdataanddimensions
Now – Distributedsystemscreatingmillionsofrowsasecond
14 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
VisualizationisGoodfor:
§ Exploration– Findtheunknown,unexpected– Hypothesisgeneration
§ Analysis– Confirmorrejecthypotheses– Informationdrill-down
§ Presentation– Communicate/disseminateresults
15 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
https://fluidi.wordpress.com
World®’16©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD15
16 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
SoWhatisDataVisualization?
§ Datavisualizationistheprocessofcreatinggraphicalabstractionsofdata
§ Usevisualisationonthedailybasis(i.e.Tubemap,weatherreport,stockmarket,webtraffic…)
§ Techniqueshaveenormousvaluetoallaspectsoftheworldwelivein– todaywefocusontesting&testdata!
17 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
CATestDataVisualizer
18 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
LargeRelationalDatabases
Protein Data Bank – pdb.org
19 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DataCombinations
§ Itisimpossibletoconsider“AllCombinations”ofdata
n timesn timesn timesn =verylarge
Eachspinofthelockisadataattribute40possiblepositions4inputsrequired
40x40x40x40=2,560,00040x40x40x40x40=102,400,000
40x40x40x40x40x40=4,096,000,000
20 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DataConcepts
§ Dataconcepts– onlysomecombinationsarerelevantforatestcase(testrequirements).
§ Howdotestsrelatetothedata?
§ Notallcolumnsmatter,buttheircombinedeffectdoes
§ Wecreateameta-layeroftestdataattributes
21 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DataView– FlattentheDataandPickRelevantAttributes
22 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Demo
23 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Conclusion– TestDataVisualizer
§ Avisualizationtool– designedtoanalyse &andassistinbuilding‘better’testdata
§ Useadvancedspotdiagramsandparallelcoordinates– Comparedataforvalidandinvalidsetsofcombinations– Identifymissingcombinationsofdata– Indentify overandunder-testing– Compareenvironmentsforcoverage(QA1,QA2)– Measuredatacoverageaccurately– Reservedataamongstteammembers
24 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
TestDataVisualizer
“WhatdatadoIhave?”
“Whatdatadon’tIhave?”“DoIhave thedataIneed
formytests?”
“WhereamIundertesting?”
“WhereamIovertesting?”
“Howeffectiveismytestdata?”
“Whatismydatacoverage?”
25 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
RecommendedSessions
SESSION# TITLE DATE/TIME
DO5T17SCaseStudy:Nationwide'sCATestDataManagerSuccessStory 11/17/2016at1:45PM
DO5T07TTechTalk:WhatHappenedintheBackend?ThePowerofDBCompare 11/17/2016at3:00PM
DO5X42STechVision:TestDataonDemand:DeliveringtheRightData,totheRightPlace,attheRightTime 11/17/2016at4:30PM
26 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
Stayconnectedatcommunities.ca.com
Thankyou.
27 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD
DevOps– ContinuousDelivery
FormoreinformationonDevOps– ContinuousDelivery,pleasevisit:http://cainc.to/PiTFpu