@nicolefv

WhatILearnedfromFourYearsof Sciencing theCrapOutof

DevOps

NicoleForsgren,PhDDirector,OrganizationalPerformanceandAnalytics

@nicolefv

Outline

Howtomakeyourdatasuckless• Writinggoodsurveyquestions• Makingsurethesurveyquestionsaregood-withSCIENCE

• (Thesemethodsapplytoyoursystemandlogdata)

Whatwefound…thatwedid(ANDdidn’t)expectThingsaboutContinuousDeliveryThingsaboutManagement

@nicolefv

Notalldataiscreatedequal

Whoherethinkssurveysaresh*t?

@nicolefv

Notalldataiscreatedequal

Whoherethinkssurveysaresh*t?

WhohereLOVESthedatafromtheirlogfiles?

@nicolefv

WhatisaLatentConstruct?

@nicolefv

Weuse

PSYCHOMETRICS

tomakeoursurveydatagood*

*orgiveusareasonableassurancethatit’stellinguswhatwethinkit’stellingus(&someofthiscanalsoapplytoyourlogdata)

@nicolefv

Psychometricsincludes:

Constructcreation(manual)• Whenpossible:usepreviouslyvalidatedconstructs• Basedondefinitionsandtheory,carefullyandpreciselyworded,cardsortingtask,pilottested

Constructevaluation(statistics)• EstablishingValidity:discriminantandconvergent• EstablishingReliability

@nicolefv

PsychometricsWritingExample:Culture

• Doesitmattertoourstudy?• Morethanjustintuition?

• WhatKINDofculture?• National identityandnorms• Adaptiveculture• Value learning(2014study)• Value informationflowandtrust(2014and2015

studies -- Westrumculture)

@nicolefv

PsychometricsWritingExample:Culture

• Doesitmattertoourstudy?• Morethanjustintuition?

• WhatKINDofculture?• National identityandnorms• Adaptiveculture• Value learning(2014study)• Valueinformationflowandtrust(2014and2015

studies-- Westrumculture)

@nicolefv

WestrumtypologyPathologicalPower-oriented

BureaucraticRule-oriented

GenerativePerformance-oriented

Low cooperation Modest cooperation High cooperation

Messengers shot Messengers neglected Messengers trained

Responsibilities shirked Narrow responsibilities Risks are shared

Bridging discouraged Bridging tolerated Bridging encouraged

Failure leads to scapegoating

Failure leads to justice Failure leads to inquiry

Novelty crushed Novelty leads to problems Novelty implemented

Try writing items

yourself!

Use strong statements with clear language.

Westrum, R. (2004). A typology of organisational cultures. Quality and safety in health care, 13(suppl 2), ii22-ii27.

@nicolefv

WestrumCultureItems

• Onmyteam,informationisactivelysought.• Onmyteam,failuresarelearningopportunities,andmessengersofthemarenotpunished.

• Onmyteam,responsibilitiesareshared.• Onmyteam,cross-functionalcollaborationisencouragedandrewarded.

• Onmyteam,failurecausesinquiry.• Onmyteam,newideasarewelcomed.

Found to be valid & reliable

Predictive of IT Performance & Organizational Performance

@nicolefv

PsychometricsAnalysisExample:NotificationofFailureAtmyorganization…

• Weareprimarilynotifiedoffailuresbyreportsfromcustomers.

• WeareprimarilynotifiedoffailuresbytheNOC.• Wegetfailurealertsfromloggingandmonitoring systems.• Wemonitorsystemhealthbasedonthresholdwarnings(ex.

CPUexceeds 100%).• Wemonitorsystemhealthbasedonrate-of-changewarnings

(ex.CPUusagehas increasedby25%overthelast10minutes).

Original in 2014, but there was a surprise.

Can you spot it?

@nicolefv

PsychometricsAnalysisExample:NotificationofFailureAtmyorganization…

• Weareprimarilynotifiedoffailuresbyreportsfromcustomers.

• WeareprimarilynotifiedoffailuresbytheNOC.• Wegetfailurealertsfromloggingandmonitoring systems.• Wemonitorsystemhealthbasedonthresholdwarnings(ex.

CPUexceeds 100%).• Wemonitorsystemhealthbasedonrate-of-changewarnings

(ex.CPUusagehas increasedby25%overthelast10minutes).

Notification from

NEAR

Notification from FAR

@nicolefv

Moredatatests!

Plus,wetesttomakesurethesurveydoesn’thaveotherproblems.• Commonmethodvariance(CMV)(akaCMBforBias)• Earlyvs.lateresponders• Surveydrop-offratesandbias

@nicolefv

OkayNOW wecanlookatthedataandhowitrelatestoeachother

@nicolefv

Anoteaboutanalysismethods

Oneofthreeconditionsmustbemet:

1. Longitudinal(no,thisiscross-sectional)

2. Randomized,experimentaldesign(no,thisisanon-experimental)

3. Theory-baseddesign

Whenthisconditionwasnotmet,onlycorrelationsweretestedandreported

@nicolefv

KEYFINDING:ITPerformance anditsbehavior

Acombinationofthroughput andstability

leadtimeforchangesreleasefrequency

timetorestoreservicechangefailrate

Forsgren, N., J. Humble (2016). "DevOps: Profiles in ITSM Performance and Contributing Factors." In the Proceedings of the Western Decision Sciences Institute (WDSI) 2016, Las Vegas, NV.

@nicolefv

KEYFINDING:ITperformancematters!“Firmswithhigh-performingITorganizationsweretwiceaslikelytoexceedtheirprofitability,marketshareandproductivitygoals.”

ITPerformanceispredictiveoforganizationalperformance.http://bit.ly/2014-devops-report/

http://bit.ly/2015-devops-report/

Forsgren, N., J. Humble (2016). “The Role of Continuous Delivery in IT and Organizational Performance.” In the Proceedings of the Western Decision Sciences Institute (WDSI) 2016, Las Vegas, NV.

@nicolefv

Continuousdelivery

OkayNOWwecanlookatthedata.

@nicolefv

Andnowwithaddedmath

https://devops-research.comForsgren, N., J. Humble (2016). "The Role of Continuous Delivery in IT and Organizational Performance." In the Proceedings of the Western Decision Sciences Institute (WDSI) 2016, Las Vegas, NV. Available at SSRN: http://ssrn.com/abstract=2681909

@nicolefv

@nicolefv

somesurprises

@nicolefv

What’snot stronglycorrelatedw/ITPerf?

Third-partyscriptsHomegrownscriptsCommercialconfigurationmanagementtoolsOpensourceGoldenimagesManualconfigurationmanagement

@nicolefv

What’snot stronglycorrelatedw/ITPerf?

Third-partyscriptsHomegrownscriptsCommercialconfigurationmanagementtoolsOpensourceGoldenimagesManualconfigurationmanagement

@nicolefv

Whichofthesemeasureeffective testpractices?

Developersprimarilycreate&maintainacceptancetests

QAprimarilycreate&maintainacceptance tests

Primarilycreated&maintainedbyoutsourcedparty

Whenautomatedtestspass,I’mconfidentthesoftwareisreleasable

Testfailuresarelikelytoindicatearealdefect

It’seasyfordeveloperstofixacceptance tests

Developersshareacommonpooloftestserverstoreproducefailures

Developerscreateondemandtestenvironments

Developersusetheirowndev environmentstoreproducefailures

@nicolefv

Whichofthesemeasureeffective testpractices?

Developersprimarilycreate&maintainacceptancetests

QAprimarilycreate&maintainacceptance tests

Primarilycreated&maintainedbyoutsourcedparty

Whenautomatedtestspass,I’mconfidentthesoftwareisreleasable

Testfailuresarelikelytoindicatearealdefect

It’seasyfordeveloperstofixacceptance tests

Developersshareacommonpooloftestserverstoreproducefailures

Developerscreateondemandtestenvironments

Developersusetheirowndevenvironmentstoreproducefailures

@nicolefv

Changemanagement

Allproductionchangesmustbeapprovedbyanexternalbody(e.g.changeapprovalboard,manager,etc.)beforedeploymentorimplementation(R)

Onlyhigh-riskchanges,suchasdatabasechanges,requireapproval

WehavenochangeapprovalprocessWerelyonpeerreviewtomanagechanges

@nicolefv

Changemanagement

Allproductionchangesmustbeapprovedbyanexternalbody(e.g.changeapprovalboard,manager,etc.)beforedeploymentorimplementation(R)

Onlyhigh-riskchanges,suchasdatabasechanges,requireapproval

WehavenochangeapprovalprocessWerelyonpeerreviewtomanagechanges

@nicolefv

SurpriseswithCulture

•Wewantedtoaddadditionalmeasuresofculture

• Googlestudy• Identity• RetainWestrum culture

@nicolefv

IdentityandGoogleitems• Iamglad Ichosetoworkforthisorganizationratherthananothercompany.

• Italkofthisorganizationtomyfriendsasagreatcompanytoworkfor.

• Iamwillingtoputinagreatdealofeffortbeyondwhatisnormallyexpected tohelpmyorganizationtobesuccessful.

• Ifindthatmyvaluesandmyorganization'svaluesareverysimilar.

• Ingeneral, thepeopleemployedbymyorganizationareworkingtowardthesamegoal.

• Ifeelthatmyorganizationcaresaboutme.AdaptedfromadaptedfromAtreyi Kankanhalli,BernardC.Y.Tan,andKwok-Kee Wei(2005),“ContributingKnowledgetoElectronicKnowledgeRepositories:AnEmpiricalInvestigation,“MISQuarterly,29,113-143.

@nicolefv

IdentityandGoogleitems• Iamglad Ichosetoworkforthisorganizationratherthananothercompany.

• Italkofthisorganizationtomyfriendsasagreatcompanytoworkfor.

• Iamwillingtoputinagreatdealofeffortbeyondwhatisnormallyexpected tohelpmyorganizationtobesuccessful.

• Ifindthatmyvaluesandmyorganization'svaluesareverysimilar.

• Ingeneral, thepeopleemployedbymyorganizationareworkingtowardthesamegoal.

• Ifeelthatmyorganizationcaresaboutme.AdaptedfromadaptedfromAtreyi Kankanhalli,BernardC.Y.Tan,andKwok-KeeWei (2005),“ContributingKnowledgetoElectronic KnowledgeRepositories:AnEmpiricalInvestigation,“MISQuarterly,29,113-143.

Westrum items

@nicolefv

Nowformanagementstuff

WeallknowmanagingWIPisimportant,right?

@nicolefv

Nowformanagementstuff

WeallknowmanagingWIPisimportant,right?CorrelationbetweenWIPandITPerfisnegligible

@nicolefv

Nowformanagementstuff

WeallknowmanagingWIPisimportant,right?CorrelationbetweenWIPandITPerfisnegligibleWhat’sgoingon?

@nicolefv

LeanmanagementSEM

@nicolefv

Alsoleanproductmanagement

@nicolefv

Conclusions

• Evenifyouthinkit’sobvious,TESTWITHDATA.• (iftheresults don’tsurpriseyou,you’redoingitwrong)• (ifyoudon’talsoconfirmsomethingsyouexpected, you’redoingitwrong)

•WeCANhaveitall,oratleastthroughputANDstability.

• ITmatters(butyouhavetodoitright)• DevOpsculture&practiceshaveameasurableimpactonIT&orgperf

@nicolefv

SignupforourROIwhitepaper&getpeer-reviewedresearchdevops-research.com

Formorescience-ing…

@nicolefv

Thankyounicolefv.com