Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and...
Transcript of Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and...
![Page 1: Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and Dev •*DD –Behavior DD (Agile QA?) –Pulls non-functional from Ops –Pushes](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed7a93048b98015c20215c0/html5/thumbnails/1.jpg)
MaestroDev
Quality Assurancein a
DevOps World
Enabling Trust
Kit Plummerhttp://[email protected]
1Tuesday, February 22, 2011
![Page 2: Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and Dev •*DD –Behavior DD (Agile QA?) –Pulls non-functional from Ops –Pushes](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed7a93048b98015c20215c0/html5/thumbnails/2.jpg)
MaestroDev
DevOps is an ecosystem of...
•People (relationships)
•Processes (activities)
•Product(s)
2Tuesday, February 22, 2011
![Page 3: Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and Dev •*DD –Behavior DD (Agile QA?) –Pulls non-functional from Ops –Pushes](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed7a93048b98015c20215c0/html5/thumbnails/3.jpg)
MaestroDev
Ecosystem…
• Applied on top of columnar organizations:–Development/SW eng./design/etc.–Quality assurance/testing/delivery–Product eng./operations/customers
3Tuesday, February 22, 2011
![Page 4: Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and Dev •*DD –Behavior DD (Agile QA?) –Pulls non-functional from Ops –Pushes](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed7a93048b98015c20215c0/html5/thumbnails/4.jpg)
MaestroDev
Current DevOps thinking is...
• aligned with the Agile Manifesto,
• and prescribes itself as a Cultural (organizational) and Professional (personal) movement.
4Tuesday, February 22, 2011
![Page 5: Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and Dev •*DD –Behavior DD (Agile QA?) –Pulls non-functional from Ops –Pushes](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed7a93048b98015c20215c0/html5/thumbnails/5.jpg)
MaestroDev5Tuesday, February 22, 2011
![Page 6: Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and Dev •*DD –Behavior DD (Agile QA?) –Pulls non-functional from Ops –Pushes](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed7a93048b98015c20215c0/html5/thumbnails/6.jpg)
MaestroDev6Tuesday, February 22, 2011
![Page 7: Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and Dev •*DD –Behavior DD (Agile QA?) –Pulls non-functional from Ops –Pushes](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed7a93048b98015c20215c0/html5/thumbnails/7.jpg)
MaestroDev7Tuesday, February 22, 2011
![Page 8: Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and Dev •*DD –Behavior DD (Agile QA?) –Pulls non-functional from Ops –Pushes](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed7a93048b98015c20215c0/html5/thumbnails/8.jpg)
MaestroDev8Tuesday, February 22, 2011
![Page 9: Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and Dev •*DD –Behavior DD (Agile QA?) –Pulls non-functional from Ops –Pushes](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed7a93048b98015c20215c0/html5/thumbnails/9.jpg)
MaestroDev
Trust is the cornerstone...
• Patrick Debois' math - based on Covey's "4 Cores":
• trust = ƒ(character,competence)• character = ƒ(integrity, intent)• competence = ƒ(capabilities, results)
9Tuesday, February 22, 2011
![Page 10: Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and Dev •*DD –Behavior DD (Agile QA?) –Pulls non-functional from Ops –Pushes](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed7a93048b98015c20215c0/html5/thumbnails/10.jpg)
MaestroDev
Trust is a QA deliverable...
•Results are the indicator
•Testing is the apparatus
•Proof of quality is the target
10Tuesday, February 22, 2011
![Page 11: Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and Dev •*DD –Behavior DD (Agile QA?) –Pulls non-functional from Ops –Pushes](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed7a93048b98015c20215c0/html5/thumbnails/11.jpg)
MaestroDev11Tuesday, February 22, 2011
![Page 12: Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and Dev •*DD –Behavior DD (Agile QA?) –Pulls non-functional from Ops –Pushes](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed7a93048b98015c20215c0/html5/thumbnails/12.jpg)
MaestroDev
The Continuum (borked)
12Tuesday, February 22, 2011
![Page 13: Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and Dev •*DD –Behavior DD (Agile QA?) –Pulls non-functional from Ops –Pushes](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed7a93048b98015c20215c0/html5/thumbnails/13.jpg)
MaestroDev
Proof == Trust
13Tuesday, February 22, 2011
![Page 14: Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and Dev •*DD –Behavior DD (Agile QA?) –Pulls non-functional from Ops –Pushes](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed7a93048b98015c20215c0/html5/thumbnails/14.jpg)
MaestroDev
Qualty Assuance
• Product Control–Requirements–Change–Release
• Catch “bugs” before presentation–Functional AND Non-Functional
14Tuesday, February 22, 2011
![Page 15: Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and Dev •*DD –Behavior DD (Agile QA?) –Pulls non-functional from Ops –Pushes](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed7a93048b98015c20215c0/html5/thumbnails/15.jpg)
MaestroDev
Bi-Directional (required)
15Tuesday, February 22, 2011
![Page 16: Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and Dev •*DD –Behavior DD (Agile QA?) –Pulls non-functional from Ops –Pushes](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed7a93048b98015c20215c0/html5/thumbnails/16.jpg)
MaestroDev
QA Frontend
• Agile–QA/Testers on
team–Stories -> UAT–Continuous
Integration
16Tuesday, February 22, 2011
![Page 17: Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and Dev •*DD –Behavior DD (Agile QA?) –Pulls non-functional from Ops –Pushes](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed7a93048b98015c20215c0/html5/thumbnails/17.jpg)
MaestroDev
QA Backend
• Test harness(es)– Integration–Functional–Non-functional
17Tuesday, February 22, 2011
![Page 18: Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and Dev •*DD –Behavior DD (Agile QA?) –Pulls non-functional from Ops –Pushes](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed7a93048b98015c20215c0/html5/thumbnails/18.jpg)
MaestroDev
QA’s DevOp Priorities
• Testing must be a first-class citizen
18Tuesday, February 22, 2011
![Page 19: Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and Dev •*DD –Behavior DD (Agile QA?) –Pulls non-functional from Ops –Pushes](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed7a93048b98015c20215c0/html5/thumbnails/19.jpg)
MaestroDev
QA’s DevOp Priorities
• Testing must be a first-class citizen
“We’re not worried about testing...our biggest
problem is automating deployment”
18Tuesday, February 22, 2011
![Page 20: Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and Dev •*DD –Behavior DD (Agile QA?) –Pulls non-functional from Ops –Pushes](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed7a93048b98015c20215c0/html5/thumbnails/20.jpg)
MaestroDev
QA’s Priorities (cont.d)
• Release management
19Tuesday, February 22, 2011
![Page 21: Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and Dev •*DD –Behavior DD (Agile QA?) –Pulls non-functional from Ops –Pushes](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed7a93048b98015c20215c0/html5/thumbnails/21.jpg)
MaestroDev
Driving Trust via Releases
• Proof (Testing) Results (Deliverables)
Unit and IntegrationLibraries/Apps(promoted)
FunctionalValidated &Versioned(released)
Functional &Non-FunctionalIntegrated &Deployed
20Tuesday, February 22, 2011
![Page 22: Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and Dev •*DD –Behavior DD (Agile QA?) –Pulls non-functional from Ops –Pushes](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed7a93048b98015c20215c0/html5/thumbnails/22.jpg)
MaestroDev
Dev-QA-Ops Choreography
21Tuesday, February 22, 2011
![Page 23: Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and Dev •*DD –Behavior DD (Agile QA?) –Pulls non-functional from Ops –Pushes](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed7a93048b98015c20215c0/html5/thumbnails/23.jpg)
MaestroDev
Push QA into Ops and Dev
• *DD–Behavior DD (Agile QA?)–Pulls non-functional from Ops–Pushes non-functional to Devs
• Less Unit, Mo Integration, Less Mock• Automate functional, but to a point• Ops and QA enable Dev testing
22Tuesday, February 22, 2011
![Page 24: Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and Dev •*DD –Behavior DD (Agile QA?) –Pulls non-functional from Ops –Pushes](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed7a93048b98015c20215c0/html5/thumbnails/24.jpg)
MaestroDev
Push QA into Ops and Dev
• *DD–Behavior DD (Agile QA?)–Pulls non-functional from Ops–Pushes non-functional to Devs
• Less Unit, Mo Integration, Less Mock• Automate functional, but to a point• Ops and QA enable Dev testing
“BDD has reduced our prerelease bugs by 80%,
in a side-by-side comparo”
22Tuesday, February 22, 2011
![Page 25: Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and Dev •*DD –Behavior DD (Agile QA?) –Pulls non-functional from Ops –Pushes](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed7a93048b98015c20215c0/html5/thumbnails/25.jpg)
MaestroDev
Alternative Perspective
• Building missiles– Platform shall have a maximum range of
100km
23Tuesday, February 22, 2011
![Page 26: Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and Dev •*DD –Behavior DD (Agile QA?) –Pulls non-functional from Ops –Pushes](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed7a93048b98015c20215c0/html5/thumbnails/26.jpg)
MaestroDev
Validation and Verification
Modeling & SimsAlgorithms/Designs
FunctionalLibrariesApps/Services
Functional &Non-FunctionalIntegrated &Deployed
24Tuesday, February 22, 2011
![Page 27: Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and Dev •*DD –Behavior DD (Agile QA?) –Pulls non-functional from Ops –Pushes](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed7a93048b98015c20215c0/html5/thumbnails/27.jpg)
MaestroDev
Quality Quality
• Tests must be good–More to it than just assertions
• True negatives? False Positives?–Testing for Failure
• Brittle Tests? Avoid testing content• Changes should come with tests
(open sourcey way)
25Tuesday, February 22, 2011
![Page 28: Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and Dev •*DD –Behavior DD (Agile QA?) –Pulls non-functional from Ops –Pushes](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed7a93048b98015c20215c0/html5/thumbnails/28.jpg)
MaestroDev
Truth isn’t Good Enough
• Regressions (read blame)–A single version change affects truth
• Measuring/Monitoring–Earlier the better
• Trending (continuous improvement)–Continuous improvement
26Tuesday, February 22, 2011
![Page 29: Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and Dev •*DD –Behavior DD (Agile QA?) –Pulls non-functional from Ops –Pushes](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed7a93048b98015c20215c0/html5/thumbnails/29.jpg)
MaestroDev
Dev/QA Reporting
27Tuesday, February 22, 2011
![Page 30: Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and Dev •*DD –Behavior DD (Agile QA?) –Pulls non-functional from Ops –Pushes](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed7a93048b98015c20215c0/html5/thumbnails/30.jpg)
MaestroDev28Tuesday, February 22, 2011
![Page 31: Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and Dev •*DD –Behavior DD (Agile QA?) –Pulls non-functional from Ops –Pushes](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed7a93048b98015c20215c0/html5/thumbnails/31.jpg)
MaestroDev
DevOps
29Tuesday, February 22, 2011
![Page 32: Quality Assurance in a DevOps World - SCALE › scale9x-media › scale... · Push QA into Ops and Dev •*DD –Behavior DD (Agile QA?) –Pulls non-functional from Ops –Pushes](https://reader034.fdocuments.in/reader034/viewer/2022042310/5ed7a93048b98015c20215c0/html5/thumbnails/32.jpg)
MaestroDev
Questions
http://devopsweekly.com/
30Tuesday, February 22, 2011