Continuous Automated Testing - Cast conference workshop august 2014
-
Upload
noah-sussman -
Category
Software
-
view
863 -
download
1
description
Transcript of Continuous Automated Testing - Cast conference workshop august 2014
![Page 1: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/1.jpg)
Noah Sussman and Joanna Burgess! ! CAST NYC, August 11, 2014
Continuous Automated Testing: A Communication System That Scales!
Jared Tarbell, Substrate
![Page 2: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/2.jpg)
Continuous Automated Testing (C.A.T.)
Schedule
✤ 9:00-10:30: CAT!
✤ 10:30-10:50: Break !
✤ 10:50-12:30: CAT!
✤ 12:30-1:30: Lunch!
✤ 1:30-3:00: CAT!
✤ 3:00-3:30: Snack!
✤ 3:30-5:00: CAT
![Page 3: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/3.jpg)
Load it Up!
✤ Jenkins http://jenkins-ci.org/ !
✤ Golly http://golly.sourceforge.net/
![Page 4: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/4.jpg)
![Page 5: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/5.jpg)
“When I use a word it means just what I choose it to mean-neither more nor less.”
What does QA mean to you?
![Page 6: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/6.jpg)
Safety is not about the breakage or lack of quality of single components. Managing quality is about single components, about seeing how they meet particular specifications.
!
—Sidney DekkerGuy Sie on Flickr
![Page 7: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/7.jpg)
In other words…when are we, and when aren’t we speaking the same language? !
More to the point…how much does it matter?
Rolling the Dice
![Page 8: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/8.jpg)
Contested Terminology
✤ Quality Assurance!
✤ Testing!
✤ Manual Testing !
✤ Bugs!
✤ Accountability!
✤ QA Environment !
![Page 9: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/9.jpg)
4REALZ: “When I use a word it means just what I choose it to mean” (ymmv)
✤ Like Humpty Dumpty, I am using contested terminology to support my own viewpoints!
✤ When I use a word, it means what I intend it to mean!
✤ If you are unsure what I mean, raise your hand!!!
![Page 10: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/10.jpg)
ETTO
✤ Efficiency-to-thoroughness-trade-offs (ETTO)!
✤ We do necessarily not pick the best option, we pick the option that best satisfies our immediate needs namely:!
✤ efficiency!
✤ thoroughness
![Page 11: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/11.jpg)
Automated Tests Are…
✤ Part of the software design process
✤ Runnable documentation
✤ NOT useful for debugging releases
✤ NOT useful for catching bugs in code
![Page 12: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/12.jpg)
Tractable Systems
A system is tractable if the principles of the functioning are known and have simple descriptions with few details. Tractable systems do not change while being described.!!
![Page 13: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/13.jpg)
Intractable SystemsIntractable systems are complex systems. A defining feature of intractable systems is that they are unsafe and defy documentation.
![Page 14: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/14.jpg)
Defining Terminology
✤ Abstraction!
✤ Orders of Magnitude Growth !
✤ Cellular Automaton!
✤ Finite Automaton!
✤ Halting Problem !
✤ CAP Theorem !
✤ Acyclic Directed Graphs !
✤ Path reduction
![Page 15: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/15.jpg)
Core Concepts
✤ Ironies of automation !
✤ Hawthorne Effects!
✤ Goodhart’s Law!
✤ Diseconomies of scale
![Page 16: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/16.jpg)
Such dev. Very ops. Wow.
![Page 18: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/18.jpg)
devops: where is the QA?
✤ STOP ASKING ME where QA fits into devops.!
✤ ROFLMAO if QA and testing aren’t ALREADY part of dev for you!!
✤ The idea that QA is distinct from development is a convenient fiction.!
✤ QA and testing have always been software engineering disciplines.
![Page 19: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/19.jpg)
Sufficiently Advanced Monitoring is Indistinguishable from Testing
✤ A Google engineer proposed this idea in 2007.!
✤ Nothing much has changed since then.!
✤ Sufficiently advanced technology is indistinguishable from magic. —Arthur C. Clarke!
✤ Sufficiently advanced technology is indistinguishable from a rigged demo. —Silicon Valley aphorism
![Page 20: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/20.jpg)
Monitoring vs. Testing
✤ Yes you should monitor all the things.!
✤ Yes you should build real time dashboards.!
✤ Yes you should deploy continuously.!
✤ Yes you should fix production bugs on the fly.!
✤ No, none of the above replaces QA and testing.!
✤ Really, no.
![Page 21: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/21.jpg)
QA and Testing are part of the software design process
✤ Monitoring has nothing to do with design.!
✤ Monitoring provides visibility into implementation.!
✤ QA and testing address design flaws.!
✤ That is, QA and testing are design tools.
![Page 22: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/22.jpg)
Each Necessary but Only Jointly Sufficient
✤Monitoring is part of the picture of overall system safety
✤QA and testing are part of the picture of overall system safety
✤Manual and automated testing — part of the overall picture
✤Developers following standards — part of the picture ✤People cooperating — part of the picture ✤The system safety picture is intrinsically incomplete!
![Page 23: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/23.jpg)
Limiting Risk
Nancy Leveson on Limiting Risk:!!✤ Oversight!
!✤ Limit complexity!!✤ Systems thinking
![Page 24: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/24.jpg)
Apply the best software engineering principles… quality assurance, testing… the highest standards. It's not going to be enough. —Nancy Leveson
Guy Sie on Flickr
![Page 25: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/25.jpg)
Law of Leaky Abstractions
“…tools which pretend to abstract out something, like all abstractions, leak, and the only way to deal with the leaks competently is to learn about how the abstractions work and what they are abstracting. So the abstractions save us time working, but they don't save us time learning.” !
—Joel Spolsky
![Page 26: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/26.jpg)
Failure is Scary But Also Inevitable
Everyone—even cats—must deal with failure.
![Page 27: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/27.jpg)
Leaky Abstractions in Software
✤ How does this relate to QA? ✤ in the new view of systems safety, safety is
derived from being able to predict the behavior of the system.
✤ Since abstractions leak, no general abstraction can ever provide granular insight into system behavior.
![Page 28: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/28.jpg)
The Mechanistic Fallacy: A Leaky Abstraction Pushed Too Far✤ Systems Thinking!
✤ Normal Failure!
✤ ETTO (Efficiency to Thoroughness Tradeoff)!
✤ Intractable Systems!
✤ The New View of system safety
![Page 29: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/29.jpg)
Complex Systems are Intractable
—Erik Hollnagel
![Page 30: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/30.jpg)
“You can take a bicycle apart and put it back together…you cannot do that with a frog.” —Liz Keogh
Bicycle vs. Frog
![Page 31: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/31.jpg)
How is a Software System Like a Frog?
✤ It’s a running system.!
✤ It’s changing all the time.!
✤ Conceptually can’t be decomposed into discrete components.!
✤ A frog is intractable.
![Page 33: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/33.jpg)
Let’s compare a bicycle to the universe
✤ How complex is a bicycle?
![Page 34: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/34.jpg)
Bicycles and universes are not on the same scale of complexity
✤ About 30 parts ~ 1/30 = 0.03
![Page 35: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/35.jpg)
✤ How many components does a program have?!
✤ Consider the range of computational “parts” that exist between 1 microsecond and a half hour of computational time [Edsger Dijkstra]!
✤ 0.001 / 1800 = 5.55-7!
✤ Notice how there’s an exponent this time?
Now let’s compare a the universe to a computer program
![Page 36: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/36.jpg)
Computers and universes are on the same scale of complexity
![Page 37: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/37.jpg)
What does n-7 levels of hierarchy look like?
![Page 38: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/38.jpg)
The Pesticide Paradox and the Complexity Barrier
“Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual.”!
“Software complexity (and therefore that of bugs) grows to the limits of our ability to manage that complexity.”!
! ! ! ! ! ! ! ! ! ! ! ! ! ! ! —Boris Beizer
![Page 39: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/39.jpg)
The Pesticide Paradox✤ Roaches have decided to terraform your apartment to suit
their needs. Bug terminator summoned; pesticides released. !
✤ Dead roaches. Problem solved or another problem created? !
✤ An unintended consequence: the more efficient the pesticide is the greater the chance that the genes for pesticide resistance will dominate the gene pool going forward. !
✤ This killing “all the bugs” actually results, over the long term, in large generations of bugs that will not be so easy to kill.
![Page 40: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/40.jpg)
The Complexity Barrier
✤ There is an upper limit to how much complexity the human brain can handle. !
✤ Simplifying an existing system is a very difficult problem.!
✤ There are no general solutions for simplification of systems.
![Page 41: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/41.jpg)
!!
!
✤ Simplicity and design are equally important until you have to pick one; then pick simplicity. !
!
✤ Modern business practices assume that there is no functional upper limit on system complexity. !
!
MIT vs. New Jersey! a.k.a “worse is better”
![Page 42: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/42.jpg)
Want to Reduce Cost?
Communicate!
![Page 43: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/43.jpg)
Conway’s Game of Life
1. Survivals. Every counter with two or three neighboring counters survives for the next generation.
2. Deaths. Each counter with four or more neighbors dies (is removed) from overpopulation. Every counter with one neighbor or none dies from isolation.
3. Births. Each empty cell adjacent to exactly three neighbors--no more, no fewer--is a birth cell. A counter is placed on it at the next move.
Delightfully Simple Rules
! ! -Martin Gardner
![Page 44: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/44.jpg)
In real systems the rules aren’t as delightfully simple.
Jonathan Borofsky, Counting 1 to 3227146, 1969/1976!
![Page 45: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/45.jpg)
Jenkins lab (frequently-running job)
1. We will need test data. So we will create two jobs: a frequently running job and a long running job!
2. Set up a Jenkins instance (download the installer from jenkinsci.org)!
3. Go to http://localhost: 8080/!
4. Click “new item”!
5. Type a name for your frequently-running project!
6. Click “build a freestyle software project” and click “OK”!
7. Create a new build step and type the code for your frequently running project !
8. Schedule the job to run once a minute (using the Cron syntax as described in the inline help)
![Page 46: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/46.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
A Review of My Previously Published Approach to Jenkins API Automation
![Page 47: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/47.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
The Jenkins JSON API
To read the documentation, go to http://ci.example.com/api
You can append /api/json to the end of nearly any Jenkins URL to get JSON data. http://ci.example.com/api/json for latest builds http://ci.example.com/job/unit-tests/api/json for history of a specific build. http://ci.example.com/computer/api/json for slave information.
![Page 48: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/48.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Using depth= to get more granular data
If the API response doesn't contain some data that you expected, try appending ?depth=1 to the URL. If you still don't get what you want, increase the integer value. Usually you'll keep getting more data up until around ?depth=5 Exactly what and how much data you'll get is dependent on the configuration of your Jenkins instance.
![Page 49: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/49.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Drawbacks of using depth=
Depending on how deep you go into the API response, you can wind up with a lot of data. Such large responses can be expensive to download. In some cases you can request a response so large that you will wind up DDoSing Jenkins!
![Page 50: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/50.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Using tree= to filter the API response
The tree= URL parameter is like a SQL query.
Use depth= to look at the wealth of information available. Then use tree= to select only the information you actually need. This can dramatically reduce the size of your API responses.
![Page 51: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/51.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
![Page 52: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/52.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
?tree=busyExecutors
![Page 53: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/53.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
?tree=computer[displayName]
![Page 54: Continuous Automated Testing - Cast conference workshop august 2014](https://reader036.fdocuments.in/reader036/viewer/2022062404/553a3ffd550346b4378b4ab8/html5/thumbnails/54.jpg)
Suggested Reading✤ Drive: The Surprising Truth About What Motivates Us by Dan Pink with RSAnimate https://www.youtube.com/watch?v=u6XAPnuFjJc!
✤ Ironies of Automation by Lisanne Bainbridge http://www.ise.ncsu.edu/nsf_itr/794B/papers/Bainbridge_1983_Automatica.pdf!
✤ Big Ball of Mud by Brian Foote and Joseph Yoder http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.316.433&rep=rep1&type=pdf!
✤ The Rise of Worse is Better by Richard Gabriel http://www.jwz.org/doc/worse-is-better.html!
✤ Worse is Better by Richard Gabriel http://dreamsongs.com/worseisbetter.html!
✤ To Err is Human: ETTO Principle by Erik Hollnagel http://www.namahn.com/we-share/interviews/erik-hollnagel-err-human-etto-principle!
✤ The Law of Leaky Abstractions by Joel Spolsky http://www.joelonsoftware.com/articles/LeakyAbstractions.html!
✤ Designers and Women in Open Source by Vitorio Miliano http://old.vi.to/designers-and-women-in-open-source.html!
✤ The fantastic combinations of John Conway’s new solitaire game “life” by Martin Gardner http://www.ibiblio.org/lifepatterns/october1970.html!
Books!✤ The Field Guide to Understanding Human Error by Sidney Dekker http://amzn.to/1mdwNdU!
✤ How Google Tests Software by James Whittaker http://amzn.to/XvMtUl!
✤ Software Testing Techniques (2nd edition) by Boris Beizer http://amzn.to/YqtevB!
✤ The Timeless Way of Building by Christopher Alexander http://amzn.to/1tgW3UJ!