Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by:...

37
BT8 Session 6/6/2013 2:15 PM “How to Survive the Coming Test Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 8882688770 9042780524 [email protected] www.sqe.com

Transcript of Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by:...

Page 1: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

 

 

BT8 Session 6/6/2013 2:15 PM 

       

“How to Survive the Coming Test Automation Zombie Apocalypse"

   

Presented by:

Dale Emery DHE

         

Brought to you by:  

  

340 Corporate Way, Suite 300, Orange Park, FL 32073 888‐268‐8770 ∙ 904‐278‐0524 ∙ [email protected] ∙ www.sqe.com

Page 2: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

Dale Emery DHE

Since 1980, Dale Emery has worked in both IT organizations and software product development companies as a developer, manager, process steward, trainer, and consultant. He helps people apply the agile values of communication, feedback, simplicity, courage, and respect to software development. Dale's combination of deep technical expertise and extensive organizational development experience makes him particularly effective in working with software teams. In 2007 Dale received the Ward Cunningham Gentle Voice of Reason Award, which the Agile Alliance created to recognize Dale’s unique contribution to the agile community. Dale's personal mission is to help people create joy, value, and meaning in their work. Learn more about Dale at dhemery.com.

 

Page 3: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

How to Survive the Coming Test Automation

Zombie Apocalypse

Dale Emeryhttp://dhemery.com

@dhemery

Page 4: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

Too gruesome to display

Page 5: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

The Trouble With Zombies

LegionRelentlessInfectious

Eat your brains

Usually not as prettyas the one

on the previous slide

Page 6: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

Test Automation ZombiesLegion

Relentless

Infectious

Eat your brains

Look just like ideas

Oddly attractive

Page 7: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

The Trouble with Test Automation Zombies

Sometimes the best you can do

They are easy

Lull you into doing them by habit

Focus on the short term

At the expense of the (not-so-)long term

Page 8: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

Pound FoolishZombies

Page 9: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

The Record and Playback Zombie

AppealLots of tests quickly, (almost) for free

DangerRecorded tests are

unexpressive, brittle, unmaintainable

Can’t record testsuntil the software is done

Page 10: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

The Automation is Easy Zombie

AppealInexpensive automators

DangerTests written by unskilled programmersare unexpressive, brittle, unmaintainable

Page 11: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

Surviving Pound Foolish Zombies

Remember:Test automation is real software development

Test automation takes timeBig cost is maintenance

Writing maintainable code is hard

Page 12: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

More Surviving Pound Foolish Zombies

Stop record and playback

Refactor recorded tests to make themexpressive, resilient, maintainable

Use real programmers...

with the experience to value maintainability...

and the programming skillto write maintainable test code

Page 13: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

DisplacementZombies

Page 14: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

The Automate Last ZombieAppeal

“Can’t automate tests before system is written”

DangerFeedback is delayed

The system is harder to test(not designed for testability)

Missed opportunityto use tests as guidance

Page 15: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

The Dedicated Test Automator Zombie

AppealKeep developers focused on developing

DangerTest automation lags development

Feedback is delayedTypically uses less-skilled programmers

Page 16: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

The Test Automation Group Zombie

AppealKeep entire development team

focused on developing

DangerSame as dedicated test automators

Organizational boundariesmake delays even worse

Page 17: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

Surviving Displacement Zombies

Create automatable examplesbefore development begins

Developers automate testsDone includes test automation

Page 18: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

MathZombies

Page 19: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

The Test Case Count Zombie

AppealConfidence that the system is tested

DangerFocuses on ease of automating each test

Instead of on the value of the tests

Page 20: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

The Code Coverage Zombie

AppealConfidence that the system is tested

DangerConfidence is unwarranted

(executed does not mean tested)

Page 21: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

Surviving Math Zombies

Focus on automating teststhat provide value

Use code coverage toolsonly to identify what is not tested

Page 22: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

StealthZombies

Page 23: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

The Dedicated First Responder Zombie

AppealRelieves the development team from having

to run, maintain, respond to tests

DangerDelays feedback

(through first responders)

Page 24: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

The Hide the Broken Feature Zombie

AppealReduce distraction by known failures

DangerReduces urgency to fix the features

(Disable tests for known-broken features)

Page 25: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

The Hide the Flaky Test Zombie

AppealReduce wasted effort diagnosing tests

DangerDiscards potentially useful test results

Reduces urgency to make tests more robust

(Disable or ignore flaky tests)

Page 26: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

The Automatic Rerun Zombie

AppealReduce wasted effort diagnosing tests

DangerDiscards potentially useful test results

Reduces motivation to make tests more robustIncreases suite run time

Page 27: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

Surviving Stealth ZombiesFix flaky tests now

Fix broken features now

Bypass flaky technology

Analyze every failure

Include automated testsin code promotion procedures

Reserve automatic reruns for exceptional cases

Page 28: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

HackerZombies

Page 29: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

The Fixed Wait Zombie

AppealEasy way to deal with variable response times

DangerFixed waits always increase

Test suite execution timebecomes maximally pessimistic

Thread.sleep(8000);

Page 30: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

Surviving The Fixed Wait Zombie

Bypass slow, variable technologies

Poll for the relevant condition

Arrange to be notifiedwhen the condition becomes satisfied

Create a utility for fixed waits

Instrument the utilityto gather information

Page 31: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

The Dependency Chain Zombie

AppealSpeed up test suites

DangerPrevents running tests independently

Skips subsequent testseven if they would provide value

Harder to understandwhat responsibility each test is testing

Page 32: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

Surviving The Dependency Chain Zombie

Make tests independent

Bypass slow technologiesfor setup and verification

Let each testestablish its own preconditions

Page 33: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

The Predefined Test Database Zombie

AppealSpeed up test suites

Often already exists for manual testing

DangerHides essential details from test code

Harder to understand what responsibilityeach test is testing

Page 34: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

Surviving The Predefined Test Database Zombie

Express every essential detaildirectly in the text of the test code

Let each testcreate its own data

Use the builder patternto create non-trivial test data

Page 35: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

The Chewy GUI Zombie

AppealAvailable and accessible to automators

Tests whole system

DangerMakes tests dependent on most volatile interface

Increases test execution timeHarder to diagnose failures

(Testing only through the GUI)

Page 36: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

SurvivingThe Chewy GUI Zombie

Bypass the GUI whenever it isnot essentialfor the test

Page 37: Test Automation Zombie Apocalypse · 2014-01-23 · Automation Zombie Apocalypse" Presented by: Dale Emery DHE Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073

How to Survive the Coming Test Automation

Zombie Apocalypse

Dale Emeryhttp://dhemery.com

@dhemery

Now You Know