5/13/16  

1  

DRAFT

Lessons learned on end to end testing from Matt Barbour Sr. Director Engineering Comcast Silicon Valley matt_barbour@cable.comcast.com STAREAST 2016 – May 4th, 2016

Confidential and Proprietary

What can we LEARN from a ZOMBIE MOVIE?

•  Characters in zombieland live by a set of rules in order to survive the chaos

•  Many of these apply in non-zombie situations too! Specifically:

End-to-end testing in real world chaos.

FOL 2014 Process Improvement Project – IDD Process 1 Confidential and Proprietary

5/13/16  

2  

Confidential and Proprietary

End To End Testing as compared to distributed computing

•  Distributed systems expect failure because the real-world is chaotic

•  Strive to be fault tolerant by optimizing for desired characteristics of the end application according to the CAP Theorum

2

•  End to end automated tests should have similar expectations about the environment they are operating in

Automated Tests

Performant

Meaningful

Repeatable

Accurate

PARM THEORY Tests must balance being:

Performant, Accurate, Repeatable, and Meaningful

Confidential and Proprietary

KNOW YOUR WAY OUT��

BE FORWARD LOOKING AND �OWN THE PROBLEM!!

•  Aim for excellence – don’t tolerate mediocrity •  “that’s the way we have always

done it”™ •  Low pass rate •  Tests cant finish quickly enough to

give feedback on a checkin

• Don’t know what a test is doing or why it exists

•  Inconsistent results • Aim for being in the center of the

PARM matrix

3

Automated Tests

P

M

R

A

5/13/16  

3  

Confidential and Proprietary

BUDDY SYSTEM��LEARN AND APPLY DEVELOPMENT BEST PRACTICES

Developing E2E tests should be no different than developing an application

GTAC 2015 4

•  Design Patterns •  Code Review

•  Unit Tests •  Static Code Analysis

Automated Tests

P

M

R

A

Confidential and Proprietary

RULE #31CHECK THE BACK SEAT

MAKE SURE YOURE NOT BRINGING ALONG UNWANTED DEPENDENCIES

• Use a dependency management tool! • Explicitly declare all dependencies • Beware of depending on uber jars in a project…

5

Automated Tests

P

M

R

A

5/13/16  

4  

Confidential and Proprietary

A.B.T. è ALWAYS BE TESTING

Trim the fat

Look for opportunities to obtain useful test metrics

outside of a traditional test

Even idle devices can tell you something!

6

Automated Tests

P

M

R

A

Confidential and Proprietary

MAKE SURE YOUR TEST TOOLS SCALE

•  Use a test framework which allow you to scale in both directions

(more devices in parallel) (more tests in parallel)

•  Use open source tools where possible

•  WebDriver isnt just for web and mobile!!

7

Automated Tests

P

M

R

A

5/13/16  

5  

Confidential and Proprietary

TRYING TO DO TOO MUCH IN A TEST IS A RECIPE FOR FAILURE!

•  Tests should focus on one thing, and one thing only.

•  BDD is helpful here • BDD scenarios should favor

DAMP over DRY

• DAMP = Descriptive and Meaningful Phrases

• DRY = Don’t repeat yourself

8

Automated Tests

P

M

R

A

Confidential and Proprietary

Choose your test methodologies wisely

•  You can lose the war against unreliable, slow tests before you even start by your choice of test methodology

•  Known offenders –  Image Comparison –  Xpath –  Record/Playback –  Not controlling your data –  The list goes on and on

9

RULE #7 TRAVEL LIGHT

Automated Tests

P

M

R

A

5/13/16  

6  

Confidential and Proprietary

RULE #5 �NO ATTACHMENTS

CONTROL ALL THE DATA IN

YOUR SYSTEM�

•  If you have a data dependent test where you are not controlling the data you probably have an unstable test, or worse…

•  Mocking production data is time consuming… so is triaging false failures

•  Tests must be able to start from a known state

10

Automated Tests

P

M

R

A

Confidential and Proprietary

WORK AROUND OR ELIMINATE ARCHITECTURAL CONGESTION

11

• Shared resources are the enemy of test parallelism.

• Your system’s architecture can get in the way here… don’t assume the application is infallible. Either work with it or modify it

• Cloud deployments can mitigate architectural problems

• VMs are cheap and easy to spin up and tear down

Automated Tests

P

M

R

A

5/13/16  

7  

Confidential and Proprietary

Avoid Strangers This one is just plain wrong •  The only way we can survive the slow unreliable test zombie apocalypse is by sharing

tools and techniques.

12

All the test tools used by Comcast Silicon Valley are open-sourced: https://github.com/Comcast

•  Cucumber testing in parallel •  Zucchini

•  Device Management •  DAWG

•  Selenium Driver Management •  Magic Wand

•  Theres a great deal of knowledge in the community out there, use it!!

DRAFT

THANK YOU!!