Post on 16-Apr-2017
Test Automation PyramidT. Alexander Lystad, Chief QA Architect, VESD
Test AutomationWhy?
Allows you to test more with less effortAllows you to deploy/release more quickly and more frequentlyHigher quality software
How?As a natural part of the development processA story is not done until appropriate automated tests are passing
Test Automation PyramidShows different technical levels on which
you can have automated testsA modelGoing up
Scope increasesExecution time increasesEffort increases
Guideline for relative distribution of your testing effort
Unit TestsTesting smallest possible parts of the system
in isolationUse of test doubles to replace dependencies
Dummies, stubs, spies, fakes, mocksSmallest scope
Easier to locate and understand errorsBest performance
Execute early and oftenWon’t catch all defects
Component Integration Tests●Testing that multiple components work together●Test doubles are not used for the integration(s)
you want to test●Varying scope●Good performance●Won’t catch all defects
API TestsTesting that your APIs behave as expected
RESTSOAP...
Big scopeInvolves many components
Decent performanceWon’t catch all defects
UI TestsEnd-to-end testingMaximum scope
Can catch many problemsWorst performance
Don't use UI for setup and teardownParallelize
Can be brittle, flaky and a lot of work to maintainTest critical user workflows
Won’t catch all defects
Exploratory TestingManual testingNot regression testingExperience + creativityUsed to learn about the system, discover defects and
to improve automated testingCan be based on a mission/test charter or persona
Test for accurate and helpful error messages when registering an expense incorrectly
Test for unexpected results when configuring company details in two different tabs
Impatient manager new to the system
Where is…?●Acceptance testing●BDD●Smoke testing●Security testing●Performance testing●System testing●System integration testing●…
What should I aim for?Depends, discuss in your teamAs an example, Google suggests
70% unit tests20% integration tests10% end-to-end tests
Tools
Selenium WebDriver (VAFT)JAutomate, Sikuli
SoapUI, RestSharp
JIRA Capture
Unit Test Frameworks
BDD Frameworks
Hour-glassIce cream cone
Anti-patterns
Other Test Automation Pyramids
Testing Week TaskConfluence: Test Automation Pyramid - Testing Week 2015How does your team's test automation effort fit into the Test Automation Pyramid,
how does your pyramid look today, and how can it improve going forward?
Example workflowIdea → Production
Example workflowDesign process
Acceptance criteria drafted
Example workflowRefinement process
3 amigos: PO/BA + dev + QA/tester
Acceptance criteria finalizedAcceptance tests finalized
Risks and mitigationWhat should be tested, on
which level
Example workflowImplementation
Developers have a clear picture of what is expected
Decided automated tests are implemented as part of story development
Example workflow: Test executionBalance between quick feedback
and discovering problems earlyDuring implementationAfter implementationAt the beginning of your delivery
pipeline (CI server)After deployments
Internal TestUser AcceptanceProduction
Exploratory testing
SummaryThe test automation pyramid is a useful thinking tool and discussion referenceMake conscious decisions about what to test on which level, and also where you
run which testsTry to distribute your tests optimally
High qualityHigh velocityMedium effort
Discuss with your teamDo the Testing Week task
Questions or comments?