Post on 13-Apr-2017
T21 Test Automation 5/5/16 15:00
Don't Be Another Statistic! Develop a Long-term Test Automation Strategy
Presented by:
Lee Barnes
Utopia
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073 888-‐-‐-‐268-‐-‐-‐8770 ·∙·∙ 904-‐-‐-‐278-‐-‐-‐0524 -‐ info@techwell.com -‐ http://www.stareast.techwell.com/
Lee Barnes Utopia
Founder and CTO of Utopia Solutions Lee Barnes is responsible for the firm's delivery of software quality solutions which include process improvement, performance management, test automation, and mobile quality. Lee has more than twenty years of experience in the software quality assurance and testing field. He has successfully implemented test automation and performance testing solutions in hundreds of environments across a wide array of industries. A recognized leader in his field, Lee speaks regularly on related topics.
© Utopia Solutions
Don’t Be Another Statistic! Develop a Strategy for Long-term Test Automation Success
Lee Barnes, CTO
Utopia Solutions
© Utopia Solutions
Agenda
• What is the end goal of test automation?
• Why do we need a strategy?
• How do we know if we’re ready to succeed?
• What should a strategy look like?
Questions to Answer Today…
© Utopia Solutions 5
Test Automation Success Criteria
• Issue detection and recovery
• Accurate verification
• Unattended execution Reliable
• Minimum sensitivity to application and test case changes
• Test case definition separate from automation code
Maintainable
• Test coverage expanded efficiently
• Automated test cases created by non-automation resources
Scalable
© Utopia Solutions 7
Why Test Automation Fails
Organizational
• Lack of support
• Unrealistic expectations
• Disjointed org structure
• Missing skill sets
Process
• Immature testing process
• Segregated processes
• Lack of test environment / data control
Technical
• Poor tool support for AUT
• Under or over design automation approach
© Utopia Solutions 10
Test Automation Readiness Assessment
Org
• Structure
• Support
• Goals and Expectations
• Resources and Skills
Process
• Test Maturity
• Test Assets
• Process Integration
• Test Environment / Data
Tech • AUT Technology
• AUT / Tool Integration
Assessment Areas
© Utopia Solutions
Organizational Questions
• Expectations
– What are our goals for test automation?
– Do we understand the effort and cost to implement and maintain test
automation
– Do we understand the required skill sets for successful test automation?
• Structure
– How is the testing function organized?
– Where are the automation resources relative to everyone else?
• Resources
– Are our automation engineers allowed to focus on automation?
– Do our automation engineers have the appropriate skills?
© Utopia Solutions 12
Common Assessment Findings – Organization
Distributed Testing /
Automation Efforts
App
App
App
App • Uneven automation
resource demand
• Inconsistent approach
• Duplicated efforts
• Restricted skill growth and knowledge sharing
Leads to…
© Utopia Solutions
Process Questions
• Test Process
– What are the testing activities that can benefit from automation?
– Do we have testing assets that can promote automation?
• Development Process
– What type of development process is used for the AUT?
– Are the development and testing processes integrated?
• Test Environment
– What environments do we have available for test automation?
– Do we have control over the test data in these environments?
© Utopia Solutions 14
Common Assessment Findings – Process
Analyze Design Build
Test Deploy
No ability to identify impact of app changes on automated test suite
Unexpected / Unpredictable Maintenance
© Utopia Solutions 15
Process
Analyze Design Build Test Deploy
Capture potential impacts as early as possible to enable proactive maintenance
Capture App Changes
Analyze Impact
Maintain Execute
Automation Activities
Application SDLC
© Utopia Solutions 16
Common Assessment Findings - Environment
Shared Test Environment
Uncontrolled Test Data
Integration
Functional
Performance
Integration Test
• Schedule conflicts
• Unreliable execution
• Unnecessary “hardening” of the test suite
Leads to… Automation
© Utopia Solutions
Technical Questions
• Applications Under Test
– Do we know which AUTs should be targeted for test automation?
– Do we know the technologies of the AUTs in scope for test automation?
– Where are the AUTs in their application lifecycle?
• Test Automation Tool
– Do we have access to an automation tool (or tools)?
– Does our automation tool appropriately support our AUTs?
– If we don’t have a tool, do we understand how to select one?
© Utopia Solutions 19
Mobile Test Strategy Topics
Test Automation
Strategy
Process & Organization
Environment
Technical Resources
Scope & Roadmap
© Utopia Solutions
Ideal Scope Attributes
Testing Support Activity
• Environment / data setup
• Tedious output verification
Test Coverage
• Focused on high frequency test testing activity
• NOT focused on automating manual test scripts verbatim
• Test cases distilled into levels
AUTs
• Supported completely by test tool
• Multiple AUTs from same development framework / technology
• Frequent release cycles
• Large regression effort
© Utopia Solutions
Ideal Organizational Attributes
Structure
• Centralized test automation function
• Automation roles are defined
Goals and Expectations
• Automation is an investment
• 100% automation is not likely
• Automation requires specialized skills
• Automation will free up resources to work on high-value tasks
Support
• Executive management has visibility to automation
• Automation has a dedicated budget
• Automation is represented appropriately on project plans
Resources and Skills
• Resources are dedicated to automation
• Resources are skilled in automation as a process AND technology
© Utopia Solutions
Ideal Process and Environment Attributes
Test Process
• Test objectives are clear and measurable
• Test cases are defined and documented
• Expected results are predictable
• Basic metrics for testing and automation
SDLC Integration
• AUT in a frequent release state with visibility to future roadmap
• Visibility to system changes as early as possible in release cycle
Test Environment
• Test environment dedicated to automation
• Test data controlled
© Utopia Solutions 23
Automation Strategy – Roadmap
• Begin with required organizational and process changes
• Overlap with base framework and minimal coverage
• Enhance and add test coverage over small iterations
Create Implementation Timeline
Phase n Phase 1
Continuous Process & Organizational Improvements
Base Framework
Initial Coverage
AUT 1
Framework Enhancements
Expand Coverage
Initial Coverage
Phase 2
AUT 2
Expand Coverage
Expand Coverage
Initial Coverage
Phase 3
AUT 3
© Utopia Solutions 25
Test Results
Test Automation Approach
Test Automation Framework
Automated Test Cases
Test Definition
Test Execution
Non-technical / Test Analysts & Subject Matter Experts
Technical / Automation Engineers
© Utopia Solutions 26
Regression Test Automation and Agile Cycles
App Dev
Cycle (Sprints)
User Stories
Application
Automation Requirements
Automation Dev Cycle
Automated Test Suite
Development and automation cycles NOT necessarily in
sync
© Utopia Solutions 27
Key Summary Points
• Test automation is NOT a tool!
• Understand your readiness to succeed from these perspectives…
• Organization
• Process & Environment
• Technical
• Develop a strategy to address readiness gaps and implement in small phases
© Utopia Solutions 28
Questions…
…. and Answers!
Direct future questions to: Lee Barnes Founder and CTO Utopia Solutions, Inc. Email: lee.barnes@utopiasolutions.com Twitter: twitter.com/USI_LeeBarnes LinkedIn: linkedin.com/in/leebarnes Blog: www.RaiseYourMQ.com