Building a world class mobile testing strategy
Transcript of Building a world class mobile testing strategy
Welcome.@propelics
Building a World-Class Mobile Testing StrategyMay 7th, 2015
We will start at 1:02pm EDT
Propelics creates mobile strategies and world-class Apps for the Enterprise.
Propelics: Background
• 15+ year track record• Founded in 2011• Fortune 500 focus• Global delivery experience
Trusted by some of the largest global brands:
Pre Built Mobile Apps
Mobile Strategy Research & Advisory
Custom AppS & Managed Services
• 100% focused on digital and mobile for the Enterprise
• San Jose, Boston, Pittsburgh, Guadalajara
2
MOBILE RESEARCH COUNCIL
We will give you the edge.
Our Objective:Driving innovation and business value with mobile software and solutions.
Our executive guidance helps members to quickly and effectively deliver high-impact mobile experiences to employees, partners and customers.
ü MRC has no sales angle or vendor bias. ü Members benefit from insider access to real-
world mobile experience.ü Members include: American
Airlines, AmerisourceBergen, Bank ofMontreal, Carnival Corporation, Family Dollar, UPS, and many more…
From Market Trends to Action at the Speed of Mobile
Research. Advisory. Community.
Shahab Choudhry: Partner - Propelics
Shahab is a visionary thought leader who has been at the forefront of trying to make sense of the various technology shifts in the past two decades.
He has advised several startups, early stage companies and Fortune 500 companies in making the most of these technology shifts. He continues to be a “Trusted Advisor” to some of the leading brands around the world. Shahab is frequent speaker at conferences related to mobile.
4
Q:What are the key considerations for my MobileTesting Strategy for the Enterprise?
5
The mobile reality…
6
Methods of customer interaction will change
Source: Morgan Stanley, 20117
8
Testing is not a new concept…
…Mobile Testing is another storyWorld Quality Report 2012 - 2013
• What is the readiness of the team to test new features in rapid sprints?
• How ready is the organization to test every app against new OS releases?
• What aspects of our testing plan can be automated?
• What type of tools are available to help with the various types of testing?
• How can organizations test for user centric design?
• How can we test every app in a BYOD environment?
• How do you plan on integrating Mobile Testing into your Project Plan?
• Are the testing methodologies consistent and repeatable across OSes and devices?
• Will our app drain a user’s battery life or use up all its memory?
• How can we procure and maintain the devices needed for testing?
• How do we test for mobile specific considerations such as phone calls, switching apps etc.
• How can we test gestures?
• …and many more10
Testing Math101
11
12
Mobile Testing: The Complexity Equation
# of supported devices # of supported OSesOS updates per year
Number of Apps # of test cases per appApp release frequency
The “I can barely keep up” syndrome
X
Disruptive Change
Ready?
Q:What are companies doing to address this need to focus on quality?
14
Enterprise Mobility Testing
Strategy
Business Drivers
mobile testing philosophy
Supported mobile
Devices / OS
Mobile App Portfolio
Reliability
App Adoption
Continuous development
Security
Quality ImprovementEliminate manual work
Mobile specific testing tools
Mobile testing skill sets
Mobile Testing: The strategy
6 Considerations for Enterprise Mobile Testing Strategy
1. Be mindful of the unforgiving attitude of the mobile user
The Unforgiving UserQuick to download - Quick to delete
74% of downloaded apps are opened only once40% of apps are deleted after first use
Average expected wait time for a app to launch is 2 secTop reason for deleting apps – app crashed
Enterprise Expectation = Consumer Experience
Gift of Consumerization
2. Know the types of testing
20
Installation Testing
• New App Install• Uninstall and Reinstall
• Upgrade testing
Interoperability (IOP)
• Voice / SMS interrupts• Notifications
• Battery /Cable Removal
Functional• Validation of Functionality
• Smoke / Regressions Testing
• Offline access testing• Negative Testing
Usability Testing
• User Experience• Competitive Analysis
• Expert Review
Security Testing• OWASP Vulnerabilities
• Dynamic Testing• Static Code Analysis
• Data Encryption
Non Functional• Network Strength / Outage /
Recovery• Different NW Types• Peripheral Testing
Language Testing
• Validation for Locales• Images and Text
• Currencies, time zones etc.• Context
Mobile Testing: the big blocks
Memory Leak• Memory Usage• Memory Leaks
• Garbage Collection
Performance Testing
• CPU Usage testing• Network Usage
• Page Render time or activity Render time
3. Understand the impact of the “New Math”
22
Mobile Testing: The Complexity Equation
# of supported devices # of supported OSesOS updates per year
Number of Apps # of test cases per appApp release frequency
The “I can barely keep up” syndrome
X
# of Supported Devices
23
Source: Open Signal
Diversity of Supported OSes
24
iOS Android
Devices and OSes
25
• B2C• Select based on analytics based on
% of sessions from each device type• Cutoff at devices with less than 1%
of sessions• Cutoff at 75% - 80%
• B2B• ALL Corporate Owned• BYOD – Approved devices
•
• Maintain a Pool of Devices• Setup a Device Cloud• Simulator as a last resort
# of Apps
26
4. Establish a baseline of where you stand
Mobility Testing Readiness: Details
28
Sta
ge
4
Sta
ge
3
Sta
ge
2
Sta
ge
1
Sta
ge
5
Mobility Testing Strategy
SDLC
Mobile Testing Tools
Types of TestingFunctional
Non-FunctionalMemory Leak
Installation TestingPerformance Testing
Security TestingInteroperability Testing
Usability TestingLanguage Testing
Mobile Testing Skills
Mobile Testing Organization
Security & compliance
Testing Maturity Ranking
Testing Readiness Report Card
5. Build a comprehensive plan of attack
App Portfolio
Testing capabilities
Testing Roadmap
implementation
Mobile Testing Roadmap
Business Drivers
Philosophy and Quality Posture
Types and Number of Apps
Complexity of Apps
Best Practices
Skills, Roles & Responsibilities
Testing Tools
Testing Types
Alignment with SDLC
Ongoing Testing
EnterpriseMobility Testing
Strategy
Mobile Testing: Philosophy
Mobility Testing Roadmap
31
• Tactical Roadmap to Improve Maturity
• Each Tactic Ranked On: 1. Value and Alignment to Goals2. Organizational Readiness3. Ease of Implementation
• All Tactics Prioritized & Phased• Prioritize Apps for mobile testing
based on• Criticality• Complexity• Effort
• Select Apps for initial phase of testing
Mobile Testing Roadmap
32
Required
Short Term
Mid/Long Term
Future
Quick Wins
6. Automated functional testing is essential
Mobile Testing Automaton Roadmap
34
Mobile Testing Automaton Roadmap
35
Business Criticality (H,M,L)Core/frequently used functionality, Security/data breach/privacy/legal, Dependence on 3rd Party data or technology, User experience impact
Value of Automation (H,M,L)Increases quality, Reduces resource load, High volume/Repetitive/Data driven, Feature life-span, Smoke test/Build verification
Automation Readiness (H,M,L)Test Script Complexity, Stable Functionality, Measurable results, Recordable actions, Explicit list of observable steps, Test Script Maintainability
Ease of Automation (H,M,L)Data or logic driven, Object level user interface, Known expected results around user experience, Clear acceptance criteria per test case, Minimal performance issues interacting with external end points or services
1. Be mindful of the unforgiving attitude of the mobile app user
2. Know the types of testing
3. Understand the impact of the “New Math”
4. Establish a baseline of where you stand
5. Build a comprehensive plan of attack
6. Automated functional testing is essential
6 Considerations for Enterprise Mobile Testing
Propelics Mobile Testing Services
• Mobile Testing Readiness Kickstart• Mobile Testing Roadmap Implementation• Mobile Test Automation Kickstart• Mobile Testing as a Managed Service
Action Plan and testing strategy recommendations
Mobile Testing maturity
assessment
Step I: Direction Setting
Step IV: Next Steps and
Recommendations
Mobile Testing Readiness Kickstart: The methodology
Step III: Prioritization & Phasing
Mobile Test
strategy scope
Mobile Testing
Roadmap
1-2 WeeksSize and Complexity of App Portfolio
Types of Testing
Testing Scope
Finalized
Step II: Current State Assessment
& Recommendations
Recommended Apps for initial
Phasing
Business Drivers
App Portfolio
App portfolio Phasing
Mobile Test Automation kickstart
Ongoing Testing
Testing Setup &
Initiation
Step I: Direction Setting
Step IV: Ongoing Testing
Mobile Test Automation: The methodology
Step III: Setup,
Initiation & Initial Testing
Step II: Planning &
Phasing
Number & complexity
of Apps
Device Types
Test Automation
Scope
Supported OS
Mobile Test
Automation Roadmap
Release frequency
User stories
Testing Tools
5-10 Days
Test Plan
1- 2 Months
Ongoing
Mobile testing as a managed service
Webinar Attendee Special Offer
40
Special OfferFree 1 Hour Mobile
Advisory [email protected]
888-405-2820
TestingStrategy for
MobileKickstart
App Scoping & Prototype
EnterpriseMobile Roadmap
IT Strategy for Mobile
Mobile UI/UXDesign
Mobile COE
41
Thank you for attending!