W8 Session 4/15/2015 1:00 PM
"Mobile Test Automation with
Big Data Analytics"
Presented by:
Tarun Bhatia
Microsoft
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073 888-‐268-‐8770 ·∙ 904-‐278-‐0524 ·∙ [email protected] ·∙ www.sqe.com
Tarun Bhatia
Microsoft Tarun Bhatia is a technical program manager in charge of driving the best breed of performance measurements and analysis for Microsoft Online Office Division. Tarun leads innovative strategies—analytics, performance, benchmarking, and compatibility—and guides the team to create an effective, reliable, and robust monitoring architecture. With more than seven years of software development experience in quality and service assurance, Tarun shows that taking initiative and thinking outside the box can deliver big results—both personally and for the company.
4/8/15
1
Tarun Bhatia
Mobile Test Automation Using Big Data Analytics
Introduction quality as·∙sur·∙ance:
A program for the systema<c monitoring and evalua<on of the various aspects of a project, service, or facility to ensure that standards of quality are being met Source: hCp://www.merriam-‐webster.com/dic<onary/quality%20assurance
4/8/15
2
Staged Rollout with Active Monitoring
• Crash Reports
• User Reviews
20% User Base
• Crash Reports
• User Reviews
50% User Base
• Crash Reports
• User Reviews
100% User Base
Manage
Analyze
Extract Value Value
What is Big Data ?
MB, GB, TB, PB
Records
Transactions
Tables, Files Volume
Batch
Near-time
Real-time
Streams
Velocity
Structured
Unstructured
Semi-Structured
All the Above
Variety
Source: Celent
The 4 Vs of Big Data
4/8/15
3
Data Everywhere
Trends in Tech Salary Reaffirm
Source: http://marketing.dice.com/pdf/Dice_TechSalarySurvey_2015.pdf
4/8/15
4
“
” If you think you can, or if you think you can’t, you are correct. – Henry Ford
Question
Your confidence level in current mobile automation architecture?
Cost of Finding Bugs
0
20
40
60
80
100
120
140
160
Req Design Code Unit Testing Integration Testing
System Testing
Test Prod
Cost
4/8/15
5
How it Starts!
Stage 1 • Company needs mobile presence • They hire Mobile Devs and Testers (usually manual)
Stage 2
• App becomes too complex to cover all the permutations via manual testing
• Company hires Automation Engineers (SDET) and are told to “automate everything”!
Stage 3 • Full-on effort to catch-up and automate all features • SDET burnout!
Creating a Plan
Successful Automation
Plan
Device Lab
Automation Framework
Prioritize Feature Test
Cases
Stress/ Performance/
Other Additional Testing
4/8/15
6
Creating a Device Lab
Creating a Device Lab (Using Big Data)
Total # of Devices
Devices with most # of
reported bugs
Your most Popular Devices
Time box and add bug to
your backlog
Buy/Loan/Rent device
and bring it in-house
4/8/15
7
Creating a Device Lab
30%
17%
13%
5%
4%
4%
3%
3%
3%
3%
2%
2% 2%
2% 2%
5% Apple
LG MS770
Samsung Galaxy SIII
Microsoft
Coolpad Quatro 4G
ZTE N9210
Samsung Galaxy Admire 4G
Droid RAZR
Samsung Galaxy Note II
LG Esteem
LG MS870
Samsung Admire
Samsung Epic 4G
Samsung Galaxy SII
Samsung Omnia II
Other
Total # of Devices > 1850!!
Pick an Automation Test Framework
4/8/15
8
Prioritize
KPI
Customer Usage Data
Finance (Revenue Stream)
Data
Marketing/ Social Data
User Usage Pattern
Home Screen, 40%
1 Detail Screen, 20%
2 Detail Screen, 15%
3 Detail Screen, 10%
All Other Values, 15%
4/8/15
9
Tests
Real User, Marketing and Finance Data
Stress Server Vs. UI Data
New Features
Performance
System Under Test
Production data
Test Results
Run Tests
Quality Assessment
Stress Testing � Find Resource Leaks � Find App’s Capacity and Capabili:es
� Find Memory and Ba>ery Consump:on Trends
4/8/15
10
Server Vs. UI Testing
Server
Client Test Framework
• Verify data is in-‐sync during tes:ng
• Ensure no data loss during test progress
• Detect UI TTL (Time to Load) on devices under various condi:ons
Performance Testing (Analyze and Record KPIs)
Top Related