W8 Session  4/15/2015  1:00  PM  

     

"Mobile Test Automation with

Big Data Analytics"  

Presented by:

Tarun Bhatia

Microsoft    

           

Brought  to  you  by:    

   

340  Corporate  Way,  Suite  300,  Orange  Park,  FL  32073  888-­‐268-­‐8770  ·∙  904-­‐278-­‐0524  ·∙  sqeinfo@sqe.com  ·∙  www.sqe.com

Tarun Bhatia

Microsoft Tarun Bhatia is a technical program manager in charge of driving the best breed of performance measurements and analysis for Microsoft Online Office Division. Tarun leads innovative strategies—analytics, performance, benchmarking, and compatibility—and guides the team to create an effective, reliable, and robust monitoring architecture. With more than seven years of software development experience in quality and service assurance, Tarun shows that taking initiative and thinking outside the box can deliver big results—both personally and for the company.  

4/8/15  

1  

Tarun Bhatia

Mobile Test Automation Using Big Data Analytics

Introduction quality  as·∙sur·∙ance:  

A  program  for  the  systema<c  monitoring  and  evalua<on  of  the  various  aspects  of  a  project,  service,  or  facility  to  ensure  that  standards  of  quality  are  being  met      Source:  hCp://www.merriam-­‐webster.com/dic<onary/quality%20assurance  

4/8/15  

2  

Staged Rollout with Active Monitoring

•  Crash Reports

• User Reviews

20% User Base

•  Crash Reports

• User Reviews

50% User Base

•  Crash Reports

• User Reviews

100% User Base

Manage

Analyze

Extract Value Value

What is Big Data ?

MB, GB, TB, PB

Records

Transactions

Tables, Files Volume

Batch

Near-time

Real-time

Streams

Velocity

Structured

Unstructured

Semi-Structured

All the Above

Variety

Source: Celent

The 4 Vs of Big Data

4/8/15  

3  

Data Everywhere

Trends in Tech Salary Reaffirm

Source: http://marketing.dice.com/pdf/Dice_TechSalarySurvey_2015.pdf

4/8/15  

4  

“

” If you think you can, or if you think you can’t, you are correct. – Henry Ford

Question

Your confidence level in current mobile automation architecture?

Cost of Finding Bugs

0

20

40

60

80

100

120

140

160

Req Design Code Unit Testing Integration Testing

System Testing

Test Prod

Cost

4/8/15  

5  

How it Starts!

Stage 1 •  Company needs mobile presence •  They hire Mobile Devs and Testers (usually manual)

Stage 2

•  App becomes too complex to cover all the permutations via manual testing

•  Company hires Automation Engineers (SDET) and are told to “automate everything”!

Stage 3 •  Full-on effort to catch-up and automate all features •  SDET burnout!

Creating a Plan

Successful Automation

Plan

Device Lab

Automation Framework

Prioritize Feature Test

Cases

Stress/ Performance/

Other Additional Testing

4/8/15  

6  

Creating a Device Lab

Creating a Device Lab (Using Big Data)

Total # of Devices

Devices with most # of

reported bugs

Your most Popular Devices

Time box and add bug to

your backlog

Buy/Loan/Rent device

and bring it in-house

4/8/15  

7  

Creating a Device Lab

30%

17%

13%

5%

4%

4%

3%

3%

3%

3%

2%

2% 2%

2% 2%

5% Apple

LG MS770

Samsung Galaxy SIII

Microsoft

Coolpad Quatro 4G

ZTE N9210

Samsung Galaxy Admire 4G

Droid RAZR

Samsung Galaxy Note II

LG Esteem

LG MS870

Samsung Admire

Samsung Epic 4G

Samsung Galaxy SII

Samsung Omnia II

Other

Total # of Devices > 1850!!

Pick an Automation Test Framework

4/8/15  

8  

Prioritize

KPI

Customer Usage Data

Finance (Revenue Stream)

Data

Marketing/ Social Data

User Usage Pattern

Home Screen, 40%

1 Detail Screen, 20%

2 Detail Screen, 15%

3 Detail Screen, 10%

All Other Values, 15%

4/8/15  

9  

Tests

Real User, Marketing and Finance Data

Stress Server Vs. UI Data

New Features

Performance

System Under Test

Production data

Test Results

Run Tests

Quality Assessment

Stress Testing � Find  Resource  Leaks  � Find  App’s  Capacity  and  Capabili:es  

� Find  Memory  and  Ba>ery  Consump:on  Trends  

4/8/15  

10  

Server Vs. UI Testing

Server

Client Test Framework

•  Verify  data  is  in-­‐sync  during  tes:ng  

•  Ensure  no  data  loss  during  test  progress  

•  Detect  UI  TTL  (Time  to  Load)  on  devices  under  various  condi:ons  

Performance Testing (Analyze and Record KPIs)

4/8/15  

11  

Effective Testing

Write Once, Test Anywhere

Active Monitoring

Test Re-Use Performance

Availability

Conclusion