T16 Mobile Testing

5/2/2013 1:30:00 PM

Introducing Mobile Testing to Your

Organization

Presented by:

Eric Montgomery

Progressive Insurance

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Eric Montgomery

A system test engineer for Progressive Insurance, Eric Montgomery joined Progressive as a C/C++/C# developer, programming their quoting, rating, and renewal logic. On a project to replace Progressive's mainframe servicing system with a .NET version, Eric transitioned to testing, first developing headless test harnesses for business logic functions and then becoming test lead of one of the largest test events ever run at Progressive. Since 2011, Eric has been the test lead for Progressive’s mobile transformation program, which created several native apps, including the award-winning Progressive flagship app and the “mobilization†of many of Progressive's web sites.

4/19/2013

1

Introducing Mobile Testing

to Your Organization

Eric W Montgomery

System Test Engineer | Progressive Insurance

Who is Progressive?

» Mayfield Village, OH

» Offering insurance since 1937

» Web awards (Webby)

» Mobile awards (Keynote Brand Innovation)

4/19/2013

2

Who Am I?

» Cleveland native

» Miami University, Oxford, OH

» Progressive since 2003

» Mobile Test Lead, 2011-2013

How it all Began

» 2 developers (1 iOS/iPhone, 1 Android)

» 2 devices (1 iPhone, 1 Android)

» 1 tester

» Mobile Program

4/19/2013

3

» People

» Devices, tools

» Testing techniques

» Stories, experiences

Agenda

Overcoming the Fear of Mobile

» Common issue

» Practice testing mobile with a device

» Discuss terminology

– App types: native, web, hybrid

– Device features

– Operating systems: Android, iOS

– so many more

4/19/2013

4

People

» Key component

» Be willing to adjust quickly

» Constant

learning

Mobile Testers’ Forum

» Meets bi-weekly

» Variety of topics

– Project work

– Technology trends

– Testing techniques

» Network

» Answer questions without

asking

» Presentations

» Learn/teach technologies

4/19/2013

5

Devices

» 2 devices (1 iPhone, 1 Android), supplement with

emulators

» How we learned what phone our CEO had

» Emulators are insufficient for System Testing

– Not exactly the same

– Not handling a device

– Different hardware: camera,

location services

– Android: manufacturers add code

» Still good for unit, smoke testing

Device Selection

MOTXT912B, 8.98%

SPH-L710, 5.50%

GT-I9300, 4.99%

SGH-I747, 4.39%

SPH-D710, 4.06%

SGH-T999, 3.80%

SGH-989, 3.26%

XT875, 2.27%

SGN, 2.18%

PC36100, 2.13%

SGH-I727, 1.89%

HDH, 1.82%

ADR6400L, 1.57%

MDX2, 1.51%

X325C, 1.49%

SCH-I500, 1.48%

MDX2, 1.47%PJ83100, 1.47%

ADR6350, 1.44%

SCH-I535, 1.41%

SGH-I717, 1.41%

X515, 1.37% SCH-I510, 1.35% XT894, 1.27%

XT907, 1.26% SGH-I997, 1.24%

SGH-I777, 1.01%

PH39100, 0.94%

ADR6425LVW, 0.94%

SCH-I405, 0.92%

GT-I9000, 0.90%

XT862, 0.87%

VILLE, 0.87%

Z710, 0.85%

GLAC, 0.84%

SGH-T769, 0.83%

SPH-D700, 0.80%VS840, 0.80%

MB855, 0.79%

SGH-T679, 0.74%

MB886, 0.74%MS770, 0.72% N860, 0.70%MB865, 0.57%

P999, 0.55%SPH-930, 0.55%MD2, 0.54%

GNEX, 0.51%

VS920, 0.50%

ADR6300, 0.48%

C771, 0.47%

P9070, 0.47%AMZ, 0.47%ELC, 0.46%SCH-I200, 0.45%MB860, 0.41%VS910, 0.41%

SCH-R760, 0.39%E739BK, 0.38%MYTSLD, 0.37%SGH-I897, 0.37%PH44100, 0.36%C800, 0.35%ADR6330, 0.34%G2TCH, 0.33%MS910, 0.32%

SCH-R720, 0.30%ADR6410LVW, 0.29%

SCH-R930, 0.28%P925, 0.27%

SPH-D600, 0.26%M886, 0.25%LS840, 0.25%

SGH-T959, 0.24%SCH-R920, 0.23%SCH-T839, 0.23%SGH-I577, 0.23%

Other, 8.70%

4/19/2013

6

Device Selection

» Consider project goals

» Ask questions

» Eliminate options!

» Equivalence classes

» Crowdsourcing

A47%

B25%

C10%

D10%

Other8%

Device Selection

4/19/2013

7

Device Selection: Using Data

» Data

» Market trends

» Making decisions jointly with customer

Devices: Owning

» Warehousing/library

» Maintaining

– Cleaning

– Operating systems

» Cords

4/19/2013

8

Testing Tools, Tips & Techniques

Old Techniques Made New Again

» Error handling

– Edits

– Nonsense messages

– Error logging

» Dates

– Valid ranges

– Formats

Testing Tools, Tips & Techniques

Automation

» How it works

» Native

– Tree structure

– Unit/Integration level tests

» Web

4/19/2013

9

Testing Tools, Tips & Techniques

New Techniques

» Brain

– Perspective

– Pair testing

» Devices

» Chrome

– Developer panel

– Remote debugging

» View source bookmarklet for iOS

» Screen shots

– Recording apps

– Camera, video

» Bug reports/exploratory sessions

Testing Tools, Tips & Techniques

New Techniques

» Settings

– User settings/airplane mode

– Moving between networks

– Cookies, JavaScript

» Navigation

– Service calls

– Links

» Keyboards

– Numeric

– Email

– iOS vs. Android

» Data Usage

4/19/2013

10

Testing Tools, Tips & Techniques

New Techniques

» Mobile

– Rotations

– Location services, camera

– Moving around

– Interruptions

» Look and Feel/UX

– Consistency

– Font, colors, headings, images

– Spelling, grammar

– Alignment, margins

» Change browsers, devices

» Deployment

Summary

4/19/2013

11

Eric W Montgomery

emontgo1@progressive.com