Why a

Mobile Test Strategy

is just Test Strategy

27 | 06 | 2015

HCMC Software Testing Club 2015

2

3

4

Are we scared of Mobile?

Challenges:

• Creating separate teams

• Managing multiple devices

• Webservers don’t run low on battery get taken home

• Browser, OS, Device, Platform combinations are more than we have ever had to manage

But it still boils down our mission – Validate products, report risk

5

Our role is greater than a mobile test strategy

Building a mobile approach is a key point in your overall strategy….

…But we must do more!

Our mission, to report risk, must include a focus on where risk is introduced

We must have a seat at the table:

- What is the mobile strategy of the organization?

- What will the next technology be?

- Where will our content and products be accessed?

6

What must we understand

7

What must we understand

8

What must we understand

• For public and consumer applications,

mobile does not just mean an “app”

• For internal, private applications, our

users or still customers and usability

can’t be taken for granted

9

There are two primary drivers to Mobile Quality

10

There are two primary drivers to Mobile Quality

11

But our apps aren’t really meant for mobile

12

What are we really testing?

• Native apps – Yes

• Hybrid apps – Yes

• Mobile web – Yes

• Responsive design – Yes

• What about:

- TV’s?

- Cars?

- Refrigerators!!!

13

What are we really testing?

14

What are we really testing?

15

What are we really testing?

16

What are we really testing?

17

But, I just want to make sure it renders

We have a lot more to test than functionality and

usability

18

But, we don’t have the devices

19

What does this mean for the tester?

• Defect escape risk is exponentially higher with

changes in platform and technology

• Our challenges are increasing and so are our

options

• Working with product teams on strategy can manage the

risk

• Knowing that you’re mobile even when you’re not

• Communicate cost of defects and customer loss

• Building for future needs will strengthen non-mobile

aspects of your strategy

20

Mobile is not only about mobile

• Building for future mobile needs will strengthen

non-mobile aspects of your strategy:

- Usability is not a uniquely mobile trait

- Customer adoption and community acceptance

isn’t either

- Size, performance, battery consumption, and

connectivity options all affect functionality

- Biometrics

- Augmented reality

21

How my strategy should change for mobile

• Clearly, new techniques are needed

• The functional tester is no longer only able to

focus on functionality!

• But, we can still break this down into similar

buckets:1. Estimation

2. Requirements

3. Environment

4. Test Phases

5. Risk Analysis

22

Strategy – Estimating for Mobile

The complexity of combinations is a significant drain on test estimation

We must focus on production statistics (if Available) or expected usage to drive initial, risk based estimations

Permutations of device, OS, Browser will create distraction – These are important, but must be based on target usage

AUT loading, charging, OS selection, and setup are also factors

23

Strategy – Managing Mobile Requirements

This is a change in the role of the tester:

• Functional requirements will not address the

infrastructure and technical needs of a mobile

application or web experience

• We must create best practices for product owners

and architects to follow that drive quality solutions

• Testing must become the voice of the mobile strategy

24

Strategy – Managing Mobile Requirements

Functional Requirements:

• Usability, accessibility, notifications and interactions with OS

Infrastructure Requirements:

• Wi-Fi vs. Network, lost transmission, security, and on device

storage

Consumption requirements:

• Battery, data, bandwidth

Input requirements:

• Camera, biometrics, keyboard types

Error Handling:

• Call interruption, multi-tasking, background usage

25

Strategy – Managing Mobile Environments

Traditionally, the test environment is managed by the

infrastructure or technical team

But with mobile, who do we “request” an environment

from?

And what constitutes a mobile environment?

It is now the tester’s job to load OS, AUT, prepare the

environment, and ready for testing

26

Strategy – Managing Mobile Test Phases

Traditional web\desktop testing has included a standard

approach that usually consists of: Functional, System,

Integration, NFR, Regression and UAT (along with many

other phases)

Often, with mobile, we are forced to only consider

functional (does it render) when we must also include:

• Usability

• Performance

• Security

27

Strategy – Managing Mobile Quality\Risk Analysis

Between native\hybrid apps, “m.”, responsive design,

and mobile web, our risk reporting must be focused on

financial impact, customer adoption, and usability

Risk reporting now includes intended

platform\OS\Device\browser combinations

Our roles in this process will be continuing to change

but we must define risk criteria as part of our strategy

from the onset

28

Summary - Quick Hits

• Mobile Strategy is as much a part of test strategy as regression, especially for customer\public portals

• Responsive design must be implemented in any external website and can be tested outside of the device

• Test strategy must evolve to include influence to design and development based on intended use

• Risk mitigation includes the identification of core aspects of product usage

• Mobile is not just for phones and tablets!

29

Brian Bernknopf

bbernknopf@kms-technology.com

30