Build Your Mobile Testing Knowledge

TL PM Tutorial

4/30/13 1:00PM

Build Your Mobile Testing

Knowledge

Presented by:

Karen Johnson

Software Test Management, Inc.

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Karen Johnson

Karen N. Johnson is a software test consultant, working on-site and remotely to help her clients build testing teams and strengthen their testing practice. Karen is considered an expert in software testing and has been part of the context-driven software testing community for a number of years. She is a frequent speaker at testing conferences; a contributing author to the book, Beautiful Testing; and co-founder of WREST, the Workshop on Regulated Software Testing. Karen has published numerous articles and blogs about her experiences with software testing. Visit Karen’s website at karennjohnson.com.

4/16/2013

1

Build Your Mobile Knowledgesection 1

Build Your Mobile Testing Knowledge v6 © Karen N. Johnson, 2013 1

Mobile manufacturers

• Manufacturers change

continually.

• Manufacturers vary by

country.

• Understand the market your

app is in vs. the overall

global market.

• Expect to update your

strategy frequently.


4/16/2013

2

Device models

• Device models change

frequently as well.

• Devices also vary by

country.

• Device detection: do you

know what your users

have?


Operating systems

• Android from Google, Inc.

• BlackBerry OS from RIM

• iOS from Apple

• Symbian OS from the

Symbian Foundation

• WebOS from HP

• Microsoft Windows Mobile


4/16/2013

3

Are you testing with mobile browsers

or testing a mobile app?


Mobile apps

iPhone App Store Android Market


4/16/2013

4

Mobile browsers

• Android browser

• BlackBerry browser

• Safari

• Internet Explorer

• Firefox

• Opera

• Dolphin

• And more -


Network carriers and settings

• Connection is based on

country, carrier, signal and

the device.

• Network carriers can impact

SMS texting and access to

websites.


4/16/2013

5

Network Carriers

• China Mobile

• Vodafone

• AT&T

• O2

• Orange

• Sprint

• T-Mobile

• Tracfone

• U.S. Cellular

• Verizon

• Virgin Mobile

Sources: Mobile Forum: http://www.mobileforum.com/

And Wikipedia

“ A mobile network operator (MNO), also known as mobile phone operator (or simply mobile operator or mobo ), carrier service provider (CSP), wireless service provider, wireless carrier, or cellular company, is a telephone company that provides services for mobile phone subscribers.”

- Source: Wikipedia

© Karen N. Johnson, 2013 9Build Your Mobile Testing Knowledge v6

GSM vs. CDMA

• Global System for Mobile

Communications (GSM, around

80–85 % market) share)

• GSM is used everywhere but the

US, India and China

Source:

http://en.wikipedia.org/wiki/List_of_mobile_networ

k_operators

• Code division multiple access

(CDMA) is a channel access

method

• CDMA is used in the US, India and

China


4/16/2013

6

Connectivity

We are not stationary

when we use mobile.

So why are we testing from

the inside of our quiet

well-connected offices

with fully charged

batteries?


Mobile interruptions

• Forced updates

• Incoming calls & texts

• Battery issues

• Connectivity issues


4/16/2013

7

InstallationApplication install & updates.


UX: User interface considerations on mobile are radically different than on the desktop.


4/16/2013

8

User scenarios

Navigational paths on

mobile are not the same

as on the desktop.


OS & firmware updates


4/16/2013

9

App Permissions

Your app asks for what?


Device settings


4/16/2013

10

Functional testing

We still have functional testing.

The UPS example:

– Track packages

– Ship packages

– Locate a store

– Get a quote

– Give feedback


Search testing:classic testing with a new twist: search with barcodes or where device location is

factored into the results.


4/16/2013

11

Security: New testing considerationsM-Commerce: the shopping cart with barcodes, location-based (NFC)

coupons, card scanning/swiping and Google wallet.


Cookies

• Remember me?

• Forgotten password

• Auto-login

• Third party login access (i.e.

Use Facebook account to

login)

• Password changed on the

desktop.


4/16/2013

12

Location aware

• Testing while moving or

traveling.

• Testing with connectivity

switching, dropped and

intermittent.

• Testing with mobile

interruptions such as

incoming calls.


Mobile statistics

• Source: Which source do

you trust?

• Country: Usage statistics

vary greatly by country.

• Analytics: Are you capturing

device usage for your own

application?


4/16/2013

13

OS market share


What source do you use to gather information?

• StatCounter

http://gs.statcounter.com/#mobile_os-ww-monthly-201012-201112

• ComScore

http://www.comscore.com/Press_Events/Press_Releases/2011/12/comScore

_Reports_November_2011_U.S._Mobile_Subscriber_Market_Share

• MobiThinking

http://mobithinking.com/mobile-marketing-tools/latest-mobile-stats

Device Matrixsource: Mobile Design and Development, by Brian Fling

Class Markup CSS Javascript

Class A

Class B

Class C

Class D

Class F


4/16/2013

14

What devices do I need to test?

• Brad Frost - blog

post:http://bradfrostweb.com/blog/mobile/test-on-real-

mobile-devices-without-breaking-the-bank/

• My comment on his post is that his view is somewhat North

American focused.

• While the iPhone and Android dominate the market, those

two options are not the only players in the market.


Device Selection

How will you decide which

devices to test on?

• User statistics

• Country statistics – based

on user population

• Budget

• What the team owns


4/16/2013

15

Mobile User Interface, Design &

User Experiencesection 2


MENUS, FORMS & NAVIGATION


4/16/2013

16

Menu Styles

• Springboard or Grid Menu

• List menu

• Tab menu

• Gallery

• Dashboard

• Metaphor

• Page Carousel

Source: Theresa Neil.

http://www.mobiledesignpatterngallery.com/mobile-patterns.php?colid=65438029-72157627607680275

http://www.slideshare.net/theresaneil/navigation-patterns-for-mobile-apps


Menu Styles

Springboard List


4/16/2013

17

Menu Styles

Tab Gallery


Menu Styles

Carousel Metaphor

Build Your Mobile Testing Knowledge v6© Karen N. Johnson, 2013 34

4/16/2013

18

Navigating

Giving the user a unique way to move forward & back.


Navigating

• Paging

• Forward & backward

• Cookie crumb trail

(websites)


4/16/2013

19

Navigating

• Rotate

• Pinch/zoom

• Accelerometer

• Vertical/horizontal

• Paging forward & backward


Input Selector/Navigation

• Rotate

• Touchscreen

• Click wheel

• Rollerball

• Pinch & zoom

• Typing/keyboard inputs

• Accelerometer

• Paging backwards/forwards

• Cookie crumb trail

• Gestures


4/16/2013

20

UI controls

• Entry fields

• Lookup buttons

• Radio buttons

• Drop downs


Custom Controls

• The slider bar on AA’s

mobile app is an unusual

custom UI control.

• Does your app have any

custom controls?


4/16/2013

21

Links

• Do links work?

• Does your site have links to

other sites?

• What about the sitemap

links?


Lists

• Alignment

• Consistency

• Disruptions from ads or 3rd

party feeds


4/16/2013

22

Color-alternating lists

• Grouped

rows

• Zebra

stripping lists

• Text

Alignment

• User

Accessibility


Colors & Symbols

• Symbols

• Spacing

• Rotating

• Columns


4/16/2013

23

Barcodes and Scanning

• Visibility

• Image tolerance

• Connecting to a

service


Currency

User interface considerations

with multiple currencies: is the

correct symbol for the

currency shown?


4/16/2013

24

Images

• Where’s the image?

• You might use your desktop

to see if the issue occurs

outside of the mobile

environment.


Images

• Size of files

• Cache

• Refresh rate

• Full screen

• Wi-Fi drops connection

• Rotate

• Pinch/zoom

• Accelerometer

• Vertical or horizontal


4/16/2013

25

Video & Streaming

• Flash?

• Videos

• Connection speed

• Availability


Completing Forms

At some point, a user

needs to enter

information. Minimizing

input – such as offering

smart fill fields on location

awareness helps to cut

down on the amount of

information a user has to

enter.


4/16/2013

26

Completing Forms

• Sign-in and search are

two of the most

common forms on

mobile devices.


Login


4/16/2013

27

Passwords

• Case sensitive

• Changing the case

• Seeing the text entry

• Staying logged in (cookies)


SEARCH, SORT, FILTER


4/16/2013

28

Search

• Wildcards

• Text entry

• Case sensitivity

• Alpha and numeric entry

• Dropdowns

• Location based searching


Search & Filter

• Location-based

filtering

• UX considerations

on filters, applying

& clearing

• UX considerations

– scrolling through

many search

results


4/16/2013

29

Search & Sort

When results are pages

long on a mobile device,

offer the user a means to

sort or refine the result

set.


Search

• Are there fields that are prefilled based on date, location or

time?

• OpenTable is an example of time and location.

• Are there fields that are not offered on mobile but are offered

on the desktop version?

• Is there a search button? Is there another method to start or

run the search?

• Is there a way to refine the results?


4/16/2013

30

LOCATION AWARENESS


Find location

Consider app-

dependencies on phone

settings - like location-

based searching.


4/16/2013

31

Search: filtering results


M-Commerce: restaurants

• Find a restaurant (local-based search or otherwise), book a reservation, order for pickup service, and booking via mobile.

• From a testing perspective, what information does the app have? Credit cards and personal information attached to an account on a mobile device open a new security concern.


4/16/2013

32

RESPONSIVE DESIGN


What is “responsive?”

Menus or other pages whose presentation or behavior is altered on different devices and screen widths. Flexible grids, flexible images and the ability to “ebb and flow” a design based on screen size.

Ethan Marcotte in 2010 coined the term in a landmark article called: A List Apart. See:

http://alistapart.com/article/responsive-web-design


4/16/2013

33

A short lesson on

responsive design

http://blog.creativemagma.com/featur

ed/2209/


Media queries

• Definition: “A media query consists of a media type and zero

or more expressions that check for the conditions of particular

media features. Among the media features that can be used

in media queries are ‘width’, ‘height’, and ‘color’. By using

media queries, presentations can be tailored to a specific

range of output devices without changing the content itself. “

- Source: W3C, http://www.w3.org/TR/css3-mediaqueries/#media0


4/16/2013

34

PHONE VS. TABLET


Phone vs. tablet

People often use both at the

same time.

Testing by planning user

scenarios is a good way to

uncover and identify

situations to test.


4/16/2013

35

Phone vs. tablet

• App versions

• Apps adapted for tablets

• What about phablets?

• Functionality and form factor – does the tablet app work the

same as the mobile phone app?


App Version


• A version page is an

opportunity to once

again show the app icon

and/or company logo.

• What about other

maintenance pages?

4/16/2013

36

App Version


Tablet Version View


• Does your app have a

version number?

• Does your app have

unique tablet or device

settings?

4/16/2013

37

Sitemaps

“They visually represent the relationship of

content to other content and provide a map for

how the user will travel through the

informational space.” Brian Fling, author Mobile Design and Development


TOUCH, PINCH, ZOOM,

ONE THUMB, GESTURES

Uniquely mobile


4/16/2013

38

Touch

Touch Gesture Icons

http://www.mobiletuxedo.com/t

ouch-gesture-icons/

Input Guidelines

http://uxdesign.smashingmagazin

e.com/2011/10/06/not-your-

parents-mobile-phone-ux-design-

guidelines-smartphones/

Touch Gesture Reference Cards

http://www.lukew.com/ff/entry.a

sp?1370


One thumb design

Fat Thumb: A One-Handed

Alternative To Pinch-To-

Zoom

http://www.fastcodesign.c

om/1671741/fat-thumb-a-

one-handed-alternative-to-

pinch-to-zoom#1

Testing One Thumb, One

Eyeball Mobile Use

http://www.lukew.com/ff/

entry.asp?1664


4/16/2013

39

Mobile App & Device Settingssection 3


app, device & browser settings

• Overall app permissions

• Specific app permissions

• App permission-dependencies

• Device settings

• Browser settings

• Cookies

• Wireless & network settings

• Data Roaming

• SD card settings

• Swype

• Display settings

• Battery Settings/Status

• Task Killers


4/16/2013

40

Overall App Permissions

Know the app permissions

your mobile app requires.

Consider what the user may or

may not have enabled on the

device for all applications.


Specific App permissions

What permissions does

your application require?


4/16/2013

41

App permission-dependencies

If your app interfaces with 3rd

party data such as location

finding, what happens when

that access is off, down or

unavailable?


Device settings

• Consider the environment of the device: what has been enabled or not.

• Consider the ability or restrictions to downloading and installing applications.

• Consider upgrade scenarios and whether permissions have been changed or reset.

• Install testing has returned.


4/16/2013

42

Browser settings


Language settings

• Finding what languages are available on the device

• Choosing a different language

• Accessing and using a site or an app that’s multilingual

• Using different keyboards on devices


4/16/2013

43

Cookies

• Does your site use cookies?

• Check the login process

with cookies on/off.


Wireless & Network Settings

Have you ever tested your

website when connectivity is

down or disrupted? This

scenario is more likely with

mobile devices.


4/16/2013

44

Network Mode

CDMA is used in the US (also in China

and India) but everywhere else uses

GSM or other modes of connection.

It can be challenging to test like a

user when you live in a different

country and some options are not

available on devices where you are

located.


Network search

Users can switch from

and to an application or

browser session while

other activities are

taking place (including

incoming phone calls)

does your app handle

that disruption?


4/16/2013

45

Connecting to a network

A user could switch to

your app or website when

they’re not connected.


Connection errors

If your app uses the SD

card for storage – what if

the card is already full or

another app writes to the

card?


4/16/2013

46

Data roaming

Would you enable your

testing team to leave the

office and experience

mobile as they travel

around the office?


SD card settings

• Does your app store

data on a card?

• What if the card is

removed?

• What if a different card

is inserted?


4/16/2013

47

SIM cards

• A SIM card or Subscriber

Identity Module is a chip.

• SIM holds personal identity

information, cell number,

phone book, text messages

and other data.

• Again, what happens if the

card is full? Changed?

Removed?


Swype settings

• Does your app work

with swyping or voice

input?


4/16/2013

48

Display settings

Consider whether a user

has made device settings

that can impact how your

application appears.


Battery conditions

• Have you tested your

app under low battery

conditions?

• Is your application

responsible for battery

drainage?- See Julian Harty’s work

- See Android Forum


4/16/2013

49

Battery conditions: power save


Is this a variable you need to

consider in testing?

Task Killers


• What background tasks are

running?

• What data is being synced?

• Does a Task Killer help or not?

Android

http://www.pcworld.com/article/24

4067/taskkiller_apps_will_they_help

_or_hurt_your_battery_life.html

IOS

http://www.youtube.com/watch?v=

nEJA1yUgR80

4/16/2013

50

The Competitive Mobile

Marketplacesection 4


What do mobile apps compete on?

According to uTest’s white paper from the product

Applause, there are intuitive metrics such as:

• content

• elegance

• pricing

• usability

• and satisfaction


4/16/2013

51

How steep is the competition?

• Apple Store: 700,00+ apps

• Google Play: 700,000+ apps

• Blackberry App World: 100,000+ apps

• By 2015, the International Data Corporation (IDC) predicts

182.7 billion mobile apps will be downloaded

• In 2010, 10.7 billion apps were downloaded

- source: uTest Applause white paper


MARKET RANKING


4/16/2013

52

Market Rankings

Applyzer is a product that watches

market rankings. Notice how they

segregate data by country and mobile

app category.


Market Rankings

Distimo is another market watching

application.


4/16/2013

53

Market Rankings


Market Rankings

RankedNow is another product

tracking the competitive market.


4/16/2013

54

MOBILE COMPETITORS


Competitive Analysis – Vertical Competition

mobile app to app

AccuWeather The Weather Channel


4/16/2013

55

Competitive Analysis – Vertical Competition

tablet app to app


Competitive Analysis – Mobile Competition

the app Shazam vs. other ways to spend time

Notice how competitors might not

be offering the same product but

compete equally well for a user’s

time, purchase and use.


4/16/2013

56

Mobile competition:

Shazam’s “competitors”

See how this “competitor” isn’t offering

the same product but is vying for the

user’s time.

This competitor is a more typical

“competitor.”


Mobile competition:

a sample of differences by store

Competitors vary from store to store.


4/16/2013

57

MARKET COMMENTS


Charmin – Sit or Squat

This Charmin app was liked until the app was tied to Facebook and requires a FB

Account in order to login or comment. Users have scathing comments about the change.


4/16/2013

58

Tide Stain Brain

The Tide app made a change when the company removed user comments on how to

Remove stains and in turn infuriated the user community.


Chlorex myStain

The app gains respect and ratings by now being solely focused on pushing their product.


4/16/2013

59

Out of Milk

Note the number of users for the free version. And also note the overall ratings by users.


Spotify

Note how users comment on updates that cause issues or comment on missing

functionality.


4/16/2013

60


Competitive Analysis

Exercise:

1. Select two mobile applications that

would seem to be competitors such as

AccuWeather and the Weather

Channel.

2. Review each app for the “intuitive

metrics” listed as factors.

3. Make notations on each of the factors

for both apps.

4. For satisfaction, you can use either your

satisfaction with the app, app store

comments or a combination of both.

5. If you were to apply a scoring to each

app, which app would you choose over

the other app?


Build Your Mobile Testing Knowledge

Technology

Transcript of Build Your Mobile Testing Knowledge