Build Your Mobile Testing Knowledge

TL PM Tutorial

4/30/13 1:00PM

Build Your Mobile Testing

Knowledge

Presented by:

Karen Johnson

Software Test Management, Inc.

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Karen Johnson

Karen N. Johnson is a software test consultant, working on-site and remotely to help her clients build testing teams and strengthen their testing practice. Karen is considered an expert in software testing and has been part of the context-driven software testing community for a number of years. She is a frequent speaker at testing conferences; a contributing author to the book, Beautiful Testing; and co-founder of WREST, the Workshop on Regulated Software Testing. Karen has published numerous articles and blogs about her experiences with software testing. Visit Karen’s website at karennjohnson.com.

4/16/2013

Build Your Mobile Knowledgesection 1

Build Your Mobile Testing Knowledge v6 © Karen N. Johnson, 2013 1

Mobile manufacturers

• Manufacturers change

continually.

• Manufacturers vary by

country.

• Understand the market your

app is in vs. the overall

global market.

• Expect to update your

strategy frequently.

4/16/2013

Device models

• Device models change

frequently as well.

• Devices also vary by

country.

• Device detection: do you

know what your users

Operating systems

• Android from Google, Inc.

• BlackBerry OS from RIM

• iOS from Apple

• Symbian OS from the

Symbian Foundation

• WebOS from HP

• Microsoft Windows Mobile

4/16/2013

Are you testing with mobile browsers

or testing a mobile app?

Mobile apps

iPhone App Store Android Market

4/16/2013

Mobile browsers

• Android browser

• BlackBerry browser

• Safari

• Internet Explorer

• Firefox

• Opera

• Dolphin

• And more -

Network carriers and settings

• Connection is based on

country, carrier, signal and

the device.

• Network carriers can impact

SMS texting and access to

websites.

4/16/2013

Network Carriers

• China Mobile

• Vodafone

• AT&T

• O2

• Orange

• Sprint

• T-Mobile

• Tracfone

• U.S. Cellular

• Verizon

• Virgin Mobile

Sources: Mobile Forum: http://www.mobileforum.com/

And Wikipedia

“ A mobile network operator (MNO), also known as mobile phone operator (or simply mobile operator or mobo ), carrier service provider (CSP), wireless service provider, wireless carrier, or cellular company, is a telephone company that provides services for mobile phone subscribers.”

- Source: Wikipedia

GSM vs. CDMA

• Global System for Mobile

Communications (GSM, around

80–85 % market) share)

• GSM is used everywhere but the

US, India and China

Source:

http://en.wikipedia.org/wiki/List_of_mobile_networ

k_operators

• Code division multiple access

(CDMA) is a channel access

method

• CDMA is used in the US, India and

4/16/2013

Connectivity

We are not stationary

when we use mobile.

So why are we testing from

the inside of our quiet

well-connected offices

with fully charged

batteries?

Mobile interruptions

• Forced updates

• Incoming calls & texts

• Battery issues

• Connectivity issues

4/16/2013

InstallationApplication install & updates.

UX: User interface considerations on mobile are radically different than on the desktop.

4/16/2013

User scenarios

Navigational paths on

mobile are not the same

as on the desktop.

OS & firmware updates

4/16/2013

App Permissions

Your app asks for what?

Device settings

4/16/2013

Functional testing

We still have functional testing.

The UPS example:

– Track packages

– Ship packages

– Locate a store

– Get a quote

– Give feedback

Search testing:classic testing with a new twist: search with barcodes or where device location is

factored into the results.

4/16/2013

Security: New testing considerationsM-Commerce: the shopping cart with barcodes, location-based (NFC)

coupons, card scanning/swiping and Google wallet.

Cookies

• Remember me?

• Forgotten password

• Auto-login

• Third party login access (i.e.

Use Facebook account to

login)

• Password changed on the

desktop.

4/16/2013

Location aware

• Testing while moving or

traveling.

• Testing with connectivity

switching, dropped and

intermittent.

• Testing with mobile

interruptions such as

incoming calls.

Mobile statistics

• Source: Which source do

you trust?

• Country: Usage statistics

vary greatly by country.

• Analytics: Are you capturing

device usage for your own

application?

4/16/2013

OS market share

What source do you use to gather information?

• StatCounter

http://gs.statcounter.com/#mobile_os-ww-monthly-201012-201112

• ComScore

http://www.comscore.com/Press_Events/Press_Releases/2011/12/comScore

_Reports_November_2011_U.S._Mobile_Subscriber_Market_Share

• MobiThinking

http://mobithinking.com/mobile-marketing-tools/latest-mobile-stats

Device Matrixsource: Mobile Design and Development, by Brian Fling

Class Markup CSS Javascript

Class A

Class B

Class C

Class D

Class F

4/16/2013

What devices do I need to test?

• Brad Frost - blog

post:http://bradfrostweb.com/blog/mobile/test-on-real-

mobile-devices-without-breaking-the-bank/

• My comment on his post is that his view is somewhat North

American focused.

• While the iPhone and Android dominate the market, those

two options are not the only players in the market.

Device Selection

How will you decide which

devices to test on?

• User statistics

• Country statistics – based

on user population

• Budget

• What the team owns

4/16/2013

Mobile User Interface, Design &

User Experiencesection 2

MENUS, FORMS & NAVIGATION

4/16/2013

Menu Styles

• Springboard or Grid Menu

• List menu

• Tab menu

• Gallery

• Dashboard

• Metaphor

• Page Carousel

Source: Theresa Neil.

http://www.mobiledesignpatterngallery.com/mobile-patterns.php?colid=65438029-72157627607680275

http://www.slideshare.net/theresaneil/navigation-patterns-for-mobile-apps

Menu Styles

Springboard List

4/16/2013

Menu Styles

Tab Gallery

Menu Styles

Carousel Metaphor

Build Your Mobile Testing Knowledge v6© Karen N. Johnson, 2013 34

4/16/2013

Navigating

Giving the user a unique way to move forward & back.

Navigating

• Paging

• Forward & backward

• Cookie crumb trail

(websites)

4/16/2013

Navigating

• Rotate

• Pinch/zoom

• Accelerometer

• Vertical/horizontal

• Paging forward & backward

Input Selector/Navigation

• Rotate

• Touchscreen

• Click wheel

• Rollerball

• Pinch & zoom

• Typing/keyboard inputs

• Accelerometer

• Paging backwards/forwards

• Cookie crumb trail

• Gestures

4/16/2013

UI controls

• Entry fields

• Lookup buttons

• Radio buttons

• Drop downs

Custom Controls

• The slider bar on AA’s

mobile app is an unusual

custom UI control.

• Does your app have any

custom controls?

4/16/2013

• Do links work?

• Does your site have links to

other sites?

• What about the sitemap

links?

• Alignment

• Consistency

• Disruptions from ads or 3rd

party feeds

4/16/2013

Color-alternating lists

• Grouped

• Zebra

stripping lists

• Text

Alignment

• User

Accessibility

Colors & Symbols

• Symbols

• Spacing

• Rotating

• Columns

4/16/2013

Barcodes and Scanning

• Visibility

• Image tolerance

• Connecting to a

service

Currency

User interface considerations

with multiple currencies: is the

correct symbol for the

currency shown?

4/16/2013

Images

• Where’s the image?

• You might use your desktop

to see if the issue occurs

outside of the mobile

environment.

Images

• Size of files

• Cache

• Refresh rate

• Full screen

• Wi-Fi drops connection

• Rotate

• Pinch/zoom

• Accelerometer

• Vertical or horizontal

4/16/2013

Video & Streaming

• Flash?

• Videos

• Connection speed

• Availability

Completing Forms

At some point, a user

needs to enter

information. Minimizing

input – such as offering

smart fill fields on location

awareness helps to cut

down on the amount of

information a user has to

enter.

4/16/2013

Completing Forms

• Sign-in and search are

two of the most

common forms on

mobile devices.

4/16/2013

Passwords

• Case sensitive

• Changing the case

• Seeing the text entry

• Staying logged in (cookies)

SEARCH, SORT, FILTER

4/16/2013

Search

• Wildcards

• Text entry

• Case sensitivity

• Alpha and numeric entry

• Dropdowns

• Location based searching

Search & Filter

• Location-based

filtering

• UX considerations

on filters, applying

& clearing

• UX considerations

– scrolling through

many search

results

4/16/2013

Search & Sort

When results are pages

long on a mobile device,

offer the user a means to

sort or refine the result

Search

• Are there fields that are prefilled based on date, location or

• OpenTable is an example of time and location.

• Are there fields that are not offered on mobile but are offered

on the desktop version?

• Is there a search button? Is there another method to start or

run the search?

• Is there a way to refine the results?

4/16/2013

LOCATION AWARENESS

Find location

Consider app-

dependencies on phone

settings - like location-

based searching.

4/16/2013

Search: filtering results

M-Commerce: restaurants

• Find a restaurant (local-based search or otherwise), book a reservation, order for pickup service, and booking via mobile.

• From a testing perspective, what information does the app have? Credit cards and personal information attached to an account on a mobile device open a new security concern.

4/16/2013

RESPONSIVE DESIGN

What is “responsive?”

Menus or other pages whose presentation or behavior is altered on different devices and screen widths. Flexible grids, flexible images and the ability to “ebb and flow” a design based on screen size.

Ethan Marcotte in 2010 coined the term in a landmark article called: A List Apart. See:

http://alistapart.com/article/responsive-web-design

4/16/2013

A short lesson on

responsive design

http://blog.creativemagma.com/featur

ed/2209/

Media queries

• Definition: “A media query consists of a media type and zero

or more expressions that check for the conditions of particular

media features. Among the media features that can be used

in media queries are ‘width’, ‘height’, and ‘color’. By using

media queries, presentations can be tailored to a specific

range of output devices without changing the content itself. “

- Source: W3C, http://www.w3.org/TR/css3-mediaqueries/#media0

4/16/2013

PHONE VS. TABLET

Phone vs. tablet

People often use both at the

same time.

Testing by planning user

scenarios is a good way to

uncover and identify

situations to test.

4/16/2013

Phone vs. tablet

• App versions

• Apps adapted for tablets

• What about phablets?

• Functionality and form factor – does the tablet app work the

same as the mobile phone app?

App Version

• A version page is an

opportunity to once

again show the app icon

and/or company logo.

• What about other

maintenance pages?

4/16/2013

App Version

Tablet Version View

• Does your app have a

version number?

• Does your app have

unique tablet or device

settings?

4/16/2013

Sitemaps

“They visually represent the relationship of

content to other content and provide a map for

how the user will travel through the

informational space.” Brian Fling, author Mobile Design and Development

TOUCH, PINCH, ZOOM,

ONE THUMB, GESTURES

Uniquely mobile

4/16/2013

Touch Gesture Icons

http://www.mobiletuxedo.com/t

ouch-gesture-icons/

Input Guidelines

http://uxdesign.smashingmagazin

e.com/2011/10/06/not-your-

parents-mobile-phone-ux-design-

guidelines-smartphones/

Touch Gesture Reference Cards

http://www.lukew.com/ff/entry.a

sp?1370

One thumb design

Fat Thumb: A One-Handed

Alternative To Pinch-To-

http://www.fastcodesign.c

om/1671741/fat-thumb-a-

one-handed-alternative-to-

pinch-to-zoom#1

Testing One Thumb, One

Eyeball Mobile Use

http://www.lukew.com/ff/

entry.asp?1664

4/16/2013

Mobile App & Device Settingssection 3

app, device & browser settings

• Overall app permissions

• Specific app permissions

• App permission-dependencies

• Device settings

• Browser settings

• Cookies

• Wireless & network settings

• Data Roaming

• SD card settings

• Swype

• Display settings

• Battery Settings/Status

• Task Killers

4/16/2013

Overall App Permissions

Know the app permissions

your mobile app requires.

Consider what the user may or

may not have enabled on the

device for all applications.

Specific App permissions

What permissions does

your application require?

4/16/2013

App permission-dependencies

If your app interfaces with 3rd

party data such as location

finding, what happens when

that access is off, down or

unavailable?

Device settings

• Consider the environment of the device: what has been enabled or not.

• Consider the ability or restrictions to downloading and installing applications.

• Consider upgrade scenarios and whether permissions have been changed or reset.

• Install testing has returned.

4/16/2013

Browser settings

Language settings

• Finding what languages are available on the device

• Choosing a different language

• Accessing and using a site or an app that’s multilingual

• Using different keyboards on devices

4/16/2013

Cookies

• Does your site use cookies?

• Check the login process

with cookies on/off.

Wireless & Network Settings

Have you ever tested your

website when connectivity is

down or disrupted? This

scenario is more likely with

mobile devices.

4/16/2013

Network Mode

CDMA is used in the US (also in China

and India) but everywhere else uses

GSM or other modes of connection.

It can be challenging to test like a

user when you live in a different

country and some options are not

available on devices where you are

located.

Network search

Users can switch from

and to an application or

browser session while

other activities are

taking place (including

incoming phone calls)

does your app handle

that disruption?

4/16/2013

Connecting to a network

A user could switch to

your app or website when

they’re not connected.

Connection errors

If your app uses the SD

card for storage – what if

the card is already full or

another app writes to the

4/16/2013

Data roaming

Would you enable your

testing team to leave the

office and experience

mobile as they travel

around the office?

SD card settings

• Does your app store

data on a card?

• What if the card is

removed?

• What if a different card

is inserted?

4/16/2013

SIM cards

• A SIM card or Subscriber

Identity Module is a chip.

• SIM holds personal identity

information, cell number,

phone book, text messages

and other data.

• Again, what happens if the

card is full? Changed?

Removed?

Swype settings

• Does your app work

with swyping or voice

input?

4/16/2013

Display settings

Consider whether a user

has made device settings

that can impact how your

application appears.

Battery conditions

• Have you tested your

app under low battery

conditions?

• Is your application

responsible for battery

drainage?- See Julian Harty’s work

- See Android Forum

4/16/2013

Battery conditions: power save

Is this a variable you need to

consider in testing?

Task Killers

• What background tasks are

running?

• What data is being synced?

• Does a Task Killer help or not?

Android

http://www.pcworld.com/article/24

4067/taskkiller_apps_will_they_help

_or_hurt_your_battery_life.html

http://www.youtube.com/watch?v=

nEJA1yUgR80

4/16/2013

The Competitive Mobile

Marketplacesection 4

What do mobile apps compete on?

According to uTest’s white paper from the product

Applause, there are intuitive metrics such as:

• content

• elegance

• pricing

• usability

• and satisfaction

4/16/2013

How steep is the competition?

• Apple Store: 700,00+ apps

• Google Play: 700,000+ apps

• Blackberry App World: 100,000+ apps

• By 2015, the International Data Corporation (IDC) predicts

182.7 billion mobile apps will be downloaded

• In 2010, 10.7 billion apps were downloaded

- source: uTest Applause white paper

MARKET RANKING

4/16/2013

Market Rankings

Applyzer is a product that watches

market rankings. Notice how they

segregate data by country and mobile

app category.

Market Rankings

Distimo is another market watching

application.

4/16/2013

Market Rankings

RankedNow is another product

tracking the competitive market.

4/16/2013

MOBILE COMPETITORS

Competitive Analysis – Vertical Competition

mobile app to app

AccuWeather The Weather Channel

4/16/2013

Competitive Analysis – Vertical Competition

tablet app to app

Competitive Analysis – Mobile Competition

the app Shazam vs. other ways to spend time

Notice how competitors might not

be offering the same product but

compete equally well for a user’s

time, purchase and use.

4/16/2013

Mobile competition:

Shazam’s “competitors”

See how this “competitor” isn’t offering

the same product but is vying for the

user’s time.

This competitor is a more typical

“competitor.”

Mobile competition:

a sample of differences by store

Competitors vary from store to store.

4/16/2013

MARKET COMMENTS

Charmin – Sit or Squat

This Charmin app was liked until the app was tied to Facebook and requires a FB

Account in order to login or comment. Users have scathing comments about the change.

4/16/2013

Tide Stain Brain

The Tide app made a change when the company removed user comments on how to

Remove stains and in turn infuriated the user community.

Chlorex myStain

The app gains respect and ratings by now being solely focused on pushing their product.

4/16/2013

Out of Milk

Note the number of users for the free version. And also note the overall ratings by users.

Spotify

Note how users comment on updates that cause issues or comment on missing

functionality.

4/16/2013

Competitive Analysis

Exercise:

1. Select two mobile applications that

would seem to be competitors such as

AccuWeather and the Weather

Channel.

2. Review each app for the “intuitive

metrics” listed as factors.

3. Make notations on each of the factors

for both apps.

4. For satisfaction, you can use either your

satisfaction with the app, app store

comments or a combination of both.

5. If you were to apply a scoring to each

app, which app would you choose over

the other app?

Build Your Mobile Testing Knowledge

Technology

Transcript of Build Your Mobile Testing Knowledge

REAL MOBILE DEVICE TESTING - BUILD VS BUYinfo.saucelabs.com/.../real-mobile-device-testing-build-vs-buy.pdfREAL MOBILE DEVICE TESTING - BUILD VS BUY Modern day development tools have

testing knowledge

Build Agile Knowledge - Participate in a sprint!api.ning.com/files/mzLbifxKbCRUAPpjQHCdOoiZwT7I8EErdqtJ79... · Build Agile Knowledge - Participate in a sprint! [Review] 1 ... handoffs,

[10 on Tuesday] Build Your National Register Knowledge

Build-in Quality!? SAFe® Testing im Finnova-Express (Swiss Testing Day 2017)

Build a Searchable Knowledge Base

Build molecular-testing-lab-toc

rcaa knowledge testing cpl/atpl

Build Your Agile Testing Skill Set

XPages build automation and testing

Exploratory Testing approach Personal knowledge as a test ... · Exploratory Testing approach – Personal knowledge as a test oracle Juha Itkonen ... testing approach report: 31.1.2012

CS52600: Information Security - Purdue University · Types of Penetration Testing • Black Box –External attacker has no knowledge of target system –Attacks often build on human

Build Your Mobile Testing Expertise

How to build knowledge through SNS

Regional Testing and Knowledge Centre, Nepal

Why does build acceptance testing fail?

Expand Your Knowledge and Build Relationships

Teaching with wikis to build and share knowledge

Andreas Weitl, System Engineer Automation for … Weitl, System Engineer Automation for Nightly Build Testing ... Automation for Nightly Build Testing ... • First steps: automate

Build Your Custom Performance Testing Framework