Picking The Right Set of Mobile Devices

By Brian KitchenerSoftware Quality Architectbkitchener@prototest.com

Overview• About me• Some Background• The Problem• Understanding Android• How Apps Work• Building a Device Matrix• Example Matrices• Conclusion

About Me• Software Quality Architect at ProtoTest

• We're a mobile test lab that combines usability testing with quality assurance to cultivate a great user experience

• Project Architect, Technical Lead, Trainer.

• Started in QA in 2001• BA in Applied Computing from University of Denver• Testing background : Functional, Performance, UAT,

Security, API, Database.• Automation : Selenium, WebDriver, WatiN, MonkeyTalk,

SOASTA, Fitnesse, QTP, EggPlant, Squish• Languages : C#, Java, Ruby, Javascript

BACKGROUND INFORMATION

Some Stats for 2012• Mobile Apps achieved $17 billion in sales• 5.2 Mobile Subscribers– 1.2 Billion PC’s– 4.2 Billion people use a toothbrush– 1 Billion Smartphones

• 722 Million Smartphones sold • 1.4 Million iOS + Android Apps• 25 developers = half of app revenue

iPhone - June 2007

About the iPhone• Steve Ballmer : Microsoft CEO

– “There’s no chance the iPhone is going to gain significant market share. No chance.”

• Patrick Stewart: – “Last Wednesday, I stupidly dropped my iPhone in the bath, and my life

has sort of spiraled almost out of control.”• Jon Rubinstein – Palm CEO

– Is there a toaster that also knows how to brew coffee? There is no such combined device, because it would not make anything better than an individual toaster or coffee machine. It works the same way with the iPod, the digital camera or mobile phone: it is important to have specialized devices.

• Mike Lazaridis – Blackberry CEO– And so what [the iPhone] has actually done is increased our sales.

ANDROID IS THE PROBLEM

And Then There Were Two…• Android unveiled November 2007• First device was sold in October

2008.• Over 11,000 models have been

released.• 48 Billion app installs• Over 1 Billion Android devices

activated• 8 OS Revisions

OS Fragmentation

Device Fragmentation

Let’s do some math!• 16 device display categories• 20 different common resolutions• 8 OS versions• 6 Hardware Manufacturers• 4 Major cellular networks• 16 x 20 x 8 x 6 x 4 = 76,800 permutations• Pairwise approach = over 30 permutations• Who can afford to increase testing by 30X?

Our Approach • Efficiency, not coverage• Flexible: support small or large number of devices• Understand how apps work to logically select criteria• Use Market research to pick most common

configurations• Pick minimum and maximum boundary values for each

criteria• Choose a value that matches an edge case or abnormal

configuration.• Pick values that stress or tax the system

UNDERSTANDING ANDROID

Android• Built and Maintained by Google• Open Source• Built on Linux kernel• ARM• X86 Ports• Built to support almost any type of device– Phones, tablets, phablets, media players, tv’s,

watches, etc.• Device Manufacturers customize code.

Example: Kindle Fire• Forked Android 2.3– Not updateable

• Customized UI• Separate App store• Not all android apps work• Custom web browser

The Operating System• Google Releases “stock”

versions• 10 Major Releases since

2008– API Level, not Version

• Device manufacturers like to customize the OS– Drivers, libraries, UI

• “Stock” OS available in Nexus devices or an Emulator

Simulators / Emulators• Simulator imitates the software layer – OS and Libraries– Apple provides a simulator in xCode IDE

• Emulator duplicates the hardware and software– Processor and Memory– Cannot mimic GPU, GPS, accelerometer

• Always run stock OS• Can be used to test some functionality• Should always test on a physical device too

The Processor• ARM RISC-based instruction set• Specification defined by ARM holdings• 32 bit• Same as iOS• X86 patches and ports• It’s only a spec, can be modified.

SoC – System On A Chip• Main Board• Processor• RAM Bus • GPU • May include : – Cellular– WiFi– NFC– GPS– Bluetooth

2 Samsung Galaxy S4sQuallcomm Snapdragon• Quallcomm Krait 300– Quad core ARMv7

Cortex A15 Architecture

• Adreno 320 GPU • Dual Channel 533Mhz

Bus• Integrated LTE

Samsung Exynos 5 Octa• Samsung Big.little

processor– Quad core Cortex A 15– Quad core Cortex A7

• PowerVR SGX 544 GPU• Dual Channel 800Mhz

Bus• No Integrated LTE

Common SoCManufacturer Device Name Cores Processor GPU

Qualcomm Snapdragon S4 2 or 4 ARM Cortex-A15 Adreno

Nvidia Tegra 3 4+1 ARM Cortex-A9 Geforce

Samsung Exynos 4 2 or 4 ARM Cortex-A9 Mali

Intel Medfeld 1 Intel x86 PowerVR

Texas Instruments OMAP 4 2+2 ARM Cortex

A9 PowerVR

ST-Ericcson NovaThor 2 ARM Cortex-A9 PowerVR

Resolution is not enough• Unlimited number of screen sizes available• Screens range from 3” to 11”• Each screen has a resolution, same as a

monitor– If you increase the resolution everything shrinks!

• Pixels per Inch = Density• Screen Size + Density = Display Bucket• Resolution is not enough!

The Display BucketsSize : • Xlarge : 8” - 10” tablet. • Large : 5” - 7” tablet. • Normal : 3.5” - 5” phones.• Small : 3” - 3.5” phones. Density :• ldpi = Low DPI (~120)• mdpi = Medium DPI (~160)• hdpi = High DPI (~240)• xhdpi = Extra High DPI

(~320)

Low density (120), ldpi

Medium density (160), mdpi

High density (240), hdpi

Extra high density (320), xhdpi

Small screen

QVGA (240x320)

480x640

Normal screen

WQVGA400 (240x400)WQVGA432 (240x432)

HVGA (320x480)

WVGA800 (480x800) WVGA854 (480x854) 600x1024

640x960

Large screen

WVGA800** (480x800) WVGA854** (480x854)

WVGA800* (480x800) WVGA854* (480x854) 600x1024

Extra Large screen

1024x600 WXGA (1280x800)†

1024x7681280x768

1536x11521920x1152 1920x1200

2048x15362560x1536 2560x1600

Display Buckets• Galaxy S3

– 1280 x 720– Xhdpi density (331ppi)– Normal screen (4.7”)

• Galaxy Tab 10.1– 1280 x 800 – ldpi density (149ppi)– Xlarge sceren (10.1”)

• Galaxy Note LTE – 1280 x 800– hdpi density (285ppi)– Large Screen (5.5”)

Market Analysis

Aspect Ratio• UI is manipulated from code• Density Pixels adjust for screen size– But can use regular pixels!

• Need to take both X and Y into account!– Easy to overlap or hide things

• Includes orientation• Some devices include an aspect ratio changer!

(LG Optimus Vu)

Cellular Carrier• Four Major US Networks– Verizon, Sprint, AT&T, T-Mobile– Some phone interoperability– 2 protocols

• GSM – T-Mobile AT&T• CDMA – Verizon and Sprint

– Carriers assigned specific frequency bands– LTE will be new standard - But spectrum issues will prevent

cross-network phones• So if the phone supports the carrier’s protocol and

band it can theoretically connect.

HOW APPS WORK

How Apps work• Apps need to work on all screen sizes– May not be functional– May be wasted space– May not make sense

• Apps define XML layouts similar to HTML– Node structure– Static Content – Images, etc– Dynamic Content – Color, Text, etc.

Layouts and Fragments• XML Fragments are

reusable components• Layouts stitch together

fragments for a specific sized device

• App may need different flow for tablet vs phone

BUILDING THE DEVICE MATRIX

Our Criteria• Operating System

– OS customizations, missing libraries, driver issues, • Screen Size

– Rendering issues, usability, missing layouts• Pixel Density

– Density Independence, missing layouts.• Aspect Ratio

– X,Y calculations, overlapping panels, display issues• SoC

– Hardware performance, Instruction set, battery, signal• Carrier

– Network protocol, speed, responsiveness, packet loss

The Goal• Efficiency, not coverage!• Build a set of devices to be used for app and

website testing.• Know when to update them• Define a list of simple categories of devices• Pick devices that offer broad coverage• Adjust the number of devices based upon

needed coverage

Categorical Approach• Define scope– Android, iOS, phone, tablet, etc.

• Understand Testing requirements• Self-descriptive Names• Help to broaden coverage• Will adjust devices chosen to cover our criteria• Should be apparent when to update a device• Spread coverage :– Usage -> Edge Cases -> Strange -> Stress

Example Categories• Common

– Matches most common display configuration

• Newest– Latest OS version, largest

screen, highest resolution

• Oldest– Oldest, slowest, smallest

device.

• Abnormal– Non-standard OS, aspect

ratio, orientation, size

• Popular– Most popular device in

terms of sales

• Budget– Low-priced new model.

Tend to have strange specs

• Flagship– Nexus device running

stock Android OS

• Catch-All– Cover any missing criteria

Android Phone Matrix March 2012

Device Name OS Display Aspect

SoC Carrier

Newest HTC Droid DNA 4.2 Normal-xhdpi 9:16 Snapdragon S4 Verizon

Oldest HTC Tattoo 1.6 Small-ldpi 3:4 Snapdragon S1 AT&T

Common Motorola Droid 3 2.3 Normal-hdpi 9:16 TI OMAP 4 Verizon

Popular Samsung Galaxy S3

4.1 Normal-xhdpi 9:16 Exynos 4 Sprint

Abnormal LG Optimus VU 4 Large-hdpi 3:4 Nvidia Tegra 3 Tmobile

Flagship LG Nexus 4 4.2 Normal-xhdpi 3:5 Snapdragon S4 TMobile

Budget Dell Venue 2.2 Normal-mdpi 3:5 Snapdragon S3 AT&T

Catch-All Sony Xperia P 2.3 Normal-hdpi 9:16 Sony NovaThor AT&T

iOS Matrix March 2012

Device Name

OS Display Aspect SoC Carrier

Newest iPhone 5S 7 4” 1136 x 640 326ppi 9:16 Apple 64bit A7 T-Mobile

Oldest iPhone 3g 6 3.5” 320 x 480 165ppi 2:3 Apple A3 AT&T

Common iPhone 5 6 4” 1136 x 640 326ppi 9:16 Apple A5 Verizon

Popular iPhone 4 6 3.5” 640x960 330ppi 2:3 Apple A4 Sprint

iPad (Retina)

iPad 3 7 10” 1536x2048 264ppi 3:4 Apple A5X Verizon

iPod iPod Touch (4th gen)

5 3.5” 640x960 326ppi 2:3 Apple A4 WiFi

Mini iPad Mini 6 7” 1024 x 768 162ppi 3:4 Apple A5 AT&T

Conclusion• Understanding how everything works allows us

to logically select devices.• A large number of permutations can be covered

in few devices• If additional coverage is needed additional

devices can be added• White Paper : – http://www.prototest.com : – Building the Ultimate Device Matrix