Mobile DeviceTesting Strategies

Stephen FreinComcast

5/8/2014

About Me• Director of QA for Comcast.com• Adjunct @ Drexel University• https://www.linkedin.com/in/stephenfrein• stephen.frein@gmail.com• www.frein.com

• Largest mass media and communicationscompany in the world (by $)

• Largest cable company / ISP in USA• Largest converged network in the world

(data / voice / video)• Comcast.com in April of 2014:

o 200 million + page viewso 25+ million unique visitors

Stuff We'll Talk About• Importance of mobile• What should I test on?• Where do I keep devices?

Mobile Is BIG

Mobile Matters (More Than Ever)• US retail mobile ecommerce sales increased

68% from 2012 to 2013 (1) *• US mobile web usage rose by a similar

amount (2)• More than half of Amazon holiday shoppers

used a mobile device in 2013 (3)

* Psst! The (1) is an endnote.

But Desktop Still Rules Web• Most mobile Internet usage (80+%) is

driven by apps (4)• Mobile usage accounts for about 30% of

web traffic (5)

Punchline

Mobile is growing in importance, but yourweb testing efforts should probablyemphasize desktop for the foreseeablefuture. (Apps are their own animal.)

Desktop > Mobile

When Will That Change?

Not in the next year . . .

Where Should I Focus?

7.1

7.06.1

Gingerbread

KitKat

Jellybean

What My Team Does• Establish coverage thresholds (browser /

OS / device type must have x% traffic)• Tune test coverage based on usual factors:

o Historical defect discovery rateso Nature of changeso Risk of leakageo Schedule pressure

• Pray we got it right

Balancing Stats

Your Stats(the focus)

Industry Stats(a counterweight)

Google Analytics, Adobe, Piwik, etc

StatCounter, Chitika, Android, etc

Industry Stats Useful Linkshttp://gs.statcounter.com/#mobile+tablet-browser-US-monthly-201304-201404

http://www.netmarketshare.com/ (some $)

http://developer.android.com/about/dashboards/index.html

http://david-smith.org/iosversionstats/

General Observations• Today, iOS and Android are the real show• New iOS versions have rapid adoption• Android versions more fragmented• Android world is heavily phone-skewed• “Default” browsers the focus• Mobile Chrome creeping up in usage• Don’t ignore Windows entirely, but it is a

lower priority• Other flavors may not be worth it

Where to Test?

Physicals

Cloud

Emulators

Help Me ChooseApproach Pros Cons

Physicals • Realism• Audio / Video

• Geo-lock• Management

Cloud • Geo-flex• Management• Automation

• Cost• Audio / Video• Speed

Emulator • Cheap• Fast

• Realism• Realism!!!

We Use Cloud (Mostly)• International team• “Hide and seek” got old• Audio / video not a priority for us• Good automation support• Private cloud helps

o Automationo Test environment access

“Who has the &%*@! iPad?”

Whose Cloud Offering?Start by looking at these two . . .

. . . but there are many others.

Remember• Mobile matters, but desktop owns web• Stats key to your device selection

– Android phones and iPhone / iPad are must-do– Set percentage standards, but then tune based

on other factors• Don’t rely on emulators – consider cloud

solutions for device hosting

Wrap-up

Questions?

Thanks!

Endnotes1. http://www.forbes.com/sites/chuckjones/2013/10/02/ecommerce-is-growing-nicely-while-mcommerce-is-on-a-tear/

2. http://bgr.com/2013/11/08/mobile-devices-internet-traffic-q3-2013/

3. http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-newsArticle&ID=1886961&highlight=

4. http://www.smartinsights.com/mobile-marketing/mobile-marketing-analytics/mobile-marketing-statistics/

5. http://www.walkersandsdigital.com/Walker-Sands-Mobile-Traffic-Report-Q3-2013