The Mobile App Quality Challenge · uTest | on-demand testing The Mobile Boom 3 •$6.2 billion to...

The Mobile App Quality Challenge:How To Build Quality Into Your Mobile Apps

Doron Reuveni | uTest CEO & Co-Founder | [email protected]

uTest | on-demand testing

Agenda

• The Mobile Boom

• A Strategy For Mobile App Quality

• Mobile Testing Challenges

• Crowdsourcing 101

• How Crowdsourced Testing Works

• Sneak Preview of Mobile Testing Tool

1


MOBILE MARKET BOOM

2


The Mobile Boom

3

• $6.2 billion to be spent in 2010 at mobile app stores

• Mobile app stores expected to exceed 4.5B downloads in 2010

• Expected to surpass $21.6 billion in 2013

• US subscribers:

204K (1985)

277MM (2009)


Why Is Mobile Booming?

4

• Mobile is always on, always accessible

• Takes the web with you

Connectivity

Publishing

Sharing

• It’s smarter than traditional web

Geo-intelligent

Contextual

• Combines the best of traditional web and phones

• Continues to innovate

Laptops smart phones net books tablets


Mobile Meets The Enterprise

5

• Not just for kids (or consumers)

• Mobile applications in the enterprise: Business apps: CRM, ERP and HR systems

Productivity apps: docs, spreadsheets and presentations

Collaboration apps: email, IM and publishing

• Clinical, financial and customer data being accessed

• Growing pressure for QA to ensure mobile apps work properly and are private and secure Handsets makers & models

Operating systems

Carriers

Locations


STRATEGY FOR MOBILE APP QUALITY

6


First Things First

7

• Mobile app quality strategy begins with a decision


The Big Decision

8

• Mobile Web vs. Native Apps


Mobile Web Path

9

• Argument For Single platform (the web)

One app to build, launch and maintain

All that’s needed for some companies

• Argument Against Less controlled user experience

Slower user experience

No app store distribution

Lack of standards across mobile browsers


Native Apps Path

10

• Argument For Rich media functionality

Controlled user experience

Faster experience for users

App store distribution

• Argument Against Multi-platform

Must build, test and maintain multiple apps

Not necessary for some companies


Alternative Paths

11

• Wholesale Applications Community (WAC) Consortium of dozens of mobile leaders

– Carriers: AT&T, China Mobile, Orange, Verizon and Sprint

– OEMs: LG, Samsung and Sony Ericsson

Goal: build app once and have it work on all devices, carriers & OS

Good theory and wonderful on paper… but will it work?

– J2EE all over again?

– Competing interests?

– Trying to be all things to all people?

• Hybrid apps Native app “shell” or “wrapper”

Shared back-end with existing web app (data, functionality, etc)


Before You Begin

12

• Regardless of which path you choose, back-end work remains API / web services

Architecture

Database

Back-end functionality

• The important lesson: app quality begins before coding or testing


Word To The Wise

13

• Start with your users – what do they want? Reach users on their terms (web; mobile web; native apps)

What are your users trying to access?

– Static content vs. rich content

– Commerce vs. functionality

• Key fact Consumer conversion (from surf to purchase) on iPhone or Android native

apps is 3X - 10X times that of mobile web sites

Why?

– User comfort with device

– Usability of the native app

– Page load speed

• Forget about making the mobile web; make the web mobile


TOP MOBILE TESTING CHALLENGES

14


1. Coverage

• Web Testing Matrix Is Tough

15

OS & Browser

Features


1. Coverage

• Mobile Testing Matrix Is Insane

16

OS & Browser

Features

Handset Makers& Models


1. Coverage


17

OS & Browser

Features


WirelessCarriers


1. Coverage


18

OS & Browser

Features


WirelessCarriers

Location


2. Managing Resources & Costs

• Managing peaks & valleys; unexpected twists & turns

• Enough to make QA leaders sick


3. Time-to-Market

• Never-ending race for each new app and version

• Can exhaust the best test managers and teams


4. Mobile App Quality

• Becoming impossible for QA leaders to assure quality


What’s At Stake

22

• Billions of dollars and the future of apps

• Mobile app competition hotter than ever Media attention

User adoption

App store approvals

• Pressure to build, test & launch apps never been higher

• Apps device sales


CROWDSOURCING DEFINED:

23

“Taking a service traditionally performed by a person or team and sourcing it to a large online group of people as an open project.”

- Jeff Howe Wired journalist & author of Crowdsourcing


Crowdsourcing Across Industries

24


Mob Mentality

BUT crowds often look (and act) like unruly mobs

25


From Mob to Community

And mobs don’t work in every category

Example: delivering a skilled service like software testing or development requires an orderly “community” capable of consistently producing the desired results

26


The challenge is to turn this –

27


Into this

28


HOW CROWDSOURCING WORKS IN TESTING

29


Crowdsourced Testing 101

• Traditionally, companies had two options: Option 1: Build robust in-house staff & lab

– Cost prohibitive for many companies

Option 2: Outsource or offshore

– Loss of control, communication challenges and quality risks

• A third, alternative trend has emerged to complement: Option 3: Crowdsourcing

– On-demand extension of in-house testing efforts

– Coverage across location, language, OS, browser, handset or carrier

30


Where Crowdsourcing Fits

• In-House vs. Offshoring

31

In-House

Pros:• More control• More visibility• App/biz knowledge

Cons:• Fixed costs• Scalability• Coverage• Higher commitment

Offshoring

Pros:• Perceived cost• Scalability• Lower commitment

Cons:• Adaptability• Diversity of testers• Transparency• Communication• Hidden costs

Crowdsourcing

Pros:• Coverage• Agility• Creativity• Total cost

Cons:• Coordination• Learning curve• Signal-to-noise ratio



• Manual vs. Automated

32

Manual

Pros:• Agility• Creativity• Explores the edges

Cons:• Scalability• Repetitive tasks• Human error• Exhaustive coverage

Automated

Pros:• Good for mainstream UX• Scalability• Cost• Speed for repetitive tasks

Cons:• In-the-lab vs. In-the-wild• Ignores the unexpected• Limits imagination• Hinders adaptability

Crowdsourcing



• Exploratory vs. Scripted

33

Exploratory

Pros:• Agility• Creative• Explores the edges

Cons:• Scale• Repetitive• Human Error

Scripted

Pros:• Predictability• Scalability• Control

Cons:• In-the-lab vs. In-the-wild• Ignores the unexpected• Limits testers’ creativity

Crowdsourcing



• On-Device vs. Off-Device

34

On-Device

Pros:• Agility• Creative• Explores the edges

Cons:• Time-consuming• Cost-prohibitive• Coverage

Off-Device

Pros:• Cost• Speed• Convenience

Cons:• In-the-lab vs. In-the-wild• Limited user expertise• Detached from real world UX

Crowdsourcing


How Crowdsourced Testing Works

35

Customers Testing Community

Testers paid

Specify Testing Needs- Handset Maker, Model & OS- Carrier, Location & Language

Customer approves

Selected testers invited to project

Testers report in real-time


Crowdsourced Testing Benefits

36

• Crowdsourced testing benefits:

Higher quality apps: – Real-world testing;

– Invite favorite testers back

Get to market faster: – 2-3 day test cycles instead of 2-3 weeks

– 24x7 testing cycles

Contain costs: – Costs a fraction of traditional outsourced/offshore testing

– No long-term contracts or fixed costs


REAL-WORLD MOBILE EXAMPLES

37


Mobile Firms Who Use The Crowd

38

http://www.1cast.com/index.html

http://seesmic.com/


Case Study: 1Cast

• Seattle-based mobile company

• iPhone & Android apps for news & videos Challenge:

– Launch leading apps in a multi-platform world -- with a startup team

– Between Wi-Fi, 3G, Edge, and others, 1Cast needed more hands-on testing resources than they could afford to hire

Strategy:– Using only in-house QA was unrealistic financially

– Unwilling to commit or relinquish control to an offshore testing firm

– Researched creative alternatives and discovered crowdsourced testing

Results:– Global testing coverage at a fraction of the cost of alternatives

– Specified testing requirements and built a virtual testing team

– Integrated crowdsourced testing into product development methodology to support testing of weekly builds

39


Case Study: Tapulous

• Palo Alto-based mobile company

• Gaming apps for iPhone and Android Challenge:

– Ensure functionality for Tap Tap Revenge across OS, carrier, device and location ahead of a major upcoming launch

– Testing specific features, including multi-player online mode, post-game screens, offline usage and 3G connectivity

Strategy:– In-house team was too familiar with app; needed a fresh set of eyes

– Outsourcing firms were prohibitively expensive and excessive

– Crowdsourcing was an ideal blend of beta users, in-house and outsourcing

Results:– Testing coverage across all of North America; usability and functionality

– Several high-priority bugs discovered before high-profile launch

– Achieved comprehensive testing coverage in weeks, not months over a blend of devices, locations and users

40


IN CONCLUSION

41


Summary

• Market growing exponentially – including B2B & enterprise

• Testing mobile is more complex than web or desktop

• Top 4 mobile testing challenges Coverage

Peaks & valleys

Time-to-market

App quality

• Crowdsourcing enables coverage across complex testing matrices Geographic Locations & Languages

Handset Makers & Models

Wireless Carriers

OS & Browsers

42


SNEAK PREVIEW: REMOTE MOBILE ACCESS

43


Remote Mobile Access

• A story… A tester – let’s say Roy from Boston – reports a critical bug on your

Blackberry app

Your QA manager and developer review it, reproduce it and ultimately, the bug is fixed

The developer or tester wants to see the bug fix in the wild – ON the actual device that first encountered the bug

Now, you can “borrow” that tester’s device remotely

• Can also be used for Reproducing bugs on the device

Running automated diagnostics scripts on the device

44


QUESTIONS?

45

Doron Reuveni | CEO | uTest, Inc

w: www.utest.com e: [email protected]

p: 1.800.445.3914

The Mobile App Quality Challenge · uTest | on-demand testing The Mobile Boom 3 •$6.2 billion to...

Documents

Transcript of The Mobile App Quality Challenge · uTest | on-demand testing The Mobile Boom 3 •$6.2 billion to...