Mobile App Assurance: Yesterday, Today, and Tomorrow.
-
Upload
bob-binder -
Category
Technology
-
view
359 -
download
3
description
Transcript of Mobile App Assurance: Yesterday, Today, and Tomorrow.
A foundation of quality
Robert V. Binder
Director of Innovation, Olenick and Associates
IMBUS Software QS Tag — November 8, 2012
The Big Picture
Overview
• The mobile deluge
• Mobile app testing retrospective
• Mobile app assurance challenges
• State of the art, 2012
• Crafting a mobile app assurance strategy
• Q&A
THE MOBILE DELUGE
SECURELY
RELIABLY SEEMLESSLY
MOBILE APP TESTING
RETROSPECTIVE
© 2004 mVerify Corporation 9
Mobile Test Technology, 2002
Typical
Mobile App,
End to End
Mobile
Switching
Center
Public
Switched
Network
Internet
BackBone
First Tier
Server
Business
Server
Base Station
Mobile User
Device
Parametric
Desktop
Load Test
Desktop
Serial Port
Interface
Critical Capabilities
Application Server
Functionality
Response Time
Server Resource Utilization
Billing/Provisioning/Security
Background contention
Dispatch/Allocation
Background IP Load
Client transaction saturation
End-to-End
Response time
Capacity
Reliability
Availability
Geographic Coverage
Handheld
Functionality
Response Time
MUD Resource Utilization
Airlink Variation
QOS Edge Combinations
In-cluster Hand Offs
Multiple Base St Protocol
Roaming
Location Services
Server Interaction
Server Exception
Configuration
Mobile Infrastructure
Op/Admin/Maintenance
Background load (“breathing”)
Packet Load
Weather, solar, etc.
© 2004 mVerify Corporation 11
Hand Held Testing
Mobile
Switching
Center
Public
Switched
Network
Internet
BackBone
First Tier
Server
Business
Server
Base Station
Mobile User
Device
Handheld
Functionality
Response Time
MUD Resource Utilization
Airlink Variation
QOS Edge Combinations
In-cluster Hand Offs
Multiple Base St Protocol
Roaming
Location Services
Server Interaction
Server Exception
Configuration
Base Station
Op/Admin/Maintenance
Background load (“breathing”)
Packet Load
Weather, solar, etc.
Application Server
Functionality
Response Time
Server Resource Utilization
Billing/Provisioning/Security
Background contention
Dispatch/Allocation
Background IP Load
Client transaction saturation
End-to-End
Response time
Capacity
Reliability
Availability
Geographic Coverage
•No Load
•No Mobility
•No Interaction
Transport Layer
Network Layer
Data Link Layer
Physical Layer
Parametric Testing
Handheld
Functionality
Response Time
MUD Resource Utilization
Airlink Variation
QOS Edge Combinations
In-cluster Hand Offs
Multiple Base St Protocol
Roaming
Location Services Server Interaction
Server Exception
Configuration
Base Station
Op/Admin/Maintenance
Background load (“breathing”)
Packet Load
Weather, solar, etc.
Application Server
Functionality
Response Time
Server Resource Utilization
Billing/Provisioning/Security
Background contention
Dispatch/Allocation
Background IP Load
Client transaction saturation
End-to-End
Response time
Capacity
Reliability
Availability
Geographic Coverage
Mobile
Switching
Center
Public
Switched
Network
Internet
BackBone
First Tier
Server
Business
Server
Base Station
Mobile User
Device
•Single connection
•No Back-end
•No Server
•No App Function
•Mobility & Load ?
Server Testing
Mobile
Switching
Center
Public
Switched
Network
Internet
BackBone
First Tier
Server
Business
Server
Base Station
Mobile User
Device
Handheld
Functionality
Response Time
MUD Resource Utilization
Airlink Variation
QOS Edge Combinations
In-cluster Hand Offs
Multiple Base St Protocol
Roaming
Location Services
Server Interaction
Server Exception
Configuration
Base Station
Op/Admin/Maintenance
Background load (“breathing”)
Packet Load
Weather, solar, etc.
Application Server
Functionality
Response Time
Server Resource Utilization
Billing/Provisioning/Security
Background contention
Dispatch/Allocation
Background IP Load
Client transaction saturation
End-to-End
Response time
Capacity
Reliability
Availability
Geographic Coverage
•No Mobility
•No Network
•Emulated Handheld
Mobile
Switching
Center
Public
Switched
Network
Internet
BackBone
First Tier
Server
Business
Server
Base Station
Mobile User
Device
End-to-end: Manual/Live Network Handheld
Functionality
Response Time
MUD Resource Utilization
Airlink Variation
QOS Edge Combinations
In-cluster Hand Offs
Multiple Base St Protocol
Roaming
Location Services
Server Interaction
Server Exception
Configuration
Base Station
Op/Admin/Maintenance
Background load (“breathing”)
Packet Load
Weather, solar, etc.
Application Server
Functionality
Response Time
Server Resource Utilization
Billing/Provisioning/Security
Background contention
Dispatch/Allocation
Background IP Load
Client transaction saturation
End-to-End
Response time
Capacity
Reliability
Availability
? Geographic Coverage
•Inconsistent
•Can’t Scale
•Time Consuming
•No Load
•Mobility (Drive test )
AMATE Project
Advanced Mobile Application Test Environment Goal: Achieve realistic end-to-end mobile testing
Model-based, Mobile-Centric
Signal variation related to mobility
User behavior related to mobility
Traffic related to mobility
NIST/ATP funded R&D
2002-2004
© 2005 mVerify Corporation 16
Load Model
• Vary aggregate input rate
• Arc
• Flat
• Internet fractal
• Negative ramp
• Positive ramp
• Random
• Spikes
• Square wave
• Waves
Actual “Waves” Loading
Signal Propagation Map
Mobility Model
1 Bar Signal
2 Bar Signal
3 Bar Signal
• Map generates real time itinerary for N virtual users • Location-specific signal strength • Location-specific end-user behavior • Controls Airlink Emulator
Virtual Users
© 2006 mVerify Corporation A Million Users in a Box ®
The Mobile Testing Nightmare
• Intense, high-stakes race to
market
• Configurations (platforms x
devices x airlinks) increase
exponentially
• More testing necessary for
competitive quality,
reliability, performance
• Ad hoc manual testing is
slow, costly, ineffective
© 2007 mVerify Corporation A Million Users in a Box ®
MTS: Any App, Any Platform
Agent Host
Client Host Under Test
Agent HostConsole Host
MTS
Console
MTS Test
Agent/STE
MTS Test
Agent/STE
Host Under Test may be
TEST RUN
REPORTS
MTS Remote
Agent
Client
Under Test
Server Host Under Test
MTS Remote
Agent
Server
Under Test
ü Cell Phone, PDA
ü Desktop or Server
ü Embedded Processor
ü Comm Interface
ü Network Equipment
ü Access Point
ü Base Station
Client Host Under Test
MTS Remote
Agent
Client
Under Test
Server Host Under Test
MTS Remote
Agent
Server
Under Test
Plug In
Plug In
MTS Model
Editor MTS
Repository
Channel
Emulator
MOBILE APP ASSURANCE
CHALLENGES
Functionality and Robustness
Using Launchpad 2.6 with a mobile
app, when the app is deactivated then
reactivated, all the elements of the
home view list are duplicated.
When the phone rings, the incoming
call screen pops up with options
accept, ignore and ignore-with-text.
Selecting ignore-with-text doesn’t
disconnect, pops up two more times,
and doesn’t send the text.
Same App, many platforms, locales …
• Android
• BlackBerry
• iPhone, iPad
• Mobile Web (HTML5)
• Windows Mobile
• Dozens of languages
Configuration Coverage
• Form factors
• Screen sizes
• Platforms and versions
• Accessories
• Wireless stacks
• Bandwidth
• Carrier policies and technology
• Logo/Store Certification
• Locales/localization
• Server-side performance
• How many ways can your
app be deployed?
Usability
Security
Scalability: Amazon Cloud Outage
• Amazon web service Elastic Compute Cloud crashes (power failure)
• The Elastic Load Balancing (ELB) system frantically tries to assign workloads to available servers.
• As Amazon’s cloud rebooted, “a large number of ELBs came up in a state which triggered a bug we hadn’t seen before”
• ELB reaction: try to allocate more larger servers.
• Backlog in the “control plane” results.
• Demand from customers in unaffected availability zones continues
• System swamped and crashes again.
Latent Positive Feedback = Dragon King
Sornett calls these exceptional events dragon
kings “to stress that we deal with a completely
different kind of animal, beyond the normal,
whose extraordinary characteristics [have]
profound significance.”
Dragon Kings
Here be Dragons
ü Latent positive feedback
External disruptors
Malicious attack target
ü Partial degradation
Provoked panic
ü Low testability
Multi-stack
Field-infeasible
Uncontrollable Inputs
STATE OF THE ART, 2012
Handheld Testability
• Many UI Test Tools
• UI event capture/replay
• Image capture/bitmap compare
• API for widgets/controls
• Invasive
• Brittle
• Several test suite composers
• Drag and drop
• Natural language/keyword
• No multi-endpoint capability
• Most are platform-specific
Crowd-sourced Testing
• Crowd Testing
• UTest
• MobTest
• Mob4Hire
• Others
“60787 people (mobsters) have 34142
different mobile handsets on 448 carriers
in 156 countries”
Test Environments as a Service
Remote Device Array
SUT
Mobile Client Emulation
Scalable Client Emulation
Live Target
SUT
Testing Technology for Mobile apps
What’s new
• Testable UI APIs
• Crowdsourced testing
• Cloud-based Test in
Production
• More stable infrastructure
What’s not new
• UI capture/replay
• Image capture/compare
• Virtual users
• Hand-crafted test suites
• Massively manual testing
• Remote device array
• Fragmented test harness
• No Model-based Testing
CRAFTING A MOBILE APP
ASSURANCE STRATEGY
System of Programs •Interfaces
•End-to-end
Program Systems Product
Program Product •Testing
•User Support
•Maintenance
3X 10X
3X
Brook’s Scope
Program
Tiered App •Interfaces
•End-to-end
Mobile System Product
Published App •Testing
•User Support
•Maintenance
3X 10X
3X
Brook’s Scope, Today’s Environment
One Billion Smartphones
2.5 Billion Endpoints
Mobile App
State of the Art
• Handheld testability
• A little better
• Mobile testing nightmare
• Remote Device Arrays
• Crowdsourced testing
• End-to-end coverage
• Infrastructure much more robust
• Capacity much improved
• No end-to-end test harness
• No support for Internet of Things
• Systems are bigger, more complex, and more critical
• Hic Sunt Dracones
Mobile App Strategy
“The best answer from 3 candidates with
over 10 years of claimed mobile application
testing experience listed on their CV, and all
ISEB qualified was, and I quote:”
“It’s more easy on the mobile app".
Hiring Manager:
“To what should you pay special attention
when testing a mobile application, in
comparison with say a standard web
application?”
Mobile App Strategy
• Design for testability
• Minimize variations
• You’ll need tooling for each
platform
• Cost of failure is very high
• App Store reject adds months
• No second chance with users
• Space is moving very fast
Mobile App Strategy
• Cover features and events
• Use and Abuse Cases
• Event Plan, pairwise
• Manual, automated, RDA
• Cover Configurations
• Remote Device Array
• Crowdsource if necessary
• Capacity Test
• Cloud Test Services
• Assess Dragon King Risk
• Model-based multi-dimensional
UC01 UC01 UC02 … UC99
Foregrnd/Backgrn
Background load
Power Sleep Cycle
Battery Drain
Incoming Call
Incoming Text
Camera Usage
Bar code scan
Accelerometer
Reboot
GPS Impairments
WiFi Impairments
Cellular Impairments
Content and Image Sources
Big Picture
http://apod.nasa.gov/apod/ap071021.html
How do people want to use their devices
http://network.cisco.com/how-do-people-want-to-use-their-devices.html
Internet of Things
http://blogs.cisco.com/news/the-internet-of-things-infographic/
Functionality … Configuration Coverage, Event Plan
http://www.udemy.com/how-to-test-mobile-apps
Usability
http://www.pagetrafficbuzz.com/google-survey-reveals-75-mobilefriendly-sites/14736/
NativeDriver
Matt DeVore, Tomohiro Kaizu, Dezheng Xu, Daigo Hamura. Native Driver Native App UI Automation with WebDriver API, 2011 Selenium Conference, San Francisco.
http://www.nativedriver.googlecode.com/files/NativeDriver_introduction.pdf
Amazon Outage
Hidden bugs that made Amazon Web Service outage worse. Cnet News, July 3, 2012.
http://news.cnet.com/8301-1009_3-57465761-83/hidden-bugs-that-made-amazon-web-service-outage-worse/
Dragon-Kings, Black Swans and the Prediction of Crises. Didier Sornette. International Journal of Terraspace Science and Engineering 2009
http://www.youtube.com/watch?v=FlTSbzOvKZI
Latent Positive Feedback = Dragon King
http://www.everythingselectric.com/forum/index.php?topic=244.0
Dragon King
http://wallpaper4me.com/wallpaper/Dragon-King-of-The-Land/
Crowdsourced Testing
http://mob4hire.com/
Hic Sunt Dracones
http://tomlytle.com/
All other content
Copyright © 2012, Robert V. Binder
Why are mobile apps so popular?
• Low cost
• Any time, any where, any one
• Connectivity to everything
• Complete control
• Personal space
Current Technology
Test Goal Plus Minus
Functionality and
Robustness
Simple Tools for all
platforms
Stack Silos
Brittle testware
Deployment Coverage Crowdsourced Testing
Remote Device Array
Systematic coverage?
Superficial
Usability Crowdsourced Testing
Security Abuse Cases Manual/technical
Scalability Proxy and cloud load
testing
Happy paths can’t find
Dragon Kings
Network of Things No framework for adapters
Sophistication Platform IDEs Mostly manual
No Model-based Testing
Attitudes Seen as “easier”
© 2005 mVerify Corporation 46
Mobile Device
Testing with AMATE
Model Builder
Simulator
Digital Airlink
Emulator
Test Driver
AUT
WiFi
Access Point
1 Generate Model
2 Generate Test Objects
4 Test Driver Controls AUT
DALE Controls Airlink
Test Agent
3 Run Test Objects
Then and Now Handheld M Infras Server E2E
Scalability 1 2 4 2
Mobility 1 2 0 0
Connectivity 1 1 5 1
Usability 1 3 4 1
Security 1 3 4 1
Controllability 1 2 5 1
Observability 1 2 5 1
Coverage 0 3 5 0
Handheld M Infras Server E2E
Scalability 4 3 4 2
Mobility 2 3 3 2
Connectivity 4 3 5 2
Usability 4 3 4 3
Security 2 4 4 2
Controllability 2 2 5 2
Observability 2 2 5 2
Coverage 4 3 5 3
0
2
1
2
1
1
1
3
Beyond Manual Testing
• Automated Testing
• Device side
• Server side
• Test management
• Seat, open source, cloud
• Automated
performance/stress
• Remote device sharing
• Crowd-sourced testing
services
© 2011, Robert V Binder. All Rights Reserved