Four best practices for performance testing mobile apps soasta and utopia
23
© 2014 SOASTA. All rights reserved. March 4, 2015 1 CONFIDENTIAL – Not for Distribution 6 5 4 3 2 1
-
Upload
soasta -
Category
Technology
-
view
1.139 -
download
3
Transcript of Four best practices for performance testing mobile apps soasta and utopia
© 2014 SOASTA. All rights reserved. March 4, 2015 1CONFIDENTIAL – Not for Distribution
65
43
21
© 2014 SOASTA. All rights reserved. March 4, 2015 2CONFIDENTIAL – Not for Distribution
Lee Barnes – Founder and CTO of Utopia Solutions
Tom Chavez– SOASTA Sr. Product Evangelist
© 2014 SOASTA. All rights reserved. March 4, 2015 3CONFIDENTIAL – Not for Distribution
o Device/OS diversity
o Network variability
o Device performance
o User location and volume
© Utopia Solutions
Best Practices for Performance Testing Mobile Apps
Lee Barnes, CTO
Utopia Solutions
© Utopia Solutions
Apology
I don’t believe in “best practices”…
I do believe in guidelines and better approaches for a given situation
© Utopia Solutions6
Key Questions
How is mobile different?
How can I adapt?
Why is performance of mobile important?
© Utopia Solutions7
Why Should I Care?
M-Commerce
66%OF
SHOPPERSABANDO
NTRANSACTIO
NS
5 OUT OF 10BECAUSE OF
POORPERFORMAN
CE
1DELA
Y
SECOND 7% DROP
INCONVERSIO
NS
equals a
2013 Mobile Commerce Insights study conducted by Jumio
© Utopia Solutions8
Still Not Convinced?
2013-2014 World Quality Report
© Utopia Solutions9
Evolution of Performance Testing
1990 2000 2010
2-TierClient Server Web Mobile
Challenges
State of the Practice
Testing Large Loads
Complex / Integrated
Systems
ProprietaryProtocols
Skilled Resources
In-Lab Testing
ToolMaturation
Open SourceTools
Specialized Resources
Cloud-based Testing
© Utopia Solutions
Mobile Challenges & Adaptations
© Utopia Solutions11
Mobile Challenges – User Access
Native App Mobile SiteFull / ResponsiveSite
Your users have options…
© Utopia Solutions12
Assess the User and Load Profile
1. Determine the user profile
• Business processes• Native app, mobile site, full site• Preferred browser• Network conditions• Geographic location
2. Determine the load profile
• Volume of users by business process and location
• Volume of users by access type• Frequency / throughput
Incorporate constant feedback from application logs and other monitoring solutions
© Utopia Solutions13
Mobile Challenges - Network
Mobile User Connected User
3G 4G InATunnelG
Network bandwidth and quality affect both end user experience and system load
© Utopia Solutions14
Assess Impact of Network Variability
1. Assess baseline performance(single user)
Observe UX with real devices behind network virtualization solutions
2. Assess performance under load
Put system under virtual user load
Observe UX with real devices behind network virtualization solutions
NV
Virtual Users
NV
Real
Devices
© Utopia Solutions15
Network Virtualization Tools
Single User (Device and/or Emulator)• Apple Network Link Conditioner• Android Emulator• Charles Proxy
Multiple Virtual Users• Charles Proxy
© Utopia Solutions16
Mobile Challenges – User Location and Volume
A high volume of global users is difficult to reproduce in a lab…
© Utopia Solutions17
Utilize Cloud-based Testing Solutions
• Achieve production level loads• Test entire infrastructure (vs.
behind the firewall)
• Distribute load geographically• Assess user impact on real
devices
© Utopia Solutions18
Mobile Challenges – Device Performance
Measuring System / Network Metrics is Not Enough…
Memory
CPU
Battery Drain
Graphics
I / O
© Utopia Solutions19
Measuring Device Performance
Critical device metrics:• CPU• Memory• Battery• Storage
Mobile development platforms• Android DDMS• iOS Instruments
Mobile test automation tools• TouchTest (SOASTA)Mobile lab / cloud solutions• deviceConnect (Mobile Labs)• Remote TestKit (NTT Resonant)
Capture metrics while exercising app
Include appropriate devices / configurations
Incorporate automation for consistency
Device Performance Measurement Solutions
© Utopia Solutions20
Summary
Key takeaways…
• Mobile users are not the same as connected users
• Performance is more than backend / network
• Push performance analysis to the left in the development cycle
© 2014 SOASTA. All rights reserved. March 4, 2015 21CONFIDENTIAL – Not for Distribution
SOASTA PRODUCT DEMO
© 2014 SOASTA. All rights reserved. March 4, 2015 22CONFIDENTIAL – Not for Distribution
o CI and Mobile Lab Scorecard
• 2 days
• Evaluate People, Process, Products, CI
• TouchTest and CI POC with customer tests
• Same mobile automation engineers that have implemented
mobile test automation for Fortune 5 and Fortune 10 retailers
© 2014 SOASTA. All rights reserved. March 4, 2015 23CONFIDENTIAL – Not for Distribution
UTOPIA
o Lee Barnes
o Founder and CTO
o Utopia Solutions, Inc.
o Email: [email protected]
o Twitter: twitter.com/USI_LeeBarnes
o LinkedIn: linkedin.com/in/leebarnes
o Blog: www.RaiseYourMQ.com
SOASTA
o Tom Chavez
o Sr. Product Evangelist
o SOASTA
o Email: [email protected]
o Twitter: @TomChavez
o LinkedIn: linkedin.com/in/tomchavez
