Four Best Practices for Modern Performance Testing

© 2014 SOASTA. All rights reserved. April 18, 2023 1CONFIDENTIAL – Not for Distribution

4 Best Practices for Mobile Performance TestingStep 6 of the 7 Steps of Pragmatic Mobile Testing

654321


4 Best Practices for Mobile Performance TestingPresentors

Lee Barnes – Founder and CTO of Utopia Solutions

Tom Chavez– SOASTA Sr. Product Evangelist


o Device/OS diversityo Network variabilityo Device performanceo User location and volume

Step 6 – Know your mobile performanceAgenda

© Utopia Solutions

Best Practices for Performance Testing Mobile Apps

Lee Barnes, CTOUtopia Solutions

© Utopia Solutions

Apology

I don’t believe in “best practices”…

I do believe in guidelines and better approaches for a given situation

© Utopia Solutions6

Key Questions

How is mobile different?

How can I adapt?

Why is performance of mobile important?


Why Should I Care?

M-Commerce

66%OF SHOPPERSABAND

ONTRANSACTIONS

5 OUT OF 10BECAUSE OF

POORPERFORMANCE

1DELAY

SECOND 7% DROP IN

CONVERSIONS

equals a

2013 Mobile Commerce Insights study conducted by Jumio


Still Not Convinced?

2013-2014 World Quality Report


Evolution of Performance Testing

1990 2000 2010

2-TierClient Server Web Mobile

Challenges

State of the Practice

Testing Large Loads

Complex / Integrated

Systems

ProprietaryProtocols

Skilled Resources

In-Lab Testing

ToolMaturation

Open SourceTools

Specialized Resources

Cloud-based Testing

© Utopia Solutions

Mobile Challenges & Adaptations


Mobile Challenges – User Access

Native App Mobile Site Full / ResponsiveSite

Your users have options…


Assess the User and Load Profile

1. Determine the user profile

• Business processes• Native app, mobile site, full site• Preferred browser• Network conditions• Geographic location

2. Determine the load profile

• Volume of users by business process and location

• Volume of users by access type• Frequency / throughput

Incorporate constant feedback from application logs and other monitoring solutions


Mobile Challenges - Network

Mobile User Connected User

3G 4G InATunnelG

Network bandwidth and quality affect both end user experience and system load


Assess Impact of Network Variability

1. Assess baseline performance (single user)

Observe UX with real devices behind network virtualization solutions

2. Assess performance under load

Put system under virtual user load

Observe UX with real devices behind network virtualization solutions

NV

Virtual Users

NV

RealDevices


Network Virtualization Tools

Single User (Device and/or Emulator)• Apple Network Link Conditioner• Android Emulator• Charles Proxy

Multiple Virtual Users• Charles Proxy


Mobile Challenges – User Location and Volume

A high volume of global users is difficult to reproduce in a lab…


Utilize Cloud-based Testing Solutions

• Achieve production level loads• Test entire infrastructure (vs.

behind the firewall)

• Distribute load geographically• Assess user impact on real

devices


Mobile Challenges – Device Performance

Measuring System / Network Metrics is Not Enough…

Memory

CPU

Battery Drain

Graphics

I / O


Measuring Device Performance

Critical device metrics:• CPU• Memory• Battery• Storage

Mobile development platforms• Android DDMS• iOS Instruments

Mobile test automation tools• TouchTest (SOASTA)Mobile lab / cloud solutions• deviceConnect (Mobile Labs)• Remote TestKit (NTT Resonant)

Capture metrics while exercising app

Include appropriate devices / configurations

Incorporate automation for consistency

Device Performance Measurement Solutions


Summary

Key takeaways…• Mobile users are not the same as

connected users• Performance is more than backend /

network• Push performance analysis to the left

in the development cycle



Getting Started: Mobile Readiness Scorecard

o CI and Mobile Lab Scorecard

• 2 days

• Evaluate People, Process, Products, CI

• TouchTest and CI POC with customer tests

• Same mobile automation engineers that have implemented mobile test automation for Fortune 5 and Fortune 10 retailers



Presentors

UTOPIA

o Lee Barnes

o Founder and CTO

o Utopia Solutions, Inc.

o Email:[email protected]

o Twitter: twitter.com/USI_LeeBarnes

o LinkedIn: linkedin.com/in/leebarnes

o Blog:

www.RaiseYourMQ.com

SOASTA

o Tom Chavez

o Sr. Product Evangelist

o SOASTA

o Email:[email protected]

o Twitter:@TomChavez

o LinkedIn: linkedin.com/in/tomchavez

Four Best Practices for Modern Performance Testing

Documents

Transcript of Four Best Practices for Modern Performance Testing