©2 016 S OA S TA , A l l r i gh t s r es e rv ed .

Tom Chavez @tomchavezMike Ostenberg @mikeostenberg

How to Load Test for your Mobile Peak

©2 016 S OA S TA , A l l r i gh t s r es e rv ed .

• How important is Mobile?• Mobile Web and Apps• The need for load testing• Measure everything• What to test?• Test early and often – shift left and right• How to test?

Importanceof Mobile

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 4

Mobile is growing!

• In 2015, mobile commerce

accounted for $115B worldwide

• In 2016, mobile commerce

is project to reach $142B

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 5

How is your site now?

• mCommerce is globalo US is not the benchmark

• 39% of mobile users

are unhappy with their online experienceso Top reasons: slow pages and site freezes/crashes

• 57% have experienced problems when using mobile

• 3 out of 4 consider mobile slower than desktop

• 46% will not return to a site that performs poorly

• 27% of smartphone users exceed data cap every month

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 6

When is your Peak Season?

• Peak is year-roundo Super Bowl Sunday – 110 million viewers

o Bi-weekly or monthly payroll processing

o Valentine’s Day, President’s Day, Mother’s Day, Father’s Day

o Quarterly processing and filing

o U.S. Tax filing

o Spring, Summer, Fall & Winter season sales

o July 4th, Christmas in July

o Back to School

o Olympics

o Halloween We are here

o Election Day

o Singles Day – world’s biggest shopping day, $9B in 12 hours

o Black Friday, Cyber Monday and Holidays

Back to top to repeat again next year!

Mobile Weband Mobile Apps

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 8

Mobile Web

Responsive web site WebSite

• Desktop Users

• Mobile Users

• Same or similar contento Adapts to different

screen sizes

Mobile optimized

• “m-dot” site for mobile

• Custom built for mobile devices

All have html, images, JavaScript, css, plus third-party content

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 9

Native App

• Android – Java

• iOS – Objective C/Swift

• Mobile app frameworko Same source code

Hybrid App

• Mix of native code + html

All have backend calls for login/authentication, account

settings, database calls, transactions, plus third-party SDK

calls

Mobile Apps

WebSite/MobileBackend

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 10

The Need for Load Testing

• Everyone runs load testing on their website, right?o Load and Performance testing is often not done or too late

o Black Friday, Cyber Monday, Holiday failures

• What’s the need for mobile web load testing?o Mobile is growing faster than desktop

o Responsive sites and mobile sites deliver different content to

mobile devices over mobile network

• What about mobile app load testing?o Mobile app back ends serve different content to mobile apps

o Remember Pokemon Go? It wasn’t the app that failed.

Measure Everything

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 12

Measure Real and Virtual Users

• Real Users (RUM) provide deep data about your visitorso No other way to know everything about your users on your site

and in your app

SOASTA mPulse collects and analyzes RUM data from your web

and mobile app users

• Virtual Users tell you how your site or app will do at scaleo You can’t get a million friends to visit your site or app at the same

time!

SOASTA CloudTest lets you easily build and execute load tests at

scales from hundreds to millions of virtual users

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 13

Other Things to Measure with RUM

• Session pathso How are your users going through your site

• Mobile App Performanceo How are your mobile apps performing?

o What are users doing in your mobile apps?

• Revenueo How much money are you making from each user and where?

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 14

Average Performance

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 15

Performance by Device

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 16

Creating a Test Plan – Peak Load

What was the busiest Day, Hour, Minute, Second?

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 17

Creating a Web Test Plan – Page Flows

What were most popular page flows with think times? Which APIs?

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 18

Network Emulation

• Testing to match realistic situationso Users are not all from one location on a fast network

o Packets get delayed, lost, re-ordered

• Apps perform differently as network conditions worseno Reduced throughput

o Timeouts occur, connections are lost

o Transactions take longer, longer sessions

Adding real network modeling to load and performance testing

Test Earlyand Often

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 20

Don’t Wait Til the Last Minute to Test!

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 21

Test Left and Test Right, even into Production

• Performance Testing in developmento Set performance SLAs for new code and test in development,

same as unit and functional tests which code must pass

• Performance Testing in stagingo Higher loads in near-production systems

• Performance Testing in Productiono Full scale load tests to expected load and beyond

o Find weak and stress points and prioritize fixes into project

Performance Testing is Continuous

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 22

Test in Production

Some issues will only be found at scale load on production systems

Don’t forget your third party providers: CDNs, external scripts

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 23

Measure Mobile App Performance Under Load

All mobile apps perform well when there is no load

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 24

Automate Your Testing

Use Continuous Integration tools to automate testing

Architectureand Demo

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 26

Now That the Test is Over

• Analyze results for future areas to addresso Weak or stress areas that aren’t likely this time but …

• Identify third parties that were affected by loado Coordinate with suppliers to address issues

o Align schedules or at least be aware of their changes

Even if the test passed, you’re not done

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 27

Issues Will Happen

• “I am 100% confident that everything we tested will work

just fine.” – Nordstrom Performance Leadero When loads are different than what you modeled, may have

problems

o E.g. At Nordstrom, during a recent anniversary sale, mobile

sign-ins were 9x greater than anticipated

o Used RUM to get to the root cause of the issue

Realize that Issues Will Happen Where You Can’t Predict Them

Q & AResourcesNext Steps

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 29

CloudTest – Load Testing for mobile web, apps

• Load testing from development to productiono Continuous testing at any scale, on-prem or in the cloud

• Real-time analytics with customizable dashboardso Safely test in production with full visibility

• Run tests live or automaticallyo Full hands-on control or hands-off automation

• Full end-to-end viewo Front-end performance to back-end systems

Fast, scalable, affordable load testing for performance optimization

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 30

SOASTA Digital Performance Management Platform

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 31

Questions, Resources & Next Steps

• Holiday Readinesso soasta.com/holiday-readiness

• The Performance Beacono soasta.com/blogo soasta.com/blog/2015-popular-posts-web-performance

• Webinarso soasta.com/webinars

o Load Testing for Peak; RUM-Based Testing

• CloudTest on-Demando soasta.com/cloudtest: SOASTA will build & run tests for you

Resources

©2 016 S OA S TA , A l l r i gh t s r es e rv ed . 32

“The leader in performance analytics.”

www.soasta.com

SOASTA