Grand National 2015Mobile application performance insightsHow to get monitoring right with millions of users

Perfecto Mobile The world leader in mobile application continuous quality helping the world’s biggest brands to deliver better apps faster

“Perfecto Mobile currently has the strongest 3rd party position in the market” [ Thomas Murphy]

Intechnica• Vendor/Technology Independent• Enable Performance by Design• Provide Performance Management Services• Specialists in Agile Performance• Enable and deliver Performance Best Practice

Your presenters:

Larry HaigSenior Performance Consultant, Intechnicalarry.haig@intechnica.co.uk+44 (0)845 054 2381Contributing author: The Art of Application Performance Testing[Molyneaux, 2nd Ed 2014 Pub O’Reilly]

Amir RozenbergDirector, Product Management, Perfecto Mobileamirr@perfectomobile.com

Agenda

Grand National: Fast Facts

Our Findings

The Next Race

Best Practices

Demonstration

vv v

v v

Grand National 2015 – fast facts

• OpenBet record 19,300,000 bets within 24 hours– Peak 45,000 bets pm– 54,000,000 account transactions [UK population c63m (2011)]

• + 24% y-o-y– Average bet size is £8.00 – 63% of remote bets via mobile, – [up from 54% 2014]

• UK Transaction volume +12.6% y-o-y– (+83.6% vs typical April trading day)

• UK Transaction value + 20.3% y-o-y Refs: OpenBet; Worldpay

Survey & caveats• Focus on Mobile perspectives, end user/client side issues

– Android and iOS– Complementary testing:

• Mobile emulation / public carrier (4G) / webkit browser (m. sites)• Real device testing (native apps)

• Raceday & 2 day buildup• High vs low traffic• Performance and Quality of Service• Caveats:

– Snapshot testing – indicative, not absolute / continuous– High level monitoring only

• No device metric correlation – Real device test redundancy limited

• Native App availability not quoted

Example – race day errors log

Summary of key findings• Overall performance improving y-o-y, BUT• Still evidence of traffic related stress• All m. sites (and some native apps) showed some evidence of Quality of Service

issues– 85%+ likely to be revenue relevant– Brand impact

• m.sites:– iPad2 Tablet slower than Smartphone– 3rd party content source of failure– Lack of traffic resilience– Transaction payload highly correlated with performance

Summary – traffic & payload:

Summary – traffic & payload:

NATIVE MOBILE APPLICATIONS

Native App Response Times by Device- The Good

4.23s 2.59s 2.91s 8.06s

5.16s 9.32s 5.67s 1.48s

Web Emulation = 1.15s

Poll

v

Impact of 5 second delay in a revenue transaction

• I would call customer support• I would complain on social networks• I would transition to competitor application• I would try again

Poll Results

v

Impact of 5 second delay in a revenue transaction

I would call customer support

5%

I would complain on social networks

5%

I would transition to a competitor application

56%

I would try again35%

Native mobile applications – maximized revenue?

Mobile Advertising on app launch takes user attention

Application behavior in absence of network- opportunity to optimize experience

Native applications Panel performance – ‘Time to Bet’

Panel # Android iOS

10 5.0

7 8.9

8 5.1 3.4

9 5.9 1.5

13 10.2 5.0

< 5 sec

>5 - < 10 sec

> 10 secRaceday median (seconds)

Performance consistency (raceday):

0600hrs 11 AprilTran

sacti

on re

spon

se (s

econ

ds)

Example root cause analysis

0

10

20

30

40

50

60

70

80

90

100

0

50000

100000

150000

200000

250000

300000

350000

400000

CPU User (%)#Device CPU Total (%)#DeviceBytes In - WLAN#Device

Rapid find & fix - root cause analysis - drilldown

• Network wait: Client side processing

• Large JavaScript D/L: Repeated downloads

• Significant advertising/tracking/3rd party activity

Brand impact [race day errors]:

PLANNING FOR THE NEXT PEAK EVENT

Digital Experience Unhappy PathNew Use Cases

What will the next race look like?

Quality of Experience- Adopt the user “glass”

Server Architecture Delivery Chain Devices & User

3rd party

Device Emulation UX Timer App-level CPU Peak

iPhone 6 0.9s 1.15s 67%

iPhone 4S 0.9s 8.9s 85%

Typical Application Launch Time

Recommendation 1

Daily Weekly

Sensors (NFC, BT)

Events

Incoming (Call/SMS/Alert)

Location

Rotation

WiFi On/Off

Airplane mode

Dynamic Backend Switching

Light/Heavy Load

Device Stress

Restricted Permissions

Environments

Network Conditions

Sensor allocation

UI AssetsOS & App-level Vitals

Optimization Directions

Reporting

Rich Media QoE

KPIs

Network Traffic

Continuous Quality Lab

Performance TestingAdvanced Functional Testing

Recommendation 2

Test, Optimize & Get Ready

Recommendation 3

Alerting Drives Early Awareness (MTTK) &

correction (MTTR)Real DevicesCoverage

Geos/Scenarios/Carriers

Remain in control:Know early, resolve early, and keep winning!

Key suggestions for monitoring1. Understand customer devices, locations, trends2. Monitor appropriately and manage all digital delivery channels

– PC, m.– Native mobile applications– Other (in store kiosks, etc.)

3. Lean is fast – manage page weights in m. sites4. Audit & optimise client side/application components5. Explicitly monitor object rendering6. Understand device specific effects 7. Manage 3rd party inclusions

– SLAs, monitoring8. Plan ahead

– Understand traffic load effects ahead of time– Baseline key metrics

www.intechnica.co.ukwww.perfectomobile.com

Thank you.