Why Load Testing your Website from the Cloud Doesn’t WorkWhy Load Testing your Website from the Cloud Doesn’t Work

Imad Mouline - CTO, Compuware Gomez

Customer Expectations Are Rising For Web & Mobile Applications

47% of consumers expect a Web page to load in 2 seconds or less

58% of mobile phone users expect websites to load almost as quickly or faster on their mobile phone than their PC

The Impact of Web Performance on Page Abandonment

-8%

-25%

-10

-5

0

5

10

2 secs4 secs

6 secs 8 secs

Performance

improvement

(seconds)

-25%

-33%

-38%

-40

-35

-30

-25

-20

-15Percentage

change in

page

abandonment

Source: Gomez 2010 Study of 500 Million End-User Interactions Across 200+ Web Sites

The Browser Is Becoming THE Integration Platform

Number of hosts accessed directly by the browser, per user transaction, averaged across 3,000 companies 8.87

Local ISP3rd Party/

Cloud ServicesBrowsers

and devices UsersUsers

Web Servers

Load Balancers

Mobile Components

The Web Application Delivery Chain

The Challenge Of Delivering Quality Web Experiences

• Inconsistent geo performance• Bad performance under load• Blocking content delivery• Incorrect geo-targeted content

• Network peering problems

• Bandwidth throttling

• Poorly performing JavaScript

• Inconsistent CSS rendering

Systems management

tools: “OK”

…user is NOT happy

MajorISP

Mobile Carrier

Internet

Content DeliveryNetworks

Storage

App Servers

DB Servers

Mainframe

Mobile Components

Network

• Network peering problems

• Outages

• Configuration issues• Oversubscribed POP• Poor routing optimization• Low cache hit rate

throttling• Inconsistent

connectivity• Configuration errors

• Application design issues

• Code defects• Insufficient

infrastructure

• Network resource shortage

• Faulty content transcoding

• SMS routing / latency issues

CSS rendering• Browser/device

incompatibility• Page size too

big• Conflicting

HTML tag support

• Too many objects

• Content not optimized for device

• Low cache hit rate

Local ISP3rd Party/

Cloud ServicesBrowsers

and devices UsersUsers

Web Servers

Load Balancers

Mobile Components

Pinpointing Problems and Optimizing Performance

The Web Application Delivery Chain

Is it my Is it an Is it a Is it a

MajorISP

Mobile Carrier

Internet

Content DeliveryNetworks

Storage

App Servers

DB Servers

Mainframe

Mobile Components

Network

Is it my data

center?

Is it anISP or the Internet?

Is it a 3rd party

provider?

Is it a browser or

device?

Test Across the Entire Web Application Delivery Chain

The Web Application Delivery Chain

Local ISP3rd Party/

Cloud ServicesBrowsers

and devices UsersUsersLoad Balancers

Load Testing 1.0

Load Testing 1.5

Load Testing 2.0 (Gomez)

MajorISP

Mobile Carrier

Internet

Content DeliveryNetworks

Storage

Web Servers

App Servers

DB Servers

Mainframe

Mobile Components

Network

Local ISP3rd Party/

Cloud ServicesBrowsers

and devices UsersLoad Balancers

Company: Online presence for a popular TV show

• Following episodes of the TV show the web site sees high traffic spikes

• Goal was to achieve 1500 logins per minute

• Load tested DB to improve performance in anticipation of another traffic spike

Load Testing 1.0 Works… Some of the Time

MajorISP

Mobile Carrier

Internet

Content DeliveryNetworks

Storage

Web Servers

App Servers

DB Servers

Mainframe

Mobile Components

Network

• As users were added, the response time of step 3 (the login) climbed immediately

• The test bottlenecked at 160 logins per minute (Goal 1500)

Application Bottleneck Causes Response Time Issue

logins per minute (Goal 1500)

• But quickly dropped off as users received server errors

• New login query was not optimized and was bottlenecking the database servers’ CPUs

Summary:

•Problem found inside firewall

•After tuning- application performance

improved.

•New Bottleneck occurred 1300 logins

per minute.

•Bandwidth limit reached at 90 Mbps

Application Bottleneck – Re-test

•Problem found inside firewall

•Fixes made for application issue

•Retest shows second issue-bandwidth

•First test

•Second test

���� ����1.0 1.5 2.0

���� ����

���� ����1.0 1.5 2.0

���� ���� ����

Local ISP3rd Party/

Cloud ServicesBrowsers

and devices UsersLoad Balancers

Company: Online Gaming Site Testing a new rollout in support of a new sports se ason

• Support anticipated traffic increases• Load tested something using cloud and Last Mile to validate

performance for real users in new geographies.

Load Testing from the cloud misses the point… end users matter

MajorISP

Mobile Carrier

Internet

Content DeliveryNetworks

Storage

Web Servers

App Servers

DB Servers

Mainframe

Mobile Components

Network

View from the Cloud

• First 20 minutes Cloud testing shows acceptable performance

• After 2500 users Response time climbs Availability drops Error rate climbs

View from the Last Mile

• Last Mile shows different story

• Availability is terrible even at minimal load for real users

Summary:Cloud-only testing may give

misleading availability dataCloud starts with 100%

availability Less than 25% for the Last

Mile

1.0 1.5 2.0

����

Difference in browsers

• Varied commercial browsers show different response times

• Response time change as a function of load is dependent on the browser.

Summary:Understanding response

time under load requires the use of multiple browsers

1.0 1.5 2.0

����

Company : Regional Online News Source• Began testing for the election season• Goal was to validate overall performance focusing in 2 key regions

Local ISP3rd Party/

Cloud ServicesBrowsers and

devices UsersLoad Balancers

Load testing from the cloud ignores real locations

MajorISP

Mobile Carrier

Internet

Content DeliveryNetworks

Storage

Web Servers

App Servers

DB Servers

Mainframe

Balancers

Mobile Components

Network

Page response times stayed under 4 seconds, outside of one brief blip

There was only 1 page error and 11 errors total out of 60000+ transactions

Increase and hold load and not exceed response times of 4 seconds and Success Rate of 99%

No Performance Issues Detected From Data-Center

1.0 or 1.5 load testing shows tests passed

Last Mile Case Study: Primary Geographies

Key geographies for this customer are New York and Pennsylvania.

The response time never met the 4 second average goal

Summary:Last Mile shows goal not reachedCloud can’t detect the end user issue

Availability was Less than 99%

1.0 1.5 2.0

����

Company: Online Retailer• Several 3rd Parties now involved in serving up key content• Goal was to validate performance of entire applicat ion

Local ISP3rd Party/

Cloud ServicesBrowsers and

devices Users

Web

Load Balancers

Don’t trust third parties to perform…test the whole WADC

MajorISP

Mobile Carrier

Internet

Content DeliveryNetworks

Storage

Web Servers

App Servers

DB Servers

Mainframe

Mobile Components

Network

Understand 3rd Party Performance Under Load

Load Test Response Time Per Page During the course of a During the course of a

Load Test, Page 3 is Load Test, Page 3 is

identified as the culpritidentified as the culprit

Page Level Drill Down shows performance by HostSummary:3rd party vendor did not have capacityOnly component to fail

1.0 1.5 2.0

��������

Host & Object Level Host & Object Level

TrendingTrending•• As the Load test As the Load test

progressed, the 3progressed, the 3rdrd

party search tool party search tool

contributed to over contributed to over

90% of the response 90% of the response

time time

Local ISP3rd Party/

Cloud ServicesBrowsers and

devices Users

Web

Load Balancers

Company: International Hotel chain• New reservations system rollout• New global server load balancing rolled out across multiple data

centers• Validate that system works globally

The Internet is global – where your customers are matters

MajorISP

Mobile Carrier

Internet

Content DeliveryNetworks

Storage

Web Servers

App Servers

DB Servers

Mainframe

Mobile Components

Network

Major Hotel Reservation System unavailable in 4 countries

0% availability in UK, Germany, Japan

99%+ availability in US, Canada, France

Summary:Internal U.S. test looked good

Distributed testing fails in key locations.

1.0 1.5 2.0

����?

Company: eRetailer fashion • 100% virtual store• Daily sales spike driving 90% of revenue stream

Local ISP3rd Party/

Cloud ServicesBrowsers and

devices Users

Web

Load Balancers

Load Testing 2.0 shows you what your customer sees

MajorISP

Mobile Carrier

Internet

Content DeliveryNetworks

Storage

Web Servers

App Servers

DB Servers

Mainframe

Mobile Components

Network

Load Testing with multiple browsers shows discrepancies

Availability vastly different between browsers

Comparison of Performance across the country - Firefox

Using Firefox browser – shows 100% availability for website

Wide variations in response time based on geography

Comparison of Performance across the country – IE

IE Browser : shows under 12 percent availability

Availability and performance tied to geography

Page Element Downloads: IE Versus Firefox- Order Varies

Summary:Only real locations can show what really happens.

1.0 1.5 2.0

����

Explanation

•Third party ad provider

modifying the DOM

•Depending on the load order

of the third party the java

script in the ad would

overwrite the DOM but only

on IE

Gomez Load Testing: Most Accurate Load Test for User Experience

HTTP : Behind the

FirewallHTTP : Data Centers Browser : Data

Centers

Real World

Desktops

Last Mile

Traditional

Client/

Server TestDatacenter Testing

Accuracy of End-User

Response Time

Incomplete Incomplete Indicative Most Accurate

Accuracy of

Application Availability

Invalid Indicative Indicative Most Accurate

Load Test 1.0 Load Test 2.0

Only Gomez SpansOnly Gomez Spans

Load Test 1.5

Application Availability

Ability to drive large

load volume

Yes-requires

substantial

hardware

Best Better Good

Understand CDN

Impact

No Misleading Misleading Most

Accurate

Understand 3rd Party

(ads, feeds, etc…)

No Minimal Some Most

Accurate

Realistic object

download

No No

Static Only

Yes Yes

Visibility behind the

firewall

Best Good Good Good

Questions

Increased conversions 10%

Reduced homepage load time from 11.3 seconds to 3.4 seconds

Improved page load times 23%

Saved 50%+ in staff and fees

Gomez Customers Enjoy Measurable Benefits

Reduced downtime 45%

Reduced seven-step transaction time by 50%

Validated decision to consolidate three data centers

Achieved under 3 second response time and 99%+ availability

Saved 50%+ in staff and fees

For more information visit Gomez.com or contact us at +1 781.778.2700