The Cloud A game changer to test, at scale and in production,

SOA based web and mobile applications. Fred Beringer – VP Business Development Europe – SOASTA

International SOA Conference 2011- Rome

PERFORMANCE

matters more than ever before

Social media is highly viral and drives massive amount of

traffic.

#1

500 million active users spending 700 billion minutes per month

13 million hours uploaded in 2010

2 billion video viewed per day

106 million accounts

10 billion tweets in 2010

Your new marketing mix

Web Architecture is complex

#2

CDN Content

Switch Ad Server

Analytics

Shopping Cart CMS Content

Search Session Info

Databases

Load balancers

Media Server

Payment Server

Inside and outside the firewall

Application servers

Content is king

#3

1980

1992

1996

2011

Access devices explosion

#4

Different performance on each device

HIGHER EXPECTACTIONS

#5

of consumers say speed is more important than functionality 39%

Source: Forrester Research

Expect a web page to load in two seconds or less 47%

Source: Forrester Research

Will abandon a web page if it takes more than three seconds

to load 40%

Source: Forrester Research

Low performance and unreliability brings

MASSIVE BUSINESS IMPACT

#6

PayPal down 45 minutes in 2009 $45 Millions lost revenue

Amazon loses 1% of their sales for each 100ms of latency

Google loses 20% of search traffic every 500ms of additional latency

60% of companies admit that they do not have any performance management processes in place or what they have is ineffective.

DynaTrace study- 2009

50%

Of companies having performance management processes admit that they are doing it only in reactive way when problem occurs.

DynaTrace study- 2009

50% DynaTrace study- 2009

of application problems are found in production – when those problems have impacted end users and are most expensive to resolve.

What is required today to test effectively

performance of web applications?

SCALE #1

Real time analysis

of performance data

#2

SPEED

#3

AFFORDABILITY #4

Ready for some pain?

Traditional Performance Testing doesn’t work anymore

2006

PERFORMANCE TESTING

REVOLUTION

“ The massive resources of the cloud have changed the game for performance testing. Cloud Testing enables organizations to test their consumer facing Web and mobile applications fast, affordably and at scale. There’s now no excuse for the type of Web site disasters we’ve seen in the past that cost businesses millions in revenues and perhaps more in reputation. ”

Tom Lounibos CEO SOASTA

Cloud Testing

What does it bring us?

Affordable platform for infinite load

#1

Billions possible points of failure

The new testing challenge

Real-time analysis brings wisdom

#2

Global and realistic traffic

#3

Where is the needle?

#4

In your code

In your infrastructure

Outside your firewall

s

Memory leaks

Scale of Test

Sta

ge /

Team

Dev & Test

Extending Performance Beyond the Lab Discovering Issues across teams, time and scale

Release & Deploy

Network & Ops

CDN file placement

Load Balancer configuration

Network bandwidth

Network configuration

DNS routing

Inadequate server resources

Default configuration settings

Unbalanced web servers Auto-scaling failures

Latency between systems

Slow third-party plug-ins

Garbage collection

Database thread counts

Inefficient database queries Slow pages

Conflict with other apps

Test Lab (<10k Vus) Staging (10-100k) Production (> 100k)

Search technology limits

Method-level tuning

Max sockets exceeded

Firewall max capacity

Global latency variance

Security bottlenecks

Memory leaks

CloudTest Live Real-world Issue Resolution

Company Profile Company Toy Manufacturer

Size Fortune 500

Project Profile Environment Lab and Staging

Application Drupal (Apache, PHP, MySQL)

Location Private Hosted Cloud

Objective Identify Breaking Points

Project Length Five Days (non-consecutive)

Staging Lab

Lab

•  First test cycle in the lab •  239 hits per second •  40.5 MBit/second bandwidth

•  Some servers hitting 100% CPU •  Bottleneck in throughput

CloudTest Live Real-world Issue Resolution

Lab

•  Second test cycle in the lab •  Higher concurrency target •  Bandwidth stayed the same

•  Some servers hitting 100% CPU •  Performance problem over time •  Load balancer flapping servers

CloudTest Live Real-world Issue Resolution

Staging

•  Bandwidth (238Mbits/s) – ouch! •  CPU at 100%

CloudTest Live Real-world Issue Resolution

Staging

•  767 Home pages served at 650 megs total transfer. •  That’s an 850kb page weight! Yikes!

CloudTest Live Real-world Issue Resolution

•  Poor throughput on the initial tests •  CPU hits 100% •  Bandwidth cap identified in the lab •  Database connections leaking over time •  Load balancer dropping servers

•  Massive page weight on the home page •  Application unresponsive at ~2000 users

•  Customer happy with the breaking point identified •  Optimized web assets in the home page •  Offloaded assets to the CDN •  Load balancer algorithm optimizations

Staging

Lab

CloudTest Live Real-world Issue Resolution

Company Profile Company Advanced Technology

Size Fortune 50

Project Profile Environment Production

Application Java (Struts), Oracle

Location Multiple Hosted Private Data Centers

Objective 172,000 Concurrent

Project Length Two Months

Production

CloudTest Live Real-world Issue Resolution

Recurring error pattern during scenario completion

Production

CloudTest Live Real-world Issue Resolution

Awesome outcome in Production!

CloudTest Live Real-world Issue Resolution

•  Load balancer firmware bug with 2048 bit cert •  Database issues with the login process

•  Worked with hardware LB manufacturer on a fix •  Really low error rate •  Successful launch!

Production

CloudTest Live Real-world Issue Resolution

AGILE WORLD Fit for an #5

Real-time collaboration

Continuous performance testing

Cloud Testing helps companies

•  Every day at noon is a holiday spike in traffic.

•  Pages must render to the user in under 1 second.

•  50% of all revenue from Gilt sales comes in less than 1 hour after sale starts.

Cloud Testing helps dolls

Cloud Testing helps taxes

Cloud Testing helps phones

CHALLENGES Testing SOA in the cloud

It gets crowded

Loose SLAs

Harder to predict

(slightly)Different strategy

Same challenges • Services with no user interface • Data driven • Services outside the organization/cloud • End-to-end scenario • Test ownership

Same Bottom-up performance testing approach • Individual services • Set (integration) of services • End-to-end

Deal with Cloud unpredictability • Varying performance encourage continuous performance testing/monitoring. • Infrastructure upgrade could impact performance. • Expect to baseline on premise application and run benchmark in the cloud

after migration. • Test the full bottom-up cycle for increased confidence.

Let’s see how this website performs!

Questions/Discussion

The Cloud A game changer for web performance testing

fberinger@soasta.com

twitter.com/fredberinger

http://www.fredberinger.com

Photos Credits: Slide 2: http://www.sxc.hu/photo/1219246 Slide 18, 19, 20: http://www.sxc.hu/photo/1267744 Slide 21: http://www.sxc.hu/photo/1181194 Slide 22,23,24: http://www.sxc.hu/photo/396185 Slide 25: http://www.flickr.com/photos/tsoumplekas/4495778365/ Slide 32: http://www.sxc.hu/photo/1160562 Slide 34: http://www.sxc.hu/photo/1331143 Slide 46: http://www.sxc.hu/photo/995000 Slide 48: http://www.sxc.hu/photo/560250 Slide 54: http://www.sxc.hu/photo/561622 Slide 59: http://www.flickr.com/photos/caitlinator/4006198703/ Slide 60: http://www.sxc.hu/photo/1221951 Slide 62: http://www.sxc.hu/photo/1019388