Webinar issues we_find_slideshare

Best Practices in Performance TestingWebinar Series

Issues Uncovered as Scale Increases

The Test Lab

Goals:• Isolate bugs in code• Optimize page performance• Plug memory leaks• Tune databases• Establish performance expectations

Limitations:• Production-quality hardware• IT support and resources• Bandwidth and network infrastructure• 3rd party application components• Time

The Staging Environment

Goals:• Verify capacity• Establish configurations• Stress available resources• Verify performance expectations

Limitations:• Availability / scheduling• Not full-scale• Excludes key components• Time

The Full Production Environment

Goals:• Test to and beyond expected limits • Verify entire infrastructure• Confirm load balancing and firewall integrity• Certify performance in normal and extreme conditions

Benefits:• All components• 3rd party apps• All integrations• Continuous • Global latencies• Reality

Performance isn’t One-Dimensional“You have a fever.” Now what?

User Experience

OPSQA

External web load

Performance is Multi-DimensionalFind problems fast when metrics are viewed in context, at scale

User Experience

Complex Web Applications : AJAX, HTML5, video, mobile, web services

Multi-Dimensional Views

External and Internal user load

Deployment Infrastructure

(Internal)

DeploymentInfrastructure

(External)Network User

Experience

Real-TimeEnd to End

Performance Data

In-Memory OLAP

Multi-Dimensional Views

ApplicationComponent &

Code

External and Internal user load

Extract Transport Load

The CloudTest platform is built for scalability and real-time identification of issues

Complex Web Applications : AJAX, HTML5, video, mobile, web services

Performance is Multi-DimensionalFind problems fast when metrics are viewed in context, at scale

Memory leaks

Scale of Test

Sta

ge

/ Te

am

Dev & Test

Extending Performance Beyond the LabDiscovering issues across teams, time and scale

Garbage collection

Inefficient database queriesSlow pages

Test Lab (a few thousand users)

Method-level tuning

Memory leaks

Debugging code

Database thread counts

Memory leaks

Scale of Test

Sta

ge

/ Te

am

Dev & Test


Release& Deploy

Inadequate server resources

Default configuration settings

Auto-scaling failures

Slow third-party plug-ins

Garbage collection



Test Lab Staging (thousands to 10’s of thousands)

Search technology limits

Method-level tuning

Max sockets exceeded Global latency variance

Security bottlenecks

Memory leaks

Configuration Setting & Sizing

s

Memory leaks

Scale of Test

Sta

ge

/ Te

am

Dev & Test


Release& Deploy

Network& Ops

CDN file placement

Load Balancer configuration

Network bandwidth

Network configuration

DNS routing

Inadequate server resources

Default configuration settings

Unbalanced web serversAuto-scaling failures

Latency between systems

Slow third-party plug-ins

Garbage collection



Conflict with other apps

Test Lab Staging Production (100% +++)

Search technology limits

Method-level tuning

Max sockets exceeded

Firewall max capacity

Global latency variance

Security bottlenecks

Memory leaks

Tuning for full-scale…and well beyond

CloudTest LiveReal-world Issue Resolution

Company Profile

Company Toy Manufacturer

Size Fortune 500

Project Profile

Environment Lab and Staging

Application Drupal (Apache, PHP, MySQL)

Location Private Hosted Cloud

Objective Identify Breaking Points

Project Length Five Days (non-consecutive)

StagingLab


Lab• First test cycle in the lab• 239 hits per second• 40.5 MBit/second bandwidth

• Some servers hitting 100% CPU• Bottleneck in throughput


Lab• Second test cycle in the lab• Higher concurrency target• Bandwidth stayed the same

• Some servers hitting 100% CPU• Performance problem over time• Load balancer flapping servers


Lab• Duration problem was crippling – could not recover


Staging• Bandwidth – ouch! • CPU at 100%


Staging

• 767 Home pages served at 650 megs total transfer.• That’s an 850kb page weight! Yikes!


Staging

• Poor throughput on the initial tests• CPU hits 100%• Bandwidth cap identified in the lab• Database connections leaking over time• Load balancer dropping servers

Mission Debriefing - Fortune 500 Toy Manufacturer

Lab

• Massive page weight on the home page• Application unresponsive at ~2000 users

Outcome• Customer happy with the breaking point identified• Optimized web assets in the home page• Offloaded assets to the CDN• Load balancer algorithm optimizations


Company Profile

Company Media and Broadcasting

Size Fortune 500

Project Profile

Environment Production

Application Java

Location Private Data Center

Objective 25,000 Concurrent

Project Length Three Days

Production


Production• Steadily increasing response times• Constant stream of errors (< 1%)• Variety of error types


Production

• One rogue page – streaming the TV episode


Production

• Connection reset errors starting the video stream• Entitlement license server calls failing


Production

Mission Debriefing - Fortune 500 Media Company

• Connection issues with CDN• One problem page• Massive bandwidth usage

Outcome

• Worked with CDN provider to reinforce certain geographies• Added capacity to the license server cluster


Company Profile

Company Advanced Technology

Size Fortune 50

Project Profile

Environment Production

Application Java (Struts), Oracle

Location Multiple Hosted Private Data Centers

Objective 172,000 Concurrent

Project Length Two Months

Production


• Recurring error pattern during scenario completion Production


Awesome outcome in Production!


Production

Mission Debriefing - Fortune 50 Technology Company

• Load balancer firmware bug with 2048 bit cert• Database issues with the login process

Outcome

• Worked with hardware LB manufacturer on a fix• Really low error rate• Successful launch!

Q&A

Please join us for the next webinar in this series:

Testing Mobile Applications

May 3rd at 10AM PDT, 1PM EDT

Register at: http://bit.ly/ezq3fP

Contact SOASTA:www.soasta.com/cloudtest/[email protected] us:

twitter.com/cloudtest

facebook.com/cloudtest

http://bit.ly/ezq3fP

http://bit.ly/ezq3fP

Webinar issues we_find_slideshare

Technology

Transcript of Webinar issues we_find_slideshare