Production Performance Testing in the Cloud
25
TK PM Tutorial 4/30/13 1:00PM Production Performance Testing in the Cloud Presented by: Dan Bartow SOASTA, Inc. Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
-
Upload
techwellpresentations -
Category
Technology
-
view
72 -
download
3
description
Testing in production for online applications has evolved into a critical component of successful performance testing strategies. Dan Bartow explains the fundamentals of cloud computing, its application to full-scale performance validation, and the practices and techniques needed to design and execute a successful testing-in-production strategy. Drawing on his experiences, Dan describes the methodology he has used for testing numerous online applications in a production environment with minimal disruption. He explains how to create a performance testing strategy to give your team critical data about how your online application performs and scales. Learn how to create a robust lab-to-production ecosystem that delivers the answers about what will happen when peak traffic hits your site. Take back practical approaches to mitigate the three most common problems—security, test data, and potential live customer impact—that arise when embarking on testing in production.
Transcript of Production Performance Testing in the Cloud
TK PM Tutorial
4/30/13 1:00PM
Production Performance
Testing in the Cloud
Presented by:
Dan Bartow
SOASTA, Inc.
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
Dan Bartow
At SOASTA Dan Bartow is vice president of product management. Prior to joining SOASTA, Dan was senior manager of engineering at Intuit where his team was responsible for the speed and stability of TurboTax Online. During the past decade he has been responsible for the performance of websites for leading consumer brands. Dan has set multiple industry precedents including launching the world's largest stateful JBoss cluster and using more than 2,000 cloud computing cores to generate load against a live website. He is credited with authoring the industry’s first testing in production methodology. A frequent industry presenter, Dan has spoken at leading testing and cloud computing conferences.
1
Production Performance
Testing in the Cloud
Dan Bartow
SOASTA
Objectives
1. Explain the fundamentals of cloud computing and its application to full-
scale performance validation
2. Describe how to create a performance testing strategy to give your team
critical data about how your online application performs and scales
3. Learn how to create a robust lab-to-production ecosystem that delivers
the answers about what will happen when peak traffic hits your
application
4. Share the best practices and techniques needed to design and execute
a successful testing-in-production strategy
5. Give practical approaches to mitigate the three most common problems -
security, test data, and live customer impact - that arise when designing
production test plans
2
Agenda
4. Production Performance TestingSecurity
Test data in production
Potential live customer impact
Discipline and rigor
Most common bottlenecks
5. Case Studies and DiscussionTurboTax Online
Myspace
Green Button
1. Performance EngineeringMethodology
Categorization of Sites by Traffic
Effort vs. Gain
2. Cloud Computing OverviewWhat is cloud computing?
The killer app for cloud
Does it work for testing?
3. Lab to Launch EcosystemEssential components - lab and live
Evolution of the testing cycle
Designing good test plans
Monitoring
About the Speaker
SOASTAVP Product Management
CloudTest Evangelist
IntuitSr. Manger, Engineering
TurboTax Online and E-com
ATGSr. Deployment Engineer
American Airlines, Best Buy, Target, Turbotax Online, QuickenOnline, MySpace,
Dennys, Dominos, Mattel, Hallmark, FAA, US Army, AT&T Wireless, Alcatel,
Newsweek, Oprah, NeimanMarcus, SBC, Plantronics, Kodak, JCrew, Cingular,
Newell Rubbermaid
Work
3
Poor performance.
How Deep Should We Go?
4
http://www.internetretailer.com/2011/11/28/thanksgiving-weekend-brings-multiple-site-headaches
http://www.internetretailer.com/2011/11/26/outage-hits-victoriassecretcom-black-friday
Black Friday | Cyber Monday - 2011
No Outages in the Top Ten US
Online Retailers!
5
At Least Six of the Top Ten Online
Retailers in the US
Tested Their Apps – In Production
– from the Cloud
In 2011
http://www.stores.org/2011/Top-100-Retailers
Wal-Mart
Kroger
Target
Walgreen
The Home Depot
Costco
CVS Caremark
Lowe’s
Best BuySears Holdings
DefineDefine
IntegrateIntegrate
StrategyStrategy ImplementationImplementation ExecutionExecution MeasurementMeasurement
SOASTA CloudTest Methodology
PeoplePeople
ProcessesProcesses DesignDesign
DefineDefine
TestTest
AssessAssess
AnalyzeAnalyze
AdjustAdjust
Methodology
6
Defining Performance
Capacity?
Page Speed?
Reliability?
Scalability?
Transaction Speed?
Code Execution Speed?
What does it mean?
Defining Performance
1. Capacity
2. Customer Experience
3. Reliability
4. Scalability
5. Transaction
6. Code Level
7
Performance Matters
Capacity
8
Categorizing Sites by Traffic
Categorizing Sites by Traffic
9
Categorizing Sites by Traffic
*SOURCE: http://royal.pingdom.com/2010/07/05/what-it-takes-to-be-a-top-100-website-
charts/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+RoyalPingdom+%28Royal+Pingdom%29
The top 10 websites get 42% of the visitors to the top 100, and 21% of the visitors
to the top 1,000*
The top 100 get 50% of the visitors to the top 1,000. I.e. the top 100 together get
as many visitors as the following 900 websites counted together*
If you want to break into the top 100 websites, you’ll need at least 22 million
visitors per month. If you “only” aim for the top 1,000, you’ll need at least 4.1
million visitors per month*
Gee, that can’t be so hard, can it…?
Categorizing Sites by Traffic
Top 10 – Mega Sites*
1. Google
2. Facebook
3. Youtube
4. Yahoo!
5. Windows Live
6. Baidu.com
7. Wikipedia
8. Blogger.com
9. Twitter
10. QQ.com
*SOURCE: Alexa Top 500 Worldwide - http://www.alexa.com/topsites
10
Categorizing Sites by Traffic
Top 10-500 – Massive Sites
MSN
Amazon.com
Wordpress
eBay
Craigslist
MySpace
Flickr
Apple
Paypal
Mashable
Skype
Amazon WS
Yelp
WSJ
Match.com
Walmart
Salesforce
AT&T
*SOURCE: Alexa Top 500 Worldwide - http://www.alexa.com/topsites
Categorizing Sites by Traffic
Top 500 – 2000 XL Sites
Orbitz
Travelocity
NBA
United
AmericanAirlines
Marriott
NHL
*SOURCE: Alexa - http://www.alexa.com/
Top 2000-10000 Large Sites
Dominos
Avis
Mint
WhiteHouse.gov
11
The Point?
Where should we focus the performance engineering efforts?
Testing in Production
Most common problems during peak
traffic periods
Load Balancers
Bandwidth
Database Scale
12
Production Performance Testing
Anyone can do it - the NYSE did production performance
testing in the eighties
Anyone can do it - the NYSE did production performance
testing in the eighties
Some enterprises have teams dedicated
to production performance testing
Some enterprises have teams dedicated
to production performance testing
Most Commonly Used
13
Typical Approach to Testing
Extrapolation
Test in a lab at a smaller scale and then useA.
Extrapolation
Extrapolation
Extrapolation
Extrapolation
Extrapolation to Prod Breaks Down
Invisible walls can only be identified out in the production environment and through testing at scale – as load and environment changes, so do problems
Load
Environment
14
Thank You, Public Cloud!
Some Cloud Providers and Locations
GoGrid – San Francisco
RackSpace – London, Dallas, Chicago
Amazon EC2 – Arlington, San Francisco, Portland, Dublin, Japan, Singapore
IBM BlueCloud – Toronto, Denver, Raleigh
Microsoft Azure – Chicago, San Antonio, Amsterdam, Dublin, China, Singapore
15
Cloud Computing
Q: What is Cloud Computing?
A: Access to affordable infinite compute power. Accessibility, Availability and Affordability.
SAAS
PAAS
IAAS
A killer application (commonly shortened to killer app), in the jargon of technologists, has
been used to refer to any computer program that is so necessary or desirable that it proves
the core value of some larger technology…*
*source: wikipedia
Lab to Launch EcosystemEssential components – lab and live
16
Lab to Launch EcosystemEvolution of the testing cycle
Lab Stage Production
1,500
10,000
100,000
Lab to Launch EcosystemDesigning good test plans
Heat Map
Scenario Guide
Test Plan Guide
17
Lab to Launch EcosystemMonitoring
What
Where
How
Production Performance Testing
Security
Test data in production
Potential live customer impact
Tools and techniques
18
Production Performance TestingDiscipline and rigor
The Play Book
Production Performance TestingMost common bottlenecks
19
Practice #1
Test accounts in production
Practice #2
Place orders in production
20
Practice #3
Test on the live site
Practice #4
Test the top 6-8 critical transaction flows
21
Practice #5
Don’t test from the same cloud you’re hosted in
Case Studies
Source: http://www.soasta.com/customers/financial/ss_turbotax/
22
Case Studies
Source: http://highscalability.com/blog/2010/3/4/how-myspace-tested-their-live-site-with-1-million-concurrent.html
Case Studies
Source: http://www.greenbutton.net/blog#post349
23
Contact Information
Dan BartowVP Product Management
CloudTest Evangelist
SOASTA Inc.
Email: [email protected]
Twitter: www.twitter.com/PerfDan
LinkedIn: www.linkedin.com/in/danbartow/
Blog: www.soasta.com
