Copyright © 2014 SOASTA, Inc. - All rights reserved.Webinar
andPresent
Automating the hard parts, made easier
2Copyright © 2014 SOASTA, Inc. - All rights reserved.
Continuous Load Testing
TODAY’S PRESENTERS
Mark Prichard: CloudBees, Dir. Product ManagementMike Ostenberg: SOASTA, Dir. Systems EngineeringBrad Johnson: SOASTA, VP Product
TWO COMPANIES, A SINGLE SOLUTION
- the leader in Mobile and Cloud Testing
- the home of Jenkins in the Cloud
Together, we offer a complete cloud-based service:
Continuous Delivery for Web & Mobile
4Copyright © 2014 SOASTA, Inc. - All rights reserved.
• Why should we load test earlier? • Building tests and preparing the test environment• Connecting your load tests to your CI (Jenkins)• Establishing and managing to a performance
baseline• Q&A
Questions: Submit in the question box during event
7Copyright © 2014 SOASTA, Inc. - All rights reserved.Copyright © 2014 SOASTA, Inc. - All rights reserved.
8Copyright © 2014 SOASTA, Inc. - All rights reserved.
“Because it’s hard.”oTools require coding skills we don’t have (C, VB?)oWe don’t have a scalable test environmentoWe don’t have hardware for load generationoIt’s a different tooloThere are too many other tests to do
9Copyright © 2014 SOASTA, Inc. - All rights reserved.
“Because it’s hard.”oTools require coding skills we don’t have (C, VB?)oWe don’t have a scalable test environmentoWe don’t have hardware for load generationoIt’s a different tooloThere are too many other tests to do
“Because it takes too long.”oScripting takes foreveroResults are slowoFast setup of environments is impossibleoWe’re dependent on IT or other teamsoWe’re too agile
10Copyright © 2014 SOASTA, Inc. - All rights reserved.
“Because it’s hard.”oTools require coding skills we don’t have (C, VB?)oWe don’t have a scalable test environmentoWe don’t have hardware for load generationoIt’s a different tooloThere are too many other tests to do
“Because it takes too long.”oScripting takes foreveroResults are slowoFast setup of environments is impossibleoWe’re dependent on IT or other teamsoWe’re too agile
“Because no one ever told us to.”oThere are no “performance coverage” requirementsoThere isn’t a performance baseline for referenceoThere isn’t a defined performance targetoNo one is managing to a performance metric
11Copyright © 2014 SOASTA, Inc. - All rights reserved.
“Because it’s hard.”oTools require coding skills we don’t have (C, VB?)oWe don’t have a scalable test environmentoWe don’t have hardware for load generationoIt’s a different tooloThere are too many other tests to do
“Because it takes too long.”oScripting takes foreveroResults are slowoFast setup of environments is impossibleoWe’re dependent on IT or other teamsoWe’re too agile
“Because no one ever told us to.”oThere are no “performance coverage” requirementsoThere isn’t a performance baseline for referenceoThere isn’t a defined performance targetoNo one is managing to a performance metric
Until Now.
13Copyright © 2014 SOASTA, Inc. - All rights reserved.
Iteration 1
Automated unit tests
The Issue:
• High expectations from tests run with diminishing time windows
• Code-level performance bugs found late set entire project back
• Separate team and schedule conflict with delivery goals
• Major issues escape to production
Iteration 2
Automated unit & functional tests
Iteration 3
Regression test suite
Iteration n
Regression test suiteLoad & Performance Testing
Deploy
Issues?
14Copyright © 2014 SOASTA, Inc. - All rights reserved.
Iteration 1Automated unit tests
Deploy
Production perf tests
Iteration 2Automated unit & functional tests
Iteration 3Regression test suite
Incl:
Iteration nRegression test suite
Incl: End to End Production Scale Test
Small perf test Small perf testsPerf smoke tests
Perf smoke tests
The Solution:
• Run incremental performance tests automatically with each iteration
• Establish a trend line for performance early
• Developers find code and architecture issues early
• Performance teams focuse on end-to-end infrastructure and user related testing
• Minimize production escapes and implement production testing
Development/Test team Performance/Ops teamDevOps
15Copyright © 2014 SOASTA, Inc. - All rights reserved.
I JUST MADE A CODE CHANGE
CloudTest runs load tests via Jenkins
Daily (or more frequent) updates keep teams “performance focused”
Identify bottlenecks
& inform teams, just
like unit and functional
tests in Jenkins
Jenkins launches suitable cloud-based test environment
Manage with a new metric!
o#1 OSS CI server
oWritten in Java
oHighly Extensible
oAbout 8 years old
oEasy to install/use
oExtensible via 1000+ plugins
oWidely adopted
o 70K+ installations
oCloudBees knows Jenkins
oOn-premise, cloud and hybrid
oExpert support
oEnterprise plugins
Have you met Jenkins?
oLet developers focus on developing
o Leave building, packaging, testing, distributing to a computer
oMake the right thing the easiest thing – don’t think about it
oMany, widely-adopted plugins
o Safety in numbers – hugely active community
oDashboard view of the state of your builds
o You don’t have to be an expert at everything
oTest Anywhere
oDeploy on-premise or in the cloud for testing
o Spin up VMs or EC2 Instances as build executors
oWe can do it for you, if you like…
Continuous Integration and Automated Testing
• CloudBees ClickStarts – A complete project in just two clicks!• Bees Shop Demo
• Git repo, database, build and deploy in seconds• Online examples: https://partnerdemo.ci.cloudbees.com
Setting up the Test Envrionment
21Copyright © 2014 SOASTA, Inc. - All rights reserved.
o Demo:
• Bees Shop Overview
• SOASTA Cloud Test: Setting up a load test up for unattended runs
• Jenkins Performance Data:
• Reviewing Performance Trending Data: Track performance of each build
• Reviewing Performance Pass/Fail Data: Failing
• Setting up Continuous Load Testing
• Install Plot Plugin
• Download GitHub: https://github.com/mostenberg/SOASTAPERFPERL
• Add steps to plot build data for any metrics of interest.
• Publish Junit test results to PASS/FAIL based on performance thresholds: SOASTA/5_PERF_THRESHOLD_RESULTS.xml
23Copyright © 2014 SOASTA, Inc. - All rights reserved.
TouchTestContinuous
Testing
CloudTestLoad &
Performance
mPulseReal User Monitoring
• Open Source: online at jenkins-ci.org
• Automatically available on CloudBees DEV@cloud
• Complete control over CloudTest/TouchTest Automation from within your Jenkins builds:
• Totally integrated into Jenkins environment
SOASTA Jenkins Plug-in
• CloudBees ClickStarts – A complete project in just two clicks!• Example configurations all online:
• https://partnerdemo.ci.cloudbees.com/• Bees-Shop-PerformanceTest
Try it out yourself!
26Copyright © 2014 SOASTA, Inc. - All rights reserved.
Thanks & Q&A
Contact SOASTA:www.soasta.com/cloudtest/[email protected] us:
twitter.com/cloudtesttwitter.com/cloudtest
facebook.com/cloudtestfacebook.com/cloudtest
Knowledge Center•White Papers•Webinar Recordings•Case Studies
CloudLink Community•Support•Tutorials•Video
RESOURCES
Contact [email protected] us:
twitter.com/cloudbeestwitter.com/cloudbees
facebook.com/CloudBeesfacebook.com/CloudBees
www.SOASTA.com www.CloudBees.com
blog.cloudbees.com•Look for Mark’s post on this topic
developer.cloudbees.com•Platform Services•Frameworks•ClickStarts
Top Related