Continuous Testing for Optimal Mobile Peroformance - STPCon Spring 2014
33
1 © 2013 SOASTA. All rights reserved.
-
Upload
soasta -
Category
Technology
-
view
405 -
download
0
Transcript of Continuous Testing for Optimal Mobile Peroformance - STPCon Spring 2014
1© 2013 SOASTA. All rights reserved.
2© 2013 SOASTA. All rights reserved.
• Continuous Integration for mobile performance testing
• The cloud to help test for mobile scale
• Real devices to validate user experience with actual device
performance
• Access and distribution of the right information to the right people at
the right time
3© 2013 SOASTA. All rights reserved.
Load tests run
via Jenkins
Daily (or more frequent) updates
keep teams “performance focused”
Identify
bottlenecks
& inform
teams, just
like unit and
functional
tests in
Jenkins
Suitable test
environment * Manage with a new metric!
…while functional automation
runs on real devices capturing
device-side perf metrics
4© 2013 SOASTA. All rights reserved.
5© 2013 SOASTA. All rights reserved.
Does it work as
designed?
Does it perform
under extreme
circumstances?
Raw MaterialsReal Drivers
What
happens in
actual use?
Assembly Line
6© 2013 SOASTA. All rights reserved.
Functional Testing
Performance,
Reliability,
Scalability?
Development
Real Business
Flow and
Devices
What
happens in
actual use?
Build
7© 2013 SOASTA. All rights reserved.
Source Code Repository
Devs
Build Server
Check out
Test Results
Check in
Run
Tests
Unit Tests
☐Pass
☐ Fail
✓
To
QA or
Users
8© 2013 SOASTA. All rights reserved.
http://www.soasta.com/knowledge-center/whitepapers/
9© 2013 SOASTA. All rights reserved.
o Small Batch Sizes
o Comprehensive Version Control
o Simple Branch Strategies
o Automation
o Continuous Feedback
o Emphasis on working builds
o Consistent environments
o Developer – Tester Collaboration
10© 2013 SOASTA. All rights reserved.
o Device test automation
o API/Web test automation
o Version control
o Build automation
o Artifact management
o Configuration automation
o Test clouds
o Device clouds
11© 2013 SOASTA. All rights reserved.
Test tool launches
tests on the devices4
CI Server kicks off
test compositions
for updated apps
3
Updated App sent to
CI Server, compiled
and made testable
1
Devices poll test tool
over Wifi
or carrier network
CI server polls
version control
for latest code
JENKINS SERVER
Test Tool
Analytics
Main
Database
Apps
pushed
to devices
(!)
2
Test tool setup
CI setup
Mobile devices (anywhere)
USB Hub
(slave)
Results
sent to
testers &
developers
5
Version Control
System
12© 2013 SOASTA. All rights reserved.
13© 2013 SOASTA. All rights reserved.
Customers
SUT/AUT
Web
Server
Web
Server
App ServerWeb
Server
Web
Server
App Server
App Server
App Server
Load
Balancer
Cache
Cache
Databas
e
14© 2013 SOASTA. All rights reserved.
“Record”
Analytics
Test Tool
Databas
e
Main
Perf Tester 1
SUT/AUT
Web
Server
Web
Server
App ServerWeb
Server
Web
Server
App Server
App Server
App Server
Load
Balancer
Cache
Cache
Databas
e
15© 2013 SOASTA. All rights reserved.
Analytics
Test Tool
Databas
e
Main
Perf Tester 1
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
East Coast (AWS)
SUT/AUT
Web
Server
Web
Server
App ServerWeb
Server
Web
Server
App Server
App Server
App Server
Load
Balancer
Cache
Cache
Databas
e
16© 2013 SOASTA. All rights reserved.
Analytics
Test Tool
Databas
e
Main
Perf Tester 1
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
East Coast (AWS)
SUT/AUT
Web
Server
Web
Server
App ServerWeb
Server
Web
Server
App Server
App Server
App Server
Load
Balancer
Cache
Cache
Databas
e
17© 2013 SOASTA. All rights reserved.
Analytics
Test Tool
Databas
e
Main
Perf Tester 1
SUT/AUT
Web
Server
Web
Server
App ServerWeb
Server
Web
Server
App Server
App Server
App Server
Load
Balancer
Cache
Cache
Databas
e
Data sources:
Native (SSH), JMX, PerfMon, CA Introscope,
AppDynamics, New Relic, CloudWatch,….
Backend Metrics
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
East Coast (AWS)
18© 2013 SOASTA. All rights reserved.
Analytics
Test Tool
Databas
e
Main
Perf Tester 1
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
East Coast (AWS)
SUT/AUT
Web
Server
Web
Server
App ServerWeb
Server
Web
Server
App Server
App Server
App Server
Load
Balancer
Cache
Cache
Databas
e
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
San Francisco (GoGrid)
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
Chicago (Rackspace)
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
Tokyo (AWS)
Analytics
Load
GeneratorLoad
Generator
Amsterdam (Azure)
Analytics
Load
GeneratorLoad
Generator
Virginia (IBM)
19© 2013 SOASTA. All rights reserved.
Analytics
Test Tool
Databas
e
Main
Perf Tester 1
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
East Coast (AWS)
SUT/AUT
Web
Server
Web
Server
App ServerWeb
Server
Web
Server
App Server
App Server
App Server
Load
Balancer
Cache
Cache
Databas
e
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
San Francisco (GoGrid)
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
Chicago (Rackspace)
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
Tokyo (AWS)
Analytics
Load
GeneratorLoad
Generator
Amsterdam (Azure)
Analytics
Load
GeneratorLoad
Generator
Virginia (IBM)
Federated
Cloud
20© 2013 SOASTA. All rights reserved.
Analytics
Test Tool
Databas
e
Main
Perf Tester 1
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
East Coast (AWS)
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
San Francisco (GoGrid)
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
Chicago (Rackspace)
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
Tokyo (AWS)
Analytics
Load
GeneratorLoad
Generator
Amsterdam (Azure)
Analytics
Load
GeneratorLoad
Generator
Virginia (IBM)SUT/AUT
Web
Server
Web
Server
App ServerWeb
Server
Web
Server
App Server
App Server
App Server
Load
Balancer
Cache
Cache
Databas
e
Data sources:
Native (SSH), JMX, PerfMon, CA Introscope,
AppDynamics, New Relic, CloudWatch,….
Federated
Cloud
Backend Metrics
21© 2013 SOASTA. All rights reserved.
Analytics
Test Tool
Databas
e
Main
Perf Tester 1
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
San Francisco (GoGrid)
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
Tokyo (AWS)
Analytics
Load
GeneratorLoad
Generator
Amsterdam (Azure)
SUT/AUT
Web
Server
Web
Server
App ServerWeb
Server
Web
Server
App Server
App Server
App Server
Load
Balancer
Cache
Cache
Databas
e
Analytics
Load
GeneratorLoad
Generator
Virginia (IBM)
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
Chicago (Rackspace)
Access
for all
Analytics
Load
GeneratorLoad
GeneratorLoad
Generator
East Coast (AWS)
Federated
Cloud
Data sources:
Native (SSH), JMX, PerfMon, CA Introscope,
AppDynamics, New Relic, CloudWatch,….
Backend Metrics
22© 2013 SOASTA. All rights reserved.
23© 2013 SOASTA. All rights reserved.
24© 2013 SOASTA. All rights reserved.
o Select your critical set
o Determine who controls/manages
o Consider private and public requirements
o Agile teams like hands-on
o Integrate to perf testing
o Add a performance metric
25© 2013 SOASTA. All rights reserved.
o Manage a private device cloud installed
behind the firewall
o Provide real-time device inventory, status
and availability
o Look for online access for all your teams
o Enable 24x7, secure, remote access to
devices
o Remotely view and interact with a
selected mobile device and any mobile
app
• Use devices for manual testing and monitoring
• Remote screen-share devices for collaboration on
design or issues
26© 2013 SOASTA. All rights reserved.
27© 2013 SOASTA. All rights reserved.
o Include real mobile devices and their “system” metrics
o Test in the production environment
o Test to 2-3x the expected peak traffic levels
o Test across geographies
o Test corner cases
o Measure in real time
http://www.soasta.com/knowledge-center/whitepapers/
28© 2013 SOASTA. All rights reserved.
29© 2013 SOASTA. All rights reserved.
30© 2013 SOASTA. All rights reserved.
o Pass / Fail is the first step
o Facts, Jack.
o Manage with a new…metric
31© 2013 SOASTA. All rights reserved.
32© 2013 SOASTA. All rights reserved.
Load tests run
via Jenkins
Daily (or more frequent) updates
keep teams “performance focused”
Identify
bottlenecks
& inform
teams, just
like unit and
functional
tests in
Jenkins
Suitable test
environment * Manage with a new metric!
…while functional automation
runs on real devices capturing
device-side perf metrics
33© 2013 SOASTA. All rights reserved.
• Continuous Integration for mobile performance testing
• The cloud to help test for mobile scale
• Real devices to validate user experience with actual device
performance
• Access and distribution of the right information to the right people at
the right time
• Information? http://www.soasta.com/knowledge-center/whitepapers/
• [email protected] - @bradjohnsonsv
