Post on 21-Jan-2018
8/19/2015
1
© 2013 IBM Corporation
Continuous delivery of software innovation on hybrid cloud
Testing ApplicationsFor the Cloud and in the Cloud
@Al_Wagner
IBM Technical Evangelist
awagner@ca.ibm.com
1
Why do we care?
In the Cloud; For the Cloud
8/19/2015
2
2
Delivering
in today’s DevOps world
In the Cloud; For the Cloud
3
DevOps – the three principles
Left to Right – Continuous Delivery
Emphasize the performance of the entire system
Right to Left – Continuous Feedback
Create the right to left feedback loops
Culture of Experimentation and Learning
Continually experiment, take risks and learn from failure;
Understand that repetition and practice is the pre-req to mastery.
Continuous Delivery
Whole team approach;
No blame culture;
Encourage experimentation & learning
Continuous Feedback
In the Cloud; For the Cloud
8/19/2015
3
4
The 1st step is solving the “legacy” challenges
Before DevOps, test
teams were fairly
successful at keeping
up with development
velocity
Test teams need to
change their behavior
in order to keep up
with increased
development velocity
Traditional test teams
can no longer keep up
with continuous delivery
In the Cloud; For the Cloud
5
Today’s reality for Testers
Testers want to spend LESS time
ibm.biz/TestingSurveyResults
In the Cloud
8/19/2015
4
6
Virtualizing today’s test labs
Modern test labs consist of…
Test Data
Management
Service
Virtualization
Deployment
Automation
Dynamic Infrastructure
In the Cloud
7
Service virtualizationmaking the unavailable available for testing
Virtual components simulate the behavior of a service or application during testing
– System dependencies
– Unavailable/inaccessible services
– Costly 3rd party access fees
Virtual components run on commodity hardware, private cloud, public cloud
Each developer and tester can easily have their own test environment
Developers and testers can continue to use current testing procedures and tools
Streamline creation of production-like environments and negative testing
Heterogeneous Environments
Public CloudPrivate Cloud
Data Warehouse MainframeEnterprise
Service Bus
Directory
Identity
File
systems
Collaboration
App Under TestRouting
Service
Third-party
Services Portals
Content
ProvidersEJB
Shared
ServicesArchives
Business
Partners
Messaging
Services
Databases Mainframe
applications
App Under Test
Third-party
Services
Packaged apps, messaging services, etc.
Virtual Components
In the Cloud
8/19/2015
5
8
Continuous deploymentautomating testing includes automating deployment
Consistent deployments and rollbacks of applications
Orchestration of changes across servers, tiers and components
Configuration and security differences across environments
Clear visibility: what is deployed where and who changed what
• Multi-Tier Application Models
• Scalable distributed automation
• Quality gates and approvals
Deployment Automation
In the Cloud
9
In the Cloud
“yes...BUT!”
I still need networking
and computing
resources to run the
deployed software!
Dynamic infrastructure
8/19/2015
6
10
Clouds are rolling in
Clouds are not a commodity.
Cloud computing doesn’t come out of the sky.
Clouds comes from physical hardware inside brick and
mortar facilities connected by hundreds of miles of
networking cable.
And no two clouds are built the same way.
In the Cloud; For the Cloud
11
Clouds deliver networking and computing power with speed!
In the Cloud; For the Cloud
privatepublic
On-Premises IT
hybrid
8/19/2015
7
12
Cloud services save you time!
Infrastructure (IaaS) Platform (PaaS) Software (SaaS)
more consumer control more vendor control
In the Cloud; For the Cloud
13
Automated test execution in the cloud
Many organizations are looking to the clouds to execute:
Functional Tests Drive execution of functional tests in datacenters around the world without having
to deploy physical servers
Reduce latency issues and long running tests
Performance Tests Drive load around from datacenters around the world to measure global user
experience
Scale user load on demand
In the Cloud
• Setup; tear down test execution
environments on demand at lower cost
• Distribute test execution around the
world with ease
Launch test execution
Log execution results
8/19/2015
8
14
Deployment automation and cloud accelerates testing
Virtualizing Hybrid Test Labs - deploy applications in hybrid clouds
Simultaneous execution of test stages – integration, functional,
performance, security testing
Eliminates scheduling delays
Eliminates hardware delays
Setup; Tear Down on demand
In the Cloud
15
Deployment automation and clouds eliminate test lab bottlenecks
15
Rational Test
Virtualization
Server
- virtualized services
CICS
Tx
SAP External
Services
DB
Service
Virtualization
Deployment Automation
Env1
Env2
Customers
DB
App
Server
SAP
CICS
PaaS(private cloud)
External
Services
SaaS(public cloud)
Off premise
Off premise
DB
App
Server
App
Server
SAP
CICS
PaaS
3rd Party
Service
3rd Party
ServiceDatabase
CICSSAP
Physical devices,
emulators, devices
clouds
Development Test Production
On premise
In the Cloud
8/19/2015
9
16
Deployment automation and cloud accelerates testing
Virtualizing Test Workstations - deploy pre-configured virtual
desktops in the cloud
Eliminate need to install, configure, license and maintain
workstations
Standardize on a test automation platform
Increases productivity; reduces down time
Setup, Tear Down on demand
In the Cloud
17
Virtualizing testing workstations
Image
Repository
Artifact
Repository
Deployment
Automation
Private
Request Cloud
instanceRequests
Testing
Workstation
Returns
Workstation
I/P address
Dynamic Infrastructure
Request
workstation
image
Request specific
workstation configuration
Remote access to
virtual workstation
Operating system
& software
Configuration
files
Test
Assets
In the Cloud
8/19/2015
10
18
For the Cloud
What is quality software
in the cloud?
Clouds can be quickly stood up in Data Centers
anywhere in the world. This capability can expose risk to
the business if not considered in the test effort.
You need to make sure the software being released is:
• Resilient
• Reliable
• Secure
19
Testing for high availability
• Does failover routing work as expected?
• Does the system automatically scale?
• Are databases redundant and backed up?
• Are faults logged & people notified of outages?
Data Center
Pod
Cluster
Server/Data
For the Cloud
8/19/2015
11
20
Testing for resiliency
Quality is not only measured by how well code survives during
a long test run, but how quickly normal operations can be
returned after an error.
Testers need to assume remote services:
will respond slowly,
will respond with an error,
or won’t respond at all
The most important production issue metric is
not “root cause” but “time to recovery”.
For the Cloud
21
Patterns for the cloud - Retry
For the Cloud
8/19/2015
12
22
Testing Retry pattern
Service A Service B500ms (timeout)
500ms (timeout)
RetryCount=2RetryCount=1
Server LogOperation Exception – (date/time)
….
Server LogOperation Exception – (date/time)
Operation Exception – (date/time)
….
For the Cloud
Timeout = 500ms
Max # of Retries = 3
extends Retry
23
extends Retry
Testing Retry pattern
Service A Service B
Server LogOperation Exception – (date/time)
Operation Exception – (date/time)
….
500ms (timeout)
500ms (timeout)
200ms (passed)
• Use a virtual service to simulate
“Service B” responses along with
timeouts and retry attempts
• Confirm log entries are captured
RetryCount=0
For the Cloud
Timeout = 500ms
Max # of Retries = 3
8/19/2015
13
24
Patterns for the cloud – Circuit Breaker states
For the Cloud
25
Patterns for the cloud – Circuit Breaker
For the Cloud
8/19/2015
14
26
Testing Circuit Breaker pattern
Service A Service B
For the Cloud
State: closed
Circuit
Breaker
success
Parameters
invocationTimer = 0.1
resetTimer = 60_000
retryThreshold = 1
Testing the “closed” state:• Straight forward and easy to test
• Validate “closed” circuit breaker state using real
service implementation or virtual service
requestIDState = closed
getState(requestID) returnState()
27
Testing Circuit Breaker pattern
Service A Service B
For the Cloud
State: closed
Circuit
Breaker
failure
Parameters
invocationTimer = 0.1
resetTimer = 60_000
Service A Service B
State: half open
Circuit
Breaker
timeout
testCall()
success
success
updateState(halfopen)
updateState(closed)
resetTimer()
success
executeCall()
Testing for “half open” state:• Hard to test negative in test labs and production
• Virtual services can help simulate the necessary
“negative” responses required
8/19/2015
15
28
Testing Circuit Breaker pattern
For the Cloud
Service A Service B
State: open
Circuit
Breaker
fast fail
Parameters
invocationTimer = 0.1
resetTimer = 60_000
retryThreshold = 1
requestIDState = open
getState(requestID) returnState()
Testing the “open” state:• Little harder if Service B is deployed
• Validate “open” circuit breaker state
• Setting requestIDState to “open”
• Routing request to stopped virtual service
29
Testing feature toggles in a continuous integration world(aka feature flags; dark launches)
Release Toggles
Hides incomplete features (dark launch)
Short lived (removed when feature deployed)
Pre-decided values across environments
Business Toggles
Enables A/B testing
Release to small control groups of people to get
feedback
Enable and disable on the fly
For the Cloud
8/19/2015
16
30
Release toggles
For the Cloud
//pseudo-code
featureToBeReleased = true;
If “featureToBeReleased” {
do this():
} else {
do that();
}
Hide features from being exposed until development and testing
is “Done”.
Remove feature toggle from code base across all modified
components.
Check to ensure “feature” is hidden or available as intended.
Be sure to toggle the UI layer!
31
Business toggles
For the Cloud
//pseudo-code
checkUser() {
boolean testgroup = checkDatabaseForUserGroup(userID);
if (testgroup) {
accessNewFeature();
} else {
useLegacyFunctionality();
}
}
Enables organizations to “test” features before releasing to the
entire user community. (A/B Testing)
Test for all user groups.
Need to determine what combinations of features need to be
tested. It could be difficult and time consuming to test them all.
Be sure to check the UI presentation as well!
8/19/2015
17
32
Continuous feedback delivers insight on software quality
Build a system that allows you to make decisions
based on real data instead of just market direction.
User behavior
Customer sentiment analysis
Performance metrics
For the Cloud
33
Testing for security
Data Breaches
Encrypt the data exchanged between services
Data Loss
Deploy redundant systems and test backups
Insecure APIs
Develop and Test 3rd party access using authorization services
Denial of Service attacks
Your organization could be billed for all the cloud services
Malicious Insiders
Protect your encryption keys
Shared Technology
You are not alone in “public” clouds
to name a few…
For the Cloud
Don’t forget to scan your code for insecure coding practices.
Penetration test your applications.
If you are hacked; ALL your customers are impacted!
8/19/2015
18
34
Remember
It is important to test:
what a product/service does
how a product/service performs
as well as how that product/service is managed
For the Cloud
35
ibm.com/devops