T11 Cloud Testing

5/2/2013 11:15:00 AM

A Year of Testing in the Cloud: Lessons

Learned

Presented by:

Jim Trentadue

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Jim Trentadue

Jim Trentadue has more than fourteen years of experience as a coordinator/manager in the software testing field. Jim’s various roles in testing have focused on test execution, automation, management, environment management, standards deployment, and test tool implementation. In the area of offshore testing, he has worked with multiple large firms on developing and coordinating cohesive relationships. As a speaker, Jim has presented at numerous industry conferences, chapter meetings, and at the University of South Florida's software testing class, where he mentors students on the testing industry and discusses trends for establishing future job searches and continued training.

4/16/2013

1

A Year of Testing in the Cloud:

Lessons Learned

Jim Trentadue

Jim_Trentadue@hotmail.com

May 2nd, 2013

“Cloud” is a new consumption and delivery model inspired by consumer Internet services.

Enabled by Virtualization, (Service) Automation, Standardization

Cloud enables:

� Self-service

� Sourcing options

� Economies-of-scale

Multiple Types of Clouds will co-exist:

� Private, Public and Hybrid

� Workload and / or Programming Model Specific

Cloud Services

Cloud Computing

Model

What is the ‘Cloud’?

Defining the terminology behind the Cloud and listing its components

4/16/2013

2

Regular stats and environment reports

Leveraging an automated Cloud Management solution enables the following

Deploying Cloud Services Managing Cloud Services

Secure User Centric Self-Service Portal, Automation Engine and Catalog

Automated Provisioning and Image Management

Monitoring and Metering

4/16/2013

3

1) Image Library 2) Software Stack 3) Server Specifications

Operating System (plus potentially component)

•Red Hat Linux 5.3

•Red Hat Linux 5.3\Oracle 10gR2

•Windows 2003 server

•Windows 2008 server – 32bit

•Windows 2008 server – 32bit \

SQL Server 2008

•Windows 2008 server – 32bit \

Application

�Verify your O\S is supported by

the Cloud provider

Software components

•Oracle 10g

•Oracle 10g R2

•Oracle 11g

•SQL Server 2008

•JBOSS v

� Must be available via a silent

install method

Hardware requirements

•Number of CPU Processors

•Amount of memory

•Storage size (through SAN)

� This can be a standard amount

or can be custom on demand

Overview for a ‘Cloud’ test environment

A high-level overview of the constructs of the new virtual test configuration

Physical

environment

structure

Virtual

environment

structure (Cloud)

Key attributes

• Significant monetary investment upfront

• Fixed CPU Processor, Memory, and Storage

• Support costs are charged by the server

• Maintenance would need to occur on each of

the servers for O/S or software upgrades

Key attributes

• Blade technology that can be flexible for the various

layers (Application, Middle-Tier, Database)

• Concept of storing one master image library based

on O/S, adding in software stack, then request for

Processor, Memory and Storage requested

• Support costs are centralized, especially for

licensing, patches and upgrades

Benefits of a virtual test environment – why move from physical?

Compare and contrast of the two different environment structures is done below

4/16/2013

4

● Very low utilization of development and test servers—usually less than 15%

● Having to dedicate a substantial number of servers within a typical IT environment to test –

sometimes 50% or more

● Finding instant access to available IT infrastructure resources (tools and platform) to perform tests

● Provisioning of new environments is a manual process that can take up to 6 weeks or more

● Very long testing backlogs, usually the single largest factor in the delay of new application deployments

● Test environments are often seen as expensive and providing little ‘real’ business value

● Inability to follow best practices due to expense of additional IT resources required.

● High risk of defects caused by wrongly configured test environments

Why Development and Test Clouds?

High costs and poor utilization of non-production environments creates the need to consider alternatives

� UAT \ End User Training environment

� Functional Test Automation environment

� Performance Test Automation environment

� Multiple System Test environments if needed

� Prototype environment for Architects \ Business Analysts \ Technical Leads

� Pre-production \ Staging environment for deployment tests

� Production Support environment for immediate break fixes

� Release-1 environment for a specified duration

� IT User Training environment for new associates to the application

� Upgraded environment for O\S or vendor patch level upgrades

New opportunities for testing

Implementation of a Cloud computing model can open other test avenues

4/16/2013

5

Business Case Risk Return on investment

Need to increase the

number of test

environments to align

with parallel project

initiatives

Reliability testing of developed code is often

compromised with multiple versions of a same

module.

Additionally, project schedules are often at risk

with defects opened due to environment issues.

Investment in additional hardware or

virtualization technology.

Leveraging a well-suited server can help

streamline additional RDBMS license costs by

storing multiple environments on one server.

Need to limit the rights

to which users have

privileges across the

various environments

Often there are times when random associates

(IT or Business), may have access to a

particular region, manipulating critical data for a

series of tests.

Investment in a source code repository.

Limiting the capability of who has access to

environments further in migration cycles (Dev,

Test, Prod).

Need to have a DBA

Service Level

Agreement agreed

upon and adhered to

The ability to refresh the data and monitor the

performance of an environment on a regular

basis is critical. Manipulated test data and a

long running query may skew results if

inadequate measures are in place.

Increased reliability and sustainability of

projects, thus expediting timelines and

deployments. Environment-related defects

should decrease and application bugs found

are worked sooner.

Need to review the buy

vs. build concept for

procuring, or running

as a service

If called upon to procure new hardware for a

new request of an additional environment -

capital, maintenance and support costs will

continually be present. Additionally, the risk is

high is if one particular server crashes.

Investigate opportunities for a virtual

environment or Cloud computing model to

avoid repeatedly purchasing servers, storage,

processors, memory and support.

Investment in the test environment build

Building a business case for a test environment project with the right-level of support

Scope of initial environment setup

Below is the list of the type of applications scoped for virtualization of a test environment

Started with seven applications in scope with the following test environment needs

1 new application with no non-production

environments built yet

3 existing applications with faltering physical

test environments

2 existing applications with additional test

environment needs apart from the physical

servers

1 existing application looking to leverage a

virtual test environment instead of a physical

landscape

4/16/2013

6

Initial landscape after setup

The first build out of the environment looked like the following:

Started with seven applications in scope with the following test environment needs

1 new application with no non-production

environments built yet

24 servers built

3 existing applications with faltering physical

test environments

9 servers built

2 existing applications with additional test

environment needs apart from the physical

servers

6 servers built

1 existing application looking to leverage a

virtual test environment instead of a physical

landscape

1 server evaluation

Cloud test environment start-up challenges

A list of items that needs to be verified before the first server build out

� Network ports, connectivity, IP subnets, etc.

� Data refresh strategy laid out as far as personnel supporting this

� Full blade server outages for patches, upgrades, etc., need a thorough checklist to ensure

� Patches either replicated or not across existing servers, not just on the image

� User access tests to all necessary servers

4/16/2013

7

If it could be done over again, what would be done different?

Requirements following the Cloud infrastructure implementation

Some items to have advance planning on prior to creating the first virtual server

With hindsight being 20-20, these items are must have to start again

Full list of all costs

associated with the Cloud

Understand ALL of the costs like:

• Cost for server uptime

• Cost for support for the environment with various infrastructure groups

• Cost of creating images, creating servers

Pay-as –you need usage

model

Outline the costs for your customers:

• Shared cost for application teams using servers

• Shared cost for creating images

• Run rate cost for continued server maintenance

Governance model

established

Ensure there are rules of engagement for usage:

• If additional storage, CPU and Memory is required, there should be a

nominal charge

• If the servers have been idle for a defined period of time, they are

subject for deactivation

SLA defined for Cloud

support

Understanding this is a non-production environment:

• Should there be any infrastructure issues, define an SLA knowing

respective to the event and impact

4/16/2013

8

Session recap

Recapping lessons learned after the first year of Cloud implementation

� Before recommending, make sure you understand the Cloud infrastructure components and how it would be deployed and managed in your environment

� Build a strong business case for investment with industry stats on test environment usage; expanding on improving what you have and venturing into new areas for opportunities

� Outline the scope and landscape for an initial rollout, based on a risk-adverse method

� Research and study what others have had as start-up challenges, to try and avoid these pitfalls when starting

� Host a lessons learned immediately following the initial scope and landscape with the groups using the environment to establish best practices for usage

QUESTIONS?

4/16/2013

9

THANK YOU!

Jim Trentadue

Jim_Trentadue@hotmail.com