Software Testing in the Cloud

Tauhida Parveen, PhD

tparveen@tauhida.org

About Me

• University Department Chair, Software

Engineering, Keiser University

• PhD, CS, Florida Institute of Technology

• MBA, University of Central Florida

• ISTQB Certified Tester (Foundation, Manager, Agile)

• Independent consultant and trainer

• Software Quality Engineer/ Tester

• Yahoo!, Sabre, Wikimedia, …

About Me…

Vice President & COO

Motivation

• Cloud computing is here to stay

• Changing the way we engineer software

• Leverage the power of cloud in testing

Objectives

What is the cloud about?

How is it different from traditional computing?

What is testing in the cloud?

What does it mean for testers?

Where to start?

What is Cloud Computing?

An old idea whose time has finally come

If computers of the kind I have advocated become the computers of the future, then computing may someday be organized as a public utility just as the telephone system is a public utility. The computer

utility could become the basis of a new and important industry.

John McCarthy, 1961

What is Cloud Computing?

Virtualized hardware

SOA, On-demand, utility model

Rapid elasticity

Resource abstraction techniques

Multi-tenancy

Self-healing

Not in your control (mostly) / things are taken care of

Cloud Services

Cloud Services

Cloud Services

Cloud Delivery Models

Public • Services offered by a provider that are publicly available, can be used with a

contractual agreement

Private• Virtualized resources available as a service owned by individual organization

• Client managed, customization, high efficiency, privacy and security policies

Hybrid

• Mix of public and private

• Extend private cloud via VPN

• Good candidate for incremental migration

Cloud Benefits

Faster, cheaper, better

Flexible

• Multiple test beds for multiple release testing

• Instant setup, tear down

Scalable

• Virtualized environments

• Scalable resources

On-demand

• Pay-per-use

What Does it Mean for Testers?

How would you test this?

Software Testing in the Cloud

Lies at the intersection of three key areas

• Software Testing

• Cloud Computing

• System Migration

STITC (Software Testing in the Cloud)

• Testing in the cloud

• Testing of the cloud

• Migrating testing to the cloud

http://www.stitc.org/

Traditional Testing

Low asset utilization

Scalability: Long time to increase capacity

Long time to build datacenters

Difficult to manage

Duplicate test systems

Creates unnecessary waste

Testing in the Cloud

Improved asset utilization

Less time (instantaneous) increase and reduction in capacity

Purchased as a service from cloud providers

Better management and increased productivity

Aggregated system

Cleaner, greener testing

Case Studies

Case Study #1: Concurrent execution of JUnit test cases

Case Study #2: Concurrent execution of GUI testing of iOS applications

• It can take a large number of test cases to cover all the variations that users could experience

• Different configurations

• OS versions (iOS ?)

• Device Orientations (Portrait, Landscape)

• Screen sizes (3.5", 4")

• Form Factors (iPhone, iPad, Universal)

Case Study #2

Software Testing in the Cloud

Software Testing in the Cloud

Management

• Process

• Technology

• People

Cloud testing strategy

Roles and responsibilities

Can’t be achieved as an execution-only

activity

Management: Process

Strengths

•Faster

•Cheaper

•Better

Weakness

•Network connectivity

•Security

•Standards

•Availability

•Performance

•Project management

Opportunities

•Faster execution

•Faster feedback

•Efficient testing process

•Value proposition

Threats

•Complexity

•Hype

•Vendor lock

•Tool chain

•Education

•Training

SWOTANAYSIS

Management: Process

Migration Strategy

• Where are you now?

• Where do you go next?

SMART-T: A decision framework to migrate testing to the cloud

Desirable

Faster Cheaper Better

Business Drivers

Feasible

YES

YES

Types of

Applications

Technical Factors

Types of

Testing

Test

Execution

Environment

Acceptable

NO

NO

YES

NO

Stakeholders

WorkshopPilot Study

Evaluation

Results

Trial Migration

Adjust

Estimates

Document

Guidelines

Implement

Migration

Plan

Actual Migration

MAYBE

MAYBE

MAYBE

Operational Results

Management: Process

Evaluate current overlapping activities

• Agile development and testing

• Test early, test often

• TDD, Unit testing

• JUnit in the cloud

• Google Test in the cloud

• HadoopUnit

• Faster feedback

• Automation

• Regression testing

Management: Process

Cost model

• How much does it cost you to test now?

• Do you have a test lab? Machines for every project?

• How often do you configure the machines?

• How often do you upgrade them?

ABC: Activity Based Cost model – Identify and put $$ to all the big and small activities

Management: Technology

Management: Technology

Testing as a Service

Distributed processing technologies

Test execution environments

Management: Technology

Amazon Web Services (AWS)

Management: People

Will your testers be able to adapt to cloud testing?

• Testing at many levels – User Interface, Services, Governance, Process, Security, Integration

Programmers?

• Beyond unit testing

System administrators?

• Cloud service providers

Training

Continuing education

Testing skills, programming skills

Communication skills

Testing in the Cloud: Strategy

Does testing in the change the foundation of testing?

Does it involve new technique, technology, and domain?

How is it going to effect your current tasks? (good/bad)

Do you need new skills?

Are there different types of testing?

Are there tools to help you?

Risks associated with cloud testing

Assumptions, constraints

Testing in the Cloud: Strategy

Critical Success Factors (CSF)

What are the CSF for testing in the cloud?

• Automation?

• Skilled testers?

• Test lab/environment?

How do your measure them?

• Post release evaluation?

• Number of bugs for the release

• Test lab setup and tear down?

Testing in the Cloud: Strategy

Roles and Responsibilities

Developers: Unit testing, Integration testing

Test Analyst: Implement testing in the cloud strategy

Technical Test Analyst: Testers

Customers: Acceptance testing

IT department: POC to cloud provider, monitoring, availability

Cloud providers: SLA, services testing

Lessons Learned

Lessons Learned

Testing Environment is a legacy system

Acceptability of partial migrations

Selecting/Preparing test execution environment

Configuration in the cloud is challenging

The nature of the test cases

False negatives

What Does it Mean for Testers?

How would you test this?

Key Points

Software testing is challenging activity

SOA, virtualization, computational grid, and cloud computing bring efficiency in development process/consumer need

Leverage these technologies in software testing

Plan, start early, know your goals

Understand the holistic system architecture

Understand the migrated test data

Establish communications with the vendor

Key Points

Cloud computing is not hype!!!

Provides the infrastructure to process Big Data

It’s changing the testing landscape

It’s changing the testing profession

We have had similar shifts in the past

Cloud computing is here to stay

Brace for change in testing!!!

Food for thought!!!

WSJ (Nov 6, 2015)

Thank Youtparveen@tauhida.org

References:• Jamsa: Cloud Computing: SaaS, PaaS, IaaS, Virtualization, Business Models, Mobile, Security• Hausman: Cloud Essentials