STPCon fall 2012: The Testing Renaissance Has Arrived

1© 2012 SOASTA. All rights reserved.

The Testing Renaissance has Arrived! (on an iPad in the Cloud)

STPCon

October 17, 2012

Session 503

2© 2012 SOASTA. All rights reserved. May 22, 2012

Who am I?

o Brad Johnson - @bradjohnsonsv

• SOASTA (“09-Present)

• Borland (“06-”09)

• Mercury Interactive (“02-”05)

• Compuware (“99-”02)

…and why am I here?

o I’ve seen a few testing market trends

• Client-server to Web

• Web to BIG Web

• Custom to packaged and back to custom apps

• Waterfall to agile to continuous

• Cloud computing

• Mobile


Topics

o Mobile testing is a vacuum

o Cloud testing is in demand (and a no brainer)

o Testing in production is an accepted practice

o Agile testing has come of age…with Continuous Integration

o DevOps is a movement, just add the testing and succeed

A Rich Palette of Opportunity

o Explosion in mobile applications

o The immensity of the web & everything web

o Rapid introduction of new technologies

o Incredible pace of development and delivery

What’s Driving the Renaissance?


The Mobile Testing Vacuum

o This is the greatest opportunity since Web

o Every company has a mobile strategy

o Very few have mobile testers

o Fewer have mobile testing strategies

o Global outsourcers are seeking help

Jump into the Void!


Hovering Gestures

3D Navigation

KineticsBendable Phones

Voice

GesturesMouseKeyboards

The mUX EvolutionAdvances in information access are life-changing


o Shear Number of Devices (953M Smartphones)

o Different Operating Systems

o Scale of Global Customers (6B)

o Dynamic Content (Video, Annimation)

o Emerging People to Machine

Interfaces

Mobile ChallengeDelivering Quality In An Ever Changing Mobile Landscape

Manual Testing Can Not Deliver Adequate Coverage


Quality is Suffering!Mobile coverage needs automation that works

o Nearly 80% of all mobile apps are tested manually (IDC)

Take Action

o Learn and implement effective automation

o Focus on user experience

o Consider the impact of precision and performance on functionality and satisfaction

o Prioritize and streamline to keep pace with the rate of change

o Communicate the complexity


o Read everything you can

o Understand how mobile apps are built and how they are consumed

o Use mobile as an opportunity to “fix” software development

o Join SOASTA’s Early Access Team (see me)

Dive in head first, get sucked in, or get spit out….

The Mobile Testing Vacuum


Cloud Testing

o Testing with cloud, testing in the cloud, testing for the cloud…

o Covered by all major (and minor) analysts: Gartner, Forrester, Aberdeen, IDC, Ovum…

o Cost and scale benefits are indubitable

o Expertise is an enterprise QA requirement

• IBM testers required skills

Embrace it. It’s real…and really helpful.


What Makes a Prime Cloud Testing Project?

o Critical web launch – mobile support, new website

o Tight timeframes - no room for experimentation

o Web-scale expectations – 100s to millions of users

o Quick start, many test iterations – adapt to flexible dev schedule

o High cost of tools and infrastructure - artificial ceiling increases risk


Cloud Testing

o Read everything you can

o Open a cloud computing account

o Watch webinars

o Try a cloud testing tool

o Run a cloud test

How to Get Experience


Testing In ProductionThe “taboo” is gone

From the wall of a very traditional

retailer’s datacenter

Code


s

Memory leaks

Scale and Time

Sta

ge /

Team

Release& Deploy

Network& Ops

CDN file placement

Load Balancer configuration

Network bandwidth

Network configuration

DNS routing

Inadequate server resources

Default configuration settings

Unbalanced web serversAuto-scaling failures

Latency between systems

Slow third-party plug-ins

Garbage collection

Slow pages

Conflict with other apps

Dev & Test

Test Lab Staging Production (100% +++)

Method-level tuning

Max sockets exceeded

Firewall max capacity

Memory leaks

Caching issues and benefits

Capacity limits

Code bugs

Database thread counts

Inefficient database queries

Search technology limits

Security bottlenecks

Global latency variance

Testing In ProductionBugs Know No Borders


Testing In Production

o Read some good whitepapers

o Watch/download Seth Eliot’s session and slides

o Attend a webinar on Oct. 30 (Seth & SOASTA)

Educate Yourself


Agile Testing Has Come of Age

o Still can’t find a good online diagram, though!

…it’s part of Continuous Integration


Continuous Testing is Progressive

Dev & Test Lab Production Environment

Start small, early to root

out code-level issues

Include functional

automation &

performance in CI

Hit expected scale in

staging and maintenance windows

Test in Production

Recheck regularly for regression

and capacity

Collect real user data to feed back

(RUM)

Keys to agile testing? SPEED – SCALE – COST


TIME

TESTHW Setup FIXANALYZE RETEST REPEAT?

BUILD TEST

ANALYZE

FIX

Traditional Approaches

Eliminate constraints and TEST MORE, at SCALE, in LESS TIME

Many approaches are process and pre-test heavy

Agile approaches deliver results fast

TEST BUILDING

TEST

ANALYZE

FIX

HW

Setup

HW

Setup

TEST

ANALYZE

FIX

HW

Setup

TEST

ANALYZE

FIX

HW

Setup

TEST

ANALYZE

FIX

HW

Setup

Extra time!

Do We Spend Time Where It Counts?


Delivery Cycle

Define

Test Types Test Objective

Refinement ofExpectations

Deploy

Assess

Test

Design

Implement

Load Test Determine Limits SPRINT 1

10 hours(Benchmark)

Performance Test Measure User Experience

Targeted Infrastructure Test

Test Specific SilosSPRINT 2 8 hours

Failover Test Verify RedundancySPRINT 33 hours

Stress Test Exceed Break PointsSPRINT 45 hours

Plan for futureCapacity TestSPRINT 58 hours

Reliability over timeSoak TestSPRINT 620 hours

Hours for example only

Agile Testing must address non-functional, too!Speed is required


How Continuous Integration Works

Devs

Source Code Repository(Git, Subversion, Atlassian JiraMicrosoft VS Team Foundation Server)

Check in

Dev or Test Lab

Run Tests(unit, component, functional)

Continuous Integration server automates builds, launches tests, manages results (Jenkins, Bamboo, VSTFS, Anthill,…)

Check out

ProductionOr QA

Test Results

☐ Pass☐ Fail

✓

Test Automation


Continuous Integration for Mobile (iOS)

Source Code Repository

Devs

Integration Serveri.e. Jenkins Slave(Mac)

Check out

Check in

Real Devices

Push to Device

Run Tests

Execute on Devices

Beta Users(ad-hoc server)

Test Results☐ Pass☐ Fail

✓

Test Automation


DevOps

o ALM meets APM

o What’s old is new…just FASTER

o This is Process Compression

Collaboration includes TESTING!

Test

Deliver

Define

Design

Build

Monitor

Manage

Deploy


DevOpsCollaboration includes TESTING!

Dev OpsDevOps!

o ALM meets APM




DevOpsCollaboration includes TESTING!

DevOps!Psst…

you forgot something

Dev(TEST)

Ops!

Ahhhh.

o ALM meets APM




DevOps

o Read everything you can – ALM and APM

o Seek visibility solutions – reporting, collaboration

o Expand your sphere of contact and influence

o Join or run a production performance test

Promote Dev-Test-Ops and be a change agent!


The Best Time in History to Be a Tester!

o Champion a mobile testing team or project?

o Run a cloud test?

o Be the Continuous Integration agent-of-change?

o Put the “test” in DevOps?

What will you do?

THANKS! Questions?

(Please turn in the eval.)

STPCon fall 2012: The Testing Renaissance Has Arrived

Technology

Transcript of STPCon fall 2012: The Testing Renaissance Has Arrived