1

The Pain and Gain of Test Automation – the early days

Andy RedwoodPortman Building Society

redwooda@msn.com

2

Portman Group

• 3rd Largest Building Society in UK• 2000+ staff, 142 branches, HO in Bournemouth• 2005 Total assets increased by 15% to £17.8 billion• 2005 Residential mortgage balances up by 16% to

£13.5 billion• 2005 New residential lending up by 18% to £4.3

billion• 2005 £237m of net retail inflows (Society)• 2005 Retail savings balances increased to £11.4bn

3

Agenda

• Starting position• Evaluation• Buy-in• Solution – first stage• People• What’s going well?• Core issues• Where are we going next?

4

The first 10 days

• 35 observations• 43 suggestions • Cross-referenced to risk

table• 19 actions agreed with

Internal Audit• 18 of 19 completed to

date

5

Problem Statement June 2005• Need to be more ‘merger friendly’;• Need to accommodate a diverse and changing

architecture;• Business applications must be resilient to change;• Life-cycle processes are reactive;• Testing takes too long;• Accountability over the testing is weak;• High Defect Rate in later test phases;• High re-work rate;• Testing Skill sets are being questioned;

6

Buy-in

• Management

• Bigger Picture• CIO and executive• Mandate for change• Budget for tools• Resource Strategy• Training programme

• Team dynamic

• Test management• Flexible process• Multi-project focus• Pragmatics• Prioritising• Communications

7

Gaining Control – Defining the End Game

Define the problem – Decision point - Define the End GameDays

Life-cycle changes

Decision point – Automation (Winrunner/Quick Test Pro)

3 - 6 months

Revised Testing Strategy (by workshop)

Pilot Programme – Champion required

Decision point – good to go?

Change Programme

Staged IterativeImprovementsBy Programme

Phased Improvement

Quick Wins Action Now - Test Leadership, Proactive Risk-focus

8

2006 Test Strategy

• 57 page Test Policy• ‘The way we work’• ‘The way we want

others to work with us’• Test Approach by

exception• Supported by Standards

and User Guides

9

Testing – Red Flags

TEST EXECUTION IMPLEMENTATIONDEFINITIONANALYSIS DESIGN BUILDINITIATION

Data

Environment

Test

Manual

Tests heldIn Access

Process isLong & error

prone

Tests areduplicated

Test does notTarget risk

Defects are missed

Test can takemonths to complete

Dependenciesunknown

Test ExecutionRestarts

Every time

1 person1 test

Lack of early involvement

InsufficientReporting

EnvironmentRebuild time

Limited E2ECapability

10

June 2005 Team Structure

Test Manager

Test Coordinator Test Coordinator Test coordinatorEnvironmentCoordinator

11 Manual Testers

11

IS Testing Flexible Working

Group Business Manager

IS Testing

Test ManagerMortgageChannel

Test ManagerBAU Projects

Test Manager Environment and

Tools

Test ManagerInvestment

Projects

Test ManagerRegulatory

Projects

Test ManagerBroker Channel

Flexible Resource Pool – IS, IST, Business Secondments, External Resource

12

Stage1 Solution

• Write, publish and sign off the Strategy;

• Standardised but flexible process;

• Testing Services function;

• Accountability and ownership of the test;

• Flexible test resourcing from professional testing organisations – annual tenders;

• Integrated Test management Toolset – Test Director;

• Optimised Automated Functional and regression;

13

High Test Management focus

Lower Test Management focus

High Test Execution focus

1 Test Manager3 Coordinators

12 Testers

Defined Business RoleRetained Expertise

Hire Testers on demand

Lower Test Execution focus

Higher Cost

Lower Cost

No go zone

Flexible Resourcing

Risk-basedFlexibility

14

Time

Cum

ulat

ive

Def

ects

High Detection Rate

Low Detection Rate

Test on or implement now

Undetermined time saving

Law of Diminshing ReturnsRisk Decision Point

deploy now?

Implementation dateRequirements Analysis Design Code Test

Informed Deployments

15

Implementation dateRequirements Analysis Design Code Test

No.

of

Tes

ts R

unExtended Testing Day

Manual TestingIn 8hr day

Automated TestingIn 14-18hr day

Higher Test VolumeHigher error detection rateFewer TestersProduction StabilityLower Costs

16

TEST EXECUTION

IMPLEMENTATIONDEFINITION ANALYSIS DESIGN BUILDINITIATION

Data

Environment

Test

Automated

Where are we now?

Input Test data Can be Scripted

Tests Held in TD8

Tests defined to a multi-project Repository

Real Time Prep by many Testers

Data Load is Scripted

Environment Performance Monitoring

Automatic Environment Sanity Check

Executions can be networked

Execution is Unattendedn-desktops

On DemandJust enough Regression

Testing

Test Advise inEarly stages

EnterpriseTest Strategy

Continuity of Standards &

Process

InformedDecisions

Measures& Metrics

17

Next Steps

• Integrate more across departments

• Roll out TD8 into development – move dynamic defect focus into Unit Testing

• Comparable execution toolsets across platforms

• Test Process Toolbox – more detail on the process

• Sort out our environments

18

Thanks for inviting us to your SiGiST

Andy Redwoodredwooda@msn.com