The 4 T’s of Test Automation:The 4 T’s of Test Automation:Your Planning Guide to Success

Presented By:David DangQuestcon Technologies

September 18, 2008

The 4 T’s of Test Automation:The 4 T’s of Test Automation:Your Planning Guide to Success

Presented By:David DangQuestcon Technologies

September 18, 2008

Quality - Innovation - VisionQuality - Innovation - Vision

Slide Slide 22

BACKGROUNDBACKGROUND

Test Automation: Saves time and Increases Coverage

Tool Evaluation

Training

Team starts using the tool

Let’s review the typical test automation process . . .

Slide Slide 33

Typical Automation Project OutcomeTypical Automation Project Outcome

Adopted by QA with success at the beginning

Over subsequent release cycles, maintenance time and costs increase

Final outcome, the automation suite becomes a maintenance nightmare and is used by a few users or becomes shelfware

Slide Slide 44

ObjectivesObjectives

The primary objective of this presentation is to demonstrate how proper planning for test automation will yield greater success. Attendees will learn:  

Potential benefits of a successful automation project

Key factors to consider in the planning phase of test

automation

Proper alignment of automation tools to technology under

test

Evaluation criteria for manual test cases

Required skill sets for test automation

Timeline impacts of using different automation approaches

Slide Slide 55

While there are multiple considerations that must be addressed during the planning phase of a test automation project, these four factors have the largest impact on the overall success.

Test CasesTest CasesTest CasesTest Cases

Key FactorsKey Factors

TechnologyTechnologyTechnologyTechnology

TalentTalentTalentTalent

TimelineTimelineTimelineTimeline

Slide Slide 66

Create an inventory of the current technology that you must test

PlatformsProgramming LanguagesDatabasesThird-party controls

Determine what technology changes may occur within two yearsExamples of technology under test (TUT):

Platform – Mainframe, Client/Server, and .Net (long-term)Programming Language – Cobol II, VB 6, C ++, and C# (long-term)Database – DB2, Sybase, and MySQL (long-term)Third-party controls – VSFlexGrid and WebServices (long-term)

TechnologyTechnology

Slide Slide 77

TechnologyTechnology

ConsiderationsLevel of support for TUTAmount of customizations required for non-supported TUTEase of useSkill sets required to work with toolCommon languagesTool costTool maintenance costIntegration with other toolsReporting

Slide Slide 88

Matching TUT to the automation tool ensures faster automation script development and decreased maintenance time by:

Reduction in custom codingReduction in object mappingIncreased flexibility in framework designEasier debuggingBetter reporting

Additionally, this approach reduces learning curve of automation toolIncreases the level of testingProvides better cost justification

BenefitsBenefitsBenefitsBenefits

TechnologyTechnology

Slide Slide 99

Test CaseTest Case

Is this a good test case?

Why?

Slide Slide 1010

Test CaseTest Case

Test cases contain detailed levels of information:Test case nameTest case objectivePre-requisitesTest case priorityInput and output dataStep nameStep descriptionExpected result

Determine:Frequency of execution over a yearly periodLevel of testing required for validationComplexity of the test casesManual only steps, e.g. loading a backup tapeAutomation priority based on risk factors

Evaluate Manual Test CasesEvaluate Manual Test CasesEvaluate Manual Test CasesEvaluate Manual Test Cases

Slide Slide 1111

Test CaseTest Case

Slide Slide 1212

Scope identification of the automation effortPrioritization of automation effort based on risk factorsElimination of manual test cases that were out-datedIdentification of manual test cases that provide little to no return on investmentUnderstanding of test case factors in creating an automation framework

BenefitsBenefitsBenefitsBenefits

Test CasesTest Cases

Slide Slide 1313

QA experience in various levels of testing such as smoke, regression, and functionalSolid understanding of business flows, business rules, and data variations in the applicationExperience with multiple data strategies

Creating “new” dataGathering existing dataBuilding data

Experience with test automation conceptsUnderstanding of computer programmingGrasp of application architectureComprehension of Object-oriented programming (OOP)

Required Skill SetsRequired Skill SetsRequired Skill SetsRequired Skill Sets

TalentTalent

Slide Slide 1414

Experience in programming using Java, VB, or C languagesExperience with multiple automation approachesExperience in SQLExperience in writing classes and methodsExperience in writing functions and sub-proceduresExperience in using or interacting with window APIsProject Management

TimelinesDeliverablesResourcesMetricsBudget

Optional Skill SetsOptional Skill SetsOptional Skill SetsOptional Skill Sets

TalentTalent

Slide Slide 1515

Establishes a good baseline for skill sets required to implement test automationCan be used to determine QA members that will work on test automation

Most companies struggle in test automation when they have the QA group perform test automationRecommend creating an automation team

Reduction in trial and error approachAutomation approach designed around skill set level available

BenefitBenefitBenefitBenefit

TalentTalent

Slide Slide 1616

Automation Projects TypicallyAutomation Projects TypicallyMirror SDLCMirror SDLC

Automation Projects TypicallyAutomation Projects TypicallyMirror SDLCMirror SDLC

TimelineTimeline

The same steps that occur in the SDLC should occur in an automation projectMost automation projects only account for Implementation and TestTimeline should take every step in the SDLC into consideration

Slide Slide 1717

Factors that Impact TimelineFactors that Impact TimelineFactors that Impact TimelineFactors that Impact Timeline

TimelineTimeline

Resource availabilityMost companies assign 30 – 50% of QA member time to test automationThe reality is QA members cannot effectively perform both manual and automation testingThis greatly increases the timeline for creating and maintaining test automation suite

Resource skill setsSelected automation toolTesting environmentData strategyManual test case condition

Slide Slide 1818

Factors that Impact TimelineFactors that Impact TimelineFactors that Impact TimelineFactors that Impact Timeline

TimelineTimeline

Automation frameworkThere are five common automation frameworks

Record/PlaybackData-drivenModularAction-based/KeywordDatabase

Automation frameworks are not mutually exclusiveAutomation frameworks can be combined, e.g. Action-based and database approachesAppropriate automation framework dependant on many factors including the company goal for test automation

Slide Slide 1919

TimelineTimeline

Case StudyCase StudyCase StudyCase Study

0200400600800

10001200140016001800

Recor

d/Play

back

Data-

Driven

Mod

ular

Action

-bas

ed/K

eywor

d

Datab

ase

Development

Maintenance

Maintenance/Year @10 Releases/Year

Slide Slide 2020

Sets clear expectations for managementIdentified timeline for automation development life cycle (ADLC)Maintenance time factored into the automation framework designClear understanding of milestones and deliverableEasier measurement of the cost and benefit of test automationROI calculation for break-even point for test automation

BenefitsBenefitsBenefitsBenefits

TimelineTimeline

Slide Slide 2121

The most important aspect of any test automation project is planning. With the right strategy, you will save time and money, and set clear expectations for your organization.

During planning, remember the 4 Ts:Technology

Test Cases

Talent

Timeline

The most important aspect of any test automation project is planning. With the right strategy, you will save time and money, and set clear expectations for your organization.

During planning, remember the 4 Ts:Technology

Test Cases

Talent

Timeline

SummarySummary

Slide Slide 2222

Q & AQ & A