Test Automation Framework Design |

Designing a Robust Test Automation FrameworkWhite Paper

idexcel

IntroductionIn today's business environment, software project teams are expected to do more and deliver higher quality systems in lesser time, with fewer resources. To increase speed, accuracy and completeness of testing, testers rely on either commer-cial or open source test automation tools. Any automated testing effort based on using only a capture tool to record and play back test cases has many drawbacks, as they are created ad hoc, their functionality can be difficult to track and repro-duce, and they are costly to maintain. Running complex and powerful tests is time consuming and expensive when using only a capture tool. To make automated testing more efficient, a better option would be to build a test automation frame-work that supports scripting and extends the capabilities of the automation tool.

A test automation framework defines an organization's way of doing things. It is a set of assumptions, concepts and tools that provide support for automated software testing. A test automation framework makes test automation more efficient, provides a higher ROI, reduces the test automation effort and lowers the cost of maintenance.

A framework may be built to support either commercial or open source tools available in the market. Different tools provide different inbuilt features, and a framework must focus on extending the capabilities and features not provided by the tool. A test automation framework allows you to write a series of tests without worrying about the constraints or limitations of the underlying test tools.

The most critical aspect of building a good test automation framework is its design. This Whitepaper discusses the best practices in designing a robust test automation framework.

2 P a g e

Automation Framework Designidexcel

Objectives of a Test Automation Framework

Benefits of a Well Designed Test Automation Framework

It is very important to define the objectives of a Test Automation Framework before starting its design and development. An organization may choose some or all of the objectives listed below, depending on its business needs. The design of the test automation framework should be based on its objectives defined by the organi-zation.

3 P a g e

Enhances efficiency in development of automated test scripts through modularized, reusable, and maintain-able code and data

Provides a structured development methodology to ensure uniformity of design across multiple test scripts to reduce dependency on individual test-case develop-ers

Provides reliable issue detection and efficient root-cause analysis with minimum human intervention for the system under test

Reduces dependence on subject matter experts by automatically selecting the test to execute according to test scenarios and dynamically refining the test scope according to changes in the test strategy or conditions of the system under test

Is tool agnostic

Performs multiple levels and types of testing – function-al, regression, load, performance, unit, integration, etc

Eliminates deployment to multiple machines for testing different versions of an application

Runs tests in parallel, runs on test machines in the grid or cloud, tests across multiple environments and appli-cation versions simultaneously

Runs in a continuous integration environment.

A well designed framework increases the speed and accu-racy of the testing process, enables a higher return on investment (ROI), reduces maintenance and systematically minimizes risks. A well designed framework:

Platform & OS independence

Support different testing types (UI, database, Web Service). Data driven capabilities (input data, output data)

Framework should be driven through tables / key words Ability to parameterize (recorded) scripts End to end testing capability to cover business scenarios View and edit test cases intuitively Ability to implement dynamic conditions & dynamic loop support Test framework should be loosely / tightly coupled with application-under-test (AUT) depending on business needs Auto update of test results in a test management tool Less dependence on highly skilled automation professionals Ability to schedule and run prioritized test cases Capture screenshots for failed steps and log defects in a defect management tool Test reporting objectives including email notifica-tions.


Enhances efficiency in development of automated test scripts through modularized, reusable, and maintain-able code and data

Provides a structured development methodology to ensure uniformity of design across multiple test scripts to reduce dependency on individual test-case develop-ers

Provides reliable issue detection and efficient root-cause analysis with minimum human intervention for the system under test

Reduces dependence on subject matter experts by automatically selecting the test to execute according to test scenarios and dynamically refining the test scope according to changes in the test strategy or conditions of the system under test

Is tool agnostic

Performs multiple levels and types of testing – function-al, regression, load, performance, unit, integration, etc

Eliminates deployment to multiple machines for testing different versions of an application

Runs tests in parallel, runs on test machines in the grid or cloud, tests across multiple environments and appli-cation versions simultaneously

Runs in a continuous integration environment.

idexcel

4 P a g e

The framework should be based on open technolo-gies, such as Java, to limit the development and main-tenance costs. Openness might also allow cross-tool execution when a suitable API exits or can be devel-oped

Should be application-independent and easy to enhance and maintain

The framework and its assets should allow end-to-end testing of the application suite

Encapsulate testers from complexities of the test framework

Identify and abstract common functions used across multiple test scripts

The framework should allow easy integration with other tools and components

Decouple complex business function testing from navigation, limit-testing, and other simple verification and validation activities.

Decouple test data from the test scripts

Structure scripts with minimal dependencies - ensur-ing execution of scripts unattended even on failures.

Selection of the type of Framework : The type of automa-tion framework helps to organize test suites and in turn improves testing efficiency. Different types of framework focus on different objectives. The “Data driven frame-work” focuses on data management, the “Keyword driven framework” focuses on creation of test cases using keywords, while the “Modular framework” reduces redun-dancy.

The framework type should be chosen based on the organization’s requirements and objectives. If the require-ment is to manage data and execute test cases for multiple sets of data, the “data driven framework” would be appro-priate. Similarly, if the framework objective is to divide the project into modules and create scripts independently for each module, the “Modular” framework would be appro-priate. Sometimes, a “Hybrid framework”, combining the strengths of different types of framework would work well.

Data Management: Data plays an important role in the framework design. The best approach is to ensure that there is no hard coding of data. The user should be allowed to manipulate and input the data for test cases (input and expected data). The framework contains different data such as test scripts creation data, test input and expected data, test results and should manage the flow between

them easily. The framework should support user friendly interface files (excel, CSV, database, etc). Choosing a database may become a constraint if the organization uses different databases, while CSV is not easy to use, leaving excel workbook as the most widely used format.

Reusability: A test automation framework is a set of routines that are reused by tests, increasing the speed of test development. Each layer should be developed to promote reusability. The more generic a layer, the more reusable it is. A framework with high reusability gives project teams a platform, on which they can write their test scripts with minimal effort. An important decision in reusability is to define the scripts, functions and methods that can be reused. Test steps commonly include tasks such as - verify messages, click on objects, enter, select, verify if an object is present, verify if editable, verify data and verify pages. Creation of reusable component for these test steps enhances reusability. Functions must be created for business rules by studying the list of business layers and any specific algorithms. Similarly, error handlers, to handle known and unknown errors and logging information must be built.

Automation Governance: Automation governance will ensure consistency in scripting or execution by defining a set of rules to be applied consistency across the frame-work. Governance includes coding standards, test suite structure, test script structure, creation of test suite, error and exception handling, creation of test data, creation of test cases, naming conventions and so on. Governance will ensure modifications of scripts for future enhancements easy.

Integration: The framework design should allow flexible integration with different tools such as test management and defect management tools and ensure smooth data flow between them. When a test case fails during test execution, the framework must log a defect in a defect tracking tool, and upload screen shots. Integration with a test management tool will allow execution of test cases

High Level Framework Design GuidelinesAs in SDLC to develop software applications, framework design plays a vital role in building a test automation framework for an organization. Enterprise applications are large, complex, and ever-evolving, and to keep pace with the organization’s growth, the design of the test automation framework should conform to the guide-lines listed below.

Test Automation Framework Design ConsiderationsThe design phase starts after the test automation frame-work requirements are defined and base-lined. A good test automation framework should be flexible, application agnostic, technology agnostic and future-proof. The framework must be application independent and must have the capability to expand, with the requirements of each application.

The various factors to be considered in designing the framework are discussed in detail below.

from the test management tool and establish bidirectional traceability from requirements all the way to defects. The design for integration of tools should be based on a “plug and play” approach, allowing tools to be changed or removed without affecting the main features of the frame-work.

Execution settings: Automated test execution is a critical part of framework design and the execution settings must be flexible and user friendly. The framework should allow execution of selective test cases, scheduling of test execu-tion and support execution of individual test cases, as well as execution in a batch (combination of tests). Executing test cases on multiple browsers, in parallel and in a distrib-uted manner should be one the core design considerations for a framework. The design must also take care of the different operating systems and browsers.

Report management: Reports are the face of a test auto-mation framework and enhance the value of a framework. Customizable and user friendly reporting allows a user to decide on the type, level of reports and reporting formats (html, xml, excel, word, pdf ) to be created for the execut-ed test cases. The framework should produce both summary (high level view of results of all the test cases executed) and detailed reports (showing step by step details) reports. The framework should verify results, handle errors with screenshots and report results to suit the needs of different stakeholders.

Logging Mechanism: Log generation is an important part of execution. It is vital to generate debug information at various points in the execution of a test case. The frame-work should provide a mechanism to create logs which can help to find a problem area quickly, thereby reducing the time to fix issues.

Automation Framework Design

Selection of the type of Framework : The type of automa-tion framework helps to organize test suites and in turn improves testing efficiency. Different types of framework focus on different objectives. The “Data driven frame-work” focuses on data management, the “Keyword driven framework” focuses on creation of test cases using keywords, while the “Modular framework” reduces redun-dancy.

The framework type should be chosen based on the organization’s requirements and objectives. If the require-ment is to manage data and execute test cases for multiple sets of data, the “data driven framework” would be appro-priate. Similarly, if the framework objective is to divide the project into modules and create scripts independently for each module, the “Modular” framework would be appro-priate. Sometimes, a “Hybrid framework”, combining the strengths of different types of framework would work well.

Data Management: Data plays an important role in the framework design. The best approach is to ensure that there is no hard coding of data. The user should be allowed to manipulate and input the data for test cases (input and expected data). The framework contains different data such as test scripts creation data, test input and expected data, test results and should manage the flow between

5 P a g e

them easily. The framework should support user friendly interface files (excel, CSV, database, etc). Choosing a database may become a constraint if the organization uses different databases, while CSV is not easy to use, leaving excel workbook as the most widely used format.

Reusability: A test automation framework is a set of routines that are reused by tests, increasing the speed of test development. Each layer should be developed to promote reusability. The more generic a layer, the more reusable it is. A framework with high reusability gives project teams a platform, on which they can write their test scripts with minimal effort. An important decision in reusability is to define the scripts, functions and methods that can be reused. Test steps commonly include tasks such as - verify messages, click on objects, enter, select, verify if an object is present, verify if editable, verify data and verify pages. Creation of reusable component for these test steps enhances reusability. Functions must be created for business rules by studying the list of business layers and any specific algorithms. Similarly, error handlers, to handle known and unknown errors and logging information must be built.

Automation Governance: Automation governance will ensure consistency in scripting or execution by defining a set of rules to be applied consistency across the frame-work. Governance includes coding standards, test suite structure, test script structure, creation of test suite, error and exception handling, creation of test data, creation of test cases, naming conventions and so on. Governance will ensure modifications of scripts for future enhancements easy.

Integration: The framework design should allow flexible integration with different tools such as test management and defect management tools and ensure smooth data flow between them. When a test case fails during test execution, the framework must log a defect in a defect tracking tool, and upload screen shots. Integration with a test management tool will allow execution of test cases

from the test management tool and establish bidirectional traceability from requirements all the way to defects. The design for integration of tools should be based on a “plug and play” approach, allowing tools to be changed or removed without affecting the main features of the frame-work.

Execution settings: Automated test execution is a critical part of framework design and the execution settings must be flexible and user friendly. The framework should allow execution of selective test cases, scheduling of test execu-tion and support execution of individual test cases, as well as execution in a batch (combination of tests). Executing test cases on multiple browsers, in parallel and in a distrib-uted manner should be one the core design considerations for a framework. The design must also take care of the different operating systems and browsers.

Report management: Reports are the face of a test auto-mation framework and enhance the value of a framework. Customizable and user friendly reporting allows a user to decide on the type, level of reports and reporting formats (html, xml, excel, word, pdf ) to be created for the execut-ed test cases. The framework should produce both summary (high level view of results of all the test cases executed) and detailed reports (showing step by step details) reports. The framework should verify results, handle errors with screenshots and report results to suit the needs of different stakeholders.

Logging Mechanism: Log generation is an important part of execution. It is vital to generate debug information at various points in the execution of a test case. The frame-work should provide a mechanism to create logs which can help to find a problem area quickly, thereby reducing the time to fix issues.


6 P a g e

Idexcel has built a test automation framework, i-TAF, keep-ing in mind the design considerations articulated in this whitepaper. The framework is designed for web functional

automation using Selenium. The high level architecture of i-TAF is shown below.

IDEXCEL’s Test Automation Framework (i-TAF)

Automation Framework Design

Idexcel’s Test Automation Framework (i-TAF) advocates the creation of a driver script, independent functions and scripts for work actions and specific test cases in different phases. Business flows are captured in a Keyword driven architecture and the test data is Data driven, which provides a Hybrid Framework Architecture. It also contains data tables to maintain the scripts to be executed and the application specific test data. Test execution results are published as customized reports to validate the status of

execution. Defects encountered during execution are automatically logged in a Defect Management tool.

Built on design considerations discussed in this whitepa-per, Idexcel’s i-TAF offers an efficient framework, with a high degree of reusability and maintainability. Integration with others tools such as test management and defect management provides complete end-to -end automated test management capabilities, with customized reporting.

DOCS

XLSX

READ

A

HTMLTXT

MAIL

Customized TestResults Test

Management Tools

Defect Management

Tools

(Data Drivers)

TestData

Hybrid Engine

(Keywords Drivers)

Business Flows

0 1 0 11 1 0 01 1 1 10 0 1 1

Project Library(Reusable Business Components)

Driver ScriptsLibrary (Application

Indipendent reusable Functions)

Test DataGeneration

Tool

TestingTypes

WebServices

WebApplications

Database

Defect LogTest Result

Defect LogTest Result

idexcel

7 P a g e

ConclusionDesign of a test automation framework is a vital activity in the building of a test automation framework. A well designed framework provides a higher ROI by significantly reducing the cost of test automation, without compromising on quali-ty. Implementation of the design considerations discussed in this whitepaper will help organizations build a high perfor-mance test automation framework that meets complex and future testing requirements, as the demand for improving test efficiencies and effectiveness in test cycles through test automation increases. The key to success here will be to adopt winning strategies realized through a better approach to framework design that will keep testing costs under control, without compromising quality.

Idexcel’s Test Automation Framework (i-TAF) has helped its clients to jump start their test automation initiatives and realize higher test efficiencies and return on investment.


About the AuthorAnita Shivaprakash, is a Test Automation Lead with Idexcel and has more than 8 years of experience in test automation. Anita is a certified HP AIS Professional (QTP and QC). She has strong knowledge on SDLC and STLC fundamentals along with a sound knowledge of various QA processes and techniques. Anita has expertise in design, development and implementation of different automation testing Frameworks (Keyword driven, Data driven, Modular and Hybrid). She is the chief architect of Idexcel's Test Automation Framework, i-TAF. Anita is also well versed in the other functional tools such as Test Complete, Selenium and Coded UI.

About IdexcelIdexcel is an innovative provider of IT Products & Services focused on emerging technologies. We help world leading companies build efficiencies and stronger businesses. With more than 15 years into existence Idexcel’s main focus is client satisfaction and technology innovation. Our industry expertise and a global, collaborative workforce forms the backbone of our services. We offer high degree of skills in Enterprise Applications, Cloud Services, Data-warehousing, Big Data, Analytic, QA & Testing Services, IT consulting and Staffing. Idexcel product line includes: NDS, ERP, and Cync - A revolutionary credit monitoring application for the manufacturing and �nancial management.For more information log on to www.idexcel.com.

Global Head quarters459 Herndon Parkway Suite 11Herndon, VA 20170Tel: 703-230-2600Fax: 703-467-0218Email: [email protected]

India Operations“Crystal Plaza” 9, 10 ,11Bhuvanappa Layout, Hosur RoadBengaluru – 560 029KarnatakaTel: +91-80-2550 8830Email: [email protected]

© Copyright, Idexcel. All rights reserved. No part of this document may be reproduced, stored in a retrieval system, transmitted in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the express written permission from Idexcel. The information contained herein is subject to change without notice. All other trademarks mentioned herein are the property of their respective owners.


Test Automation Framework Design |

Technology

Transcript of Test Automation Framework Design |