AUTOMATED TEST GENERATION Muhammed İ. KALKAN. Automatic Test Generation (ATG) Index Index What is...

AUTOMATED TEST AUTOMATED TEST GENERATION GENERATION

Muhammed İ. KALKANMuhammed İ. KALKAN

Automatic Test Generation (ATG)Automatic Test Generation (ATG)

IndexIndex What is ATG?What is ATG? ATG TypesATG Types Why do we need ATG?Why do we need ATG? What are the main benefits of ATGWhat are the main benefits of ATG ATG MethodsATG Methods Testing MethodologiesTesting Methodologies Automatic Test Pattern GenerationAutomatic Test Pattern Generation Relevant ExamplesRelevant Examples

What is ATG?What is ATG?

Testing an engineered system is the most important part of the product revealed.

Engineers are working on and putting alot of effort into this issue.

Since testing process is required for all obtained engineering products, engineers decided to develop an automated test mechanism instead of generating test cases manually.

What is ATG?What is ATG?

Automated Test Generation includes test cases , scenarios and the evalution of the obtained result such as passed and failed.

Given inputs and obtained outputs are evaluated automatically.

The Main Idea : TEST COMPLETELY TEST COMPLETELY

AUTOMATICALLYAUTOMATICALLY

ATG or ATPG?ATG or ATPG?

ATG is an engineering subject which ATG is an engineering subject which can be applied to both Computer can be applied to both Computer Sciences and other engineering Sciences and other engineering specialities like Electronics or specialities like Electronics or hardware circuits.hardware circuits.

ATG is a term which is generally ATG is a term which is generally used for Computer Sciences used for Computer Sciences especially software testing.especially software testing.

On the other hand for hardware On the other hand for hardware testing Automatic Test Generation is testing Automatic Test Generation is usually referred as Automatic Test usually referred as Automatic Test Pattern Generation.Pattern Generation.

ATG or ATPG?ATG or ATPG?

ATPG ATPG is defined as follows:is defined as follows: ATPGATPG is an electronic pattern is an electronic pattern

automation method/technology used automation method/technology used to find an input (or test) sequence to find an input (or test) sequence

When applied to a digital circuit, it When applied to a digital circuit, it enables testers to distinguish enables testers to distinguish between the correct circuit behavior between the correct circuit behavior and the faulty circuit behavior and the faulty circuit behavior caused by defects. caused by defects.

ATG…….But WHY??ATG…….But WHY??

As years have passed,computer hardwares As years have passed,computer hardwares and its capabilities grow bigger and bigger. and its capabilities grow bigger and bigger.

Result 1: Software types and their Result 1: Software types and their capabilities are varied.capabilities are varied.

Result 2: New software demand on market Result 2: New software demand on market for jobs.for jobs.

Result 3: Bigger income ,more firms to Result 3: Bigger income ,more firms to compete for market ration.compete for market ration.

Result 4: Requirement for strong,bug free Result 4: Requirement for strong,bug free and developed in small amount of time and developed in small amount of time softwaressoftwares


Result 5 : Software Engineering Result 5 : Software Engineering and Testing studies.and Testing studies.


Software testing is importantSoftware errors cost the U.S.

economy about $59.5 billion each year

Improving testing infrastructure could save 1/3 cost

Software testing is costlyAccount for even half the total cost

of software development

BenefitsBenefits

Reduces Testing TimeReduces Testing Time Reduces Testing CostReduces Testing Cost Reduces human errors while Reduces human errors while

testingtesting

ATG MethodsATG Methods

Data-Driven ATGData-Driven ATG xUnit Frameworks for ATGxUnit Frameworks for ATG

Examples : JUnit and NUnitExamples : JUnit and NUnit Model-Based ATGModel-Based ATG Keyword-Driven ATGKeyword-Driven ATG

xUnitxUnit

Various code-driven testing Various code-driven testing frameworks have come to be frameworks have come to be known collectively as known collectively as xUnitxUnit..

Based on a design by Kent Based on a design by Kent Beck, originally implemented for Beck, originally implemented for Smalltalk as SUnit, but are now Smalltalk as SUnit, but are now available for many programming available for many programming languages and development languages and development platforms. platforms.

xUnitxUnit

The overall design of xUnit frameworks The overall design of xUnit frameworks depends on several components.depends on several components.

Test FixturesTest Fixtures A test fixture is the set of preconditions or A test fixture is the set of preconditions or

state needed for a test to run. Also known state needed for a test to run. Also known as a test context.as a test context.

Test SuitesTest Suites A test suite is a set of tests that all share the A test suite is a set of tests that all share the

same fixture.same fixture. Test ExecutionTest Execution

The execution of an individual unit The execution of an individual unit test proceeds as follows:test proceeds as follows:

xUnitxUnit

The setup() and teardown() methods serve to The setup() and teardown() methods serve to initialize and clean up test fixtures. initialize and clean up test fixtures.

AssertionsAssertions An assertion is a function or macro that An assertion is a function or macro that

verifies the behavior of the unit under test. verifies the behavior of the unit under test. Failure of an assertion typically throws an Failure of an assertion typically throws an exception, aborting the execution of the exception, aborting the execution of the current test.current test.

Data-Driven ATGData-Driven ATG

Data-driven testingData-driven testing is a methodology is a methodology used in Test automationused in Test automation

Test scripts are executed and verified Test scripts are executed and verified based on the data values stored in one or based on the data values stored in one or more central data sources or databases.more central data sources or databases.

Databases can range from datapools, Databases can range from datapools, ODBC sources, csv files, Excel files, DAO ODBC sources, csv files, Excel files, DAO objects, ADO objects, etc.objects, ADO objects, etc.

Data-driven testing is the establishment of Data-driven testing is the establishment of several interacting test scripts together with several interacting test scripts together with their related data results in a framework their related data results in a framework used for the methodology. used for the methodology.

Data-Driven ATGData-Driven ATG

Variables are used for both input values and Variables are used for both input values and output verification values: navigation through output verification values: navigation through the program, reading of the data sources, and the program, reading of the data sources, and logging of test status and information are all logging of test status and information are all coded in the test script. Thus, the logic coded in the test script. Thus, the logic executed in the script is also dependant on the executed in the script is also dependant on the data values. data values.

An example of xUnit Framework Automatic An example of xUnit Framework Automatic Testing by using Data-Driven Test Testing by using Data-Driven Test Methodology is as follows:Methodology is as follows:

ExampleExample Put the data that Put the data that

varies from test varies from test to test into the to test into the Data-Driven TestData-Driven Test file that the file that the interpreter reads interpreter reads to execute the to execute the teststests

For each test it For each test it does the same does the same sequence of sequence of actions to actions to implement the implement the

Four-Phase TestFour-Phase Test

Example (FPT)Example (FPT)

Set-up the test fixtureSet-up the test fixture Try the test caseTry the test case Evaluate the resultEvaluate the result Release the fixtureRelease the fixture

Keyword-Driven ATGKeyword-Driven ATG

It is also known as table-driven testing, action-It is also known as table-driven testing, action-word testing word testing

It can be used for manuel testing as well as It can be used for manuel testing as well as automated testingautomated testing

The advantages for automated tests are the The advantages for automated tests are the reusability and therefore ease of maintenance reusability and therefore ease of maintenance of tests which have been created at a high of tests which have been created at a high level of abstraction.level of abstraction.

The Keyword-Driven ATG methodology divides The Keyword-Driven ATG methodology divides test creation into two stages. test creation into two stages. Planning PhasePlanning Phase Implementing PhaseImplementing Phase


Planning PhasePlanning Phase

The application (or the The application (or the requirements for the application) requirements for the application) is analyzed to determine which is analyzed to determine which operations and objects will need operations and objects will need to be tested.to be tested.

ExampleExample

A web-questionnaire application A web-questionnaire application will require a large amount of text entries. will require a large amount of text entries. By identifying which operations should be By identifying which operations should be encapsulated into keywords, the encapsulated into keywords, the efficiency and maintainability of the tests efficiency and maintainability of the tests are maximized. are maximized.


Implementation PhaseImplementation Phase

The implementation stage differs depending on the The implementation stage differs depending on the tool or framework used.tool or framework used.

Often, automation engineers implement a Often, automation engineers implement a framework that provides keywords like “check” and framework that provides keywords like “check” and “enter”. “enter”.

Testers or test designers (who don’t have to know Testers or test designers (who don’t have to know how to program) write test cases based on the how to program) write test cases based on the keywords defined in the planning stage, which have keywords defined in the planning stage, which have been implemented by the engineers.been implemented by the engineers.

The test is executed using a driver which reads the The test is executed using a driver which reads the keywords and executes the corresponding code. keywords and executes the corresponding code.


Other methodologies use an all-in-one Other methodologies use an all-in-one implementation stage.implementation stage.

Instead of separating the tasks of test Instead of separating the tasks of test design and test engineering, the test design and test engineering, the test design design isis the test automation. the test automation.

Keywords, such as “enter” or “check” are Keywords, such as “enter” or “check” are created using tools in which the necessary created using tools in which the necessary code has already been written. code has already been written.

This removes the necessity for extra This removes the necessity for extra engineers in the test process, because the engineers in the test process, because the implementation for the keywords is already implementation for the keywords is already a part of the tool. a part of the tool.


BenefitsBenefits

This methodology requires more planning and a This methodology requires more planning and a longer initial time-investment than going directly to longer initial time-investment than going directly to the test creation stage and recording a test.the test creation stage and recording a test.

It does make the test creation and test maintenance It does make the test creation and test maintenance stages more efficient and keeps the structure of stages more efficient and keeps the structure of individual tests more readable and easier to modify.individual tests more readable and easier to modify.

The more abstract keywords are, the more reusable The more abstract keywords are, the more reusable they are, and therefore the easier a test is to they are, and therefore the easier a test is to maintain. maintain.

As well as reducing the cost and time spent As well as reducing the cost and time spent maintaining and updating tests, the modular maintaining and updating tests, the modular structure of Keyword-Driven ATG means that new structure of Keyword-Driven ATG means that new tests can easily be created from pre-existing tests can easily be created from pre-existing modules.modules.


Another advantage is the reduction Another advantage is the reduction in technical know-how required for in technical know-how required for the test automation process.the test automation process.

In the first approach, technical know-In the first approach, technical know-how is only required by the how is only required by the engineers that implement the engineers that implement the keywords. keywords.

In the second approach, even this is In the second approach, even this is not required, which means that the not required, which means that the test team is capable of entirely test team is capable of entirely automating tests, even without automating tests, even without programming knowledge.programming knowledge.

Model Based ATGModel Based ATG

Model-based testingModel-based testing is software testing in is software testing in which test cases are derived in whole or in part which test cases are derived in whole or in part from a model that describes some (usually from a model that describes some (usually functional) aspects of the system under test functional) aspects of the system under test processprocess The model is usually an abstract, partial The model is usually an abstract, partial

presentation of the system under test's desired presentation of the system under test's desired behavior. behavior.

The test cases derived from this model are The test cases derived from this model are functional tests on the same level of functional tests on the same level of abstraction as the model abstraction as the model

Can not run abstracted test cases.Need Can not run abstracted test cases.Need executable ones.executable ones.


Diagram description : Diagram description :

Model-Based Test SystemsModel-Based Test Systems


Many ways to produce test cases.Many ways to produce test cases. No best method for producing tests.No best method for producing tests. Use your test interest to define your test Use your test interest to define your test

cases.cases. Design decisions is often known as Design decisions is often known as

"test requirements", "test purpose" or "test requirements", "test purpose" or even "use case" even "use case"


Use model and test requirements to Use model and test requirements to derive abstract test casesderive abstract test cases

Implementation Extra Information to Implementation Extra Information to abstract test suit to produce executable abstract test suit to produce executable test casestest cases


Execute the testExecute the test Evaluate the reportsEvaluate the reports

Model Based ATG Model Based ATG

There are a few ways to deploy the There are a few ways to deploy the Model-Based TestingModel-Based Testing Online TestingOnline Testing: A model-based testing : A model-based testing

tool connects “directly” to a system tool connects “directly” to a system under test and tests it dynamically. under test and tests it dynamically.

Offline Generation of Executable Offline Generation of Executable TestsTests:: A model-based testing tool A model-based testing tool generates test cases as a computer-generates test cases as a computer-readable asset that can be later readable asset that can be later deployed automatically. deployed automatically.

A collection of Python classes that embodies A collection of Python classes that embodies the generated testing logic the generated testing logic


Offline Generation of Manually Offline Generation of Manually Deployable Tests:Deployable Tests: A model-based A model-based testing tool generates test cases as a testing tool generates test cases as a human-readable asset that can be later human-readable asset that can be later deployed manually. deployed manually.

A PDF document in English that A PDF document in English that describes the generated test steps describes the generated test steps

Another Step :Another Step : Test GenerationTest Generation


Model-Based Testing has a good potential in Model-Based Testing has a good potential in aspect of automation.aspect of automation.

Test case generation is one of the issues that Test case generation is one of the issues that ATG is subject toATG is subject to

To find test cases, Model-Based structure is To find test cases, Model-Based structure is interpreted as a kind of Finite State Machine to interpreted as a kind of Finite State Machine to see different states that the system might be insee different states that the system might be in

There are a few technics for generating test There are a few technics for generating test cases.Most knowns are as follows:cases.Most knowns are as follows:


Test case generation by theorem provingTest case generation by theorem proving System model is defined as predicates,kinds of System model is defined as predicates,kinds of

logical expressionslogical expressions These expression are processed and proved These expression are processed and proved

afterwardsafterwards Test case generation by constraint logic Test case generation by constraint logic

programmingprogramming Selecting test cases satisfying specific Selecting test cases satisfying specific

constraints by solving a set of constraints over a constraints by solving a set of constraints over a set of variables set of variables


Test case generation by model checkingTest case generation by model checking We provide a model of the system under test and a We provide a model of the system under test and a

property we want to test to the model checker property we want to test to the model checker Test case generation by symbolic executionTest case generation by symbolic execution

Searching for execution traces in an abstract model. Searching for execution traces in an abstract model. In principle the program execution is simulated using In principle the program execution is simulated using symbols for variables rather than actual values symbols for variables rather than actual values

Test case generation by using an event-flow Test case generation by using an event-flow modelmodel With a graphical user-interface (GUI) front-end With a graphical user-interface (GUI) front-end

is called the event-flow model that represents is called the event-flow model that represents events and event interactions events and event interactions

ExampleExample

TestMaster™ automates the generation of TestMaster™ automates the generation of tests for call processing features developed tests for call processing features developed for the 5ESS®-2000 Switchfor the 5ESS®-2000 Switch

This test uses Model-Based Test GenerationThis test uses Model-Based Test Generation Results obtained here as follows.Results obtained here as follows.

AA test generation productivityimprovement test generation productivityimprovement

of just over 90%of just over 90%

Automatic Test Pattern Automatic Test Pattern GenerationGeneration ATPG is an electronic design automation ATPG is an electronic design automation

method/technology used to find an input (or method/technology used to find an input (or test) sequence that, when applied to a digital test) sequence that, when applied to a digital circuit, enables testers to distinguish between circuit, enables testers to distinguish between the correct circuit behavior and the faulty circuit the correct circuit behavior and the faulty circuit behavior caused by defectsbehavior caused by defects

The generated patterns are used to test The generated patterns are used to test semiconductor devices after manufacture, semiconductor devices after manufacture,

In some cases to assist with determining the In some cases to assist with determining the cause of failure cause of failure

Automatic Test Pattern Automatic Test Pattern GenerationGeneration The effectiveness of ATPG is The effectiveness of ATPG is

measured by the amount of modeled measured by the amount of modeled defects, or fault models, that are defects, or fault models, that are detected and the number of detected and the number of generated patterns.generated patterns.

We can estimate test quality by We can estimate test quality by effectiveness of ATGPeffectiveness of ATGP

Automatic Test Pattern Automatic Test Pattern GenerationGeneration A defect is an error introduced into a device A defect is an error introduced into a device

during the manufacturing process.during the manufacturing process. A fault model is a mathematical description of A fault model is a mathematical description of

how a defect alters design behavior.how a defect alters design behavior. A fault is said to be A fault is said to be detecteddetected by a test pattern by a test pattern

if, when applying the pattern to the design, any if, when applying the pattern to the design, any logic value observed at one or more of the logic value observed at one or more of the circuit's primary outputs differs between the circuit's primary outputs differs between the original design and the design with the fault.original design and the design with the fault.

There are two steps that ATPG should take to There are two steps that ATPG should take to detect fault.detect fault.

Automatic Test Pattern Automatic Test Pattern GenerationGeneration Fault activation: Fault activation: Establishes a signal Establishes a signal

value at the fault model site that is value at the fault model site that is opposite of the value produced by the opposite of the value produced by the fault model fault model

Fault propagation:Fault propagation: Moves the resulting Moves the resulting signal value, or fault effect, forward by signal value, or fault effect, forward by sensitizing a path from the fault site to a sensitizing a path from the fault site to a primary output.primary output.

Sequential ATPGSequential ATPG

Searches for a sequence of vectors to detect a Searches for a sequence of vectors to detect a particular fault through the space of all possible particular fault through the space of all possible vector sequencesvector sequences Vector Sequence: A sequence of values for Vector Sequence: A sequence of values for

circuit inputcircuit input In Sequential Circuits In Sequential Circuits due to the presence of due to the presence of

memory elements, the controllability and memory elements, the controllability and observability of the internal signals (in observability of the internal signals (in general)are much more difficult than those general)are much more difficult than those in a combinational circuitin a combinational circuit

Complexity of sequential ATPG much Complexity of sequential ATPG much higher than that of combinational ATPG. higher than that of combinational ATPG.

Algorithmic MethodsAlgorithmic Methods

Testing very-large-scale integrated circuits with Testing very-large-scale integrated circuits with a high fault coverage is a difficult task because a high fault coverage is a difficult task because of complexity. of complexity.

Many different ATPG methods have been Many different ATPG methods have been developed to address combinatorial and developed to address combinatorial and sequential circuits.sequential circuits.

Some examples:Some examples: Pseudorandom test generation is the simplest Pseudorandom test generation is the simplest

method of creating tests. It uses a method of creating tests. It uses a pseudorandom number generator to generate pseudorandom number generator to generate test vectorstest vectors

Algorithmic MethodsAlgorithmic Methods

The D Algorithm was the first practical test The D Algorithm was the first practical test generation algorithm in terms of memory generation algorithm in terms of memory requirements. The D Algorithm introduced D requirements. The D Algorithm introduced D Notation which continues to be used in most Notation which continues to be used in most ATPG algorithms.ATPG algorithms.

Fan-Out Oriented Algorithm: It limits the Fan-Out Oriented Algorithm: It limits the ATPG search space to reduce computation ATPG search space to reduce computation time and accelerates backtracing. time and accelerates backtracing.

Automatic Test Pattern Automatic Test Pattern GenerationGeneration ATPG can fail to find a test for a ATPG can fail to find a test for a

particular fault in at least two cases. particular fault in at least two cases. The fault may be intrinsically The fault may be intrinsically

undetectableundetectable it is possible that a pattern(s) it is possible that a pattern(s)

exist, but the algorithm cannot find exist, but the algorithm cannot find it it

ReferencesReferences Automated Test GenerationAutomated Test Generation, , (From a Behavioral (From a Behavioral

Model)Model), , James M. ClarkeJames M. Clarke, , Lucent TechnologiesLucent Technologies Automatic Test Pattern Generation , Automatic Test Pattern Generation ,

http://en.wikipedia.org/wiki/Automatic_test_pattern_http://en.wikipedia.org/wiki/Automatic_test_pattern_generationgeneration

Data Driven Tests, Gerrard Meszaros, http://xunitpatterns.com/Data-Driven%20Test.html, 2007

Data-Driven Testing, Data-Driven Testing, http://en.wikipedia.org/wiki/Data-driven_testinghttp://en.wikipedia.org/wiki/Data-driven_testing

ReferencesReferences

Test Automation, Test Automation, http://en.wikipedia.org/wiki/Test_automationhttp://en.wikipedia.org/wiki/Test_automation

Test Automation Framework, Test Automation Framework, http://en.wikipedia.org/wiki/Test_automation_framehttp://en.wikipedia.org/wiki/Test_automation_frameworkwork

A Survay on Automatic Test Case Generation, M.Prasanna, S.N. Sivanandam, R.Venkatesan, R.Sundarrajan, Department of Computer Science and Engineering, PSG College Of Technology, http://www.acadjournal.com/2005/v15/part6/p4/

Keyword-Driven Testing , Keyword-Driven Testing , http://en.wikipedia.org/wiki/Keyword-driven_testinghttp://en.wikipedia.org/wiki/Keyword-driven_testing

ReferencesReferences

Model – Based Testing, Model – Based Testing, http://en.wikipedia.org/wiki/Model-based_testinghttp://en.wikipedia.org/wiki/Model-based_testing

Improving Automation in Developer Testing: Achievements and Challenges, Tao Xie, Department of Computer Science, North Carolina State University

xUnit , http://en.wikipedia.org/wiki/XUnitxUnit , http://en.wikipedia.org/wiki/XUnit

The EndThe End

Thank YouThank You

Q & AQ & A

AUTOMATED TEST GENERATION Muhammed İ. KALKAN. Automatic Test Generation (ATG) Index Index What is...

Documents

Transcript of AUTOMATED TEST GENERATION Muhammed İ. KALKAN. Automatic Test Generation (ATG) Index Index What is...