February 27, 2013 Douglas Hoffman, BACS, MBA, MSEE, ASQ ... · – Past Chair, Silicon Valley...

of 88 /88
Belgium Testing Days 2013 Exploratory Automated Tests (Tutorial) February 27, 2013 Page 1 Exploratory Automated Testing Belgium Testing Days February 27, 2013 Tutorial Part 1 Douglas Hoffman, BACS, MBA, MSEE, ASQ-CSQE, ASQ-CMQ/OE, ASQ Fellow Software Quality Methods, LLC. (SQM) www.SoftwareQualityMethods.com [email protected] Douglas Hoffman Copyright © 2004-13, SQM, LLC. 1 About Doug Hoffman I am a management consultant in testing/QA strategy and tactics. I help plan quality strategies and tactical approaches for organizations, especially esoteric test automation. I gravitated into quality assurance from engineering. I’ve been a production engineer, developer, support engineer, tester, writer, instructor, and I’ve managed manufacturing quality assurance, software quality assurance, technical support, software development, and documentation. My work has cut across the industry from start-ups to multi-trillion dollar organizations. Along the way I have learned a great deal about software testing and automation. I enjoy sharing what I’ve learned with interested people. Current employment President of Software Quality Methods, LLC. (SQM) Management consultant in strategic and tactical planning for software quality Education B.A. in Computer Science MS in Electrical Engineering, (Digital Design and Information Science) MBA Professional Past President, Association for Software Testing Past Chair, Silicon Valley Section, American Society for Quality (ASQ) Founding Member and Current Chair, Santa Clara Valley Software Quality Association (SSQA) Certified in Software Quality Engineering (ASQ-CSQE) Certified Quality Manager (ASQ-CMQ/OE,) Participant in the Los Altos Workshop on Software Testing and dozens of other offshoots Douglas Hoffman Copyright © 2004-13, SQM, LLC. 2

Embed Size (px)

Transcript of February 27, 2013 Douglas Hoffman, BACS, MBA, MSEE, ASQ ... · – Past Chair, Silicon Valley...

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 1

    Exploratory Automated Testing

    Belgium Testing DaysFebruary 27, 2013

    Tutorial Part 1

    Douglas Hoffman, BACS, MBA, MSEE,

    ASQ-CSQE, ASQ-CMQ/OE, ASQ Fellow

    Software Quality Methods, LLC. (SQM)

    www.SoftwareQualityMethods.com

    [email protected]

    Douglas Hoffman Copyright © 2004-13, SQM, LLC. 1

    About Doug HoffmanI am a management consultant in testing/QA strategy and tactics. I help plan quality strategies and tactical approaches for organizations, especially esoteric test automation. I gravitated into quality assurance from engineering. I’ve been a production engineer, developer, support engineer, tester, writer, instructor, and I’ve managed manufacturing quality assurance, software quality assurance, technical support, software development, and documentation. My work has cut across the industry from start-ups to multi-trillion dollar organizations. Along the way I have learned a great deal about software testing and automation. I enjoy sharing what I’ve learned with interested people.

    Current employment

    – President of Software Quality Methods, LLC. (SQM)

    – Management consultant in strategic and tactical planning for software quality

    Education

    – B.A. in Computer Science

    – MS in Electrical Engineering, (Digital Design and Information Science)

    – MBAProfessional

    – Past President, Association for Software Testing

    – Past Chair, Silicon Valley Section, American Society for Quality (ASQ)

    – Founding Member and Current Chair, Santa Clara Valley Software Quality Association (SSQA)

    – Certified in Software Quality Engineering (ASQ-CSQE)

    – Certified Quality Manager (ASQ-CMQ/OE,)

    – Participant in the Los Altos Workshop on Software Testing and dozens of other offshoots

    Douglas Hoffman Copyright © 2004-13, SQM, LLC. 2

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 2

    Today’s Topics

    • Test automation

    • Automated exploratory tests (Part 1)

    • Ancillary test support

    • Test oracles and automation

    • Automated exploratory tests (Part 2)

    • Results comparison

    Douglas Hoffman 3Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 4

    The Question About SUT

    Misbehavior

    What can happen if there is a bug in the SUT?

    Anything!

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 3

    Douglas Hoffman 5

    Automation Background

    Working Concepts

    Copyright © 2004-13, SQM, LLC.

    Opportunities For Automation

    • Program analysis

    • Test design

    • Test case management/selection

    • Input selection/generation

    • Automated test case

    • Test execution control

    • Actual results capture

    • Expected results generation

    • Results comparison

    • Report generationDouglas Hoffman 6Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 4

    Automation Defined For This Class

    Get the computer to do one or more of:

    − Input selection/generation

    − Automated test case

    − Test execution control

    − Actual results capture

    − Expected results generation

    − Results comparison

    Douglas Hoffman 7Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 8

    Automated vs. Manual Tests

    An automated test is not equivalent to the most

    similar manual test:

    – Automated comparison is typically more precise

    (and may be tripped by irrelevant discrepancies)

    – Skilled human comparison samples a wider range

    of dimensions, noting oddities that one wouldn't

    program the computer to detect

    – Automated input is consistent, while humans

    cannot closely replicate their test activities

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 5

    Douglas Hoffman 9

    Automation Narrows Our Scope

    An automated test is more limited than a

    manual test:

    − The test exercise must be automated in advance

    − People can integrate outside experience

    − People may gain insights

    − We can only check machine available results

    − We only check result values that we pre-specify

    − Cannot easily recover when something

    unexpected happens

    Copyright © 2004-13, SQM, LLC.

    Typical Automated Tests

    • Mimic a human tester’s actions to repeat and

    speed up manual testing

    • Work at the UI level

    • Are a list of test activities (a script)

    • Check results at specified points in the script

    • Are based on the functions of a test tool

    Douglas Hoffman 10Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 6

    Regression Test or Demo?

    Regression Test:• Think of an interesting test

    • Map out steps to perform the

    test

    • Run the steps manually

    − if there is a bug; report

    and wait until fixed

    − if no bugs, capture the

    steps

    • Rerun the steps to repeat the

    test

    Demo:• Think of an interesting demo

    • Map out steps to perform the

    demo

    • Run the steps manually

    − if there is a bug; report

    and wait until fixed

    − if no bugs, capture the

    steps

    • Rerun the steps to repeat the

    demo

    Douglas Hoffman Copyright © 2012, SQM, LLC. 11

    The goal is to find bugs The goal is to AVOID bugs

    Valuable Regression Automation

    • Build/smoke tests

    • Required repeatability

    • Rerun many times

    • Demo script

    • Comfort managers

    Douglas Hoffman 12Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 7

    Douglas Hoffman 13

    Questions We Should Ask

    About Testing and Automation

    • Should we limit our thinking to what a tool does?

    • Should we focus automation on things we can do

    manually and script?

    • Do speedy manual tests find more or different

    bugs than manually running tests?

    • Should we limit ourselves to UIs or APIs?

    • Are we checking everything that’s important?

    • Can inefficient or approximate tests be valuable?

    • Must tests do the same things every time?

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 14

    Exploratory Automated Tests

    Part 1: Introduction

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 8

    Douglas Hoffman 15

    Examples of Exploratory Automation

    • Cem Kaner’s “Telenova” example using random events

    • Statistical packet profiles for data link testing

    • Using “Dumb monkeys” and MS Word

    • “Sandboxed” random regression tests

    • 1/3 or 3x heuristic in a test harness

    • Periodic database unload/check

    • Database links checking

    • Database locking (single and multi-threaded)

    • Long walks for a device front panel state machine

    • Database load/unload dropouts and shifts

    • Database unbalanced splitting

    • Random machine instruction generation

    Copyright © 2004-13, SQM, LLC.

    The Easy Part of Testing

    • Driving the inputs or generating data is

    usually straightforward

    • Design of tests that can reveal bugs is more

    difficult

    • The hard part is telling whether or not the

    SUT is misbehaving (the oracle problem)

    Exploratory Automated Testing is about

    solving all three parts

    Douglas Hoffman Copyright © 2004-13, SQM, LLC. Slide 16

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 9

    Douglas Hoffman 17

    What Makes Automation

    Exploratory?

    • Enables and amplifies tester’s abilities

    • Does something different every time

    • May use massive numbers of iterations

    • May use multiple parallel oracles

    • Can find bugs we never imagined

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 18

    Enable the Exploratory Tester

    • Enables a human tester to do more things

    • Enables a human tester to work faster

    • Amplifies a human tester’s abilities

    • Can look under the covers

    • Can provide real time monitoring

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 10

    Douglas Hoffman 19

    Randomness and

    Exploratory Tests

    • The power of doing new things

    • Random number generators

    • Randomized input values

    • Randomized data generation

    • Random test selection

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 20

    Advantages of Exploratory

    Automation

    • Supplement the baseline tests

    • Does things a human tester cannot do

    • Does something different each time

    • May use massive numbers of iterations

    • May feed inputs directly to the SUT

    • Oracles may check internal information

    • May run multiple parallel oracles

    • Can uncover obscure bugs

    • Can uncover bugs impossible to find manuallyCopyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 11

    Douglas Hoffman 21

    Disadvantages of Exploratory

    Automation

    • May not be repeatable (even with seeds)

    • Difficult to capture program and system

    information for diagnosis

    • Coordination of test activities and

    multiple, autonomous, real-time oracles

    • Does not provide rigorous coverage

    • Can uncover bugs that can’t be fixed

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 22

    A Model of Test Execution

    Setting The Stage For Oracles

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 12

    Douglas Hoffman 23

    Oracles and Test Automation

    Good automated testing depends on our ability

    to programmatically detect whether the SUT

    behaves in expected or unexpected ways

    Our ability to automate testing is

    fundamentally constrained by our

    ability to create and use oracles.

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 24

    The Oracle

    • The principle or mechanism for telling

    whether the SUT behavior appears OK or if

    further investigation is required

    • Answers the question “is this expected or

    acceptable behavior?”

    • A fundamental part of every test execution

    • Expected result is NOT required in advance

    • Multiple oracles can be used for a test

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 13

    Douglas Hoffman 25

    Test Execution Model

    System

    Under

    Test

    Test Inputs

    Input Data

    Program State

    Environmental

    Inputs

    Test Results

    Postcondition Data

    Post-condition

    Program State

    Environmental

    Results

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 26

    Implications of the Model

    • The test exercise is usually the easy part

    • We don’t control all influences

    • We can’t check everything

    • Multiple domains are involved

    • We can’t know all the factors

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 14

    Douglas Hoffman 27

    Notes On Test Oracles

    • At least one type is used in every test

    • Implemented oracles may have some characteristics of multiple types of oracles

    • Multiple oracles may be used for one test

    • Oracles may be independent or integrated into a test

    • They may be synchronous or asynchronous

    • The oracles are key to good automation

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 28

    Important Factors

    in Outcome Comparison

    • Which outcomes do we check?

    • How do we know what to expect?

    • Do we know how to capture it?

    • Can (or should) we store the results?

    • What differences matter?

    • Are “fuzzy” comparisons needed?

    • When do we compare outcomes?

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 15

    Douglas Hoffman 29

    Exploratory Automated Testing

    Ansilary Support

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 30

    Keys to Support of EAT

    • The oracles are key to identifying program

    misbehavior

    • Logging of important data for anomaly

    analysis is critical

    • Coordination of oracles with test is desirable

    • Keep reported data consistent – a human is

    going to be involved when anomaly detected

    • Systems must have similar configurations

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 16

    Douglas Hoffman 31

    Management Support

    • If management values testing for important

    bugs, they should consider EAT as valuable

    – Understand how much effort EAT deserves

    – Keep the test count and flagged errors out of

    metrics (until a bug is confirmed)

    • If management does not understand the value

    of EAT

    – Educate them if possible

    – Possibly covert measures are appropriate

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 32

    Administrative Support

    • The systems must be configured similarly

    – Drives

    – Versions of OS (except when testing different

    platforms)

    – Data set locations

    – Permissions (except when testing them)

    – Background processes (except oracles)

    • Updating needs to be managed centrally

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 17

    Douglas Hoffman 33

    Development Guidelines

    • Close support from development is

    extremely valuable for EAT

    • Some EAT can be done without direct

    support from development

    – Ask two questions of the developers:

    • What data do you need if I find a potential error?

    • How can I get that data for you?

    – Use a black box (or grey box) approach

    – Limit background processes (except oracles)

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 34

    Software Test Automation:

    About Test Oracles

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 18

    Douglas Hoffman 35

    Defining A Test Oracle

    The mechanism or principle by which we determine whether or not the

    software under test (SUT) isbehaving reasonably.

    – Unreasonable SUT behavior requires investigation by a person.

    – We just move on when the oracle says the behavior is unremarkable.

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 36

    Test Execution Model

    System

    Under

    Test

    Test Inputs

    Input Data

    Program State

    Environmental

    Inputs

    Test Results

    Postcondition Data

    Post-condition

    Program State

    Environmental

    Results

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 19

    Douglas Hoffman 37

    The Test Oracle

    Different views on the use of oracles:

    – Reference Function: You ask the oracle what the “correct” answer is and it gives you outcomes

    – Evaluation Function: You ask the oracle whether the program behavior should be investigated independent of inputs

    – Reference and Evaluation Function: You ask the oracle whether the program behavior is nominal or abnormal given inputs and outcomes

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 38

    Types of Comparisons

    Comparing the program’s behavior to a

    reference of expected behavior can be

    inexact.

    – Deterministic oracle (mismatch means

    behavior is abnormal)

    – Probabilistic oracle (means behavior is

    probably abnormal)

    Sometimes comparison is done in the oracle.

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 20

    Douglas Hoffman 39

    Which Outcomes To Check

    • Expected results

    • Anticipated likely errors

    • Available easy oracles

    • Major environmental factors

    • Invariants

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 40

    Checking Results

    • Exact comparison− Values− Ranges − Behaviors − Key words/values

    • Heuristic comparison− Similarity− Algorithm based− Secondary characteristics

    • Check during or after the test run

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 21

    Douglas Hoffman 41

    Software Test Automation:

    Oracle Characteristics

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 42

    Oracles: Challenges

    � Completeness of information

    � Accuracy of information

    � Usability of the oracle or of its results

    � Maintainability of the oracle

    � Complexity when compared to the SUT

    � Temporal relationships

    � Costs

    � Value (ROI)

    To 127

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 22

    Douglas Hoffman 43

    Oracle Completeness

    • Input Coverage

    • Result Coverage

    • Function Coverage

    • Sufficiency of the checks

    • Types of errors it might detect

    • SUT environments it works for

    There may be more than one oracle for the SUT

    Inputs may affect more than one oracle

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 44

    Oracle Accuracy

    • How similar to SUT− Arithmetic accuracy

    − Statistical similarity

    • How extensive− The more ways in which the oracle matches

    the SUT, i.e. the more complex the oracle, the

    more errors

    • What types of possible errors − Misses

    − False alarmsCopyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 23

    Douglas Hoffman 45

    Oracle Accuracy

    • Independence from the SUT

    − Algorithms

    − Sub-programs & libraries

    − System platform

    − Operating environment

    Close correspondence makes common mode faults

    more likely and often reduces maintainability

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 46

    Oracle Usability

    • Form of information

    − Bits and bytes

    − Electronic signals

    − Hardcopy and display

    • Location of information

    • Data set size

    • Fitness for intended use

    • Availability of comparators

    • Support in SUT environments

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 24

    Douglas Hoffman 47

    Oracle Maintainability

    • COTS or custom

    − A custom oracle can become more complex

    than the SUT

    − More complex oracles make more errors

    • Keeping correspondence through changes

    − Test exercises

    − Test data

    − Tools

    • Ancillary support activitiesCopyright © 2004-13, SQM, LLC.

    Douglas Hoffman 48

    Oracle Complexity

    • Correspondence with SUT

    • Coverage of SUT domains and

    functions

    • Accuracy of generated results

    • Maintenance effort necessary to keep

    correspondence through SUT changes

    • Difficulty of ancillary support activities

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 25

    Douglas Hoffman 49

    Oracle Temporal Relationships

    • Speed of results generation

    • Speed of comparison

    • When the oracle is run

    • When results are generated

    • When results are compared

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 50

    Oracle Costs

    • Creation or acquisition costs

    • Maintenance of oracle and comparators

    • Execution cost

    • Comparison cost

    • Additional analysis of errors

    • Cost of misses and false alarms

    • Cost of ancillary activities

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 26

    Douglas Hoffman 51

    Oracle Value

    • Value and costs are compared with ROI

    • Financial ROI computed with cost savings

    • Test ROI is based on value of information

    − Total cost of a test (create, check, misses,

    false-alarms, opportunity, etc.)

    − Real value of the information (required

    knowledge, insignificant, etc.)

    − Not easily valued in monetary form

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 52

    Software Test Automation:

    Types of Test Oracles

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 27

    Types of Test Oracles

    • No Oracle

    • Independent

    Implementation

    • Consistency

    • Self-Verifying

    • Model-based

    • Constraint-Based

    • Probabilistic

    • Statistical

    • Property-Based

    • Computational

    • Diagnostic

    • Hand-crafted

    • Human

    Douglas Hoffman 53Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 54

    Notes About ‘Heuristic Oracles’

    I used to talk about heuristic oracles, which were ones that used approximations and rules of thumb. I have refined the descriptions and no longer differentiate between heuristic and other oracles. Some oracles are based on heuristic principles while others are not. But, All oracles are “heuristic” with regard to pass and fail. That is:

    – Passing a test doesn’t mean there isn’t a bug here

    – Failing a test doesn’t mean there is a bug present

    – No oracle is complete – there are always ways the SUT

    could be failing in other unnoticed ways

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 28

    Douglas Hoffman 55

    Oracle Taxonomy (1 of 4)Definition Advantages Disadvantages

    No Oracle - Doesn’t check correctness of results, (only that some results were produced)

    - Can run any amount of data

    (limited only by the time the SUT

    takes)

    - Only spectacular failures

    are noticed

    Independent

    Implementa-

    tion

    - Independent generation of all expected results

    - No encountered errors go

    undetected

    - Can be used for many different

    tests

    -Fast way to automate using an

    oracle (if available)

    - Expensive to implement

    and maintain

    - Complex and often time-

    consuming when run

    Consistency

    (Saved

    Master)

    - Verifies current run results with a previous run

    - Verification is straightforward

    - Can generate and verify large

    amounts of data

    -Original run may include

    undetected errors

    -Can take large data

    storage space

    Consistency

    (Function

    Equivalence)

    - Verifies current run with another implementation

    - Verification is straightforward in

    real time

    - Can generate and verify large

    amounts of data without storage

    - Possibility of undetected

    common-mode errors

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 56

    Oracle Taxonomy (2 of 4)

    Definition Advantages Disadvantages

    Self-Verifying - Embeds key to the answer within data

    -Can check large amounts of data

    - Allows extensive run-time and post-test

    analysis

    - Straightforward to check

    - Verification can be based solely on

    message contents

    - Test must require input

    data set with compliant

    structure

    - Must define answers and

    generate messages to

    contain them

    Model Based - Uses digital data model of SUT behavior

    - May use digital model for multiple tests

    - Digital form of model easier to maintain

    than automated test

    - Tests may work for multiple SUTs by

    using different models

    - Maintenance of complex

    SUT models is expensive

    - Model must match

    expected behavior

    Constraint

    Based

    - Verifies characteristics in compliance with constraints

    - Faster and easier than most other oracles

    - Less expensive to create and use

    - May be reusable across SUTs and tests

    - Can miss systematic

    errors

    - Can miss obvious errors

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 29

    Douglas Hoffman 57

    Oracle Taxonomy (3 of 4)

    Definition Advantages Disadvantages

    Probabilistic - Checks a highly likely characteristic

    - Faster and easier than most other oracles

    - Less expensive to create and use

    - Can cause false alarms

    - Can miss systematic

    errors

    - Can miss obvious errors

    Statistical - Uses statistical properties of outcomes

    - Allows checking of very large data sets

    - Allows checking of live systems’ data

    - Sometimes allows post-test checking

    - Can cause false alarms

    - May miss systematic

    errors

    - Can miss obvious errors

    Property

    Based

    - Checks an indirectly correlated characteristic

    - Usually simple

    - Good for straightforward

    transformations

    - Correlated variable may

    not exist

    - May be difficult to

    exploit the correlation

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 58

    Oracle Taxonomy (4 of 4)

    Definition Advantages Disadvantages

    Computational - Reverses the behavior of the SUT to revert results to inputs

    - Good for mathematical functions

    - Good for straightforward

    transformations

    - Limited applicability

    - May require complex

    programming

    Diagnostic - Programmatically checks assertions or traces activities and values

    - May be built into code as defensive

    programming

    - Provides good diagnostic data when

    exception detected

    -Can miss significant

    errors

    - Changes program

    characteristics

    - Can miss obvious errors

    Hand-Crafted - Result is carefully selected by test designer

    - Useful for some very complex SUTs

    - Expected result can be well

    understood

    - Does the same thing

    every time

    - Limited number of cases

    can be generated

    Human - Applies a person’s brain power to decide correctness

    - Available

    - Flexible (can always be applied)

    - Applies a broad spectrum of filters

    - Slow

    - Error prone

    - Easily distracted

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 30

    Douglas Hoffman 59

    ‘No Oracle’

    • Definition:

    − No check for correctness – just

    run the test exercise (watch for

    crashes)

    • Examples:

    − Random keyboard input

    − API calls with no return checks

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 60

    ‘No Oracle’

    • Advantages:

    − Easy to implement

    − Runs fast

    • Disadvantages:

    − Only spectacular events are noticed

    − Nothing found ≠ No problem

    − False sense of accomplishment

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 31

    Douglas Hoffman 61

    ‘No Oracle’ Strategy

    • Run the test using generated [usually

    random] inputs

    • Don’t worry about SUT behavior since

    we’re only looking for spectacular results

    • ROI based on really inexpensive input

    generators

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 62

    ‘No Oracle’ In Context

    • May be useful:− Early development testing

    − Robustness testing

    − Load or life testing

    − Security testing (“fuzzing”)

    • Usually avoided:− Used as a primary test mechanism

    − When results must be correct

    − Input is non-trivial to generate

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 32

    Douglas Hoffman 63

    ‘Independent Implementation’

    Oracle

    • Definition:

    − A separate program created to generate

    expected results

    • Examples:

    − Adder function written to check a calculator

    − An alternative algorithm implementation

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 64

    ‘Independent Implementation’

    Oracle

    • Advantages:

    − Customized for the SUT

    − Can be exact for some SUT characteristics

    • Disadvantages:

    − Often expensive to implement

    − May be difficult to run

    − May have high maintenance costs

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 33

    Douglas Hoffman 65

    ‘Independent Implementation’

    Strategy

    • Independent implementation

    • Complete coverage over some domain

    − Specific functions

    − Input ranges

    − Result ranges

    • Generates “Correct” results

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 66

    ‘Independent Implementation’

    In Context

    • May be useful:− Inexpensive to create or acquire

    − Extremely high reliability required

    − Reasonably good coverage of some range

    • Usually avoided:− Too expensive to create or acquire

    − Areas where exactness is not required

    − High risk that the oracle may be wrong

    − Other types of oracles are sufficient

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 34

    Douglas Hoffman 67

    ‘Consistency’ Oracle

    • Definition:− Compare results between different programs or

    from run-to-run to identify when values change − Really only indicates that something is different

    − Usually don’t know whether or not original results

    are “correct”

    • Examples:− Comparison of test results with “golden master”

    − Function equivalence testing

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 68

    ‘Consistency’ Oracle

    • Advantages:− Common, straightforward approach

    − Usually easy to implement using log files

    − Logs can be any data types

    − Can check without knowing the correct results

    • Disadvantages:− Legacy bugs may be there but undiscovered

    − Established bugs may “get tenure”

    − Nothing found means no obvious changes

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 35

    Douglas Hoffman 69

    ‘Consistency’ Oracle Strategy

    • Check for changes or differences

    − Against previous results (“Golden Master”)

    − Against alternate versions or platforms

    − Function equivalence testing

    • Results comparison with:

    − Validated outcomes

    − Unvalidated outcomes

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 70

    ‘Consistency’ Oracles In Context

    • May be useful:

    − Inexpensive to create

    − Have automated tests with logging

    − High likelihood that the oracle is correct

    − Equivalent product is available

    • Usually avoided:

    − High risk that the oracle may be wrong

    − Other types of oracles are available

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 36

    Douglas Hoffman 71

    ‘Self-Verifying Data’ (SVD) Oracle

    • Description:

    − Use a “key” to tag data when generated and later

    use the tag to verify data correctness

    − Self-descriptive data (Red)

    − Cyclic algorithms (repeating patterns)

    − Tagged data (CRC)

    − Embedded keys with shared algorithms

    − SVD can be used for data sets or input data

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 72

    ‘Self-Verifying Data’ (SVD) Oracle

    • Advantages:− Excellent for random data generation

    − Generates verifiable records

    − Can be used to quickly check large data sets

    − Allows extensive results analysis during or

    after test execution

    • Disadvantages:− Persistent, non-transformed data required

    − Requires some place to embed a key

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 37

    Douglas Hoffman 73

    ‘SVD’ Strategy

    • Embed key information within the data to verify data integrity− Human readable form (self-descriptive)− Data pattern based (cyclic)− Embed unique summary (CRC)− Embedded seed or key

    • Data verification− Recreate data− Extract the pattern− Synchronous or asynchronous checking

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 74

    ‘SVD’ In Context

    • May be useful:− High volume of inputs or referenced data

    − Key or seed can be used for data regeneration

    − Straightforward to incorporate key with data

    • Usually avoided:− Outcomes don’t reflect SVD data

    − Data not easily generated from a key

    − Key not easy to include with data

    − Overly complex data structures

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 38

    Douglas Hoffman 75

    ‘Model-Based’ Oracle

    • Definition:

    − Use a model of some SUT behavior to structure

    a test and provide the oracle

    • Examples:

    − Menu tree

    − Internal state machine

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 76

    ‘Model-Based’ Oracle

    • Advantages:

    − Decouples tests from SUT models

    − Works well for families of similar products

    − Provides test ideas and oracles

    • Disadvantages:

    − SUT must have model-based behaviors

    − Models must be simple to be practical

    − Often requires ability to detect program state

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 39

    Douglas Hoffman 77

    ‘Model Based’ Oracle Strategy

    • Identify and describe a machine readable form of a

    model of some aspect of the SUT

    • Design tests using the model as input

    • Implement the tests (by reading in the model)

    • Use the model information to verify actions

    • Update the model or use multiple models as needed

    Copyright © 2004-13, SQM, LLC.

    Slide 78

    ‘Model-Based’ Tests and Oracles

    • Describe a model of the software (e.g., menu tree)

    • Describe the model in a machine readable form

    • Verify the correctness of the model with intended

    behavior

    • Write a test using the model as input, as a guide for

    generation of events, and monitoring of responses

    • Generate events / inputs to the program from the

    model

    • Check program functions and state transitions based

    on the model

    Douglas Hoffman Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 40

    Douglas Hoffman 79

    Some Types of Models

    • State Diagrams

    • State Transition Tables

    • Flow Charts

    • Use Cases

    • Business rules

    • Entity-Relationship Diagrams

    • Activity Diagrams

    Copyright © 2004-13, SQM, LLC.

    Slide 80

    State Transition Table Model ExampleInitial State Event Result New State

    S1 E1 S1

    E2 logged in S2

    E3 SU log in S3

    S2 E4 … S4

    E5 S2

    E6 logged out Exit

    S3 E4 … S4

    E5 admin S3

    E6 logged out Exit

    S1

    S3

    S2

    Exit

    S4

    E1

    E2

    E3E4

    E5

    E6

    E4

    E5

    E6

    Douglas Hoffman Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 41

    Douglas Hoffman 81

    ‘Model Based’ Oracle In Context

    • May be useful:− When the model is simple− When the model is well defined (e.g., state

    machine or screen hierarchy descriptions)− Events can be reliably generated− Best if states can be monitored or detected

    • Usually avoided:− No identifiable models− Unstable, poorly defined, or complex models− Events are difficult to generate or manage− State of the program is not easily determined

    Copyright © 2004-13, SQM, LLC.

    Exploratory Automated Testing

    Belgium Testing DaysFebruary 27, 2013

    Tutorial Part 2

    Douglas Hoffman, BACS, MBA, MSEE,

    ASQ-CSQE, ASQ-CMQ/OE, ASQ Fellow

    Software Quality Methods, LLC. (SQM)

    www.SoftwareQualityMethods.com

    [email protected]

    Douglas Hoffman Copyright © 2004-13, SQM, LLC. 82

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 42

    Douglas Hoffman 83

    ‘Constraint-Based’ Oracle

    • Definition:

    − Check for simple valid (or invalid) individual or

    combinations of data values or characteristics

    • Examples:

    − USA Zip Codes are 5 or 9 digits, Canadian Zip

    Codes are 6 alpha-numeric values

    − Employee Start Date is after Employee Birth Date

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 84

    ‘Constraint-Based’ Oracle

    • Advantages:

    − Low cost checks for some error conditions

    − Provides both test ideas and oracles

    • Disadvantages:

    − May be code-invasive

    − Constraint conditions have to be recognized

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 43

    Douglas Hoffman 85

    ‘Constraint-Based’ Strategy

    • Identify checkable secondary characteristics− Size, form, type, illegal values, etc.

    − Values that constrain one another

    − Invariant rules

    − Coincident (not causal) values

    • Checking − Create checking mechanism to confirm

    conformance with the constraints

    − Synchronous or asynchronous checking

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 86

    ‘Constraint-Based’ In Context

    • May be useful:− Recognizable relationships

    − Straightforward to check the relationships

    − Potentially quick verification of large data

    sets

    • Usually avoided:− No recognizable relationships of value

    − Getting values is difficult

    − Checking is overly complex

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 44

    Douglas Hoffman 87

    ‘Probabilistic’ Oracle

    • Definition:− Use an approximation, heuristic (rule of thumb),

    or partial information that supports but does not

    necessarily mandate a given conclusion

    • Examples:− Employee is usually older than their dependents

    − This test should take more than 1/3 the time and

    less than 3 times the time it ran last

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 88

    ‘Probabilistic’ Oracle

    • Advantages:

    − Can provide a quick ‘sanity check’ on large data

    sets

    − May be inexpensive to implement

    • Disadvantages:

    − Misses or False Alarms may be expensive

    − Nothing found ≠ No problem

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 45

    Douglas Hoffman 89

    ‘Probabilistic’ Oracle Strategy

    • Probabilistic opportunities

    − Relationships that are usually true

    − Very unlikely conditions

    − Approximation available

    • Checking

    − Create checking mechanism to confirm

    relationship or condition

    − Synchronous or asynchronous checking

    Copyright © 2004-13, SQM, LLC.

    Some Useful Oracle Heuristics

    • Similar results that don’t always work

    • Less exact computations (32 bits instead of 64)

    • Look for subsets

    • Any relationships not explicit in SUT– Date-time/transaction number

    – One home address

    – Timings

    • Look for general characteristics

    • Harmonic or repeating patterns

    Douglas Hoffman 90Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 46

    Douglas Hoffman 91

    ‘Probabilistic’ Oracle In Context

    • May be useful:− High speed of comparison is important

    − Exact results are difficult to generate

    − Few expected exception cases (false alarms)

    − Simple heuristic is available

    • Usually avoided:− Used as the only oracle

    − Heuristic is too complex

    − More exact checks are required

    − Too many expected exceptionsCopyright © 2004-13, SQM, LLC.

    Douglas Hoffman 92

    ‘Statistical’ Oracle

    • Definition:− Compare the statistical properties of the inputs

    and results to confirm similarity

    • Examples:− The properties of the input distribution file

    records is similar to the output distribution

    − Today’s pattern of use is similar to yesterday’s

    − Test of random communication data packets by

    checking statistical properties of the packets

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 47

    Douglas Hoffman 93

    ‘Statistical’ Oracle

    • Advantages:− Can check on a high volume of well-formed data

    − May be inexpensive to implement

    − Can be used with live data

    • Disadvantages:− Only useful with large populations

    − Result properties must be described by some

    transformation of the input properties

    − Inputs require reasonably computable properties

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 94

    ‘Statistical’ Oracle Strategy

    • Principle idea:– Useful for high-volume data impractical to otherwise verify

    – Use high-volume random tests generated with particular

    statistical properties

    – Results checking based on population statistical properties

    • The oracle:– Computes the statistical properties from the inputs

    – Computes the statistical properties from the outcomes

    – Compares the statistics in light of the expected

    transformation through the SUT

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 48

    Douglas Hoffman 95

    ‘Statistical’ Oracle In Context

    • May be useful:− Inputs and outcomes can be statistically counted

    − Strong correlation between input and outcome population statistics

    − Population statistics are easily computable

    • Usually avoided:− Used as the only oracle

    − Data is not countable for statistical profiling

    − Input/outcome populations are not statistically related

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 96

    ‘Property-Based’ Oracle

    • Definition:− Compare correlated but not causal relationships

    between variables

    • Examples:− Sales Order Number should be in time-sequence

    order

    − Checking the number of pages printed by a test

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 49

    Douglas Hoffman 97

    ‘Property-Based’ Oracle

    • Advantages:

    − Can independently double-check data consistency

    − Can be used with live data

    • Disadvantages:

    − Property has to exist and be recognized

    − Property has to be easily testable

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 98

    ‘Property-Based’ Oracle Strategy

    • Principle idea:

    – Check secondary or coincidental characteristics

    – Most are ‘sanity checks’

    • The oracle:

    – Checks for conformance with expected

    characteristics

    – May miss obvious errors

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 50

    Douglas Hoffman 99

    ‘Property-Based’ Oracle In Context

    • May be useful:

    − Recognizable secondary or coincidental

    characteristics

    − In-test checking for obvious errors

    − Simple checks may identify significant problems

    • Usually avoided:

    − Unimportant characteristics

    − High expectation of false alarms

    − Difficult or invasive checking

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 100

    ‘Diagnostic’ Oracle

    • Definition:− Instrument the code (assertions or logging)

    • Examples:− Code assertions (e.g., must have a valid UID

    to be logged in)

    − Log files generated by SUT

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 51

    Douglas Hoffman 101

    ‘Diagnostic’ Oracle

    • Advantages:

    − Can test for invisible conditions

    − Can be tailored for specific types of errors

    • Disadvantages:

    − Code invasive

    − May be difficult to interpret the logs

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 102

    ‘Diagnostic’ Oracle Strategy

    • Principle idea:

    – Embed checks or logs in the code

    – Use high-volume random tests to create large number of

    conditions

    – Errors flagged by asserts or analysis of log files

    • The oracle:

    – Code assertions to flag “impossible” conditions

    – Inconsistencies identifiable in the log files

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 52

    Douglas Hoffman 103

    ‘Diagnostic’ Oracle In Context

    • May be useful:− Code available for modifications

    − Close working relationship with developers

    − Code changes do not significantly change program behavior

    • Usually avoided:− Uncooperative developers

    − Program is timing dependent

    − Changes cause program behavior to significantly change

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 104

    ‘Hand-Crafted’ Oracle

    • Definition:− Select inputs for ease of generating expected results

    − Select or compute complex result and corresponding

    inputs

    • Examples:− Construct a drawing with 8 overlapping layers

    containing different figures, location, and fill …

    − Sub-atomic particle A with mass X and velocity Y is

    hit by particle B with mass W and velocity Z,

    resulting in generation of particles C and D …

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 53

    Douglas Hoffman 105

    ‘Hand-Crafted’ Oracle

    • Advantages:

    − Can test for specific conditions (e.g., boundaries)

    − Can provide oracles for arbitrarily complex SUTs

    − Oracle is often built into the test

    • Disadvantages:

    − One-off cases

    − Has a limited ability to detect errors

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 106

    ‘Hand-Crafted’ Oracle Strategy

    • Expected result is carefully crafted (or

    selected) with the input values

    • Input and result are specified together

    • Oracle is frequently built into the test

    • The approach is most often taken for

    regression tests in complex SUTs

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 54

    Douglas Hoffman 107

    ‘Hand Crafted’ Oracle In Context

    • May be useful:

    − Complex function

    − Special cases are easily identified

    • Usually avoided:

    − Outcomes extremely difficult or time consuming to predict

    − Other oracles are available

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 108

    ‘Human’ Oracle

    • Definition:

    − Person observes SUT behavior to detect possible

    problems

    • Examples:

    − Manual GUI testing

    Note that a human oracle is applied whenever any

    other oracle flags a potential bug

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 55

    Douglas Hoffman 109

    ‘Human’ Oracle

    • Advantages:− Extremely flexible to changes

    − Tests are multi-dimensional (using all senses)

    − Can notice potential errors on many levels at once

    (e.g., placement, font, content, navigation)

    • Disadvantages:− Human speeds for computation

    − Cannot do exact comparisons on large data sets

    − Easily trained (e.g., inattentional blindness)

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 110

    ‘Human’ Oracle Strategy

    • Set a person in front of the SUT to observe

    • Human uses their judgment to decide the

    verdict

    • Works for manual or automated exercises

    • Works for scripted or unscripted tests

    Note that a human oracle is applied whenever

    any other oracle identifies a potential bug

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 56

    Douglas Hoffman 111

    ‘Human’ Oracle In Context

    • May be useful:

    − Outcome medium is not machine readable

    − Input/outcome relationship is very complex

    − Insufficient time to automate the oracle

    • Usually avoided:

    − High volume of comparisons

    − Very repetitive checking

    − High level of detail or specificity required

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 112

    Software Test Automation:

    Test Comparators

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 57

    Douglas Hoffman 113

    Evaluation Mechanisms

    • Deterministic– The comparator (or oracle) accepts the inputs and/or the

    test outcomes to compare for a match between expected

    and actual behaviors

    – If they do not match, outcome is wrong and

    investigation required

    • Non-Deterministic– The comparator (or oracle) accepts the inputs and/or the

    test outputs and evaluates whether the results are

    plausible

    – If the outcomes are implausible or not close enough,

    investigation is required

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 114

    Deterministic Evaluation

    Comparison for data set equivalence

    • Saved result from a previous run

    • Parallel generation

    – previous version

    – competitor’s product

    – alternate implementation

    – custom oracle

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 58

    Douglas Hoffman 115

    Other Deterministic Evaluation

    Comparison for outcome consistency

    • Inverse function– mathematical inverse

    – operational inverse

    • Useful functional rules– Deterministic incidental or informative attributes

    • Expected result embedded in the data– self-descriptive (blue)– embedded key (CRC, seed)

    – cyclic data

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 116

    Non-Deterministic Evaluation

    Approximate comparisons or similarities

    • Compare insufficient attributes

    – use 16 bit functions to check 64 bit functions

    – testable pattern for a set of values for a variable

    – one or a few primary attributes of outcomes

    • Statistical distributions

    – test for outliers, means, predicted distribution

    – statistical properties (predicted population statistics)

    – comparison of correlated variables’ populations

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 59

    Douglas Hoffman 117

    Non-Deterministic Evaluation

    Approximate comparisons

    • Approximate models– At the end of the exercise Z should be true

    – X is usually greater than Y

    – precondition = postcondition

    • “Fuzzy” comparisons– within a range of values

    – bitmaps

    – statistical analysis (likely properties)

    – shading

    – CRC type summaries

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 118

    Non-Deterministic Evaluation

    Similarities

    • Incidental or informative attributes– correlations (time of day and sales order number)

    – relative duration (e.g., within factor of x/3 to 3x)

    – likely sequences

    – size or shape (number of digits/characters)

    – uniqueness of SSN

    • Reordered sets (where order matters)– itemized list of sales items (taxable and nontaxable)

    – reorder asynchronous events for comparison

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 60

    Douglas Hoffman 119

    Software Test Automation:

    Verdict Analysis

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 120

    Basic Oracle Model

    Test Oracle

    System

    Under

    Test

    Verdict

    Copyright © 2004-13, SQM, LLC.

    Test Inputs

    Input Data

    Program State

    Environmental

    Inputs

    Test Results

    Postcondition

    Data

    Postcondition

    Program State

    Environmental

    Results

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 61

    Douglas Hoffman 121

    ‘Other Implementation’ Model

    Test Results

    Postcondition

    Data

    Postcondition

    Program State

    Environmental

    Results

    Test

    Oracle

    System

    Under

    Test

    Test Inputs

    Input Data

    Program State

    Environmental

    Inputs

    Compare

    Verdict

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 122

    ‘No’ Oracle Model

    Test Results

    Postcondition

    Data

    Postcondition

    Program State

    Environmental

    Results

    System

    Under

    Test

    Test Inputs

    Input Data

    Program State

    Environmental

    Inputs

    Compare

    Verdict

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 62

    Douglas Hoffman 123

    ‘Saved Master’ Model

    Test Results

    Postcondition

    Data

    Postcondition

    Program State

    Environmental

    Results

    System

    Under

    Test

    Test Inputs

    Input Data

    Program State

    Environmental

    Inputs

    Verdict

    Previous

    Results

    Compare

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 124

    ‘Function Equivalence’ Model

    System

    Under

    Test

    Copyright © 2004-13, SQM, LLC.

    Verdict

    Test OracleTest Inputs

    Input Data

    Program State

    Environmental

    Inputs

    Test Results

    Postcondition

    Data

    Postcondition

    Program State

    Environmental

    Results

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 63

    Douglas Hoffman 125

    ‘Self-Verifying Data’ Oracle Model

    Test

    Oracle

    System

    Under

    Test

    Verdict

    Copyright © 2004-13, SQM, LLC.

    Test Inputs

    Input Data

    Program State

    Environmental

    Inputs

    Test Results

    Postcondition

    Data

    Postcondition

    Program State

    Environmental

    Results

    Douglas Hoffman 126

    ‘Model Based’ Oracle Model

    System

    Under

    Test

    Model

    of SUT

    Copyright © 2004-13, SQM, LLC.

    Verdict

    Test Oracle

    Test Inputs

    Input Data

    Program State

    Environmental

    Inputs

    Test Results

    Postcondition

    Data

    Postcondition

    Program State

    Environmental

    Results

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 64

    Douglas Hoffman 127

    Software Test Automation:

    Exploratory Automated Testing

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 128

    High Volume Random Tests

    • Principle idea

    – High-volume testing using varied inputs

    – Results checking based on individual results or population’s

    statistical characteristics

    • Fundamental goal is to have a huge numbers of tests

    – The individual tests may not be not all that powerful or

    compelling

    – Input is varied for each step

    – Individual results may not be checked for correctness –

    sometimes population statistics are used

    – The power of the approach lies in the large number of tests

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 65

    Douglas Hoffman 129

    Low Volume Exploratory Tests

    • Principle idea

    – One-at-a-time testing using varied inputs

    – Use automation to make exploration easy

    • Fundamental goal is to enable exploration

    – Variations on a theme (modify automated tests)

    – Quick-and-dirty generation of tests/data/comparisons

    – Background checking

    • Memory leak detection

    • File modification

    • Etc.

    – Command line or UI based variations

    Copyright © 2004-13, SQM, LLC.

    The Easy Part

    I said before that driving the inputs or

    generating data is usually straightforward

    – Random number generators can allow us to

    create tests that do new things all the time

    – Using well-formed expressions it is possible to

    create arbitrary inputs and data sets

    – The oracles are fundamental to deciding test

    design and methodology

    Douglas Hoffman Copyright © 2004-13, SQM, LLC. Slide 130

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 66

    The Catch for the Easy Part

    Although driving the inputs may be

    straightforward, selecting the interface points

    is more difficult

    – Requires understanding of the architecture and

    protocols to identify touch points

    – Identifying the valuable of the touch points is

    critical

    – Checking for unexpected behavior is the key

    to success

    Douglas Hoffman Copyright © 2004-13, SQM, LLC. Slide 131

    Douglas Hoffman 132

    Random Number Generators

    • Randomized input values

    • Randomized data generation

    • Random test selection

    • Pseudo-Random numbers

    • Using seed values

    • Generating random seeds

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 67

    Douglas Hoffman 133

    Random Number Seeds

    Computer random numbers aren’t really

    random – they’re pseudo-random

    – Long series of statistically random values

    that repeats

    – By default it begins randomly somewhere in

    the series

    – A seed value specifies where to begin in the

    series so it repeats

    Copyright © 2004-13, SQM, LLC.

    Repeatable Random Series# RUBY code

    MAX_SEED = 1_000_000_000

    def initial_RNG_seed(myseed)

    if (myseed == nil) # Check if seed is provided

    # Create a random number to seed RNG

    puts "(no seed passed in, so generate one)"

    myseed = srand()

    myseed = rand(MAX_SEED)

    end

    # print the seed so that we know the seed used

    puts "myseed is #{myseed.to_s}\n"

    foo2 = srand (myseed) # initialize the RNG

    foo = rand() # generate the [first] random number

    return foo

    end

    Douglas Hoffman 134Copyright © 2012-13, SQM, LLC.

    RNG

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 68

    Random Series Output Example

    puts ("First run: #{initial_RNG_seed(nil)} \n \n")

    puts ("Second run: #{initial_RNG_seed(400)} \n \n")

    puts ("Third run: #{initial_RNG_seed(nil)} \n")

    (no seed passed in, so generate one)

    myseed is 144288918

    First run: 0.3705579466087263

    myseed is 400

    Second run: 0.6687289088341747

    (no seed passed in, so generate one)

    myseed is 108495905

    Third run: 0.09838898989988143

    Douglas Hoffman Copyright © 2012-13, SQM, LLC. 135

    Douglas Hoffman 136

    Well-Formed Values

    • Input values and data have well-defined

    grammars (they’re formatted)

    • Generating arbitrary values is a matter of

    applying the grammar

    • Seed values with the random number

    generator allow us to repeat the random

    series and regenerate the inputs

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 69

    Well-Formed Data#RUBY program that generates simple arithmetic phrases

    def eq_gen

    jubilee = 1 + rand(8) # 8 choices for the 4 defined cases

    case jubilee

    when 1

    "("

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 70

    Douglas Hoffman Copyright © 2012-13, SQM, LLC. 139

    “Sandboxed” Tests

    A sandboxed test gets along with itself

    • Self-contained regression test

    • Independent of other tests

    • Can be automatically launched

    • Runs to completion

    • Ends without resetting SUT or system

    • Can successfully be run repeatedly (e.g., 200x)

    They’re used by randomly selecting and

    running from among a set of them

    Random Sandboxed Tests#RUBY program that runs executables in separate threads

    # The numeric display routine courtesy of Nawwar Kabbani

    threads = []

    list = []

    list = Dir.entries ("./executables") # directory with tests

    list3 = list.reject { |s| s =~ /^\./ } # exclude ./ and ../

    TestCount = list3.length

    for y in (1..10) # launch 10 random executions from the files

    in the directory

    name = list3[rand(TestCount)] # randomly select the test

    threads

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 71

    Douglas Hoffman 141

    Software Test Automation:

    Examples of Oracles

    Copyright © 2004-13, SQM, LLC.

    ‘No Oracle’ Example

    Test functions through an API:

    • Select a random function and use random

    parameter values

    • Run the test exercise

    • Repeat without checking ‘results’

    • Watch for hangs or crashes

    Douglas Hoffman 142Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 72

    ‘No Oracle’ Example

    Test random character input to a word processor:

    • Generate a random input character

    • Run the test exercise

    • Repeat without checking results

    • Watch for hangs or crashes

    Douglas Hoffman 143Copyright © 2004-13, SQM, LLC.

    ‘Independent Implementation’

    ExampleCreate a independent sine (x) function oracle:

    • Determine the algorithm used by the SUT

    • Implement a separate sine (x) function using a

    different algorithm

    • Generate random values for x

    • The values returned from both sine (x) functions

    should match

    Douglas Hoffman 144Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 73

    ‘Consistency’ Oracle

    There are two approaches for consistency

    testing:

    1. “Golden Master”

    – Saved results from an earlier run

    – Validated or unvalidated

    2. Function equivalence

    – Competitive product

    – Alternate implementation

    Douglas Hoffman 145Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 146

    ‘Golden Master’ Approach

    • Checks for changes or differences

    • Most frequently used strategy

    • Most common method:

    − Create a test with logging to a disc file

    − Run the test and check the log carefully

    − Keep the log as a [golden] master

    − Compare subsequent runs against the log

    − Update the master when legacy bugs are fixed

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 74

    Douglas Hoffman Copyright © 2012, SQM, LLC. 147

    ‘Function Equivalence’ Tests

    Passing input values to two products for

    comparison in real-time

    • Competitive product, alternate implementation,

    earlier version, different platforms, etc.

    • Randomly generated, well-formed inputs (or not)

    • Use the same inputs for both versions

    • Test the outcomes for “equivalency”

    ‘Function Equivalence’ Example

    Create a function equivalence test for

    spreadsheet math functions:

    • Identify functions of interest and the parameter

    definitions

    • Create a test exercise that creates random, well

    formed function calls with parameters

    • Feed the calls to OOSpreadsheet and Excel

    • Compare the returned results

    Douglas Hoffman 148Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 75

    ‘Self-Descriptive’ SVD Examples

    • Describe the expected result in the data

    – The name of a font (e.g., Comic Sans MS)

    – Color (e.g., Blue)

    – Size (e.g., 36 point)

    – The following line should be “xyz”

    Douglas Hoffman 149Copyright © 2004-13, SQM, LLC.

    ‘Cyclic Algorithm’ SVD Example

    1. Use a pattern when the data is generated

    – E.g., start, increment, count

    – E.g., basic string, count of iterations

    2. Identify the pattern in the result

    3. Confirm that the actual pattern is expected

    – Build into comparator

    – Embed with the data

    Douglas Hoffman 150Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 76

    ‘Tagged Data’ SVD Example

    • Generate a value that summarizes the data

    such as a cyclic redundancy check:1. Use the first value as the starting number for

    the CRC

    2. Shift the CRC one place

    3. Add the next value

    4. Repeat steps 2 and 3 until all values have been

    included in the CRC computation

    5. Append the computed CRC to the data

    Douglas Hoffman 151Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 152

    ‘Embedded Key’ SVD Example

    • For a 128 character name field:− Use seeded RNG to generate name (up to 120

    characters)

    − Convert the seed to an 8 character value

    − Append the seed to the name

    − Regenerate the name using the seed and algorithm

    Copyright © 2004-13, SQM, LLC.

    Name(128) = … L Random characters … 8 character S

    9 to 128 characters long

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 77

    ‘Shared Keys’ SVD Example

    Create a random name:

    1. Generate and save random number seed (S) and

    convert to a string

    2. Use the first random value using RAND(S) as

    the Length (L)

    3. Generate random name (N) with L characters

    using RAND()

    4. Concatenate the seed to the name

    Douglas Hoffman 153Copyright © 2004-13, SQM, LLC.

    ‘Model-Based’ Oracle Example

    • For a state machine (such as web page

    transitions)

    – Describe the state transition table in machine

    readable form

    – Design a test exercise applying the table

    (e.g., walk all transitions, random walk, or

    generate illegal events)

    – Verify transitions within the test by sensing

    the state

    Douglas Hoffman 154Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 78

    ‘Constraint-Based’ Oracle

    Examples

    Employee data:

    • Employee birth date is before start date

    • Each employee has a unique SSN

    • Active employee start dates are before today

    • At most one current spouse

    • Only one record for an employee

    Douglas Hoffman 155Copyright © 2004-13, SQM, LLC.

    ‘Probabilistic’ Oracle Examples

    Employee database integrity check:

    1. Employee older than dependents

    2. Employee start date after company established

    3. Employee age greater than 15

    4. Zip code (if USA) 5 or 9 digits

    5. USA SSN has 9 digits

    6. No duplicate SSN entries

    Douglas Hoffman 156Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 79

    ‘Statistical’ Oracle Example

    Computing the right state taxes:

    • Generate random items, quantities, locations,

    etc. for purchase transactions

    • Track total amounts of purchases and taxes

    collected

    • Compute average tax across all locations

    • (total taxes collected) should equal (total

    purchases * average taxes)

    Douglas Hoffman 157Copyright © 2004-13, SQM, LLC.

    ‘Property-Based’ Oracle

    Example

    Sales Order Number and Date/Time Stamp:

    • Run an SQL query for all records sorted by

    Sales Order Number

    • Run an SQL query for all records sorted by

    Date/Time of Creation

    • The two sets of records should be the same

    Douglas Hoffman 158Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 80

    ‘Computational’ Oracle Example

    Computing the square root:

    • Generate a random input value (x)

    • Compute y = square root (x)

    • Compute z = y2

    • Check x = z

    Douglas Hoffman 159Copyright © 2004-13, SQM, LLC.

    ‘Computational’ Oracle Example

    Splitting a table:

    • Generate and populate a table

    • Select some row in the table

    • Split the table

    • The oracle deletes the split between the two

    tables

    • Check the original and final tables are the same

    Douglas Hoffman 160Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 81

    ‘Diagnostic’ Oracle Example

    Long random walk:

    • Put assert statements in the code for invalid or

    unusual circumstance

    • Create records of significant data for diagnostic

    purposes when assertion is violated

    • Generate a large number of valid inputs to

    exercise the code

    Douglas Hoffman 161Copyright © 2004-13, SQM, LLC.

    ‘Diagnostic’ Oracle Example

    Long random walk:

    • Put logging statements in the code to record

    internal state and value information

    • Run exploratory test

    • Evaluate logs after testing for unexpected

    circumstances

    Douglas Hoffman 162Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 82

    ‘Hand-Crafted’ Oracle Example

    Sales order processing:

    • Choose specific values for order information

    (e.g., boundary cases)

    • Identify what the order should look like and

    how the SUT should react

    • Verify actual vs. expected when the test is run

    Douglas Hoffman 163Copyright © 2004-13, SQM, LLC.

    ‘Human’ Oracle Example

    Calendar generation program testing:

    • Choose many sets of input values describing a

    calendar

    • Print the input values and corresponding

    generated calendar

    • Review the calendar to confirm appropriate

    calendar or error indication output

    Douglas Hoffman 164Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 83

    Douglas Hoffman 165

    Software Test Automation:

    Some Parting Words about the

    Design of Automated Tests

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 166

    Exploratory Test Automation

    • Extends our reach

    • Augments human capabilities

    • Does something different every time

    • Is heavily dependent on oracles

    • May access internal information

    • Can surface bugs that we didn’t consider

    • May check invariants

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 84

    Douglas Hoffman 167

    Good Automated Tests

    �Start with a known state

    �Build variation into the tests

    �Plan for the capture of data on error

    �Check for errors during the test run

    �Capture information when an error is noticed

    �Minimize error masking and cascading

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 168

    Start With a Known State • Data

    – Load preset values in advance of testing

    – Reduce dependencies on other tests

    • Program State

    – External view

    – Internal state variables

    • Environment

    – Decide on desired controlled configuration

    – Capture relevant session information

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 85

    Douglas Hoffman 169

    Build Variation Into the Tests

    • Dumb monkeys

    • Data driven tests

    • Pseudo-random event generation

    • Model driven and model based automation

    • Variations on a theme

    • Configuration variables

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 170

    Plan For Capture of Data

    • Know what data may be important to identify and

    fix errors

    • When necessary, capture prospective diagnostic

    information before an error is detected

    • Include information beyond inputs and outcomes

    • Check as many things as possible

    • Design tests to minimize changes after errors occur

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 86

    Douglas Hoffman 171

    Check for Errors

    • Periodically check for errors as the test runs

    • Document expectations in the tests

    • Capture prospective diagnostic information before an

    error is detected

    • Capture information when the error is found (don’t wait)

    – Results

    – Other domains

    – “Dump the world”

    • Check as many things as possible

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 172

    Capture Information

    When An Error is Noticed• When something unexpected is detected: “dump the

    world”

    – All “interesting” information

    – Let the users of the information determine what’s interesting

    – Since the developers are typically the information users, enlist

    their help to automate information gathering

    OR

    • Freeze – wait for a person

    – When “interesting” information is expensive to capture

    – When there isn’t time to create automated capture routines

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 87

    Douglas Hoffman 173

    Don’t Encourage

    Error Masking or Error Cascading

    • Session runs a series of tests

    • A test does not run to normal completion

    – Error masking occurs if testing stops

    – Error cascading occurs if one or more downstream tests fails as a consequence of this test failing

    • Impossible to avoid altogether

    • Should not design automated tests that unnecessarily cause either situation

    Copyright © 2004-13, SQM, LLC.

    Douglas Hoffman 174

    Recapping

    �Use automation to explore (not just repeat)

    �Extend your reach into the system

    �Think of models for test execution

    �Automate based on available oracles

    �Use different types of oracles

    �Design good automated tests

    Copyright © 2004-13, SQM, LLC.

  • Belgium Testing Days 2013

    Exploratory Automated Tests

    (Tutorial)

    February 27, 2013

    Page 88

    Douglas Hoffman Copyright © 2004-13, SQM, LLC. Slide 175