Testing Tools

Prepared by

Stephen M. Thebaut, Ph.D.

University of Florida

Software Testing and Verification

Lecture 14

Overview of Topics

• Why is test automation desirable?

• Types of test automation– test design tools– test implementation (including “testing

frameworks”) tools– test execution and other run-time tools

• Sources for testing tools

• One other “Testing Tool:” Test Plans and Planning

Why is test automation desirable?

• A recent survey result reported in SoftwareTech News† (a DoD publication) confirms that software testing activity consumes a significant fraction of the overall development lifecycle.

† October 2007 Vol. 10, Number 3

• In the same article, a study undertaken by the US Navy showed that substantial time savings can be achieved via automated testing.

Why is test automation desirable? (cont’d)

• Some types of testing are not feasible without automation, e.g.:

– load (“concurrency”) testing

– long term “soak” testing to detect memory leaks/performance degradation

• More efficient build verification (“smoke”) testing

• Stand-alone “lights out” testing

• Increased repeatability

• Testers can focus on more creative tasks

• Much higher levels of coverage

Other benefits of automated testing

Test Design Tools

Test Design Tools

• Parameter value generators

• Boundary value, equivalence class, and special value generators

• Random data generation based on probabilistic CFGs (Context-Free Grammars)

• Fault tree coverage

• Cause-Effect analysis

• Database test data generators (for performance, load, and usability test)

(cont’d)

Test Design Tools (cont’d)

• Statistical test generators (for reliability, availability, and performance testing)

• Structural test case design tools:

– utilize code instrumentation (see Run-Time Tools)

– source code specific

– may incorporate symbolic execution capabilities to aid in path sensitization

Test Implementation Tools

Test Implementation Tools

• Test Case Scripting Languages:

– test scripts automate the steps involved in carrying out tests

– suitable for tests that will be repeated (i.e., in support of regression testing)

– scripts are maintained in a change-controlled Test Database

(cont’d)

Test Implementation Tools (cont’d)

• Test Harness (scaffolding) Generators:

– automates development of test drivers, stubs, and oracles (“assertions”)

– often coupled with a test scripting capability (= “testing frameworks”)

– useful for module and integration test

Run-Time Tools

Run-Time Tools

• Code Instrumentation:– inserts additional code to

• monitor cumulative logic/dataflow coverage,• support trace and debug functions, and• detect run-time errors.

– aids in designing test cases to meet white-box coverage goals or to debug code.

– alters execution performance and may mask or introduce timing errors.

– many commercial (source code specific) tools available.

– among the oldest of testing tool types.

(cont’d)

Let’s Pause for a Moment…

Imagine that it’s summertime and that a 3-day weekend is just starting… Wouldn’t it be great to just grab a fishin’ pole and head on out to the lake!

Let’s Pause for a Moment… (Take 2)

There’s NOTHING like Paris in the the spring!

Run-Time Tools (cont’d)

• Data / Output Comparators

– used to automatically compare output or data files for discrepancies

– effective in dealing with the “Paris in the the spring” problem†

– most are not “smart enough” to ignore “don’t care” data fields.

– useful for regression testing when combined with a keystroke recorder and playback tool…

† Also know as inattentional or perceptual blindness.

(cont’d)

Run-Time Tools (cont’d)

• Keystroke Recorder and Playback Tools

– records and plays-back keystrokes, pointer movement, and mouse clicks

– useful for regression testing when combined with a data/output comparator

Sources for Tools

• An excellent source of information about testing tools is the WWW. Use your favorite search engine to find sites associated with “software testing tools” (about 94 million Google hits at last count).

• Several QA and testing related magazines, newsletters, and trade journals are available which publish tool reviews and comparisons.

(cont’d)

Sources for Tools (cont’d)

• Relevant links can also be found at the websites of several QA and testing related organizations and societies, e.g.,

– SEI– IEEE– ACM– American Society for Quality– Society for Software Quality– Quality Assurance Institute– European Organization for Quality

(cont’d)

Sources for Tools (cont’d)

• Some useful testing tools-related links:

– www.opensourcetesting.org

– www.testingfaqs.org

– www.qaforums.com

Test Plans and Planning

Pezze and Young, Software Testing and Analysis, Wiley, 2008

Motivation

• Planning – just as important in testing as in any other engineering activity.

• Allows for the effective use of limited resources and management of activities.

• In a nutshell, it involves a disciplined, timely consideration of:

– What needs to be done,

– When and How it will be done, and

– What resources (including time and effort) will be required.

The Master Test Plan

• Also called a Comprehensive Test Plan.• Must be compatible with overall project plan.• Developed early – during project planning and

requirements stages.• Defines requirements for lower level plans:

– Unit Test Plan– Component Test Plan– Product Test Plan– System Test Plan

Master Test Plan Check-List

• Defines test stages (e.g., unit, integration, system) schedules, and responsibilities.

• Establishes objectives, strategies, and procedures.

• Identifies required tools, facilities, and test libraries.

• Sets criteria for test completion and success.• Defines the integration test strategy and iden-

tifies all scaffolding required with respon-sibilities and schedules for its production.

(cont’d)

Master Test Plan Check-List (cont’d)

• Defines test coverage goals and methods for tracking progress.

• Establishes a regression test policy.

• Specifies required hardware and software, particularly non-standard or special purpose elements.

• Identifies estimated resource requirements including machine time, disk storage, physical / logical device access, etc.

• Identifies training required.

• Specifies test entry and exit criteria.

Typical Test Entry Criteria

• Test plan is complete and has been reviewed

• Design and implementation of test cases reviewed and all problems fixed

• Drivers, stubs, oracles in place

• Test environment (hardware and software) is ready

Typical Test Exit Criteria

• All problems discovered have been recorded and fixed

• All fixes have been re-tested

• Regression test(s) complete

• All test cases and associated documentation updated as necessary

IEEE Test Plan Template

1. Introduction and references2. Test items (versions, media, references, etc.)3. Features to be tested4. Features NOT to be tested (and reasons)5. Testing strategies and approach6. Dependencies7. Test case success/failure criteria8. Pass/fail criteria for the complete test9. Test entry/exit criteria

(cont’d)

IEEE Test Plan Template (cont’d)

10. Test deliverables/status communication vehicles

11. Testing tasks12. Hardware and software requirements13. Problem determination and correction

responsibilities14. Staffing and training needs/assignments15. Test schedules16. Risks and contingencies17. Approvals

Test Planning Summary

• Test planning allows for the effective use of limited resources and the effective manage-ment of test activities.

• A plan is only as good as the planning that goes into it!

Testing Tools

Prepared by

Stephen M. Thebaut, Ph.D.

University of Florida

Software Testing and Verification

Lecture 14