Excerpts from A Practitioner's Guide to Software Test Design

Advantages of Scripted Testing 1. Scripted testing provides a division of laborplanning, test case design, test case

implementation, and test case execution that can be performed by people with specific skills

and at different times during the development process.

2. Test design techniques such as equivalence class partitioning, boundary value testing, control

flow testing, pairwise testing, etc. can be integrated into a formal testing process description

that not only guides our testing but that could also be used to audit for process compliance.

3. Because scripted tests are created from requirements, design, and code, all important

attributes of the system will be covered by tests and this coverage can be demonstrated.

4. Because the test cases can be traced back to their respective requirements, design, and code,

coverage can be clearly defined and measured.

5. Because the tests are documented, they can be easily understood and repeated when

necessary without additional test analysis or design effort.

6. Because the tests are defined in detail, they are more easily automated.

7. Because the tests are created early in the development process, this may free up additional

time during the critical test execution period.

8. In situations where a good requirements specification is lacking, the test cases, at the end of

the project, become the de facto requirements specification, including the results that

demonstrate which requirements were actually fulfilled and which were not.

9. Scripted tests, when written to the appropriate level of detail, can be run by people who

would otherwise not be able to test the system because of lack of domain knowledge or lack

of testing knowledge.

10. You may have special contractual requirements that can only be met by scripted testing.

11. There may be certain tests that must be executed in just the same way, every time, in order to

serve as a kind of benchmark.

12. By creating the tests early in the project we can discover what we don't know.

13. By creating the tests early we can focus on the "big picture."

In his book Software System Testing and Quality Assurance, Boris Beizer summarizes in this way:

"Testing is like playing pool. There's real pool and kiddie pool. In kiddie pool, you hit the

balls and whatever pocket they happen to fall into, you claim as the intended pocket. It's not

much of a game and although suitable to ten-year-olds it's hardly a challenge. The object of

real pool is to specify the pocket in advance. Similarly for testing. There's real testing and

kiddie testing. In kiddie testing, the tester says, after the fact, that the observed outcome was

the intended outcome. In real testing the outcome is predicted and documented before the

test is run."

Disadvantages of Scripted Testing 1. Scripted testing is very dependent on the quality of the system's requirements. Will the

requirements really be complete, consistent, unambiguous, and stable enough as the

foundation for scripted testing? Perhaps not.

2. Scripted testing is, by definition, inflexible. It follows the script. If, while testing, we see

something curious, we note it in a Test Incident Report but we do not pursue it. Why not?

Because it is not in the script to do so. Many interesting defects could be missed with this

approach.

3. Scripted testing is often used to "de-skill" the job of testing. The approach seems to be,

"Teach a tester a skill or two and send them off to document mountains of tests. The sheer

bulk of the tests will probably find most of the defects."

Advantages of Exploratory Testing 1. Exploratory testing is valuable in situations where choosing the next test case to be run

cannot be determined in advance, but should be based on previous tests and their results.

2. Exploratory testing is useful when you are asked to provide rapid feedback on a product's

quality on short notice, with little time, off the top of your head, when requirements are

vague or even nonexistent, or early in the development process when the system may be

unstable.

3. Exploratory testing is useful when, once a defect is detected, we want to explore the size,

scope, and variations of that defect to provide better feedback to our developers.

4. Exploratory testing is a useful addition to scripted testing when the scripted tests become

"tired," that is, they are not detecting many errors.

Disadvantages of Exploratory Testing 1. Exploratory testing has no ability to prevent defects. Because the design of scripted test cases

begins during the requirements gathering and design phases, defects can be identified and

corrected earlier.

2. If you are already sure exactly which tests must be executed, and in which order, there is no

need to explore. Write and then execute scripted tests.

3. If you are required by contract, rule, or regulation to use scripted testing then do so.

Consider adding exploratory tests as a complementary technique.