Black Box Software Testing

Copyright © 1994-2004 Cem Kaner and SQM, LLC. All Rights Reserved. 527

Black Box Software Testing

Part 15.

Testing User Documentation


Copyright Notice

These slides are distributed under the Creative Commons License.

In brief summary, you may make and distribute copies of these slides so long as you give the original author credit and, if you alter, transform or build upon this work, you distribute the resulting work only under a license identical to this one.

For the rest of the details of the license, see http://creativecommons.org/licenses/by-sa/2.0/legalcode.


Documentation is anExpress Warranty

A warranty is a statement of fact, either articulated or implied by law, respecting the quality or character of the goods to be sold. Under the Uniform Commercial Code an express warranty is:2-313(a) Any affirmation of fact or promise made by the seller to the buyer which relates to the goods and becomes part of the basis of the bargain . . .2-313(b) Any description of the goods which is made part of the basis of the bargain . . .2-313(c) Any sample or model which is made part of the basis of the bargain.


Documentation is anExpress Warranty

You can’t disclaim an express warranty --you are accountable for your claims.

Uniform Commercial Code 2-316 (1):

Words or conduct relevant to the creation of an express warranty and words or conduct tending to negate or limit warranty shall be construed whenever reasonable as consistent with each other; but . . . negation or limitation is inoperative to the extent that such construction is unreasonable.


Black Box Testing:Testing Documentation

Doc testing is important because:• Errors in the manual increase risks of legal liability.• Testing the documentation improves the reliability

of the program.• The documentation may be your mainstream test

plan and your most up-to-date specification.• Confusion in the manual reflects confusion in the

program’s design.

Refer to Testing Computer Software, Chapter 10


Testing Documentation:What to Test

• Verify every statement of fact and every reasonable implication• Check the placement and accuracy of figures• Audit the completeness of the manual (check that every feature is

documented)

Track errors in the documentation in a way that is normal for the Doc group. This probably doesn’t involve the bug tracking system (but put code/doc mismatches there). If you give back marked up manuscripts, keep photocopies of your markups. Check your corrections against the next circulating draft of themanual.

On average, you will cover 4 pages per hour in a reasonably stable program. Your second pass will go more quickly because the program is in better shape, but it will still take several minutes per page because you will actually test every page.


Testing Documentation:Things to Say

• Your role is not editorial: you are not the authority on style and layout.

• Keep your tone non-judgmental.• Point out upcoming changes (design changes,

new error handling, data structures, etc.) that might affect the manual. Mark these in appropriate sections on the manuscript.

• Name experts or references to consult when the writer is confused.

• Suggest examples.• Point to useful existing data files (for examples).


Testing Documentation:Things to Say

• When appropriate, you might do some writing. The writer might or might not use what you have written. You might write in two ways:» words that you think belong in the manual “as is”

(you’re saying, “Here, say it like this and it will be right.”)

» explanations that are background material for the writer. Maybe a rough draft of what she’ll write.

• Note: the final review is a meeting just a few days before the book goes to the printer. If you have many small-detail late comments, offer the writer a chance to review them privately with you a few days before the review.


Documentation Testing:On-Line Help

• Contents of the help• Cross-reference jumps• Glossary lookups• Browse sequences• Graphic hotspots• Graphic display - color or resolution• Window size, e.g. compile at 1024x768 and display at 640x480• Procedure sequences• Balloon help / tool tips• Index• Search• Context jumps• Error messages

Refer to Testing Computer Software, Chapter 10


Publisher Liability forContent-Related Errors

Winter v. G.P. Putnam’s Sons, 938 F.2d 1033, (9th Circuit) 1991. Winter became seriously ill from picking and eating mushrooms after relying on The Encyclopedia of Mushrooms, published by Putnam. Putnam did not verify the material in the book and did not intentionally include the error.

• Putnam was not liable for errors in the book.• Noted that Jeppesen cases consistently held publisher liable for

information error when information published was to be used as a tool.• The court said that a software publisher might be liable for program that

does not work as intended.ALM v. Van Nostrand Reinhold 480 NE.2d 1263, 1985. The Making of Tools. Plaintiff used it, and a tool shattered, causing injury. VNR not liable, but author of the book might be.

Liability might also attach if the program provides professionalservices (such as tax preparation) and gives bad information.


Warranties & Misrepresentations: What Must You Test?

• Advertisements• Published specifications• Interviews• Box copy• Fax-backs• Manual• Help system• Warranty• Web pages• Readme• Advice given to customers on Internet, CompuServe or AOL


Black Box Software TestingPart 16.

Managing Automated Software Testing

Several of these slides were developed by Doug Hoffman or in co-authorship with Doug Hoffman for a course that we co-taught on software test automation.

Many of the ideas in this presentation were presented and refined in Los Altos Workshops on Software Testing.

LAWST 5 focused on oracles. Participants were Chris Agruss, James Bach, Jack Falk, David Gelperin, Elisabeth Hendrickson, Doug Hoffman, Bob Johnson, Cem Kaner, Brian Lawrence, Noel Nyman, Jeff Payne, Johanna Rothman, Melora Svoboda, Loretta Suzuki, and Ned Young.

LAWST 1-3 focused on several aspects of automated testing. Participants were Chris Agruss, Tom Arnold, Richard Bender, James Bach, Jim Brooks, Karla Fisher, Chip Groder, Elizabeth Hendrickson, Doug Hoffman, Keith W. Hooper, III, Bob Johnson, Cem Kaner, Brian Lawrence, Tom Lindemuth, Brian Marick, Thanga Meenakshi, Noel Nyman, Jeffery E. Payne, Bret Pettichord, Drew Pritsker, Johanna Rothman, Jane Stepak, Melora Svoboda, Jeremy White, and Rodney Wilson.

I’m indebted to James Whittaker, James Tierney, Harry Robinson, and Noel Nyman for additional explanations of stochastic testing.

Copyright © 1994-2004 Cem Kaner and SQM, LLC. All Rights Reserved. 539and SQM, LLC.

Overview

• About Automated Software Testing

• The regression testing paradigm

• 19 common mistakes

• 27 questions about requirements

• Planning for short-term ROI

• 6 sometimes successful architectures

• Conclusions from LAWST

• Breaking away from the regression paradigm


Overview

In the mass-market software world, many efforts to automate testing have been expensive failures. Paradigms that dominate the Information Technology and DoD-driven worlds don’t apply well to the mass-market paradigm. Our risks are different. Solutions that are highly efficient for those environments are not at all necessarily good for mass-market products.The point of an automated testing strategy is to save time and money. Or to find more bugs or harder-to-find bugs. The strategy works if it makes you more effective (you find better and more-reproducible bugs) and more efficient (you find the bugs faster and cheaper). I wrote a lot of test automation code back in the late 1970’s and early 1980’s. (That’s why WordStar hired me as its Testing Technology Team Leader when I came to California in 1983.) Since 1988, I haven’t written a line of code. I’ve been managing other people, and then consulting, seeing talented folks succeed or fail when confronted with similar problems. I’m not an expert in automated test implementation, but I have strengths in testing project management, and how that applies to test automation projects. That level of analysis--the manager’s view of a technology and its risks/benefits as an investment--is the point of this section.


About Automated Testing

Source of test cases• Old• Intentionally new• Random new

Size of test pool• Small• Large• Exhaustive

Serial dependence among tests• Independent• Sequence is relevant

and SQM, LLC.


About Automated Testing

Evaluation strategy

• Comparison to saved result

• Comparison to an oracle

• Comparison to a computational or logical model

• Comparison to a heuristic prediction. (NOTE: All oracles are heuristic.)

• Crash

• Diagnostic

• State model

and SQM, LLC.


Issues Faced in A Typical Automated Test

• What is being tested?

• How is the test set up?

• Where are the inputs coming from?

• What is being checked?

• Where are the expected results?

• How do you know pass or fail?

and SQM, LLC.


The “Complete” Oracle

Test ResultsTest Results

Postcondition DataPostcondition Data

Postcondition Program State

Postcondition Program State

Environmental Results

Environmental Results

Test Oracle

System UnderTest

Test Inputs

Precondition Data

PreconditionProgram State

EnvironmentalInputs

Reprinted with permission of Doug Hoffmanand SQM, LLC.


Automated Software Test Functions

• Automated test case/data generation• Test case design from requirements or code• Selection of test cases• Able to run two or more specified test cases• Able to run a subset of all the automated test cases• No intervention needed after launching tests• Automatically sets-up and/or records relevant test environment• Runs test cases• Captures relevant results• Compares actual with expected results• Reports analysis of pass/fail

and SQM, LLC.


Characteristics of“fully automated” tests

• A set of tests is defined and will be run together.

• No intervention needed after launching tests.

• Automatically sets-up and/or records relevant test environment.

• Obtains input from existing data files, random generation, or another defined source.

• Runs test exercise.

• Captures relevant results.

• Evaluates actual against expected results.

• Reports analysis of pass/fail.

Not all automation is full automation. Partial automation can be very useful.

and SQM, LLC.


Capabilities of Automation Tools

Automated test tools combine a variety of capabilities. For example, GUI regression tools provide:

• capture/replay for easy manual creation of tests

• execution of test scripts

• recording of test events

• compare the test results with expected results

• report test results

Some GUI tools provide additional capabilities, but no tool does everything well.

and SQM, LLC.


Capabilities of Automation Tools

Here are examples of automated test tool capabilities:• Analyze source code for bugs• Design test cases• Create test cases (from requirements or code)• Generate test data• Ease manual creation of test cases• Ease creation/management of traceability matrix• Manage testware environment• Select tests to be run• Execute test scripts• Record test events• Measure software responses to tests (Discovery Functions)• Determine expected results of tests (Reference Functions)• Evaluate test results (Evaluation Functions)• Report and analyze results

and SQM, LLC.


Tools for Improving Testability by Providing Diagnostic Support

• Hardware integrity tests. Example: power supply deterioration can look like irreproducible, buggy behavior.

• Database integrity. Ongoing tests for database corruption, making corruption quickly visible to the tester.

• Code integrity. Quick check (such as checksum) to see whether part of the code was overwritten in memory.

• Memory integrity. Check for wild pointers, other corruption.

• Resource usage reports: Check for memory leaks, stack leaks, etc.

• Event logs. See reports of suspicious behavior. Probably requires collaboration with programmers.

• Wrappers. Layer of indirection surrounding a called function or object. The automator can detect and modify incoming and outgoing messages, forcing or detecting states and data values of interest.

and SQM, LLC.


Automation Design

• Determine the goals of the automation

• Determine the capabilities needed to achieve those goals

• Select automation components

• Set relationships between components

• Identify locations of components and events

• Sequence test events

• Evaluate and report results of test events.

and SQM, LLC.


The Regression Testing Paradigm

The most commonly discussed automation approach:

• create a test exercise

• run it and inspect the output

• if the program fails, report a bug and try again later

• if the program passes the test, save the resulting outputs

• in future tests, run the program and compare the output to the saved results. Report an exception whenever the current output and the saved output don’t match.

and SQM, LLC.


Is this Really Automation?

Analyze product -- humanDesign test -- humanRun test 1st time -- humanEvaluate results -- humanReport 1st bug -- humanSave code -- humanSave result -- humanDocument test -- humanRe-run the test -- MACHINEEvaluate result -- MACHINE

(plus human if there’s any mismatch)Maintain result -- human

Woo-hoo! We really get the machine to do a whole lot of our work! (Maybe, but not this way.)

and SQM, LLC.


Common Mistakes about Test Automation

The paper (Avoiding Shelfware) lists 19 “Don’ts.”For example,

Don’t expect to be more productive over the short term.

• The reality is that most of the benefits from automation don’t happen until the second release.

• It takes 3 to 10+ times the effort to create an automated test than to just manually do the test. Apparent productivity drops at least 66% and possibly over 90%.

• Additional effort is required to create and administer automated test tools.

and SQM, LLC.


GUI Automation is Expensive

• Test case creation is expensive. Estimates run from 3-5 times the time to create and manually execute a test case (Bender) to 3-10 times (Kaner) to 10 times (Pettichord) or higher (LAWST).

• You usually have to increase the testing staff in order to generate automated tests. Otherwise, how will you achieve the same breadth of testing?

• Your most technically skilled staff are tied up in automation

• Automation can delay testing, adding even more cost (albeit hidden cost.)

• Excessive reliance leads to the 20 questions problem. (Fully defining a test suite in advance, before you know the program’s weaknesses, is like playing 20 questions where you have to ask all the questions before you get your first answer.)

and SQM, LLC.


GUI Automation Pays off Late

• GUI changes force maintenance of tests» May need to wait for GUI stablization

» Most early test failures are due to GUI changes

• Regression testing has low power» Rerunning old tests that the program has passed is

less powerful than running new tests

» Old tests do not address new features

• Maintainability is a core issue because our main payback is usually in the next release, not this one.

and SQM, LLC.and SQM, LLC.


Test Automation is Programming

Win NT 4 had 6 million lines of code, and 12 million lines of test codeCommon (and often vendor-recommended) design and programming practices for automated testing are appalling:

• Embedded constants• No modularity• No source control• No documentation• No requirements analysis

No wonder we fail


Notes

______________________________________________________________________________________________________________________ __________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

and SQM, LLC.


Requirements Analysis

Automation requirements are not just about the software under test and its risks. To understand what we’re up to, we have to understand:

• Software under test and its risks

• The development strategy and timeframe for the software under test

• How people will use the software

• What environments the software runs under and their associated risks

• What tools are available in this environment and their capabilities

• The regulatory / required recordkeeping environment

• The attitudes and interests of test group management.• The overall organizational situation

and SQM, LLC.


Requirements Analysis

Requirement: “Anything that drives design choices.”The paper (Avoiding Shelfware) lists 27 questions. For example,

Will the user interface of the application be stable or not?

• Let’s analyze this. The reality is that, in many companies, the UI changes late.

• Suppose we’re in an extreme case. Does that mean we cannot automate cost effectively? No. It means that we should do only those types of automation that will yield a fast return on investment.

and SQM, LLC.


You Can Plan for Short Term ROI

• Smoke testing• Configuration testing

• Variations on a theme

• Stress testing

• Load testing

• Life testing

• Performance benchmarking

• Other tests that extend your reach

and SQM, LLC.


Six Sometimes-Successful Automation Architectures

• Quick & dirty

• Equivalence testing

• Framework

• Data-driven

• Application-independent data-driven

• Real-time simulator with event logs

and SQM, LLC.


Quick and Dirty

• Smoke tests

• Configuration tests

• Variations on a theme

• Stress, load, or life testing

and SQM, LLC.


Equivalence Testing

A/B comparison

Random tests using an oracle

Regression testing is the weakest form

and SQM, LLC.


Framework-Based Architecture

Frameworks are code libraries that separate routine calls from designed tests.• modularity• reuse of components• hide design evolution of UI or tool commands• partial salvation from the custom control problem• independence of application (the test case) from user interface

details (execute using keyboard? Mouse? API?)• important utilities, such as error recovery

For more on frameworks, see Linda Hayes’ book on automated testing, Tom Arnold’s book on Visual Test, and Mark Fewster& Dorothy Graham’s book “Software Test Automation.”

and SQM, LLC.


Data-Driven Tests

• Variables are data

• Commands are data

• UI is data

• Program’s state is data

• Test tool’s syntax is data

and SQM, LLC.


Data-Driven Architecture

• In test automation, there are three interesting programs:• The software under test (SUT)• The automation tool that executes the automated test code• The test code (test scripts) that define the individual tests

• From the point of view of the automation software,• The SUT’s variables are data• The SUT’s commands are data• The SUT’s UI is data• The SUT’s state is data

• Therefore it is entirely fair game to treat these implementationdetails of the SUT as values assigned to variables of the automation software.

• Additionally, we can think of the externally determined (e.g. determined by you) test inputs and expected test results as data.

• Additionally, if the automation tool’s syntax is subject to change, we might rationally treat the command set as variable data as well.

and SQM, LLC.


Data-Driven Architecture

In general, we can benefit from separating the treatment of one type of data from another with an eye to:• optimizing the maintainability of each• optimizing the understandability (to the test case creator

or maintainer) of the link between the data and whatever inspired those choices of values of the data

• minimizing churn that comes from changes in the UI, the underlying features, the test tool, or the overlying requirements

and SQM, LLC.


Data-Driven Architecture:Calendar Example

Imagine testing a calendar-making program. The look of the calendar, the dates, etc., can all be thought ofas being tied to physical examples in the world, rather than being tied to the program. If your collection of cool calendars wouldn’t change with changes in the UI of the software under test, then the test data that define the calendar are of a different class from the test data that define the program’s features.– Define the calendars in a table. This table should not be

invalidated across calendar program versions. Columns name features settings, each test case is on its own row.

– An interpreter associates the values in each column with a set of commands (a test script) that execute the value of the cell in a given column/row.

– The interpreter itself might use “wrapped” functions, i.e. make indirect calls to the automation tool’s built-in features.

and SQM, LLC.


Data-Driven Architecture:Calendar Example

This is a good design from the point of view of optimizing for maintainability because it separates out four types of things that can vary independently:• The descriptions of the calendars themselves come from real-world and

can stay stable across program versions.• The mapping of calendar element to UI feature will change frequently

because the UI will change frequently. The mappings (one per UI element) are written as short, separate functions that can be maintained easily.

• The short scripts that map calendar elements to the program functions probably call sub-scripts (think of them as library functions) that wrap common program functions. Therefore a fundamental change in the program might lead to a modest change in the software under test.

• The short scripts that map calendar elements to the program functions probably also call sub-scripts (think of them as library functions) that wrap functions of the automation tool. If the tool syntax changes, maintenance involves changing the wrappers’ definitions rather than the scripts.

and SQM, LLC.


Data Driven Architecture

Note with this example:• we didn’t run tests twice• we automated execution, not evaluation• we saved SOME time• we focused the tester on design and results, not execution.

Other table-driven cases:• automated comparison can be done via a pointer in the

table to the file• the underlying approach runs an interpreter against table

entries. » Hans Buwalda and others use this to create a structure that is

natural for non-tester subject matter experts to manipulate.

and SQM, LLC.


Application-Independent Data-Driven

• Generic tables of repetitive types

• Rows for instances

• Automation of exercises

and SQM, LLC.


Real-time Simulator

• Test embodies rules for activities

• Stochastic process

• Possible monitors

• Code assertions• Event logs• State transition maps• Oracles

and SQM, LLC.


Think About:

• Automation is software development.

• Regression automation is expensive and can be

inefficient.

• Automation need not be regression--you can

run new tests instead of old ones.

• Maintainability is essential.

• Design to your requirements.

• Set management expectations with care.

and SQM, LLC.


GUI Regression Strategies:Some Papers of Interest

Chris Agruss, Automating Software Installation TestingJames Bach, Test Automation Snake OilHans Buwalda, Testing Using Action WordsHans Buwalda, Automated testing with Action Words: Abandoning Record

& PlaybackElisabeth Hendrickson, The Difference between Test Automation Failure

and SuccessCem Kaner, Avoiding Shelfware: A Manager’s View of Automated GUI

TestingJohn Kent, Advanced Automated Testing ArchitecturesBret Pettichord, Success with Test AutomationBret Pettichord, Seven Steps to Test Automation SuccessKeith Zambelich, Totally Data-Driven Automated Testing

and SQM, LLC.


Notes

______________________________________________________________________________________________________________________ __________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________



Part 17

Test Strategy Planning


Test Strategy

“How we plan to cover the product so as to develop an adequate assessment of quality.”

A good test strategy is:

• Diversified• Specific• Practical• Defensible


Test Strategy

• Makes use of test techniques.

• May be expressed by test procedures and cases.

• Not to be confused with test logistics, which involve the details of bringing resources to bear on the test strategy at the right time and place.

• You don’t have to know the entire strategy in advance. The strategy can change as you learn more about the product and its problems.


Test Cases/Procedures

• Test cases and procedures should manifest the test strategy.

• If your strategy is to “execute the test suite I got from Joe Third-Party”, how does that answer the prime strategic questions:

• How will you cover the product and assess quality?

• How is that practical and justified with respect to the specifics of this project and product?

• If you don’t know, then your real strategy is that you’re trusting things to work out.


Diverse Half-Measures

• There is no single technique that finds all bugs.• We can’t do any technique perfectly.• We can’t do all conceivable techniques.

Use “diverse half-measures”-- lots of differentpoints of view, approaches, techniques, evenif no one strategy is performed completely.


Heuristics from James Bach’s Test Plan Evaluation Model, www.satisfice.com

1. Look for the important problems first

2. Focus mainly on areas of potential technical risk

3. Address configuration, operation , observation,

and evaluation of the product

4. Diversify test techniques and perspectives

5. Specify test data design and generation

6. Don’t just run preplanned tests

7. Test against stated and implied requirements

8. Collaborate with outside people for testing ideas


Heuristics from James Bach’s Test Plan Evaluation Model, www.satisfice.com

9. Work with developers to improve product testability

10. Highlight the non-routine, project-specific aspects of the testing strategy and the testing project

11. Use people and tools for what they do best12. Identify dependencies in the testing schedule13. Keep testing off the critical path for release14. Make the feedback loop with development short15. Learn what people outside of testing think of the

product quality16. Review all test materials


Notes

______________________________________________________________________________________________________________________ __________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________


Heuristics for Test Plan EvaluationBasis for the HeuristicHeuristic

Sloppiness or neglect within any of these four basic testing activities will increase the likelihood that important problems will go undetected.

3. Test strategy should address test platform configuration, how the product will be operated, how the product will be observed, and how observations will be used to evaluate the product.

Complete testing is impossible, and we can never know if our perception of technical risk is completely accurate.

2. Test strategy should focus most effort on areas of potential technical risk, while still putting some effort into low risk areas just in case the risk analysis is wrong.

The later in the project that a problem is found, the greater the risk that it will not be safely fixed in time to ship. The sooner a problem is found after it is created, the lesser the risk of a bad fix.

1. Testing should be optimized to find important problems fast, rather than attempting to find all problems with equal urgency.


Heuristics for Test Plan Evaluation

Basis for the HeuristicHeuristic

No single test technique can reveal all important problems in a linear fashion. We can never know for sure if we have found all the problems that matter. Diversification minimizes the risk that the test strategy will be blind to certain kinds of problems.Use diverse half-measures to go after low-hanging fruit.

4. Test strategy should be diversified in terms of test techniques and perspectives. Methods of evaluating test coverage should take into account multiple dimensions of coverage, including structural, functional, data, platform, operations, and requirements.

It is common for the test strategy to be organized around functionality or code, leaving it to the testers to concoct test data on the fly. Often that indicates that the strategy is too focused on validating capability and not focused enough on reliability.

5. The test strategy should specify how test data will be designed and generated.




A rigid test strategy may make it more likely that a particular subset of problems will be uncovered, but in a complex system it reduces the likelihood that all important problems will be uncovered. Reasonable variability in testing, such as that which results from interactive, exploratory testing, increases incidental test coverage, without substantially sacrificing essential coverage.

6. Not all testing should be pre-specified in detail. The test strategy should incorporate reasonable variation and make use of the testers’ ability to use situational reasoning to focus on important, but unanticipated problems.

Testing only against explicit written requirements will not reveal all important problems, since defined requirements are generally incomplete and natural language is inherently ambiguous.

7. It is important to test against implied requirements—the full extent of what the requirements mean, not just what they say.



Other teams and stakeholders often have information about product problems or potential problems that can be of use to the test team. Their perspective may help the testers make a better analysis of risk. Testers may also have information that is of use to them.

8. The test project should promote collaboration with all other functions of the project, especially developers, technical support, and technical writing. Whenever possible, testers should also collaborate with actual customers and users, in order to better understand their requirements.

The likelihood that a test strategy will serve its purpose is profoundly affected by the testability of the product.

9. The test project should consult with development to help them build a more testable product.





Virtually every software project worth doing involves special technical challenges that a good test effort must take into account. A completely generic test plan usually indicates a weak test planning process. It could also indicate that the test plan is nothing but unchanged boilerplate.

10. A test plan should highlight the non-routine, project-specific aspects of the test strategy and test project.




Many test projects suffer under the false belief that human testers are effective when they use exactingly specified test scripts, or that test automation duplicates the value of human cognition in the test execution process. Manual and automated testing are not two forms of the same thing. They are two entirely different classes of test technique.

11. The test project should use humans for what humans do well and use automation for what automation does well. Manual testing should allow for improvisation and on the spot critical thinking, while automated testing should be used for tests that require high repeatability, high speed, and no judgment.



This is important in order to deflect pressure to truncate the testing process.

13. The test process should be kept off of the critical path to the extent possible. This can be done by testing in parallel with development work, and finding problems worth fixing faster than the developers fix them.


A monolithic test schedule in a test plan often indicates the false belief that testing is an independent activity. The test schedule can stand alone only to the extent that the product the highly testable, development is complete, and the test process is not interrupted by the frequent need to report problems.

12. The test schedule should be represented and justified in such a way as to highlight any dependencies on the progress of development, the testability of the product, time required to report problems, and the project team’s assessment of risk.




This is important in order to maximize the efficiency and speed of quality improvement. It also helps keep testing off of the critical path.

14. The feedback loop between testers and developers should be as tight as possible. Test cycles should be designed to provide rapid feedback to developers about recent additions and changes they have made before a full regression test is commenced. Whenever possible testers and developers should work physically near each other.




Tunnel-vision is the great occupational hazard of testing. Review not only helps to reveal blind spots in test design, but it can also help promote dialog and peer education about test practices.

16. All documentation related to the test strategy, including test cases and procedures, should be undergo review by someone other than the person who wrote them. The review process used should be commensurate with the criticality of the document.

By examining product quality information gathered through various means beyond the test team, blind spots in the formal test strategy can be uncovered.

15. The test project should employ channels of information about quality other than formal testing in order to help evaluate and adjust the test project. Examples of these channels are inspections, field testing, or informal testing by people outside of the test team.


Evaluating Your Plan: Context Free Questions

Based on: The CIA’s Phoenix Checklists (Thinkertoys, p. 140) and Bach’s Evaluation Strategies (Rapid Testing Course notes)• Can you solve the whole problem? Part of the problem?• What would you like the resolution to be? Can you picture it?• How much of the unknown can you determine?• What reference data are you using (if any)?• What product output will you evaluate?• How will you do the evaluation?• Can you derive something useful from the information you have?• Have you used all the information?• Have you taken into account all essential notions in the problem?• Can you separate the steps in the problem-solving process? Can you

determine the correctness of each step?• What creative thinking techniques can you use to generate ideas? How

many different techniques?• Can you see the result? How many different kinds of results can you see?• How many different ways have you tried to solve the problem?


Evaluating Your Plan: Context Free Questions

• What have others done?• Can you intuit the solution? Can you check the results?• What should be done? • How should it be done?• Where should it be done?• When should it be done?• Who should do it?• What do you need to do at this time?• Who will be responsible for what?• Can you use this problem to solve some other problem?• What is the unique set of qualities that makes this

problem what it is and none other?• What milestones can best mark your progress?• How will you know when you are successful?• How conclusive and specific is your answer?


Notes

______________________________________________________________________________________________________________________ __________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________



Part 18.

Planning Testing Projects


Planning and Negotiating the Testing Project

• The notes here focus on your process of negotiating resources and quality level.

• Please see Testing Computer Software, Chapter 13 for a detailed discussion of planning and testing tasks across the time line of the project.


Early Planning (1)

Things you can do very early in the project:• Analyze any requirements documents for testability,

ambiguity.• Facilitate inspection and walkthrough meetings.• Prepare a preliminary list of hardware configurations

and start arranging for loaners.• Ask for testing support code, such as debug monitors,

memory meters, support for testpoints.• Ask for a clear error handling message structure.• Discuss the possibility of code coverage measurement

(which will require programmer support).


Early Planning (2)

Things you can do very early in the project:• Prepare for test automation. This involves reaching

agreements in principle on breadth of automation and automation support staffing level.

• Order automation support equipment.• Order external test suites.• Learn about the product’s market and competition.• Evaluate coding tools that facilitate automation (e.g.

test 3rd party custom controls against MS-Test)


First Principles in Planning

• You can’t nail everything down.• You will face difficult prioritization choices, and

many constraints will be out of your direct control.• You can influence several constraints by opening up

your judgments to other stakeholders in the company. This is more than getting a sign-off.

• Reality is far more important than your ability to cast blame.

• Your task is to manage project-level risks. This includes risks to the project’s cost, schedule, and interpersonal dynamics, as well as to the product’s feature set and reliability.


Notes

______________________________________________________________________________________________________________________ __________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________


Deciding Your Depth of Testing

You have three key objectives:1 Achieve a uniform and well-understood minimum level of

testing.2 Be explicit about the level of testing for each area:

» mainstream» exploratory» formally planned» structured regression

3 Reach corporate agreement on the level of testing for each area Just like bug deferrals, depth-of-testing restrictions

must rest on sound business decisions.This should not be your decision to make.


Identify the Testing Tasks

Develop a project task list with your staff. Remember, you are looking for realism in estimates of complexity, difficulty, and time.

• Make the listing and estimation process public, and allow publiccomment.

• Identify all potential sources of information.• List all main functional areas, and all other cross-functional

approaches that you will take in testing the program.• List every task that appears to need more than 1/2 day

(Probably you will do this by a group brainstorming session on flipcharts, listing sources on one chart. Gather the sources.List areas and approaches on a few charts. Make one chart for each area of work, and list tasks for each chart. Tentatively assign times to each task, possibly by a museum tour.)


Estimate the Project Time

1 Assign time estimates to each task. Invite programmers, marketers, and project managers to walk through the charts and provide their own estimates. Explore the bases of differences. You might have underestimated a task’s complexity. Or you might be seeing a priority disagreement. Or wishful thinking.

2 Make your best-estimate task list. Provide the time needed to achieve formal, planned testing for each area. Include estimates for structured regression for key areas. This number is probably absurdly huge.

3 Add to the list your suggestions for time-cuts to guerrila-level and mainstream-level testing for selected areas.

4 Circulate your lists to all stakeholders, call one or more planning meetings, and reach agreements on the level of testing for each area. Keep cutting tasks and/or adding staff until you reach an achievable result.


Estimate the Project Time

Don’t forget:• Budget for meetings and administrative overhead (10-30%).• Budget for bug regression (5-15%).

• Budget for down time.• Budget for holidays and vacations.

• Never develop a budget that relies on overtime. Leave the overtime free for discretionary additional work by the individual tester.

• Don’t let yourself be bullied or embarrassed, and don’t bully or embarrass your staff, into making underestimates. To achieve a result, insist on cutting tasks, adding staff, or finding realistic ways to improve productivity.


Identify Dependencies & Milestones

You can’t start reviewing the manual until you receive it, right? List these dependencies and point out, in every project meeting, what is due to you that is not yet delivered and holding you up.

Try to reach verifiable, realistic definitions for each milestone. For example, if “gamma” is 6 weeks before release, the product can’t have more than 6 weeks of schedule risk at gamma. Negotiate a definition.


Monitor Progress

• Put the tasks and agreed-to durations on a timeline and review progress of all staff every week.

• When prerequisite materials aren’t provided, find other tasks to do instead. When you can’t meet the schedule, due to absent materials, publish new estimates.

• When design changes invalidate your estimates, publish new estimates.

• If you’re falling consistently behind (e.g. due to underestimated overhead), publish the problem and ask for fewer tasks, more staff, or some other realistic solution.

• If one of your staff has trouble with an area, recognize it and deal with it.


Control Late-Stage Issues

• Watch out for late changes. Encourage people to be cautious about late bug fixes, and super-cautious about late design changes.

• Provide interim and final deferred-bug reviews.

• Take a final inventory of your testing coverage.

• Carry out final integrity tests.

• You may have to assess and reporting the reliability of the tested product.

• Plan for post-release processes. Develop a release kit for product support.

• Don’t forget the party.


Notes

______________________________________________________________________________________________________________________ __________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________



Part 19.

Metrics and Measurement Theory


We Aren’t Collecting Data.

Capers Jones, Patterns of Software Systems Failure & Success: The number one root cause of cancellations, schedule slippages, and cost overruns is the chronic failure of the software industry to collect accurate historical data from ongoing and completed projects. This failure means that the vast majority of major software projects are begun without anyone having a solid notion of how much time will be required. Software is perhaps the only technical industry where neither clients, managers, nor technical staff have any accurate quantitative data available to them from similar projects when beginning major construction activities. . . .A result that is initially surprising but quite common across the industry is to discover that the software management community within a company knows so little about the technology of software planning and estimating that they do not even know of the kinds of tools that are commercially available.


Question

Imagine being on the job. Your local PBH (pointy-haired boss) drops in and asks:

“So, how much testing have you gotten done?”

Please write down an answer. Feel free to use fictitious numbers but (except for the numbers) try to be realistic in terms of the type of information that you would provide.


How do we Measure Extent of Testing?

Before we can measure something, we need some sense of what we’re measuring. It’s easy to come up with “measurements” but we have to understand the relationship between the thing we want to measure and the statistic that we calculate to “measure” it.

If we want to measure the “extent of testing”, we have to start by

understanding what we mean by “extent of testing.”


What is measurement?

Measurement is the assignment of

numbers to objects or events according

to a rule derived from a model or theory.


The Question is Remarkably Ambiguous

Common answers are based on the:

• We’ve worked 80 hours a week on this for 4 months. We’ve run 7,243 tests.

Effort

• We’ve discovered 593 bugs.Results

• We’ve run 80% of the test cases.Plan

• We’ve tested 80% of the lines of code.Product


The Question is Remarkably Ambiguous

Common answers are based on the:

• At this milestone on previous projects, we had fewer than 12.3712% of the bugs found still open. We should be at that percentage on this product too.

History across

projects

• Beta testers have found 30 bugs that we missed. Our regression tests seem ineffective.

Quality of Testing

• We’re getting a lot of complaints from beta testers and we have 400 bugs open. The product can’t be ready to ship in three days.

Risks

• We’ve been plugging away but we can’t be efficient until X, Y, and Z are dealt with.

Obstacles


A Framework for Measurement

A measurement involves at least 10 factors:• Attribute to be measured

» appropriate scale for the attribute» variation of the attribute

• Instrument that measures the attribute» scale of the instrument» variation of measurements made with this instrument

• Relationship between the attribute and the instrument• Likely side effects of using this instrument to measure

this attribute• Purpose • Scope


Framework for Measurement

• Are we measuring the work of one tester? One team on one project? Is this a cross-project metrics effort? Cross-departmental research?

Scope

• If we do something that makes the measured result look better, will that mean that we’ve actually increased the extent of testing?

Side Effect

• Why are we measuring this? What will we do with the number?

Purpose

• How will increasing “extent of testing” affect the reading (the measure) on the instrument?

Mechanism

• What should we count? Lines? Bugs? Test cases? Hours? Temper tantrums?

Instrument

• Extent of testing – What does that mean?Attribute


Simple measurement

You have a room full of tables that appear to be the same length. You want to measure their lengths.You have a one-foot ruler.You use the ruler to measure the lengths of a few tables. You get:• 6.01 feet• 5.99 feet• 6.05 feet

You conclude that the tables are “6 feet” long.


Simple measurement (2)

Note the variation• Measurement errors using the ruler• Manufacturing variation in the tables

Note the rule:• We are relying on a direct matching operation and

on some basic axioms of mathematics» The sum of 6 one-foot ruler-lengths is 6.» A table that is 6 ruler-lengths long is twice as long

as one that is 3 ruler-lengths long.

These rules don’t always apply. What do we do when we have something hard to measure?


Attributes and Instruments

??? Branches ???Code complexity

??? Bug count ???Tester goodness

?? Count bug reports or graph bug curves??

----Proportion of bugs found

?? Count statements / branches tested ??----Product coverage

???Extent of testing???

Sound level comparisons by humansLoudness

Sound level meterSound energy

Ruler / StopwatchSpeed

StopwatchDuration

RulerLength


Surrogate measures

"Many of the attributes we wish to study do not have generally agreed methods of measurement. To overcome the lack of a measure for an attribute, some factor which can be measured is used instead. This alternate measure is presumed to be related to the actual attribute with which the study is concerned. These alternate measures are called surrogate measures."

Mark Johnson’s MA Thesis

“Surrogates” provide unambiguous assignments of numbers according to rules, but they don’t provide an underlying theory or model that relates the measure to the attribute allegedly being measured.


Consider bug counts

• Do bug counts measure testers? • Do bug counts measure thoroughness of testing?• Do bug counts measure the effectiveness of an

automation effort?• Do bug counts measure how near we are to being

ready to ship the product?

How would we know?


Bug counts and testers

To evaluate an instrument that is supposed to measure an attribute, we have to ask two key questions:• What underlying mechanism, or fundamental relationship,

justifies the use of the reading we take from this instrument asa measure of the attribute? If the attribute increases by 20%, what will happen to the reading?

• What can we know from the instrument reading? How tightly is the reading traceable to the underlying attribute? If the reading increases by 20%, does this mean that the attribute has increased 20%. If the linkage is not tight, we risk serious side effects as people push the reading (the “measurement”) up and down without improving the underlying attribute.


Bug counts and testers: mechanism?

Suppose we could improve testing by 20%.This might mean that:

• We find more subtle bugs that are important but that require more thorough investigation and analysis

• We create bug reports that are more thorough, better researched, more descriptive of the problem and therefore more likely to yield fixes.

• We do superb testing of a critical area that turns out to be relatively stable.

The bug counts might even go down, even though tester goodness has gone up.


Bug Counts & Testers: Side effects

What if you could increase the count of reported bugs by 20%? If you reward testers for higher bug counts, won’t you make changes like these more likely?

• Testers report easier-to-find, more superficial bugs• Testers report multiple instances of the same bug • Programmers dismiss design bugs as non-bugs that

testers put in the system to raise their bug counts• No one will work on the bug tracking system or other

group infrastructure.• Testers become less willing to spend time coaching

other testers.


Bug counts and extent of testing?

What Is This Curve?

Week

Bu

gs

Per

Wee

k


Bug counts and extent of testing

• If we change testing to maximize the bug count, does that mean we’ve achieved more of the testing?

Side Effect

• If we increase the extent of testing, does that result in more bug reports? Not necessarily.

Mechanism

• Bugs found. (Variations: bugs found this week, etc., various numbers based on bug count.)

Instrument

• Not sure. Maybe we’re thinking of percentage found of the total population of bugs in this product.

Attribute

Maybe in a trivial sense, but what if we’re finding lots of simple bugs at the expense of testing for a smaller number of harder-to-find serious

bugs?


Bug curves and extent of testing

• If we do something that makes the measured result look better, will that mean that we’ve actually increased the extent of testing? No, no, no. See side effect discussion.

Side Effect

• As we increase the extent of testing, will our bug numbers conform to the curve? Not necessarily. It depends on the bugs that are left in the product.

Mechanism

• Bugs per week. A key thing that we look at is the agreement between the predictive curve and the actual bug counts.

Instrument

• We have a model of the rate at which new bugs will be found over the life of the project.

Attribute


Side effects of bug curves

Earlier in testing: (Pressure is to increase bug counts)• Run tests of features known to be broken or incomplete.• Run multiple related tests to find multiple related bugs.• Look for easy bugs in high quantities rather than hard

bugs.• Less emphasis on infrastructure, automation architecture,

tools and more emphasis of bug finding. (Short term payoff but long term inefficiency.)


Side effects of bug curves

Later in testing: (Pressure is to decrease new bug rate)• Run lots of already-run regression tests• Don’t look as hard for new bugs.• Shift focus to appraisal, status reporting.• Classify unrelated bugs as duplicates• Class related bugs as duplicates (and closed), hiding key data

about the symptoms / causes of the problem.• Postpone bug reporting until after the measurement checkpoint

(milestone). (Some bugs are lost.)• Report bugs informally, keeping them out of the tracking system• Testers get sent to the movies before measurement checkpoints.• Programmers ignore bugs they find until testers report them.• Bugs are taken personally.• More bugs are rejected.


Bug curve counterproductive?

Shouldn't We Strive For This ?

Week

Bu

gs

Per

Wee

k


Code coverage

Coverage measures the amount of testing done of a certain type. Since testing is done to find bugs, coverage is a measure of your effort to detect a certain class of potential errors:

• 100% line coverage means that you tested for every bug that can be revealed by simple execution of a line of code.

• 100% branch coverage means you will find every error that can be revealed by testing each branch.

• 100% coverage should mean that you tested for every possible error. This is obviously impossible.


Benefits of coverage

Before I attack coverage measures, let me acknowledge that they are often useful.

• Many projects achieve low statement coverage, as little as 2% at one well-known publisher that had done (as measured by tester-hours) extensive testing and test automation. The results from checking statement coverage caused a complete rethinking of the company’s automation strategy.

• Coverage tools can point to unreachable code or to code that is active but persistently untested.

Coverage tools can provide powerful diagnostic information about the testing strategy, even if they are terrible measures of the extent of testing.


Analysis: Statement/Branch Coverage

• Not specifiedScope

• If we design our tests to make sure we hit more lines, does that mean we’ll have done more extensive testing? Maybe in a trivial sense, but we can achieve this with weaker tests that find fewer bugs.

Side Effect

• Not specifiedPurpose

• If we do more testing and find more bugs, does that mean that our line count will increase? Not necessarily. Example—configuration tests.

Mechanism

• Count statements and branches testedInstrument

• Extent of testing –

How much of the product have we tested?Attribute


Statement / branch coverage just test the flowchart

You’re not testing::» data flow» tables that determine control flow in table-driven code » side effects of interrupts, or interaction with background tasks» special values, such as boundary cases. These might or might

not be tested. » unexpected values (e.g. divide by zero)» user interface errors» timing-related bugs» compliance with contracts, regulations, or other requirements» configuration/compatibility failures» volume, load, hardware faults


If we use “coverage”?

• If we improve testing by 20%, does this result in a 20% increase in “coverage”? Does it necessarily result in ANY increase in “coverage”?

• If we increase “coverage” by 20%, does this mean that there was a 20% improvement in the testing?

• If we achieve 100% “coverage”, do we really think we’ve found all the bugs?


Side effects and “coverage”

• Without a mechanism that ties changes in the attribute being measured to changes in the reading we get from the instrument, we have a “measure”that is ripe for abuse.

• People will optimize what is tracked. If you track “coverage”, the coverage number will go up, but (as Marick has often pointed out) the quality of testing might go down.


Having framed the problem . . .

Do I have a valid, useful measure of the extent of testing?• Nope, not yet.• The development and validation of a field’s measures

takes time.In the meantime, what do we do?• We still have to manage projects, monitor progress, make

decisions based on what’s going on - - - so ignoring the measurement question is not an option.

• Let’s look at strategies of some seasoned test managers and testers.


One approach: Balanced scorecard

“Coverage” measures are popular because they provide management with essential (if incorrect) feedback about the progress of testing.

Rather than reporting a single not-very-representative measure of testing progress, consider adopting a “balanced scorecard” approach. Report:

• a small number (maybe 10) of different measures, • none of them perfect, • all of them different from each other, and • all of them reporting progress that is meaningful to you.

Together, these show a pattern that might more accurately reflect progress.


One Approach: Balanced Scorecard

• For 101 examples of possible coverage measures, that might be suitable for balancing, see “Software Negligence and Testing Coverage” at www.kaner.com. These are merged in a list with over 100 additional indicators of extent of testing in the paper, “Measurement Issues & Software Testing”, which is included in the proceedings.

• Robert Austin criticizes even this balanced scorecard approach. It will still lead to abuse, if the measures don’t balance each other out.


“Extent” as a Multidimensional Problem

We developed the 8 aspects (or dimensions) of “extent of testing” by looking at the types of measures of extent of testing that we were reporting.The accompanying paper reports a few hundred “measures” that fit in the 8 categories • product coverage - plan / agreement• effort - results• obstacles - risks• quality of testing - project history

ALL of them have problems


“Extent” as a Multidimensional Problem

So, look at testing progress reports actually used in the field. Do we see focus on these individualmeasures?• Often, NOT.• Instead, we see reports that show a pattern of information

of different types, to give the reader / listener a sense of the overall flow of the testing project.

• Several examples in the paper, here are two of them:» Hendrickson’s component map» Bach’s project dashboard


Project Report / Component Map (Hendrickson)

Component Test Type

Tester Total Tests Planned / Created

Tests Passed / Failed / Blocked

Time Budget

Time Spent

Projected for Next Build

Notes

Page 1 --- Issues that need management attentionPage 2 --- Component mapPage 3 --- Bug statistics

We see in this report:

- Progress against plan - Obstacles / Risks

- Effort - Results


Bach’s Dashboard

☺MedMedLowView

1621LLowMedBlockedInsert

1345, 1410KLowHighHighFile/edit

CommentsQualityCoverage

AchievedCoverage

PlannedEffortArea

Build

32Updated

11/1/00Testing Dashboard

We see coverage of areas, progress against plan, current effort, key results and risks, and obstacles.


Suggested Lessons

• Bug count metrics cover only a narrow slice of testing progress. There are LOTS of alternatives.

• Simple charts can carry a lot of useful information and lead you to a lot of useful questions.

• Report multidimensional patterns, rather than single measures or a few measures along the same line.

• Think carefully about the potential side effects of your measures.

• Listen critically to reports (case studies) of success with simple metrics. If you can’t see the data and don’t know how the data were actually collected, you might well be looking at results that were sanitized by working staff (as a side effect of the imposition of the measurement process).


Notes

______________________________________________________________________________________________________________________ __________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________



Part 20.

Defect Life Cycles


Why Track Defects?

• Get things that should be fixed, fixed

• Capture valuable information

• Identify responsibilities

• Provide data about quality

• Focal point for decision making


Tasks In Defect Tracking

• Quickly notify appropriate people

• Avoid forgetting about errors

• Assure important issues get resolved

• Minimize errors going unfixed due to

poor communication


What Are We Tracking?

• Symptoms of possible errors

• Severity of possible errors

• Priority for fixing errors

• Events related to the reported error

• No duplicate reports (for single error)

• Responsibilities

• Resolutions


Forces Acting on Defect Tracking

• Severity and Priority• Features and Quality• Duplicate reports• Project status reporting• Project management / Developer tasking• Metrics• Issue tracking and incident reporting• Irreproducible problems• Enhancement requests• Design errors• Deferring problems• Branches and release versions


Problem of Similar Bugs

You File aDefect Report

You Ignor It

It's aNew Defect

It Gets Fixed Never Fixed(Consumer Risk)

Same OldDefect

Waste of Time(Producer Risk)

Gets FixedAnyway


Defect Tracking Activities

•Submitting

•Assigning

•Fixing

•Verifying

•Resolving

•Administering


Example Defect Life Cycle

IssuePerceived Submitted

Assigned

Deferred Fixed

Closed

Not an issue,Duplicate,Will not fix,Cannot reproduce,Withdrawn

Fix Later

It’s Later

Not fixedFinished

Fix verified

Accepted

Entered


Defect Resolutions

• Common resolutions include:• Pending: the bug is still being worked on.• Fixed: the programmer says it’s fixed. Now you should check it.• Cannot reproduce: The programmer can’t make the failure

happen. You must add details, reset the resolution to Pending, and notify the programmer.

• Deferred: It’s a bug, but we’ll fix it later.• As Designed: The program works as it’s supposed to. • Need Info: The programmer needs more info from you. She has

probably asked a question in the comments.• Duplicate: This is just a repeat of another bug report (XREF it on

this report.) Duplicates should not close until the duplicated bug closes.

• Withdrawn: The tester who reported this bug is withdrawing the report.


Notes

______________________________________________________________________________________________________________________ __________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________


Project Planning

Part 21.

Testing Impossible Software Under

Impossible Deadlines

Cem Kaner, Software Testing Analysis East, 1999


Introduction

I didn’t come up with the idea for this talk--the STAR folk suggested it as something that I might be able to provide a few insights into. After all, this is a chronic complaint among testers. . . .

Well, here goes. How should you deal with a situation in which the software comes to you undocumented on a tight deadline?

I think that the answer depends on the causes of the situation that you’re in. Your best solution might be technical or political or resume-ical. It depends on how you and the company got into this mess. If it is a mess.


What Caused This Situation?

So, let’s start with some questions about causation:

• Why is the software undocumented?

• Why do you have so little time?

• What are the quality issues for this customer?


Why Do You Have So Little Time?

A Few Possibilities

• Time-to-release drives competition• Cash flow drives release date• Key fiscal milestone drives release date• Executive belief that you’ll never be satisfied, so your

schedule input is irrelevant• Executive belief that testing group is out of control and

needs to be controlled by tight budgets• Executive belief that you have the wrong priorities (e.g.

paperwork rather than bugs)• Executive belief that testing is irrelevant• Structural lack of concern about the end customer, or• Maybe you have an appropriate amount of time.


Quality Issues For This Customer?

• In-house, “In-house” outsourced• External, custom• External, large package, packaged• External, mass-market, packaged• What is quality in this market? What are their costs of

failure? » User groups, Software stores, Sales calls, Support calls

Over the long term, a good understanding of quality in the market, and as it affects your company’s costs, will buy you credibility and authority.


Political Approaches: Buying Time

Many of the ideas in this section will help you if you’re dealing with a single project that is out of control.

If your problem is structural (reflects policy or standard practice of the company), then some of these ideas will be counter-productive.


Buying Time: Delaying a Premature Release -2

Take the high road

• “I want to see knives in the chests, not in the backs.”» Trip Hawkins, founding President, Electronic Arts

• Communicate honestly.

• Avoid springing surprises on people.

• Never sabotage the project.

• Don’t become The AdversaryThe Adversary: If you are nasty and personal, you will become a tool, to be used by someone else. And you will be disposable.



• Search for show-stoppers. If you can, dedicate a specialist within your group to this task.

• Circulate deferred bug lists broadly.• Consider writing mid-project and late-in-

project assessments of:» extent of testing (by area)» extent of defects (by area, with predictions

about level of bugs left)» deferred bugs» likely out of box experience or reviewers’

experience



Do regular, effective status reporting. List:• your group’s issues (deliverables due to you, things

slowing or blocking your progress, etc., areas in which you are behind or ahead of plan)

• project issues

• bug statistics

Find allies (think in terms of quality-related costs)


Signing Off on the Release?

• Don’t sign off.• Don’t accept the authority to sign off.• Don’t pretend to be “QA” when you (obviously) are not.• Position yourself as a technical information provider.

» Publish pre-release reports that provide data on the stability of the product and the extent of completed testing

» Publish a report that lays out the likely issues that will be raised by magazine reviewers (or other third parities) when they see the product

» Let the people who want to ship prematurely own their responsibility. Your responsibility is to provide them with the information they need to make that decision.


Technical Approaches

• Find reference docs (you can get lots of data, even if you can’t get specs.) See the notes on this in the section on Spec-Driven testing

• Re-use materials across projects.• Plan for exploratory testing.• Do cost-effective automation.• Use tools to find bugs faster

» web page syntax checkers, spiders, etc.

• Facilitate inspections » (Get those bugs out before they reach you.)

• Cover the obvious legal issues.


Reusable Test Matrix

Numeric Input Field

Not

hing

LB o

f va

lue

UB

of

valu

eLB

of

valu

e -

1U

B o

f va

lue

+ 1

0 Neg

ativ

eLB

num

ber

of

digi

ts o

r ch

ars

UB

nu

mb

er o

f di

gits

or

char

sE

mpt

y fi

eld

(cle

ar

the

defa

ult

valu

e)M

axim

um p

ossi

ble

num

ber

of c

hars

Non

-dig

its


Group Management Approaches

Can you add head count?• Will they let you?• Can you absorb them?

» Do a work flow analysis to figure out what parts of each task could be split off and delegated to a newcomer.

Stay organized• Use functional outlines• Or use a detailed project plan• Create and publish explicit coverage reports

Get critical processes under control• Example: release management


Notes

______________________________________________________________________________________________________________________ __________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________



Part 22.

Career Planning for Testers



Tracks within testing:• Technical

» Automation programmer» Automation architect» Systems analyst» User interface / human factors analyst / critic» Test planner» Subject matter expert» Black box tester

• Management» Test lead, manager, director, internal consultant, external

consultant• Process management

» Metrics» Software process improvement specialist


Certification of Testers

Several organizations will now give you “certification” in software testing or software quality, including the American Society for Quality, the Quality Assurance Institute and the British Computer Society.

Should you be certified?

Training / studying for certification may or may not train you in skills useful to your work. However, a certificate is a clear statement that you care about your role in the profession.

In a tight market, such a certificate might improve your chancesof getting a job. I think that additional courses in programmingor work samples that show you have used test automation tools (download a demo version and write your own tests if necessary) are more likely to get you a good position.



Tracks outside of testing (for example):• Programming

• Program management

• Marketing

• Technical support

• Documentation

• Sales

• Field support

• Human factors analyst

• Systems analyst



Tracks outside of testing• People move back and forth between other fields

and testing. This doesn’t happen by accident. One way to prepare to move to another field is by picking tasks within testing that will educate you in tasks that are performed in that other field.

• Management of multiple areas (e.g. testing and something else) can lead to much more senior management positions. Multi-functional managers are often paid much better (along with promotion faster) than managers of single areas.


Career Planning for Testers:Bodies of Knowledge

Based on the American Society for Quality Control’s Body of Knowledge for the Certified Software Quality Engineer, Software Testing Laboratories developed three useful documents for describing the role of testers in a black box testing organization:

• The STL Body of Knowledge describes the knowledge that they expect their staff to possess.

• The STL Job Ladder is an overview of the knowledge and skill of different levels of testing staff. The “Lead Track” is the track of someone headed toward management. The “Engineer Track” is the career track of an increasingly senior technical contributor.

• The STL Body of Knowledge Map relates the knowledge areas in the Body of Knowledge to the seniority levels in the STL Job Ladder.


Career Planning for Testers:Bodies of Knowledge

The two bodies of knowledge are quite different. Both list the knowledge areas that they expect of a senior person in the field, but ST Labs’ was developed by and for their testers, whereas ASQ’s development was dominated by process improvement specialists from (or who consult to) large corporations, and includes relatively little on testing.

The ST Labs approach is useful in several ways:

• You can work with your staff to plan their advancement as testers. (Look for assignments that will grow their skills / knowledge indesired ways).

• You can work with candidates and hires who intend to stay in testing only for a fixed time (say, a year or 18 months). They’re coming to you with a plan to move to programming, tech support, marketing,writing, something. You can review with them what they can work on within testing that will help them get and then do well in that next job.


Notes

______________________________________________________________________________________________________________________ __________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________


Job Seeking in an Expanding Market

Part 22.

Cem Kaner, J.D., Ph.D., ASQ-CQE

Contact Information: [email protected]

www.kaner.com (testing website)www.badsoftware.com (legal website)

These slides were written in 1999 and early 2000 when the market was at its peak. Some of the advice is dated now that the market has changed, but much

of it is still current, you just have to look harder for the right position.


Factors to Consider

• What are your responsibilities to your current employer?• What will make you happy?• How much money?• What about stock and benefits?• What should your resume look like?• How much should you stretch the truth?• How do you build a reputation?• Where can you find out about jobs?• How should you manage recruiters?• How can you learn about a company?• How can you look good in an interview?• How do you prepare to negotiate?• How do you negotiate?• Dealing with race, gender, age, etc.• How do you set yourself up for success on the job?


Responsibilities to Your Current Employer

Loyalty?• Many employers are trying to appeal to the loyalty of their staff, to

keep them from leaving.• Pardon my cynicism, but many of the same people bragged about

controlling costs with layoffs when the market was tighter.• There are exceptions, such as Hewlett-Packard and IBM, that

undertake a strong sense of long-term responsibility to their staff. You might reasonably feel a mutual obligation to these companies.

• For the average company, if they want to appeal to your loyalty,maybe you should ask for some golden handcuffs.

More generally• Give reasonable notice, but don’t give more than your employer

would give you, unless that serves other interests of yours• Preserve secrets• If you signed a non-compete contract, check with your lawyer• In general, what goes around comes around


What Will Make You Happy?

Great job markets are temporary. Take advantage of it while you’ve got it. Now is the time to think about what will make you happy, and go for it.

• Money, stock, benefits, money• Opportunity for specific experiences / education• Opportunity for career growth• Opportunity to work with exceptional people (or to be a big

fish in a pond)• Social value of your work• Make room for your family, social life, education• Location• Specific allowances, support for your health• Other special circumstances


Money

Salary surveys• Many of these are lowball estimates• Few of these account for software testing as a separate area• There are several at websites like careerbuilder.com• Take geographic differences seriously.

Practice interviews• Interview with many companies• In several of the interviews, try out high numbers to test the top of

your market value. You’ll lose credibility in some of these interviews (and thus lose the job) but you’ll learn a lot for your next jobs.

Employers will typically whine about how much you are about to make, during a salary negotiation. Many will lay a trip on you even if you are asking for half of the rate they expected to pay.


Stock & Benefits

Stock• Pre-IPO often means No-IPO.• We hear about stock option millionaires, but most become

hundredaires or thousandaires.• Look for factors that make it credible that this company will

go public in X timeframe:» Profitability» Consistent meeting of projections, no surprises to market» Reasonably unique niche

Benefits• The usual medical, dental, education—what is important to

you?• Think carefully about overvaluing swimming pool, fitness

center, laundry room, dry cleaning pickup at your company. Also, if they offer all this at the office, where is your life?


Resume Tips

A resume INCLUDES and EXCLUDES. Your goal is to appeal to the right sub-market.

Functional vs. historical• You need buzz words but you are at risk in a functional

resume that is essentially a collection of buzz words. Interviewers need history. Screeners need buzz.

Create several resumes for different market segments

Length (1 page, 2 page, ???)

Emphasize what is special about you. Your hobby might be your strongest selling point.

Verbosity is dangerous, people will tune out.


Resume Tips

When should you send references with your cover letter?• You are stretching to a job or are seeking work in a

discriminated situation. In general, include refs if they make it substantially more likely that potential employers will interview or hire you.

Write your COVER LETTER to specifically respond to the ad. If it lists “requirements”then point out the ones that you meet and the ones on the list that you don’t meet but really want to learn.


Stretching the Truth?

Never stretch the truth

25% of American resumes contain false data. Don’t join this 25%

Think of your manager• Test managers are used to BS from others, and we

don’t like it.• Probably completely intolerant of the false details,

and probably has a well-developed BS-detector.


How Do You Build a Reputation?

• Talks at local meetings• Teach classes at local universities / colleges• Go to conferences• Publish• Create a web site

The more attractive you are in the marketplace, the higher your final salary will be.


Where to Find Jobs

• Newspapers

• Magazines

• On-line services

• Internet search engines

• Recruiters

• Colleagues

• Spam (would you work for a

company that spams you?)


Managing Recruiters

They are salespeople, typically on pure commission. The more easily they think they can sell you, the more they’ll work on your case.

Contact them regularly • Be consistent in the day / time that you contact them• Remind them of you• Ask them of openings that have recently crossed their desk that you

might be aware of.Restrict their circulation of your resume

• You must approve all sendings of your resume. Put this restriction in writing. Refuse to deal with anyone who won’t honor this.

• Don’t tell them about other opportunities (or agree to tell them) (this would have you giving another recruiter’s secrets to this recruiter).

Be aware of differences among recruiters• Executive recruiters vs. general purpose recruiters vs. outplacement

firm


Learn About the Company

During the search (or later)• Read the company’s web site, download demo software• Read the SEC filing• Deja News• Google, askjeeves.com, northern light• Stock sites that give investor info• Credit report (knowx.com)• Your peer network•

In the phone screen• Ask for product literature• Ask for demo copies of software• Ask how else to find info about the company•



Useful questions: I ask some of these of managers and some of them of working staff. Use your judgment about who you ask what:

• What kinds of products and services does your company provide?• Can I see a demonstration of the key product?• What is special about your products and services? What are the key

strengths and weaknesses?• How did you develop the main product? What were the key

development tradeoffs? (Time vs Features vs. Cost vs. Reliability)• Who are your customers?• Who are your competition?• How do you learn about your customers?• How do you learn about your customers’ satisfaction with the overall

product, with the design, and with the defects?• Show me an organization chart (and where you are on it and where I

would be on it)



• What is it like to work here?• What do you do? What kinds of products and services

do you provide?• Can I see some examples?• Where do you fit in the product development process?• What do you like about your job?• What would you like to change?• How do you make time for your family?• How much control do you have over your own work?• Who designs the tests that you run? Who runs the tests

that you design?• Tell me about your test design process.• Can I see some test plans and test cases?• How do you feel about your pay / boss / colleagues?



• What courses or conferences did you take last year?• What other training have you received?• How do you learn new things?• Describe three key things that you learned last year.

Keep your eyes open• Are the interviewers tired?• How is their furniture? How does it compare to the

company’s executives’ furniture? • How much space / equipment / light do standard testers

get, and how much do higher ranking staff get?• Look for congruencies and incongruencies of claims

regarding working conditions (e.g. 4-day work week)


Looking Good in an Interview

What makes you attractive to them?• Skills, knowledge, aptitude, other (KSAO)• Your independence and confidence• Your reputation• If you had to look for a job today, what would be your unique

selling proposition? Are you happy with it? What kind of position will it gain for you?

What makes you look bad?• Looking desperate• Arrogance

What convinces them that you are serious about them?• Background research• Saying that you want the job• Looking interested


Some characteristics of great testers

Effective with programmers

Effective with test managers

Effective with senior testers

Effective with junior testers

Editor (criticize / improve printed materials)

Diplomatic

Not very defensive (able to take criticism)

DecisiveDecision maker (good judgment)

Customer focused Curious (inquisitive) Credible

CreativeCourageousCopes with difficult circumstances

Commitment to qualityCommitment to a task (do what it takes)

Commitment (keep promises, stick around)

AuthorAuditorAssertive

Artistic (knowledgeably critique esthetic issues)

Arrogance (usually, less is better)

Architect

Analytical problem solverAttentive to detailAlert



ProgrammerPragmaticPolicy and procedure developer

Politically perceptivePersuasiveOrganizer and planner

Multi-taskingMentorMeeting manager

Long term thinkerLeadershipInvestigative reader

InterviewerInterpersonally perceptive

Integrity (honest; keeps commitments)

Humility Glue (promotes group cohesiveness)

Goal setting

FlexibleFinds bugs (intuitive tester)

Financially aware and sophisticated

Fast abstraction skillsEnergizingEmpowering

Empirical frame of reference

EmpatheticEffective with non-testing managers



CatalystZealot (Rarely desirable in large quantities.)

Written communication Warm (makes the human environment more pleasant)

Versatile (many abilities)

UI designTolerant of different development approaches

Tolerant of ambiguity

Team builder Substance abuser (not)Subject matter expert

Strength of characterSpoken communicationSense of humor

Scholarly (collects information, can back up or evaluate arguments)

PunctualProtective (stands behind his staff)


Looking Good in an Interview

What marketing materials are you bringing?• Resume, letters of reference, papers, printout of web

pages, other stuff that shows your vision• Work samples (beware of confidentiality)• Comments on their product

Practice interviewing• Interview lots of times• Interview with companies that are non-critical• Do mock interviews with friends

Send a follow-up letter• Astonishingly few people send these.• This is a marketing opportunity for you to spin the

meeting’s results.


Preparing to Negotiate

Fisher, R. & Ertel, D. (1995) Getting Ready to Negotiate: The Getting to Yes WorkbookThe key things are

• Knowing what you want• Knowing what they want• Knowing what they think of themselves• Knowing how they will present themselves as potential employers• Knowing what your alternatives are (your best alternative to a

negotiated agreement for this job)PRACTICE NEGOTIATING

• With friends• With potential employers. Plan to blow several away.

» Learn your market value» Learn what’s out there» Learn what seems to turn employers on and off.


How Do You Negotiate?

You are negotiating a long term relationship that you have to live with

If you don’t negotiate, you leave money on the table that cuts back on your lifetime earning expectations

If you don’t negotiate, you leave your work open instead of picking your assignment

If you don’t negotiate, you don’t learn how the company will be when you do need to negotiate.


How Do You Negotiate?• Say what you want (non-monetary issues) or what

excites you, early in the interview.• Refuse to provide previous salary info. Politely

reject queries about your salary expectations until after the employer is excited about you.

• Be enthusiastic but don’t be over-eager.• Speak about their product in their vocabulary.• Find and point out ways in which you can help them.

Make proposals, show examples of your thinking.• Let them know that you want the job.• Talk about your stretch opportunity. If you have to

stretch for this job, let them see how they could make you happy with the job (how it is a fit for you) and how you could do it.

• Keep your eye out for intimidating styles from the potential employer.


How Do You Negotiate?

Chapman, J. (1996, 3rd Ed.) Negotiating Your Salary: How to Make $1000 a Minute

Fisher, R., Ury, W., & Patton, B. (1991), Getting to Yes.Freund, J.C. (1992), Smart Negotiating: How to Make

Good Deals in the Real WorldO’Malley, M. (1998) Are You Paid What You’re Worth?Miller, L.J. (1998), Get More Money on Your Next Job:

25 Proven Strategies for Getting More Money, Better Benefits, & Greater Job Security.

Tarrant, J. (1997), Perks & Parachutes: Negotiating Your Best Possible Employment Deal, from Salary and Bonus to Benefits and Protection.


Dealing with Race, Gender, Age, etc.

H1 is creating a discriminatory environment.A.D.A gives you right to notice of testsI think I see a widening pay differential. Much of the program

that I think is the problem is the negotiating style of the individual rather than a policy of the company. Some behaviors that turn out to have a severe differential impact are engaged in by people who would not call themselves (or normally be called) racist or sexist.

If you are a member of a group that is often treated as disadvantaged / easy target, then you should definitely:

1. Read books on negotiation2. Take a negotiating class (seriously consider trying Karrass, not the

nicey-nicey win/win stuff) 3. Do practice interviews4. Join a group like Toastmasters5. Find ways to compare notes with white men in comparable

positions. Knowledge is power.


How Do You Set Yourself Up for Success?

• Remember that if they don’t treat you well in the interview, it won’t get better later.

• Don’t take a low rate to get in the door unless you are brand new and plan to leave once you get experience. They perceive you as more senior if they pay you more.

• If you have special needs, you must bargain for them clearly and precisely up front. The alternative—bring it up later—is often a loser.

• Get it in writing. Get any ambiguities cleared up in the writing. Trust me doesn’t work well in many of these ingredients.

• Be specifically aware of the possibility of becoming the expendable 40+ manager who is out of work for a year. (If you’re in this state, you have no fresh skills, no hot new experience, you look easily replaced, and therefore you find it intimidating to sell yourself.)


Notes

______________________________________________________________________________________________________________________ __________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________



Part 24.

Recruiting for Testers


Warning, warning: Legal issues

I am NOT well versed in employment law. Do NOT take this as legal advice.This talk raises a FEW legal issues, but only to get you thinking. The rules vary by state. The rules are followed differently by different companies. Talk to your HR department.It is generally unwise to ask questions about age, race, sexual orientation, political preference, national origin, pregnancy orparental or marital status, disability (except for questions that are provably directly tied to ability to perform the job), or anything else that a normal person would consider private. Talk to your HR department about what questions you are allowed to ask.Be very cautious about checking a candidate’s credit record or about insisting that she take a polygraph exam.Beware of accidentally making unintended promises.


Underlying principles

• Every hire should support the short term need and the long term mission of the group.

• Your team should be diverse.• With some creativity and tolerance, you can find

great people.• Hiring decisions should be made by consensus.• Use a behavioral strategy for gathering information• Tests should predict performance on the job.


What is the mission of your group?

Every position in the testing group should help your group realize its mission.• The nature of each position is determined partially in terms

of work that has to be done over some foreseeable term and partially in terms of the overall mission of the group.

• The skills, knowledge, and abilities required for each position are partially determined by the specific nature of the tasks to be accomplished over the short or intermediate term and partially by the need to have people who can help the group fulfill its mission.

» Example—the group needs a statistician but this will occupy only 10% of that person’s time. You will hire for some “other”position, but keep an eye out for statisticians.


What is the mission of your group?

Examples of different missions:• Assess and report the quality of the product? (So what’s

quality?)(What’s assessment? Probably need a person skilled in statistics and measurement.)

• Discover, report and advocate for the repair of product defects? (So what’s a defect?)

• Investigate the product’s conformance to a written specification.

• Carry out some other limited set of functions, such as looking only for coding errors (mismatch between the programmer’s intended and actual behavior of the program).


A Sample Job Description

The second level associate engineer has skill and experience in software testing or development, and a demonstrated ability to follow our internal SQA practices. We expect successful A2’s to create and execute a systematic test plan for a routine project, producing all relevant reports. We expect them to have knowledge of typical technologies sufficient to recognize quality problems in those areas.Minimum 1 year directly relevant experience in addition to required education and other professional experience.

• (Copied from materials published by ST Labs)


What position are you trying to fill?

The notion of “essential job functions”• Job descriptions are generic. The specific

position you have to fill now has other details.• Make a list of what you absolutely need, right now.• Make a list of what else you want of a person in

this position.• Think of a good candidate as someone who

meets all of the absolutely-needs and some portion (maybe a percentage) of the want-to-haves.

• This turns into a memo that you distribute to everyone interviewing for the position.


Who are you looking for?

Knowledge—the body of information that the candidate will need in order to perform effectively. For example, a person might know or have training in test case design, maybe even a course in creating test matrices.

Skills—involve proficiency at a specific task. For example, a person might be really good at creating test matrices.

Abilities—potential to do a job. For example, a person might be a very bright, systematic, analytical thinker, who you would expect to be able to quickly become very good at creating test matrices.

Other—other characteristics that are important to the job. For example, a person might have personal integrity.


Characteristics: The Need for Diversity

Diversity is essential. We need:• subject matter experts• programmers• testers, test planners• project managers, writers

Standard credentials are not the answer• An example: Staffing for a financial application


Exercise: Characteristics of Candidates

Refer to the slides in the Career Planning section that list a few characteristics of good testers.Take 15 minutes to:

• Imagine recruiting for a mid-level tester in your test group.

• Identify at least one skill, area of knowledge, or ability that is important to the position (or to your company as a whole), that is not listed on these slides.

• Circle (or write in) the top 10 characteristics that are most important for this position


Notes

______________________________________________________________________________________________________________________ __________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________


The Opportunity Hire

Some human examples of opportunity hiring• Training requirement: Entertainment company, low pay.

Recruited Director-level employee as a tester. Key promise to teach him about American business culture, goal of opening a test lab.

• Temporary assignments:» Marketeer (wanted training in product development)» Tech support (offered expertise in my product, supervisory

experience; wanted a stint in PD)

• Work force: Left computing (project/product mgr) to do construction project management. Came back to computing without the most current skills.


The Opportunity Hire

Opportunity hiring poses challenges. These challenges may or may not be insurmountable. Here are some hypothetical examples that have a loose relationship to real problems I’ve had to manage.

• Programmer who wants to move groups today• Programmer who seeks out “spare time” programming

assignments• Executive who is too used to deferential treatment for

the rough and tumble of testing• Family commitment led to time jealousy


Consensus-driven hiring

I follow three simple rules when hiring:• Anyone in the company who wants to be part of the

interview process for a candidate is welcome.• Example, talk with manufacturing mgr about doc mgr and then

invite to do the interview• Of the people who have interviewed the candidate, anyone

in the testing group and any senior player from any other group who will work with the candidate can veto the hiring of this person.

• The veto policy must be actively managed so that vetoes will not be based on race, religion, family situation, gender, sexual orientation, age, disability, national origin, etc.


Consensus-driven hiring

• Important (almost essential) for opportunity hiring• Group acceptance of special situation, at the start and

later• Additionally

• More likely to discover serious problems• More likely to reject based on those problems• Provides support network for incoming candidate

• Key assumption:It is a more serious mistake to hire badly

than to pass up a good candidate.


Behavioral information gathering

• The general principle:• The goal of the interview is to predict how the candidate will

behave if she joins your group.• You can achieve this by (for example)

• asking questioning that elicit behavioral responses,• examining work samples,• testing, • setting up situations that let you see how the candidate responds.

• Simple questioning often fails to elicit behavioral information.

• Much of the behavior that you will see is independent of the questioning. • Example: The late interview candidate.


Behavioral Interviewing

Questioning• Closed vs. open-ended questions• Factual vs. opinion questions• Hypothetical vs. behavioral questions

Work samples

• Ask (during phone screen) if he has any work samples that he can bring for review.

• The problem of confidential documents:» Don’t request confidential documents.» Before reviewing something that looks as if it might be

confidential, ask.» Don’t review documents that are (or that you think probably are)» Be diplomatic in your refusal to read them» Listen to the candidate’s responses—this is a predictor of how

he will treat your confidential documents in the future.


Behavioral Interviewing: Testing

Testing• Puzzles

» Big practice effects» Cultural fairness issues (may have a discriminatory

impact)» I’m not convinced that they predict success in testing» They are NOT IQ tests, though many people treat

them as if they were.

• Test cases• A testing / training exercise• Bug reports


The Myers Test Case

The program reads three integer values from a card. The three values are interpreted as representing the lengths of the sides of a triangle. The program prints a message that states whether the triangle is scalene, isosceles, or equilateral.

Write a set of test cases that would adequately test this program.


The Training Exercise

File1File2File3

File4 OK


Interview Technique

You’re the tester, here’s a CD. You have four hours to test it. What will you do?Question is, how do we gather information about the product, what is the though process. “How can you generate a test case without knowing what the requirements are?” Go into a feature map after you determine the market and the requirements.

Another example, use a print dialog that is broken for interviewing. For example, leave a blank button somewhere, see what they do with it.


Notes

______________________________________________________________________________________________________________________ __________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________



Part 25.

Learning Styles --- Teaching Testers


Learning Styles

People learn in different ways. There are several models of learning styles. Different presentation methods work best for different styles of learners.I’m not up to date on this literature. Back when I was a grad student in psychology, we talked about

• Visual vs. auditory learners. Visual learners have more success in lectures if they have a copy of every slide

• Active vs. passive learners. Some people learn better in lecture or by reading / memorizing. Others learn better through exercises.

• Individual vs. group learners. Some people learn better on their own, others on project teams.

• Concrete vs. conceptual learners. Some people learn better from detailed stories, others from the formal presentation of the underlying principles.


Learning Styles

I’m still thinking about how to develop a course that works well with the different styles. (I’m especially interested in this for building on-line courses that are worth teaching/taking).For now, my approach involves:

• Lecture of key material• Hard copies of every slide• Many stories with rich detail to support key points• Several assignments, which can be done by individuals or jointly• Sample questions that can be studied by individuals or jointly.

And may soon involve• Self-study drills• A library of examples that students can read on their own.


Learning Styles

There’s nothing magical about the division that I learned, and several others are more popular in the literature.For a useful introduction and discussion of several style classifications, see R.M. Felder, Matters of Style, http://www2.ncsu.edu/unity/lockers/users/f/felder/public/Papers/LS-Prism.htmFelder makes the point that the basic lessons for instructional design end up being pretty similar across the different classification schemes. The next slides, taken verbatim from his paper, show his advice to chemistry teachers:


Examples from Felder’s Matter of Style

“Here are some strategies to ensure that your courses present information that appeals to a range of learning styles. These suggestions are based on the Felder-Silverman model.

• Teach theoretical material by first presenting phenomena and problems that relate to the theory (sensing, inductive, global). For example, don't jump directly into free-body diagrams and force balances on the first day of a statics course. First describe problems associated with the design of buildings and bridges and artificial limbs, and perhaps give the students some of those problems and see how far they can go before they get all the tools for solving them.

• Balance conceptual information (intuitive) with concrete information (sensing). Intuitors favor conceptual information--theories, mathematical models, and material that emphasizes fundamental understanding. Sensors prefer concrete information such as descriptions of physical phenomena, results from real and simulated experiments, demonstrations, and problem-solving algorithms. For example, when covering concepts of vapor-liquid equilibria, explain Raoult'sand Henry's Law calculations and nonideal solution behavior, but also explain how these concepts relate to barometric pressure and the manufacture of carbonated beverages.



• Make extensive use of sketches, plots, schematics, vector diagrams, computer graphics, and physical demonstrations (visual) in addition to oral and written explanations and derivations (verbal) in lectures and readings. For example, show flow charts of the reaction and transport processes that occur in particle accelerators, test tubes, and biological cells before presenting the relevant theories, and sketch or demonstrate the experiments used to validate the theories.

• To illustrate an abstract concept or problem-solving algorithm, use at least one numerical example (sensing) to supplement the usual algebraic example (intuitive). For example, when presenting Euler's method for numerical integration, instead of simply giving the formulas for successive steps, use the algorithm to integrate a simple function like y = x2 and work out the first few steps on the chalkboard with a hand calculator.

• Use physical analogies and demonstrations to illustrate the magnitudes of calculated quantities (sensing, global). For example, tell your students to think of 100 microns is about the thickness of a sheet of paper and to think of a mole as a very large dozen molecules. Have them pick up a 100 ml. bottle of water and a 100 ml. bottle of mercury before talking about density.



• Occasionally give some experimental observations before presenting the general principle, and have the students (preferably working in groups) see how far they can get toward inferring the latter (inductive). For example, rather than giving the students Ohm's or Kirchoff's Law up front and asking them to solve for an unknown, give them experimental voltage/current/resistance data for several circuits and let them try to figure out the laws for themselves.

• Provide class time for students to think about the material being presented (reflective) and for active student participation (active). Occasionally pause during a lecture to allow time for thinking and formulating questions. Assign "one-minute papers" near the end of a lecture period, having students write on index cards the lecture's most important point and the single most pressing unanswered question. Assign brief group problem-solving exercises in class that require students to work in groups of three or four.

• Encourage or mandate cooperation on homework (every style category). Hundreds of research studies show that students who participate in cooperative learning experiences tend to earn better grades, display more enthusiasm for their chosen field, and improve their chances for graduation in that field relative to their counterparts in more traditional competitive class settings.



• Demonstrate the logical flow of individual course topics (sequential), but also point out connections between the current material and other relevant material in the same course, in other courses in the same discipline, in other disciplines, and in everyday experience (global). For example, before discussing cell metabolism chemistry in detail, describe energy release by glucose oxidation and relate it to energy release by nuclear fission, electron orbit decay, waterfalls, and combustion in fireplaces, power plant boilers, and automobiles. Discuss where the energy comes from and where it goes in each of these processes and how cell metabolism differs. Then consider the photosynthetic origins of the energy stored in C-H bonds and the conditions under which the earth's supply of usable energy might run out.”


Blooms Taxonomy Slides

“Bloom's Taxonomy, created in 1956 by Dr. Benjamin Bloom of the University of Chicago and his group of educational psychologists, is a categorization of verbs describing cognitive skills verbs into six classes (knowledge, comprehension, application, analysis, synthesis, and evaluation). The classes are ranked from least complex (knowledge) to most complex (evaluation) in terms of the level of thinking required for students to achieve these objectives. In general, critical thinking skills encompass only the three most complex categories (analysis, synthesis, and evaluation).”(From Michael Bowen’s web page, http://207.233.105.16/curriculum/bloomtax.htm, to be distributed in class.)


Testing Training

• It is easy to teach to the first few levels (tests would have students recall what was taught, define terms, make simple applications).

• It is much tougher to teach the underlying skills of testing.• Putting students through tests, exercises and exams has

been eye-opening in terms of how much they can learn with personal involvement and how little they get from traditional lectures.


Example of a Task Breakdown

Bug Reporting• Reproduce the bug• Simplify the bug• Follow-up testing• Describe the screen• Describe the sequence• Determine customer impact• Determine technical impact

What sub-tasks and skills are involved in each of these?


Closing Notes

Given a group of motivated, bright people who have appropriate background knowledge and belong to your testing group or your testing class:• Only some testers will learn well in lectures• Only some testers will learn well by being thrown into a

project and being told to figure it out• Only some testers will learn well from books• Only some testers will learn well or work well on their own

and some will only learn well or work well on their own

Build training strategies to achieve the learning you want to convey, accommodate multiple learning styles, and push for mastery, not memory.



Part 26.

Testing Resources on the Net


Testing Resources on the NetVarious Web Sites

DOUG HOFFMAN’S HOME PAGE www.SoftwareQualityMethods.comConsulting and training in strategy and tactics for software quality. Articles on software testing, quality engineering, and management. Look here for updated links.

CEM KANER’S HOME PAGE www.kaner.comArticles on software testing and testing-related laws

JAMES BACH www.satisfice.comSeveral interesting articles from one of the field’s most interesting people.

BRETT PETTICHORD www.io.com/~wazmo/qa.htmlSeveral interesting papers on test automation. Other good stuff too.

BRIAN LAWRENCE www.coyotevalley.comProject planning from Brian Lawrence & Bob Johnson.

BRIAN MARICK www.testing.comBrian Marick wrote an interesting series of papers for CenterLine. This particular one is a checklist before automating testing. The CenterLine site has a variety of other useful papers.

ELISABETH HENDRICKSON www.QualityTree.comConsulting and training in software quality and testing.

JOHANNA ROTHMAN www.jrothman.comConsulting in project management, risk management, and people management.

HUNG NGUYEN www.logigear.comTesting services, training, and defect tracking products.



LESSONS LEARNED IN SOFTWARE TESTING www.testinglessons.comThis is the home page for Cem’, James’, and Bret’s book, Lessons Learned in Software Testing.

BAD SOFTWARE HOME PAGE www.badsoftware.comThis is the home page for Cem’s book, Bad Software. Material on the law of software quality and software customer dissatisfaction.

SOFTWARE QUALITY ENGINEERING www.sqe.comSeveral interesting articles on current topics.

SOFTWARE QUALITY ENGINEERING www.stqe.com www.stickyminds.comArticles from STQE magazine, forum for software testing and quality engineering.

QA DUDE’S QUALITY INFO CENTER www.dcez.com/~qadude“Over 200 quality links” -- pointers to standards organizations, companies, etc. Plus artic les, sample test plans, etc.

QUALITY AUDITOR www.geocities.com/WallStreet/2233/qa-home.htmDocuments, links, listservs dealing with auditing of product quality.

THE OBJECT AGENCY www.toa.comEd Berard’s site. Object-oriented consulting and publications. Interesting material.

RBSC (BOB BINDER) www.rbsc.comA different approach to object-oriented development and testing.

DILBERT www.unitedmedia.com/comics/dilbert.Home of Ratbert, black box tester from Heck.



SSQA www.ventanatech.com/ssqaSilicon Valley Software Quality Association is a local professional software QA organization with monthly meetings, newsletter, more.

AMERICAN SOCIETY FOR QUALITY (ASQ) www.asq.orgNational/international professional QA organization.

SILICON VALLEY SECTION OF (ASQ) www.asq-silicon-valley.orgISO www.iso.ch

Describes ISO (International Organization for Standardization), with links to other standards organizers

AMERICAN NATIONAL STANDARDS INSTITUTE www.ansi.orgNSSN www.nssn.org

National Standards Systems Network. Find / order various standards. Lots of links to standards providers, developers and sellers.

IEEE Computer Society www.computer.orgBack issues of IEEE journals, other good stuff.

SOFTWARE ENGINEERING INSTITUTE www.sei.cmu.eduSEI at Carnegie Melon University. Creators of CMM and CMMI.



CENTER FOR SOFTWARE DEVELOPMENT www.center.orgNon-profit in San Jose with a big test lab and various other support facilities.

RELIABLE SOFTWARE TECHNOLOGIES www.rstcorp.comConsulting firm. Hot stuff on software reliability and testability. Big archive of downloadable papers. Lots of pointers to other software engineering sites.

SOFTWARE TESTING INSTITUTE www.ondaweb.com/stiMembership-funded institute that promotes professionalism in the industry. BIG list of pointers to resources in the industry (the Online STI Resource Guide).

SOFTWARE PRODUCTIVITY CENTRE www.spc.caMethodology, training and research center that supports software development in the Vancouver BC area.

CENTRE FOR SOFTWARE ENGINEERING www.cse.dcu.ie“Committed to raising the standards of quality and productivity within Ireland’s software development community.”

EUROPEAN SOFTWARE INSTITUTE www.esi.esIndustry organization founded by leading European companies to improve the competitiveness of the European software industry. Very interesting for the Euromethod contracted software lifecycle and documents.



QUALITY.ORG www.casti.comLinks to quality control source materials.

CSST (CLIENT-SERVER SOFTWARE TESTING) www.cse.dcu.ieD. J. Mosley’s home page. Lots of client / server publications.

FORMAL TECHNICAL REVIEW ARCHIVE www.ics.Hawaii.edu/~johnson/FTRDocuments, links, listservs dealing with auditing of product quality.

SOCIETY FOR TECHNICAL COMMUNICATION www.stc.orgLinks to research material on documentation process and quality.

BUGNET www.bugnet.comLists of bugs and (sometimes) fixes. Great source for data.TRY THIS SOMETIME -- With a product you own, look up bugs in BugNet that doesn’t list a workaround or a bugfix release, and replicate it on your computer. Then call the publisher’s tech support group and ask if they have an upgrade or a fix for this bug. Don’t tell them that you found it in BugNet. The question is, what is the probability that your publisher’s support staff will say, “Gosh, we’ve never heard of that problem before.”

JPL TEST ENGINEERING LAB tsunami.jpl.nasa.govJet Propulsion Lab’s Test Engineering Laboratory. Includes the comp.software.testing archives.Additionally, there are many sites for specialized work, such as sites for HTML compatibility tests of browsers. The usual search tools lead you to the key sites (the list changes weekly.)



SOFTWARE RESEARCH INC. www.soft.comAlso a major consulting and toolbuilding firm. Organizes the Quality Week conference. Publishes the TTN-Online newsletter. Excellent links.

QUALITY ASSURANCE INSTITUTE www.qai.comQuality control focused on the needs of Information Technology groups.

AETG WEBSITE http://aetgweb.argreenhouse.com/Home page for the AETG combinatorial testing product. Includes articles describing the theory of the product.

UNIFORM COMMERCIAL CODE ARTICLE 2B www.law.upenn.edu/bll/ulc/ulc.htmThese hold the drafts of the proposed Article 2B (UCITA), which will govern all sales of software. This will become the legal foundation of software quality. Currently, the foundation looks like it will be made of sand, Jell-O, and invisible ink.

SOFTWARE PUBLISHERS ASSOCIATION www.spa.orgSoftware & Information Industry Association is the main software publishers’ trade association.


Testing Resources on the Net:Some News Groups

comp.software.testing This covers issues of general interest to software testers. Plagued by too much spamming, this is not quite as interesting a spot as it used to be.

comp.human-factors User interface issues, usability testing, safety, man-machine reliability, design tools.

comp.software.internationalInternationalization and localization issues

comp.software.config-mgmtVarious configuration management issues.

comp.software-eng Discussions of all areas of software engineering, including design, architecture, testing, etc. The comp.software-eng FAQ is the one that lists sources of bug tracking systems, for example. (You’d think it would be in comp.software.testing, but comp.software-eng got there first.)

comp.software.measurement Not much there, but the goal is picking up metrics / measurements / data.


Testing Resources on the Net:Some News Groups

comp.answersThe home of many, many FAQs (documents that answer Frequently Asked Questions).

comp.jobs, comp.jobs.offered, ba.jobs, misc.jobs, etc.A good way to check out market values for testers (or to find your new home when you need one).

comp. risksDaily discussions of newsworthy bugs.

comp.dcom.modemsLots and lots and lots of discussion of modems.

misc.industry.qualityVarious discussions of quality control paradigms and experiences.

alt.comp.virusThis covers viruses, explaining things at end-user and more technical levels. The group often has very-up-to-date stuff. And like most alt-based groups, it seems to have a lot of spam mixed in. . .


Testing Resources on the NetMailing Lists, etc.

[email protected] [email protected] with"subscribe" in the body of the message to subscribe.

baldrige, qs9000, many otherscontact bill casti The Quality Czar <[email protected]>

DEMING-Lcontact [email protected]

testing technology newslettercontact [email protected], software research assocs


Notes

______________________________________________________________________________________________________________________ __________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

Black Box Software Testing

Documents

Transcript of Black Box Software Testing