More and Better Test Ideas

Rikard Edgren

TIBCO Spotfire

EuroSTAR 2009-12-03

share one-liner test ideas

Acknowledgements

• This approach was “invented” and refined at my workplace, and at many other places as well

• A lot of good talking around test ideas:Kaner, Bach, Bolton

• Many tips & tricks at many places

• See paper for details and references

Agenda

• Test idea definition

• Two examples

• More test ideas

• Better test ideas

• Summary

Test Idea Definitions

• "a brief statement of something that should be tested.“ (Brian Marick)

• “test condition: An item or event of a component or system that could be verified by one or more test cases, e.g. a function, transaction, feature, quality attribute, or structural element.”(ISTQB 1.1)

• Test-Ideas List: “An enumerated list of ideas, often partially formed, that identify potentially useful tests to conduct” (Rational Unified Process)

More Test Idea Definitions

• ”A test idea is the thought that guides our creation of a test” (Cem Kaner)

• “Test Idea: an idea for testing something” (James Bach)

• maybe no definition is needed, you immediately understand what a test idea is.

Example #1

• Spare time project from Canada• A tool for exploratory testers• Test specification from summer 2009• Used for Beta release 0.2

Example #1

• A combination of generic and detailed test ideas

• 53 test ideas cover many important aspects• Possible to see what is missing

• It is a good idea to combine several test ideas, and execute them together

• Efficient to make the list while trying the software

Scripted & Exploratory Testing

• As approaches to software testing:– ST focus on control, precision– ET focus on learning and the freedom of the tester

• As methods/activities/techniques:– ST designs and reviews tests in advance– ET design and execute simultaneously

• Test ideas enable early feedback, add visibility, save time, and can act as a check list for ET

Example #2

• Product development in Gothenburg, Sweden• Test specification from 2008• Used for a 1.0 release• Good starting point for next project

Example #2

• Test Ideas range from vague to very vague

• Test Ideas can be converted to detailed test scripts, together with necessary details

• Test Ideas can be executed as is, trusting the tester to design the tests and choose thoroughness

More Test Ideas

• Requirements aren’t enough

• For each part and its interactions– What should be right?– What can go wrong?– What quality attributes matter?– What is important?

• Be creative, and work hard– for each test idea, there are two more

Test Idea Triggers

• So many inputs, methods, heuristics...– requirements/specifications/prototypes/code– bugs/error catalogs/support/customers– technology/tools/systems/interactions/models– quality objectives/attributes– testing techniques– test idea lists– people

• your subjectivity, knowledge, experience...

My Favorites #1

• Quality attributes– Capability– Reliability– Usability– Security– Scalability– Performance– Installability– Compatibility

– Supportability– Testability– Maintainability– Portability– Localizability

– Accessibility– Interoperability

My Favorites #2

• Delete/Remove/Empty/Null

• Read the manual

• Look at all deliverables

• Use the same data/document a very long ”time”

• ”Real” usage of the product

Better Test Ideas

• Powerful• Yield significant results• Credible• Likely• Easy to evaluate

• Useful for troubleshooting• Informative• Appropriately complex• Giving insightful information• Easy to understand*• Fast to execute*

* not in Kaner’s What is a good test case

Better Test Ideas

• Test ideas get better through collaboration• ...with people that care, and understand what’s

important

• Try to get feedback from many sources• Discard test ideas that seem weaker

• You will get even better test ideas as you learn more about the product

Test Ideas in Practice

• Many different types• One-liners are fast to read• Easy to update• Vague to leave freedom• Useful now-then, me-otherpeople

• Adjust granularity for better readability

Find Five Faults

...but there are other things than these faults that are more important...

Limitations

• Test executors might need a lot of testing and product knowledge

• Might be difficult to evaluate results anyway

• Can be difficult to get good feedback

• Generic test ideas might obscure the new context

Summary

• Many different test ideas is a good thing• More people will give better perspectives

• Test ideas are fast to read, write and update

• Test ideas could be the glue you need for using test scripts and an exploratory testing approach

• Documented test ideas is a simple idea, you need to use it to understand the value

Questions

• ???

redgren@tibco.comwww.thetesteye.com