ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-1
These slides are distributed under the Creative Commons License.
In brief summary, you may make and distribute copies of these slides so long as you give the original author credit and, if you alter, transform or build upon this work, you distribute the resulting work only under a license identical to this one.
For the rest of the details of the license, see http://creativecommons.org/licenses/by-sa/2.0/legalcode.
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-2
Planning, Documentation and Execution Outline
Introduction
Test Management and Techniques
ET Planning, Exec. and Documentation
ET Styles
ET Management
• How much planning is done with ET?
• How do you execute the tests with ET?
• What kind of documentation is produced with ET?
• How to use Pair Testing with ET?
• How do you use ET with eXtreme Programming (XP)?
1.
2.
3.
4.
5.
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-3
3. Exploratory Test Planning, Execution and Documentation.
3.1 Planning, Tasks and Documentation
3.2 Exploratory Testing in Pairs
3.3 Exploratory Testing with eXtreme Programming
1.
2.
3.
4.
5.
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-4
ET Planning“We plan as much as we can (based on the knowledge available), when we can (based on the time available), but not before.”
“When we apply Exploratory Planning to Testing, we create Exploratory Testing.”
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management
Lee Copeland, Exploratory Planning, 2001
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-5
1.
2.
3.
4.
5.
Exploratory Testing Tasks
ExploreExecute
TestsDesignTests
Product(coverage)
Techniques
Quality(evaluation)
Discover the elements
of the product.
Discover howthe product
should work.
Discover testdesign techniquesthat can be used.
Decide whichelements to test.
Observe product behavior.
Speculate aboutpossible quality
problems.
Evaluatebehavior against
expectations.
Configure & operate the
product.
Select & applytest design techniques.
Testingnotes
TestsProblems
FoundFrom Rapid Software Testing, copyright © 1996-2002 James Bach
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-6
1.
2.
3.
4.
5. PreparationDecompose the Product into elements
Areas / Groups of functions or features you expect to be able to test within a day or two
Identify the Quality Criteria
Ref. Satisfice Testing Model
Check your toolbox
Test techniques
ApplicationTo be tested
Decomposed Into elements
Element Element Element
Each element is expected to be tested within a day or two.
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-7
1.
2.
3.
4.
5.
ET Planning – Architecting Charters
Pick an element to testDecompose further – into smaller elements
Each new element should be tested within an hour or twoPrepare a charter for each element
Element
Charter
ElementTo be tested
Decomposed Into smaller
elements /Charters
Each Charter is expected to be tested within an hour or two.
Charter Charter
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-8
1.
2.
3.
4.
5. Doing Exploratory TestingKeep your mission clearly in mind.Keep notes that help you report what you did, why you did it, and support your assessment of product quality.Keep track of questions and issues raised in your exploration (logging and debriefing).To supercharge your testing, pair up with another tester and test the same thing on the same computer at the same time.
From Rapid Software Testing, copyright © 1996-2002 James Bach
Charter/Mission
Notes, Risks, Issues,Questions and Errors
Execution: Pairs & Sessions
DebriefingDebriefing
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-9
1.
2.
3.
4.
5.
Charter: A clear mission for the session
A charter may suggest what should be tested, how it should be tested, and what problems to look for.A charter is not meant to be a detailed plan.General charters may be necessary at first:
“Analyze the Insert Picture function”
Specific charters provide better focus, but take more effort to design:
“Test clip art insertion. Focus on stress and flow techniques, and make sure to insert into a variety of documents. We’re concerned about resource leaks or anything else that might degrade performance over time.”
From Rapid Software Testing, copyright © 1996-2002 James Bach
Charter/Mission
Notes, Risks, Issues,Questions and Errors
Execution: Pairs & Sessions
DebriefingDebriefing
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-10
1.
2.
3.
4.
5. Lateral Thinking
but periodically take stockof your status against your mission
Let yourself be distracted…
‘cause you never knowwhat you’ll find…
From Rapid Software Testing, copyright © 1996-2002 James Bach
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-11
1.
2.
3.
4.
5. Charter – summary…“Architecting the Charters” i.e. Test PlanningBrief information / guidelines on:
Mission: Why do we test this?What should be tested?How to test (approach)?What problems to look for?
Might include guidelines on:Tools to useSpecific Test Techniques or tactics to useWhat risks are involvedDocuments to examineDesired output from the testing
Charter/Mission
Notes, Risks, Issues,Questions and Errors
Execution: Pairs & Sessions
DebriefingDebriefing
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-12
1.
2.
3.
4.
5. Taking NotesTest Coverage Outline/MatrixEvaluation NotesRisk/Strategy ListTest Execution LogIssues, Questions & Anomalies
It would be easier to test if you changed/added…How does … work?Is this important to test? How should I test it?I saw something strange…
From Rapid Software Testing, copyright © 1996-2002 James Bach
Charter/Mission
Notes, Risks, Issues,Questions and Errors
Execution: Pairs & Sessions
DebriefingDebriefing
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-13
1.
2.
3.
4.
5. ET Execution - SessionsSession
Basic testing work unitUninterrupted
e-mail, meetings, telephone calls etc.
ReviewableA report should be produced
CharteredMission associated with this session; What are we testing? What are we looking for?
Charter/Mission
Notes, Risks, Issues,Questions and Errors
Execution: Pairs & Sessions
DebriefingDebriefing
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-14
1.
2.
3.
4.
5. Exercise 1a
Prepare a test charter for testing one selected area of StarOffice 5.2
Bullet and Numbering orTables
Work in pairs.
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-15
1.
2.
3.
4.
5. ET DocumentationPlanning:
Charters – overview of what to test (plan)Migh be a flip chart on the wall
Missions – What are we looking for?
ExecutionNotes – what happened during testing?
What did I do? Why did I do it? Used to assess product quality after test.
Data files – input data used for testingBug reports – enough details to recreate the test / bugTrack questions and IssuesDebriefing
Charter/Mission
Notes, Risks, Issues,Questions and Errors
Execution: Pairs & Sessions
DebriefingDebriefing
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-16
1.
2.
3.
4.
5.
Key Ideas of Exploratory Testing
The results from the tests you design and execute influence the next test you will choose to design and execute.You build a mental model of the product while you test it. This model includes what the product is and how it behaves, and how it’s supposed to behave.You test what you know about, and you are alert for clues about behaviors and aspects of the product that you don’t yet know about.
From Rapid Software Testing, copyright © 1996-2002 James Bach
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-17
1.
2.
3.
4.
5.
Challenges ofHigh Accountability Exploratory
Testing
Architecting the system of charters (test planning)Making time for debriefingsGetting the metrics rightCreating good test notesKeeping the technique from dominating the testingMaintaining commitment to the approachFor example session sheets and metrics see
http://www.satisfice.com/sbtm
From Rapid Software Testing, copyright © 1996-2002 James Bach
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-18
1.
2.
3.
4.
5. Problems to be Wary ofHabituation may cause you to miss problems.Lack of information may impair explorationExpensive or difficult product set-up may increase the cost of exploringExploratory feedback loop may be too slowOld problems may pop up again and againHigh MTBF may not be achievable without well defined test cases and procedures, in addition to exploratory approach.
From Rapid Software Testing, copyright © 1996-2002 James Bach
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-19
1.
2.
3.
4.
5. Exercise 1b
Prepare a test charter for testing one selected area of MiniTest
Work in pairs.
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-20
3. Exploratory Test Planning, Execution and Documentation.
3.1 Planning, Tasks and Documentation
3.2 Exploratory Testing in Pairs
3.3 Exploratory Testing with eXtreme Programming
1.
2.
3.
4.
5.
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-21
ET and Pair Testing
Observations made at Satisfice, several independent companies and Florida Tec – spectacular results:
Testing workgroups:High productivityHigh creativity
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management
Copyright © 1996 – 2002 Cem Kaner and James Bach (2001c)
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-22
What is Pair Testing?Two testers and (typically) one machineTypically (as in XP):
Pairs work together voluntarily. One person might pair with several others during a dayA given testing task is the responsibility of one person, who recruits one or more partners (one at a time) to help out.
Some stable pairs have worked together for yearsOne tester strokes the keys (keyboard might pass back and forth in a session) while the other suggests ideas or tests, pay attention and takes notes, listen, asks questions, grabs reference material etc.
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management
Copyright © 1996 – 2002 Cem Kaner and James Bach (2001c)
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-23
A Paired Testing SessionStart with a charter
Testers might operate from a detailed project outline, pick a task that will take a day or lessMight (instead or also) create a flipchart page that outlines this session’s work or the work for the next few sessions
An exploratory testing session lasts about 60-90 minutes
The charter for a session might include what to test, what tools to use, what testing tactics to use, what risks are involved, what bugs to look for, what documents to examine, what outputs are desired etc.
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management
Copyright © 1996 – 2002 Cem Kaner and James Bach (2001c)
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-24
Benefits of Pair TestingGenerate more ideas.More fun.Helps the tester stay on task.
Also, the fact that two are working together limits the willingness of others to interrupt them…
Better Bug Reporting.Great Training.Additional technical benefit.
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management
Copyright © 1996 – 2002 Cem Kaner and James Bach (2001c)
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-25
Risks and SuggestionsSenior / junior pairs: The pairs are partners; the junior tester is often the one at the keyboard, and he / she is always allowed to try out his / her own ideas.Accountability must always belong to one person.Some people are introverts. They need time to work alone and recharge themselves for group interaction.Some people have strong opinions and don’t work well with others. Coaching may be essential.Learn from XP experiences, Laurie Williams of NCSU (collaboration.csc.ncsu.edu/laurie/)
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management
Copyright © 1996 – 2002 Cem Kaner and James Bach (2001c)
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-26
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management
Pair Programming vs. Testing
Pair Testing is developed independently of paired programming, but many of the same problems and benefits apply.The eXtreme Programming community has a great deal of experience with paired work and offers many lessons:
Kent Beck, Extreme Programming ExplainedRon Jeffries, Ann Anderson & Chet Hendrickson, Extreme Programming Installed
Laurie Williams of NCSU does research in pair programming. For her publications, see http://collaboration.csc.ncsu.edu/laurie/
Copyright © 1996 – 2002 Cem Kaner and James Bach (2001c)
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-27
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management Pair Programming – Problems?
Paper by Gerold Keefer, AVOCA GmbH:“So far there is no evidence about the usefulness of this practice [pair programming] in general,….In addition there are several reports that indicate that pairing leads to faster exhaustion… On the other hand no one will deny that pairing in certain situations, such as educational settings or debugging sessions, is good common sense.”Alternative to pair programming:
Mutual Programming: A pair of developers does mutual QA on each others work products.Mutual Testing?
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-28
1.
2.
3.
4.
5. Exercise 1c
Prepare a test charter for testing the WEB-page: www.amland.no/et_test
Work in pairs.
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-29
3. Exploratory Test Planning, Execution and Documentation.
3.1 Planning, Tasks and Documentation
3.2 Exploratory Testing in Pairs
3.3 Exploratory Testing with eXtreme Programming
1.
2.
3.
4.
5.
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-30
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management ET and XP analogy…
Exploratory Testingis to Scripted Testing
whatXP is to
traditional “requirements and specifications” basedsystem development methods
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-31
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management
What is eXtreme Programming? (1)
Agile methodologyWorking in small development groupsTwo roles:
CustomerDeveloper
Everybody participate in design, programming and testIterative development
Iterations not more than 2 – 3 weeksCreate executable code every night
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-32
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management
What is eXtreme Programming? (2)
Key areas:Communication
in the team, I.e. with the customerSimplicity
Keep everything simple, should be possible to implement within hours (days)Reduce risk
FeedbackFrom customers and othersContinuous system improvement
Accept ChangesLow threshold to changesContinuously specify, design, code and test
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-33
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management
What is eXtreme Programming? (3)
TestingAutomated testingCreate test before coding
Test Driven Design
The test will document requirementsDaily Build and Test
Unit Test, Integration Test and System Test
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-34
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management
What is eXtreme Programming? (4)
Note, please, that in the XP context,
only the Customer Test can /validate/
that the program does the right thing.
The Programmer Tests merely /verify/
that the program does what the programmers
intended.
Ron Jeffries, (e-mail on agile-testing list, April 29, 2002)
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-35
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management
XP and Exploratory Testing (1)
“You develop the application test-first, where automated customer tests specify what the system is to do.
At some points, you invite James Bach in to do
exploratory testing. Perhaps at the ends of iterations.
If you’re so inclined, you say that James is part of the Customer Role.”
Brian Marick (e-mail on agile-testing list, March 7, 2002)
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-36
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management
XP and Exploratory Testing (2)
“I’ve found that exploratory testing helps me to think of
additional acceptance tests to propose to the customer.
Lots of times you don’t think of everything about a system until
you use it.”
Lisa Crispin (e-mail on agile-testing list, March 8, 2002, writing book on XP and
Testing)
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-37
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management XP and ET – Problems?Paper by Gerold Keefer, AVOCA GmbH:
Extreme Programming Considered Harmful for Reliable Software Development (2002):
Discussing pros and cons of XP (including Pair Programming)
“The most recent study … suggests that … pair programming is rather expensive technology.. and ...XP-like pair programming appears less efficient than it is reported” (with references)
Reviewing XP’s most well known reference project, C3 at DaimlerChrysle Company
Launched in 1995, Kent Beck (XP) took over 1996Supposed to be used by 87.000 employees, never used by more then 10.000The project was cancelled in 2000 after 4 years in XP mode!
How should this effect Exploratory Testing?
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-38
Introduction
Test Management and Techniques
ET Planning and Documentation
ET Styles
ET Management ET – The VanguardThe object of your mission is to explore
the Missouri river, & such principal streams of it, as, by its course and
communication with the waters of the Pacific ocean...may offer the most direct
& practicable water communication across this continent for the purposes of
commerce.
- Thomas Jefferson's letter to Meriwether Lewis, June 1803Robinson, H., Microsoft, Exploratory Modeling
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-39
1.
2.
3.
4.
5.
Planning, Documentation and Execution Summary
Introduction
Test Management and Techniques
ET Planning, Exec. and Documentation
ET Styles
ET Management
• How much planning is done with ET?
• How do you execute the tests with ET?
• What kind of documentation is produced with ET?
• How to use Pair Testing with ET?
ET Workshop v. 1.20 - How to?
©2002 Amland Consulting 3-40
1.
2.
3.
4.
5. Exercise 2
Based on previously produced charter and available information about StarOffice:
Select one (or more) testing technique(s) and test the application. Explain your mission / reasoningExplain technique(s) used and errors found.
Top Related