Reviews and Inspections
-
Upload
magee-valdez -
Category
Documents
-
view
37 -
download
0
description
Transcript of Reviews and Inspections
![Page 1: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/1.jpg)
Reviews and Inspections
Prepared by
Stephen M. Thebaut, Ph.D.
University of Florida
Software Testing and Verification
Lecture 13
![Page 2: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/2.jpg)
Required Readings
• D. Gause and G. Weinberg, “Making Meetings Work for Everybody,” Exploring Requirements: Quality Before Design, Dorset House, 1989.
• M. Fagan, “Design and Code Inspections to Reduce Errors in Program Development,” IBM Systems Journal, Vol. 15, No. 3, July 1976.
(cont’d)
![Page 3: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/3.jpg)
Required Readings (cont’d)
• B. Grady and T. Van Slack, “Key Lessons in Achieving Widespread Inspection Use,” IEEE Software, July, 1994.
• Sauer, et al., “The Effectiveness of Software Development Technical Reviews: A Behaviorally Motivated Program of Research,” IEEE TSE, Vol. 26, No. 1, 2000.
![Page 4: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/4.jpg)
About Meetings
• “Reviews and Inspections” are a form of human-based testing that involves people working together cooperatively.
• Thus, we begin with a few basic axioms (ala Gause and Weinberg) regarding meetings…
![Page 5: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/5.jpg)
D. Gause and G. Weinberg, “Making Meetings Work for Everybody,” Exploring Requirements: Quality Before Design, Dorset House, 1989.
![Page 6: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/6.jpg)
Meetings can be terrible...
• Ever been to a really BAD meeting?
• Meetings as measurements: If your meetings are terrible, then your entire process is probably sick.
• In order for meetings to be effective, they need to be made safe…
![Page 7: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/7.jpg)
Safe to attend...
• Establish a no-interruption policy, but also set time limits for individual speakers so that everyone will be able to participate.
• Outlaw personal attacks and put-downs.
• Finish on time, but schedule a continuation of the meeting if business isn’t finished.
• Use a related issues list and ensure follow-up for important off-topic matters that come up.
![Page 8: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/8.jpg)
And safe NOT to attend...
• To remove uncertainty about what will be covered, publish an agenda and stick to it.
• Handle “emergency issues” in a way that will not hurt people who were not invited.
• Be sure people who should attend are identified and explicitly invited in advance.
• Gently confront those present who should not attend immediately but unobtrusively – preferably before the meeting starts.
![Page 9: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/9.jpg)
Other Meeting Axioms
• Meetings should be as small as possible, but no smaller.
• Keep the agenda short. (A meeting that tries to do too many things does none well.)
• Design meetings to have the appropriate structure and pace.
• Identify someone to act as a facilitator.
• Be prepared! (95% of meetings that fail do so because of inadequate preparation.)
![Page 10: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/10.jpg)
M. Fagan, “Design and Code Inspections to Reduce Errors in Program Development,” IBM Systems Journal, Vol. 15, No. 3, July 1976.
![Page 11: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/11.jpg)
What are Reviews & Inspections?
• Involve people examining a work-product / document (requirements, user manuals, design, test plan, test cases, source code, etc.) with the aim of discovering anomalies and defects.
• More formal than walkthroughs or desk-checking
• Can be more effective than (machine-based) testing after system implemen-tation. (As demonstrated in many studies.)
![Page 12: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/12.jpg)
Quotes and Notes from Fagan’s Paper
• “An ingredient (of inspections) that gives max-imum play to the planning, measurement, and control elements (of software development) is consistent and vigorous discipline.”
• “...finding errors by inspection and reworking them earlier in the process reduces the overall rework time and increases productivity…”
(cont’d)
![Page 13: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/13.jpg)
Quotes and Notes from Fagan’s Paper (cont’d)
• “…we first describe…places at which inspections are important. Then we discuss factors that affect productivity and the operations involved with inspections. Finally, we compare inspections and walk-throughs…”
![Page 14: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/14.jpg)
Proposed Inspection Types and Levels
• Product
– I0: component level design
– I1: unit level design (“design complete”)
– I2: code inspection
• Test
– IT1: test plans
– IT2: test cases
• Publications
– PI0 - PI2
![Page 15: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/15.jpg)
Update on Inspection Types and Levels
• In addition to those originally proposed by Fagan, IBM now undertakes the following inspections:
– DR1 - DR3 inspections on requirements, product-level design, and system-level design, and
– IT1/2 inspections expanded to unit, component, product, and system testing levels.
![Page 16: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/16.jpg)
Code Inspections – the People Involved
1. Moderator:
– the key person; the coach
– technically competent, but preferably someone working on a different project
– Trained and experienced in facilitation
2. Designer
3. Coder/Implementer (author/owner)
4. Tester
![Page 17: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/17.jpg)
I1 (“Design Complete”) Inspection Process
1. Overview (whole team)
2. Preparation (individual) using…
– ranked distributions of error types
– checklists of clues on finding errors
(cont’d)
![Page 18: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/18.jpg)
I1 (“Design Complete”) Inspection Process (cont’d)
3. Inspection (whole team)
– a “reader” is chosen by the moderator†
– every element of logic and every branch is considered
– objective is to find errors; no specific solution hunting is permitted
– moderator prepares written report within one day
† Which team member should NOT assume this role?
(cont’d)
![Page 19: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/19.jpg)
I1 (“Design Complete”) Inspection Process (cont’d)
4. Rework (owner)
5. Follow-up (moderator)
– if > 5% of material has been reworked, the entire element is re-inspected
![Page 20: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/20.jpg)
Other Considerations/Observations
• “...people have to be taught or prompted to find errors effectively.”
• (inspection results) “...should not under any circumstances be used for programmer performance appraisal.”
• (acceptance of inspections) “...is related to the success of the inspections (people) have experienced, the conduct of the trained moderator, and the attitude demonstrated by management.”
(cont’d)
![Page 21: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/21.jpg)
Other Considerations/Observations (cont’d)• “The most marked effect of inspections on the
development process is to change the old adage that design is not complete until testing is completed…”
• “...in one case...approximately two thirds of all errors reported during development (were) found by...inspections...”
![Page 22: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/22.jpg)
Inspections versus Walkthroughs: “Comparison of Key Properties”
Properties Inspection Walk-through
Formal moderator training Yes No
Definite participant roles Yes No
Who “drives” the process Moderator Owner
Use checklists? Yes No
Formal follow-up Yes No
Analysis process problems improvements
Yes No
![Page 23: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/23.jpg)
Inspections versus Walkthroughs: “Comparison of Key Properties”
Properties Inspection Walk-through
Formal moderator training Yes No
Definite participant roles Yes No
Who “drives” the process Moderator Owner
Use checklists? Yes No
Formal follow-up Yes No
Analysis process problems improvements
Yes No
![Page 24: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/24.jpg)
Inspections versus Walkthroughs: “Comparison of Key Properties”
Properties Inspection Walk-through
Formal moderator training Yes No
Definite participant roles Yes No
Who “drives” the process Moderator Owner
Use checklists? Yes No
Formal follow-up Yes No
Analysis process problems improvements
Yes No
![Page 25: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/25.jpg)
Inspections versus Walkthroughs: “Comparison of Key Properties”
Properties Inspection Walk-through
Formal moderator training Yes No
Definite participant roles Yes No
Who “drives” the process Moderator Owner
Use checklists? Yes No
Formal follow-up Yes No
Analysis process problems improvements
Yes No
![Page 26: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/26.jpg)
Inspections versus Walkthroughs: “Comparison of Key Properties”
Properties Inspection Walk-through
Formal moderator training Yes No
Definite participant roles Yes No
Who “drives” the process Moderator Owner
Use checklists? Yes No
Formal follow-up Yes No
Analysis process problems improvements
Yes No
![Page 27: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/27.jpg)
Inspections versus Walkthroughs: “Comparison of Key Properties”
Properties Inspection Walk-through
Formal moderator training Yes No
Definite participant roles Yes No
Who “drives” the process Moderator Owner
Use checklists? Yes No
Formal follow-up Yes No
Analysis process problems improvements
Yes No
![Page 28: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/28.jpg)
Inspections versus Walkthroughs: “Comparison of Key Properties”
Properties Inspection Walk-through
Formal moderator training Yes No
Definite participant roles Yes No
Who “drives” the process Moderator Owner
Use checklists? Yes No
Formal follow-up Yes No
Analysis process problems improvements
Yes No
![Page 29: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/29.jpg)
Inspecting Modified Code
• “Since most modifications are small...they are often erroneously regarded as trivially simple and handled accordingly; ...In the author’s opinion, all modifications are well worth inspecting...”
• “Human tendency is to consider the ‘fix,’ or correction, to a problem to be error-free itself. ...The number of bad fixes can be...reduced by some simple inspection after clean compilation of the fix.”
![Page 30: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/30.jpg)
B. Grady and T. Van Slack, “Key Lessons in Achieving Widespread Inspection Use,” IEEE Software, July, 1994.
![Page 31: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/31.jpg)
Notes from the Grady and Van Slack Paper
• On the value of Inspections:
– ROI: some can achieve 100:1; based on HP data, you can expect 10:1.
– “We estimate that roughly 1/3 of all HP software costs are rework, and inspections can save 60% of these costs.”
– Lessons at HP suggest inspections are appropriate for ALL software development organizations.
![Page 32: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/32.jpg)
Influences on Inspections at HP
• History of hardware reviews (first software reviews were walk-throughs)
• Fagan’s “very influential” paper which introduced the term “software inspection.”
• Tom Gilb’s 1988 SE Manamement book
– focus on EARLY life-cycle artifacts– measures of the inspection process itself– use of these measures for process
improvement
![Page 33: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/33.jpg)
HP’s Inspection Process
• Focus on software “documents” as opposed to code
• Goal: to ensure that documents are clear, self-explanatory, correct, and consistent with all related documents
• Five roles: inspectors, scribe, owner, moderator, chief moderator (process owner and champion)
![Page 34: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/34.jpg)
HP’s Inspection Process (cont’d)
• Seven steps:
1. planning (moderator and owner plan inspection and create packet)
2. kickoff (brief team meeting)
3. preparation (individuals find issues)
4. logging meeting (team meets to find and log issues)
(cont’d)
![Page 35: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/35.jpg)
HP’s Inspection Process (cont’d)
• Seven steps:
1. planning (moderator and owner plan inspection and create packet)
2. kickoff (brief team meeting)
3. preparation (individuals find issues)*
4. logging meeting (team meets to find and log issues)*
* cf. Fagan’s description of these steps.
(cont’d)
![Page 36: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/36.jpg)
HP’s Inspection Process (cont’d)
• Seven steps: (cont’d)
5. cause/prevention (brainstorm causes and recommendations)
6. rework (verify and fix defects)
7. follow-up (moderator)
![Page 37: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/37.jpg)
HP’s Inspection Process (cont’d)
• Seven steps: (cont’d)
5. cause/prevention (brainstorm causes and recommendations)*
6. rework (verify and fix defects)
7. follow-up (moderator)
* “The process step that varies the most…” Why?
![Page 38: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/38.jpg)
Technology Transfer Lessons
• Importance of communicating successes
• Clearly defining who is responsible for process improvement
• A high-level, compelling vision (shape-up or else...)
• Readily available training (including MANAGEMENT training)
(cont’d)
![Page 39: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/39.jpg)
Technology Transfer Lessons (cont’d)
• Consulting to create an environment for success BEFORE training is begun
– adapt the objectives/process as needed
– find a chief moderator/champion
– benchmark the current process
– follow-up to ensure success
![Page 40: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/40.jpg)
Sauer, et al., “The Effectiveness of Software Development Technical Reviews: A Behaviorally Motivated Program of Research,” IEEE TSE, Vol. 26, No. 1, 2000.
![Page 41: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/41.jpg)
Questions Posed by Sauer, et al., Regarding Technical Reviews1. What makes reviews effective at defect
detection?
2. How can we improve review performance within the current review design?
3. What design alternatives would theory predict to be more effective?
Approach: applying behavioral theory of group performance
![Page 42: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/42.jpg)
Controversy Regarding the Inspection / Logging Meeting
“…20 years after Fagan’s seminal paper, debate continues over whether a key component of inspections, the group meeting, is necessary for defect detection because it is not clear whether, in the context of a process in which individual preparation focuses on defect detection, significant numbers of defects are discovered through reviewers interacting (synergy).”
![Page 43: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/43.jpg)
Results from the Behavioral Theory of Group Performance Suggest:• Increasing inspectors’ defect detection
expertise (e.g., through appropriate selection an/or training) should have the largest impact on performance.
• The interacting group meeting does not improve performance in discovering new defects.
(cont’d)
![Page 44: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/44.jpg)
Results from the Behavioral Theory of Group Performance Suggest: (cont’d)• The performance advantage of an interacting
group is a function of the level of false positives discovered by individuals.
• An expert pair performs the discrimination task (identifying false positives) as well as any larger group.
• Above a critical limit, performance declines with group size.
![Page 45: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/45.jpg)
Implications for Practice
• Under current review designs, performance may be improved by:
– selecting better reviewers,
– providing (better) training,
– fine-tuning the size of reviews, and
– providing aids to expertise (e.g., error checklists).
(cont’d)
![Page 46: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/46.jpg)
Implications for Practice (cont’d)
• Using alternative designs, performance may be improved by separating the tasks of defect collection and defect discrimination.
• Managing Reviews: past view that review performance should be collective, not individualistic, should change. Individuals’ review performance should be assessed in staff appraisals.
![Page 47: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/47.jpg)
Reviews and Inspections
Summary
![Page 48: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/48.jpg)
Reviews and Inspections Summary
• Disciplined, human-based testing (e.g., inspections) can be very effective.
• Human-based testing is applicable to code, design, requirements, testware, publications, etc.
• The exclusive objective of inspections should be defect detection.
(cont’d)
![Page 49: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/49.jpg)
Summary (cont’d)
• Inspection results are useful for process tracking and improvement.
• Performance may be improved by increasing expertise and fine-tuning group size.
• Separating the tasks of defect collection and defect discrimination may be a useful design alternative for reviews.
(cont’d)
![Page 50: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/50.jpg)
Summary (cont’d)
• Inspection results should never be used in programmer performance appraisals...
• Individuals’ review performance should be assessed in staff appraisals...
(Can you reconcile these two statements?)
![Page 51: Reviews and Inspections](https://reader036.fdocuments.in/reader036/viewer/2022062407/56812a68550346895d8ded23/html5/thumbnails/51.jpg)
Reviews and Inspections
Prepared by
Stephen M. Thebaut, Ph.D.
University of Florida
Software Testing and Verification
Lecture 13