Post on 31-Dec-2015
How to Know That What You Want Has Been Done- 1
Claire LohrMember, SESC Management Board
Chair IEEE 829 Working Grouplohrsys@erols.com
How to Know That What You Want Has Been Done
STC May 1, 2002
How to Know That What You Want Has Been Done- 2
Topics
• Sources of the information presented• Inclusion of best practices from industry• Definition of the steps for recognition of
successful completion of software projects• Applicable references
NOTE: Best practices marked with
How to Know That What You Want Has Been Done- 3
Sources of this Information
• Highly competitive commercial industries– Semiconductor manufacturing– PC manufacturing– Insurance– Finance– Telecommunications– Etc.
Some succeed
Some don’t
How to Know That What You Want Has Been Done- 4
The Challenge
Current formal definitions of quality are …
Too broad for software
Real need is to attain the customer’s own specific, detailed quality goals
How to Know That What You Want Has Been Done- 5
Steps for Recognizing Quality
• Step #1: define what you don’t want• Step #2: define what you do want with
sufficient detail• Step #3: hire and train good people• Step #4: prepare to measure the results (test
outputs) in parallel with defining the target (requirements)
• Step #5: refine with business analysts throughout lifecycle
How to Know That What You Want Has Been Done- 6
Step #1
Define what you don’t want (the business case)
“ IT buyers are mad and aren’t going to take it anymore … Up to 50% of software cost results from ongoing maintenance and integration activities … We spend so much time and money dealing with software problems, it's ridiculous … Some patches break more things than they fix.”
Karyl Scott Information Week January 14, 2002
How to Know That What You Want Has Been Done- 7
Step #2: Define What You Do Want
Define what you do want (general)– From the Scott article
• “The Software Bill of Rights, released this week [Jan. 14 2002], stipulates that buyers have the right to expect a quality product, accurate delivery schedules, explicit pricing, development accountability, and effective technical support.”
– Universally accepted definitions of quality• Meets specifications• Meets customer expectations• Fitness for use
How to Know That What You Want Has Been Done- 8
Define what you do want (specific)– Document requirements and SQA
• IEEE Std 830-1998 Recommended Practice for Software Requirements Specifications
• IEEE Std 730-1998 Standard for Software Quality Assurance Plans
– Document requirements with Use Cases for more clarity
• Visual
• Shows flows
• Includes everybody
– Plan testing in parallel from the beginning• IEEE Std 829-1998 Standard for Software Test Documentation
Step #2: Define What You Do Want
How to Know That What You Want Has Been Done- 9
Use Case Example
Step #2: Define What You Do Want
Attend IEEE Track
Use case
Actor SEPG
Member
Association Attend STC
Order SESC standards
Visit computer.org
How to Know That What You Want Has Been Done- 10
Step #3: Good People
• Hire and train good people (a good process enhances talent does not replace it)– Thorough employment screening– “Boot camp” training for entry level new hires– Employee friendly work environment ($ to $$$$)– Low turnover– Easy access to training
• Incorporate standards for process and documentation• Use real examples online and in class
How to Know That What You Want Has Been Done- 11
Step #4: Prepare Tests and Requirements
Prepare to measure the results (test outputs) in parallel with defining the target (requirements)– Helps developers clarify their thinking when
answering tester’s questions– Assures testability of the final software– Defines “pass criteria” for the software early in the life
cycle– Allows time for acquisition or development of test
tools (less use of tools leads to less testing)
How to Know That What You Want Has Been Done- 12
Test Documentation (from IEEE Std 829-1998)
TestExe-
cution
TestPlan
TestDesignSpecs
TestProc
Specs
TestCaseSpecs
TestLog
TestIncidentReports
TestSummary
Report
Always
• Tailored
• Real examples available on-line
Step #4: Prepare Tests and Requirements
How to Know That What You Want Has Been Done- 13
Test Plan Outline
1. Test-plan identifier
2. Introduction
3. Test items
4. Features to be tested
5. Features not to be tested
6. Approach
7. Item pass/fail criteria
8. Suspension criteria and resumption requirements
9. Test deliverables
10. Environmental needs
11. Responsibilities
12. Staffing and training needs
13. Schedule
14. Risks and contingencies
15. Approvals
Step #4: Prepare Tests and Requirements
How to Know That What You Want Has Been Done- 14
• Add to the front-end of the lifecycle– Requirements for testing
• Choose how many levels of testing for this project• Choose documentation for this project• Choose metrics
– Master Test Plan• Over 10 million LOC product line (MANY levels of
test to sort out)• Identify test levels• Hand-off criteria between levels• Email lohrsys@erols.com for a template example
Test Documentation Tailoring Options
How to Know That What You Want Has Been Done- 15
• Skip documents– Test Plan
• Management issues like schedule, and staffing are covered in other documents
• These issues are not under control of testers
• Combine documents– Specifications– Procedures– Log
Test Documentation Tailoring Options
How to Know That What You Want Has Been Done- 16
• Example of combination of Std 829 Specification, Procedures, and Log in MS Word on the next slide
• Email lohrsys@erols.com for an example of this template with an explanation of all of the fields
Tailoring Options
How to Know That What You Want Has Been Done- 17
Tailoring Options
Scenario #: G001Purpose: To test the screens which are used during start-up. These include the Copyright, Sign-on, Status, Idle/Lockout and Screen Map screens. Therequisite white and black (both valid and invalid) box tests are included. The navigation aspects are tested in the navigation test scenarios, but the button that areunique to the Screen Map and do not result in navigation are tested here.Special Needs and Requirements: The validation of the entered passwords needs to be set up by initializing the data structures which represent theaccepted/unaccepted user id and password combinations.Step Inputs Expected Outputs Actual Outputs TR# Special Needs/Comment
SIGN ON SCREEN AND POPUPS1. Attempt to sign on with no user id or password entry
1.1 Power up the HHD Copyright screen is displayed G21.2 Tap Beep to turn it on Highlighting goes on and beeps are
heard throughout the execution ofthis scenario
1.3 Tap Continue Sign On screen is displayed G31.4 Tap Sign On Invalid Sign On popup is displayed G4
Verify that the sign onmessage is set up to be sentOK and set the sign onparameters validation datastructure to invalid
1.5 Tap OK Return to Sign On screen G32. Attempt to sign on with a valid user id and no password
2.1 Tap the user id field The user id field is highlighted2.2 Tap in a valid user id of
BXPWJS1User id is displayed
2.3 Tap Sign On Invalid Sign On popup is displayed G4Verify that the sign onmessage is set up to be sentOK and set the sign onparameters validation datastructure to invalid
2.4 Tap OK Return to Sign On screen G3
How to Know That What You Want Has Been Done- 18
Design
Require-ments
Code
Unit andSystemTests
Accept-anceTest
Test Plan V1.0
Revise Test Plan and develop the
rest of the test documentation
Test Doc’s V2.0
Test Doc’s Vx.y
Step #4: Prepare Tests and Requirements
How to Know That What You Want Has Been Done- 19
• Plan coverage measures for tests; examples:– Source Lines of Code (SLOC)– John Musa’s Operational Profile
• Use detailed checklists for test case development (examples for web site testing follow)
• Put the checklists in an understandable framework
Step #4: Prepare Tests and Requirements
How to Know That What You Want Has Been Done- 20
Step #4: Prepare Tests and Requirements
VALIDMinimumMaximumNormalExploratory
INVALID<Minimum>MaximumCombinationsProcessesExploratory
BEST PRACTICESUsabilityAccessibilityGeneral
Web testing
checklist framework
How to Know That What You Want Has Been Done- 21
Step #4: Prepare Tests and Requirements
VALIDMinimum The shortest Use Case defined navigation sequences Use as few buttons as possibleMaximum The longest Use Case defined navigation sequences Use as many buttons as possibleNormal Meets the requirements of the Style Guide The rest of the Use Case defined navigation sequences All internal links operate as expected (and do not launch a new
browser window) All external links operate as expected (and do launch a new browser
window) Go/History list updated correctly after each link is executed Back/Forward/Home/Stop buttons correctly use data entered and go to
the correct page after a link is used No orphaned pages Site map is current Navigate each frameset menu option Print from framesets Enough of the frameset is viewable on the smallest monitor supported
How to Know That What You Want Has Been Done- 22
Step #5: Refine with Business Analysts
• Refine definition of “correct” for software throughout lifecycle– Testers have constant access to “business”
analysts (similar to JAD)• They use a combination of formal and informal process• There is a continuity of personnel (same business analyst
from project to project)• Business analysts are physically located nearby• Authoritative answers for questions (typical testing
challenge)• Business analysts help deploy to in-house user community
How to Know That What You Want Has Been Done- 23
Summary
• Define what you think you want• Keep effective communication open to refine
requirements• Develop specific tests to prove the goals
have been met• Retain good people and maximize their
potential
How to Know That What You Want Has Been Done- 24
References
• IEEE Std 730-1998 Standard for Software Quality Assurance Plans
• IEEE Std 829-1998 Standard for Software Test Documentation• IEEE Std 830-1998 Recommended Practice for Software
Requirements Specifications