How to Know That What You Want Has Been Done- 1 Claire Lohr Member, SESC Management Board Chair IEEE...

How to Know That What You Want Has Been Done- 1

Claire LohrMember, SESC Management Board

Chair IEEE 829 Working [email protected]

How to Know That What You Want Has Been Done

STC May 1, 2002


Topics

• Sources of the information presented• Inclusion of best practices from industry• Definition of the steps for recognition of

successful completion of software projects• Applicable references

NOTE: Best practices marked with


Sources of this Information

• Highly competitive commercial industries– Semiconductor manufacturing– PC manufacturing– Insurance– Finance– Telecommunications– Etc.

Some succeed

Some don’t


The Challenge

Current formal definitions of quality are …

Too broad for software

Real need is to attain the customer’s own specific, detailed quality goals


Steps for Recognizing Quality

• Step #1: define what you don’t want• Step #2: define what you do want with

sufficient detail• Step #3: hire and train good people• Step #4: prepare to measure the results (test

outputs) in parallel with defining the target (requirements)

• Step #5: refine with business analysts throughout lifecycle


Step #1

Define what you don’t want (the business case)

“ IT buyers are mad and aren’t going to take it anymore … Up to 50% of software cost results from ongoing maintenance and integration activities … We spend so much time and money dealing with software problems, it's ridiculous … Some patches break more things than they fix.”

Karyl Scott Information Week January 14, 2002


Step #2: Define What You Do Want

Define what you do want (general)– From the Scott article

• “The Software Bill of Rights, released this week [Jan. 14 2002], stipulates that buyers have the right to expect a quality product, accurate delivery schedules, explicit pricing, development accountability, and effective technical support.”

– Universally accepted definitions of quality• Meets specifications• Meets customer expectations• Fitness for use


Define what you do want (specific)– Document requirements and SQA

• IEEE Std 830-1998 Recommended Practice for Software Requirements Specifications

• IEEE Std 730-1998 Standard for Software Quality Assurance Plans

– Document requirements with Use Cases for more clarity

• Visual

• Shows flows

• Includes everybody

– Plan testing in parallel from the beginning• IEEE Std 829-1998 Standard for Software Test Documentation



Use Case Example


Attend IEEE Track

Use case

Actor SEPG

Member

Association Attend STC

Order SESC standards

Visit computer.org


Step #3: Good People

• Hire and train good people (a good process enhances talent does not replace it)– Thorough employment screening– “Boot camp” training for entry level new hires– Employee friendly work environment ($ to $$$$)– Low turnover– Easy access to training

• Incorporate standards for process and documentation• Use real examples online and in class


Step #4: Prepare Tests and Requirements

Prepare to measure the results (test outputs) in parallel with defining the target (requirements)– Helps developers clarify their thinking when

answering tester’s questions– Assures testability of the final software– Defines “pass criteria” for the software early in the life

cycle– Allows time for acquisition or development of test

tools (less use of tools leads to less testing)


Test Documentation (from IEEE Std 829-1998)

TestExe-

cution

TestPlan

TestDesignSpecs

TestProc

Specs

TestCaseSpecs

TestLog

TestIncidentReports

TestSummary

Report

Always

• Tailored

• Real examples available on-line



Test Plan Outline

1. Test-plan identifier

2. Introduction

3. Test items

4. Features to be tested

5. Features not to be tested

6. Approach

7. Item pass/fail criteria

8. Suspension criteria and resumption requirements

9. Test deliverables

10. Environmental needs

11. Responsibilities

12. Staffing and training needs

13. Schedule

14. Risks and contingencies

15. Approvals



• Add to the front-end of the lifecycle– Requirements for testing

• Choose how many levels of testing for this project• Choose documentation for this project• Choose metrics

– Master Test Plan• Over 10 million LOC product line (MANY levels of

test to sort out)• Identify test levels• Hand-off criteria between levels• Email [email protected] for a template example

Test Documentation Tailoring Options


• Skip documents– Test Plan

• Management issues like schedule, and staffing are covered in other documents

• These issues are not under control of testers

• Combine documents– Specifications– Procedures– Log

Test Documentation Tailoring Options


• Example of combination of Std 829 Specification, Procedures, and Log in MS Word on the next slide

• Email [email protected] for an example of this template with an explanation of all of the fields

Tailoring Options


Tailoring Options

Scenario #: G001Purpose: To test the screens which are used during start-up. These include the Copyright, Sign-on, Status, Idle/Lockout and Screen Map screens. Therequisite white and black (both valid and invalid) box tests are included. The navigation aspects are tested in the navigation test scenarios, but the button that areunique to the Screen Map and do not result in navigation are tested here.Special Needs and Requirements: The validation of the entered passwords needs to be set up by initializing the data structures which represent theaccepted/unaccepted user id and password combinations.Step Inputs Expected Outputs Actual Outputs TR# Special Needs/Comment

SIGN ON SCREEN AND POPUPS1. Attempt to sign on with no user id or password entry

1.1 Power up the HHD Copyright screen is displayed G21.2 Tap Beep to turn it on Highlighting goes on and beeps are

heard throughout the execution ofthis scenario

1.3 Tap Continue Sign On screen is displayed G31.4 Tap Sign On Invalid Sign On popup is displayed G4

Verify that the sign onmessage is set up to be sentOK and set the sign onparameters validation datastructure to invalid

1.5 Tap OK Return to Sign On screen G32. Attempt to sign on with a valid user id and no password

2.1 Tap the user id field The user id field is highlighted2.2 Tap in a valid user id of

BXPWJS1User id is displayed

2.3 Tap Sign On Invalid Sign On popup is displayed G4Verify that the sign onmessage is set up to be sentOK and set the sign onparameters validation datastructure to invalid

2.4 Tap OK Return to Sign On screen G3


Design

Require-ments

Code

Unit andSystemTests

Accept-anceTest

Test Plan V1.0

Revise Test Plan and develop the

rest of the test documentation

Test Doc’s V2.0

Test Doc’s Vx.y



• Plan coverage measures for tests; examples:– Source Lines of Code (SLOC)– John Musa’s Operational Profile

• Use detailed checklists for test case development (examples for web site testing follow)

• Put the checklists in an understandable framework




VALIDMinimumMaximumNormalExploratory

INVALID<Minimum>MaximumCombinationsProcessesExploratory

BEST PRACTICESUsabilityAccessibilityGeneral

Web testing

checklist framework



VALIDMinimum The shortest Use Case defined navigation sequences Use as few buttons as possibleMaximum The longest Use Case defined navigation sequences Use as many buttons as possibleNormal Meets the requirements of the Style Guide The rest of the Use Case defined navigation sequences All internal links operate as expected (and do not launch a new

browser window) All external links operate as expected (and do launch a new browser

window) Go/History list updated correctly after each link is executed Back/Forward/Home/Stop buttons correctly use data entered and go to

the correct page after a link is used No orphaned pages Site map is current Navigate each frameset menu option Print from framesets Enough of the frameset is viewable on the smallest monitor supported


Step #5: Refine with Business Analysts

• Refine definition of “correct” for software throughout lifecycle– Testers have constant access to “business”

analysts (similar to JAD)• They use a combination of formal and informal process• There is a continuity of personnel (same business analyst

from project to project)• Business analysts are physically located nearby• Authoritative answers for questions (typical testing

challenge)• Business analysts help deploy to in-house user community


Summary

• Define what you think you want• Keep effective communication open to refine

requirements• Develop specific tests to prove the goals

have been met• Retain good people and maximize their

potential


References

• IEEE Std 730-1998 Standard for Software Quality Assurance Plans

• IEEE Std 829-1998 Standard for Software Test Documentation• IEEE Std 830-1998 Recommended Practice for Software

Requirements Specifications

How to Know That What You Want Has Been Done- 1 Claire Lohr Member, SESC Management Board Chair IEEE...

Documents

Transcript of How to Know That What You Want Has Been Done- 1 Claire Lohr Member, SESC Management Board Chair IEEE...