Effective Methods of Software Testing
-
Upload
api-3700815 -
Category
Documents
-
view
125 -
download
2
Transcript of Effective Methods of Software Testing
04/08/23
Quality Assurance Institute’s (QAI)Effective Methods of Software Testing
A PerspectiveSeattle Area Software Quality Assurance Group
John Napier, CSQE, CSTEBEST Consulting
Purpose Value of Professional Associations Value of Software Certification programs QAI’s “Effective Methods of Software Testing”
Disclaimers
The opinions expressed in this presentation are my own, and not those of:
BEST Consulting The Boeing Company Quality Assurance Institute SASQAG
What its like in the trenches... No common language “In my last job…” No individual professional context No organizational professional context Can’t see the forest...
What’s a person to do? Establish a personal professional context Get to know “the forest” Help create a common language Help bring best practices into your organization
04/08/23
The Quality Assurance Institute
What’s QAI?
The Quality Assurance Institute An international association of 1000
organizations banded together to share best practices, benchmark data and experiences
Software quality focus Certification Programs
Certified Quality Analyst Certified Software Quality Engineer
QAI Certified Software Test Engineer (CSTE)
Mandatory Skills Test Planning Use of Test Tools Test Performance Defect Management
Optional Skills Risk Analysis Measurement Test Case Design Establishing and Maintaining
a Test Environment
ASQC* Certified Quality Software Engineer (CQSE)
Experience Professional Association Software Engineering Body of Knowledge
Exam Many other ASQC Certification Programs
*American Society for Quality Control
04/08/23
Effective Methods of Software Testing
Carolyn Fairbanks
Q-Star Consultants
10-14 March 1997
Seattle
04/08/23
Effective Methods of Software TestingThere are really only two issues
in software testing:
Requirements Organizational Culture
QAI Test Materials Organization Test Strategy Test Plan
Supporting Materials Metrics Test Toolbox - Techniques Test Conditions - Techniques Test Case Design - Techniques
Test Strategy Definition 1. Type of Project 2. Type of Software 3. When Testing will Occur 4. Critical Success Factors 5. Who will Conduct the Testing 6. Tradeoffs
1. Types of Project Traditional Client/Server Prototyping Object Oriented Purchased / Contracted
2. Types of SoftwareBatchProcess ControlGraphic User InterfaceObject OrientedProcedure ControlExpert SystemMulti-MediaAdvanced Math Models
Diagnostic SoftwareSensor and SignalSimulationDB ManagementData PresentationDecision AnalysisPattern & ImageComputer OperationsSoftware Development Tools
3. When will Testing Occur? Requirements Phase Activities Design Phase Activities Build Phase Activities Test Phase Activities Installation Phase Activities Maintenance Phase Activities
4. Critical Success FactorsCorrectnessAuthorizationData IntegrityAudit TrailContinuity of ProcessingService LevelsAccess Control
MethodologyReliabilityEase of UseMaintainablePortableCouplingPerformanceEase of Operations
Testing Concerns Matrix
Critical SuccessFactors Life Cycle Stages
Correctness
Requirements
Complete, Clear, Correct,
Testable, Feasible
Design
Etc
7. Tradeoffs (What can be given up?) Schedule Cost / Resources Scope Technology Quality / Critical Success Factors
04/08/23
Some QAI Philosophy
Test Life Cycle, Test Workbench
Testing Relationship Definitions -Testing Lifecycle Concept
Operational orBusiness Need
DefineRequirements
DesignSystem
BuildSystem
UnitTest
IntegrationTest
SystemTest
AcceptanceTest
Testing Relationship Definitions -Continuous Life Cycle Testing
Operational orBusiness Need
DefineRequirements
DesignSystem
BuildSystem
UnitTest
IntegrationTest
SystemTest
AcceptanceTest
Verify
Verify
Verify
Verify Validate
Validate
Validate
Validate
Testing Relationship Definitions -Test Support Activities
Operational orBusiness Need
DefineRequirements
DesignSystem
BuildSystem
UnitTest
IntegrationTest
SystemTest
AcceptanceTest
Verify
Verify
Verify
Verify Validate
Validate
Validate
ValidateTEST OBJECTIVES
TEST PLANNING
TEST EXECUTION(STATIC)
TEST EXECUTION(DYNAMIC) &TEST REPORTING
TestQualityControl
QAI Definitions - “Test Workbench” Concept
TestProcesses
Test Toolbox
TestMeasures
ValidatedTestResults
Problems
ProductsforTest
The QAI Test Process
1. Set Test Objectives2. Develop Test Plan3. Execute Tests4. Summarize and Report Results
Step 1 - Set Test Objectives Task 1 - Organize Test Team Task 2 - Perform Risk Assessment Task 3 - Set Test Objectives Task 4 - Quality Control
Set Test ObjectivesTask 1 - Organize Test Team
Define Test Responsibilities / Tasks Determine People to Perform Tasks
Worksheets 4,5
Set Test Objectives 2. Perform Risk Assessment
Dimensions of Risk Criticality / Mission Impact System Size, Scale and Complexity System Environment and Stability Reliability and Integrity Technology Integration
Impact of Failure
Likelihoodof
Failure
Simple or Complex Scoring Approach
Set Test Objectives 3. Set Test Objectives
Using: System Objectives Test Strategy Risk Score Analysis Team Assignments
Set Objectives Brainstorming Relate Objectives to Function / Output Relate Objectives to Events / System Input
Prioritize Each- Low Medium High
Worksheet 6
Set Test Objectives Quality Control
Objectives QC Checklist Objectives Incident Report (Change Request)
…lots of checklists!
Develop Test Plan 1. Define Business Processes 2. Define Structural Attributes 3. Define Tests 4. Create Test / Function Matrix 5. Define Admin Requirements 6. Formulate System Test Plan 7. QC System Test Plan 8. Develop Unit Test Plan 9. QC Unit Test Plan
Develop Test Plan1. Define Business Functions
Describe what the system should do …stands alone as an activity …is controllable …can be a transition …can be described by a process
Business Function Matrix
Function Description Evaluation Criteria
Worksheet 8
Develop Test Plan2. Define Structural Attributes
Reliability Efficiency Integrity Usability Maintainability
Testability Flexibility Portability Reusability Interoperability
Worksheet 9
Structural Attribute Matrix
Attribute Priority Evaluation CriteriaDescription
Develop Test Plan3. Define Tests
Using Test Objectives Worksheet Business Functions Worksheet Structural Attributes Worksheet
Define Tests Associate test factors, techniques and test types Associate evaluation criteria with techniques and types of
tests Determine type of test which satisfies each objective
Type of Test Techniques Functional
Requirements Regression Error Handling Manual Support Intersystems Control Parallel
Structural Stress Execution Recovery Operations Compliance (to process) Security
Test Data Sources Test Data / Test Desks Production Data Simulation / Prototyping Review / Confirmation Instrumentation / Tracing Test Script
Test Definition Each Test is defined in terms of:
Controls Inputs Outputs Procedures
...At a high level
Develop Test Plan4. Create Function / Test Matrix
TestsFunction
Test 1 Test 2 Ect.
Function #1
Function #2
25 17,30
5,7 45
Develop Test Plan5. Define Admin Test Requirements
Schedule Budget Resources Testing Materials Test Training Tests to be Conducted Defect Estimation
Worksheets 13,14
Develop Test Plan6. Formalize Test Plan
Section I - General Information Section 2 - Plan Section 3 - Specifications and Evaluation Section 4 - Test Description
See Test Plan Example
Develop Test Plan7. Test Plan Quality Control
Quality Control Checklist Test Plan Incident Report (Change Request)
…even more checklists!
Develop Test Plan8. Develop Unit Test Plan
Using: System Test Plan Program Initiation Document Program Design / Specification Documents
Develop Test Plan Administration Subfunction / Test Matrix Interface / Test Matrix
Worksheets 20,21,22
Develop Test Plan9. Unit Test Plan Quality Control
Checklists
Step 3 - Execute Tests Task 1 - Select Test Tools Task 2 - Develop Test Cases Task 3 - Execute Tests Task 4 - Quality Control
QAI Tool Definition What’s a Test Tool?
…any vehicle which assists in performing a test.
Examples- Decision Tables Automated capture / playback Checklists Program Analyzer
Execute TestsStep 1 - Select Test Tools
Using: System Test Plan Unit Test Plan Tool Selection Methodology
Execute TestsStep 2 - Develop Test Cases
Worksheet 23
Expected ResultTest Condition Test Procedures
Execute TestsStep 3 - Execute Tests
Test Activities Log Regression Testing
Full Partial
Worksheet 24
Step 4 - Summarize and Report Test Results
Task 1 - Record Defects Task 2 - Perform Data Reduction Task 3 - Develop Findings and Recommendations Task 4 - Finalize Test Report Task 5 - Test Report Quality Control
Test Defect Report Test Defect List Test Defect Log
Worksheets 25,26,27,28
04/08/23
Supporting Material Metrics Toolbox Techniques Test Condition Analysis Test Design Analysis
Test Metrics
Method -
Goal - Question - Metric
QAI Recommend Test Metrics Defect Reports Requirements Implemented Backlog of Defects (Not Corrected) Status of -
Project, Tests, Schedule, Budget Selected Test Metrics
QAI Test Metrics Selection
Types General Subject
1 Involvement in Testing4 Extent of Testing3 Resources Consumed in Testing20 Effectiveness of Testing2 Assessment of Testing
The QAI Test Toolbox Data Modeling Boundary Value Analysis Capture/Playback Cause-Effect Graphing Change Control Trackers Checklists Checkpoint Review Code Comparison Compiler-based Analysis Complexity-based Analysis
Compliance Checkers Control Flow Analysis Correctness Proofs Coverage Based Analysis Data Dictionary Decision Tables Defect Trackers Desk Checking Equivalence Partitioning Error Guessing
The QAI Test Toolbox File Comparison Flowcharts Help Desk Trackers Inspections Instrumentation Integrated Test Facility Mapping Model Animation Model Balancing Network Analyzers
Performance / Timing Analyzers Population Analysis Risk Matrix Scoring Snapshots Symbolic Execution System Logs Test Case Generator Test Data Generator Test Scripts
The QAI Test Toolbox Test Stubs and Harnesses Test Suite Managers Tracing Walkthroughs
The QAI Test Toolbox
Test Tools
Boundary Analysis
Types of Functional andStructural Tests
Population Analysis
Checklists
X
X
X
X X
Test Condition Analysis Sources -
1. Specification Decomposition 2. Population Analysis 3. Business Process Analysis 4. Structural Analysis 5. Test Transactions
Test Condition - Expected Result - Test Process
Identify Test Conditions 1. System Specifications or Production Environment 2. Population Analysis
Files and Tables Screens Field/Data Element
Worksheet 4
Test Condition - Expected Result - Test Process
Identify Test Conditions
Field Record File Relationships Error Use (Outputs) Search
Match / Merge Stress Control Attributes States Procedures Control Flow
3. Test Transaction Types
Test Condition - Expected Result - Test Process
Identify Test Conditions 4. Business Cases
…business scenarios… 5. Structural Analysis
…decision points...
Test Condition - Expected Result - Test Process
Design Test Cases Design Test Cases Develop Scripts / Sets
Expand Test Conditions
Design Test Cases Expand Conditions into Test Cases using -
Boundary Value Analysis Equivalence Partitioning Decision Tables Code Coverage
Statement Coverage Condition Coverage
Expand Test Conditions
Design Test Cases
Test Scripts
Actor/Scene Dialog
Detailed Test Script Descriptions
Script Reuse
…Sam scowlsat the report...
“Where did thiscome from?”
Expand Test Conditions
QAI Summary - Ten Key Test Concepts Use strategic test questions to set direction Develop Test Objectives Use a cross-functional “test team” Prepare a test plan, then execute it Utilize both verification and validation methods Collect and analyze defects Prepare test management reports
QAI Summary - Ten Key Test Concepts Make test efforts reusable Feed the continuous improvement process with
information from testing Continuously improve the process
04/08/23
Incorporating the Process
Models within Models
…Make them your own
Questions?