ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New...
Transcript of ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New...
![Page 1: ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New International Software Testing Standards Stuart Reid ©Stuart Reid, 2015](https://reader034.fdocuments.in/reader034/viewer/2022052122/5aacc1327f8b9a2e088d5ef6/html5/thumbnails/1.jpg)
Madrid, 26th to 28th May 2014
ISO/IEC/IEEE 29119The New International Software
Testing Standards
Stuart Reid
©Stuart Reid, 2015
![Page 2: ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New International Software Testing Standards Stuart Reid ©Stuart Reid, 2015](https://reader034.fdocuments.in/reader034/viewer/2022052122/5aacc1327f8b9a2e088d5ef6/html5/thumbnails/2.jpg)
Madrid, 26th to 28th May 2014
Scope
The purpose of standards
Motivation for software testing standards
Development of standards
Overview of ISO 29119
Applicability
Timeline
Future involvement
![Page 3: ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New International Software Testing Standards Stuart Reid ©Stuart Reid, 2015](https://reader034.fdocuments.in/reader034/viewer/2022052122/5aacc1327f8b9a2e088d5ef6/html5/thumbnails/3.jpg)
Madrid, 26th to 28th May 2014
What are Standards?
– Guidelines documents as they are not compulsory
unless mandated by an individual or an organization
– Agreements because they should reflect a certain
level of consensus
“Guideline documentation that reflects agreements on products, practices, or operations by nationally or internationallyrecognized industrial, professional, trade associations or governmental bodies”
- ISO
![Page 4: ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New International Software Testing Standards Stuart Reid ©Stuart Reid, 2015](https://reader034.fdocuments.in/reader034/viewer/2022052122/5aacc1327f8b9a2e088d5ef6/html5/thumbnails/4.jpg)
Madrid, 26th to 28th May 2014
Why use Standards?
Consumers
– Confidence in compliant products
– Authors provide expertise in standards
Manufacturers
– Conformance and Marketing
– ‘Safety’ from liability
– Guidelines on production
But not ‘Best Practice’…
![Page 5: ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New International Software Testing Standards Stuart Reid ©Stuart Reid, 2015](https://reader034.fdocuments.in/reader034/viewer/2022052122/5aacc1327f8b9a2e088d5ef6/html5/thumbnails/5.jpg)
Madrid, 26th to 28th May 2014
Quality and Standards
![Page 6: ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New International Software Testing Standards Stuart Reid ©Stuart Reid, 2015](https://reader034.fdocuments.in/reader034/viewer/2022052122/5aacc1327f8b9a2e088d5ef6/html5/thumbnails/6.jpg)
Madrid, 26th to 28th May 2014
What use are Standards?
Basis for:
– Communication – common terminology
– Professional qualifications
– Certification/compliance schemes
– Benchmark of ‘good industry practice’
– Contracts
– Interoperability and consistency……
Standards describe a current ‘body of knowledge’
that provides the basis for a professional
discipline
![Page 7: ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New International Software Testing Standards Stuart Reid ©Stuart Reid, 2015](https://reader034.fdocuments.in/reader034/viewer/2022052122/5aacc1327f8b9a2e088d5ef6/html5/thumbnails/7.jpg)
Madrid, 26th to 28th May 2014
A Case in Point
![Page 8: ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New International Software Testing Standards Stuart Reid ©Stuart Reid, 2015](https://reader034.fdocuments.in/reader034/viewer/2022052122/5aacc1327f8b9a2e088d5ef6/html5/thumbnails/8.jpg)
Madrid, 26th to 28th May 2014
Motivation for ISO 29119
Demand for existing 'standards’
Gaps in the current standards provision
Conflicts in current definitions and processes
A Baseline for the Testing Discipline
Current industry practice is lacking
Buyers unclear on what is 'good test practice'
![Page 9: ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New International Software Testing Standards Stuart Reid ©Stuart Reid, 2015](https://reader034.fdocuments.in/reader034/viewer/2022052122/5aacc1327f8b9a2e088d5ef6/html5/thumbnails/9.jpg)
Madrid, 26th to 28th May 2014
Standardization Bodies
![Page 10: ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New International Software Testing Standards Stuart Reid ©Stuart Reid, 2015](https://reader034.fdocuments.in/reader034/viewer/2022052122/5aacc1327f8b9a2e088d5ef6/html5/thumbnails/10.jpg)
Madrid, 26th to 28th May 2014
ISO 29119 – Structure
BS 7925-1
BS 7925-2 IEEE 829
Concepts & Vocabulary
Part 1
Testing
Techniques
Part 4
Documentation
Part 3Part 2
Processes
Keyword-Driven
Testing
Part 5
Process
Assessment
ISO/IEC 33063
Static Testing
Techniques
Part 6IEEE 1028
![Page 11: ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New International Software Testing Standards Stuart Reid ©Stuart Reid, 2015](https://reader034.fdocuments.in/reader034/viewer/2022052122/5aacc1327f8b9a2e088d5ef6/html5/thumbnails/11.jpg)
Madrid, 26th to 28th May 2014
SOFTWARE TESTING CONCEPTS
Scope, Conformance, Normative References
ANNEX - TESTING IN DIFFERENT LIFE CYCLES
ANNEX - ROLES AND RESPONSIBILITIES
Bibliography
DEFINITIONS
RBT
Part 1: Concepts & Vocabulary
ANNEX – TEST METRICS & MEASURES
ANNEX - TESTING IN V & V
![Page 12: ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New International Software Testing Standards Stuart Reid ©Stuart Reid, 2015](https://reader034.fdocuments.in/reader034/viewer/2022052122/5aacc1327f8b9a2e088d5ef6/html5/thumbnails/12.jpg)
Madrid, 26th to 28th May 2014
Part 2: Testing Processes
TEST MANAGEMENT PROCESSES
ORGANIZATIONAL TEST PROCESS
DYNAMIC TEST PROCESSES
![Page 13: ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New International Software Testing Standards Stuart Reid ©Stuart Reid, 2015](https://reader034.fdocuments.in/reader034/viewer/2022052122/5aacc1327f8b9a2e088d5ef6/html5/thumbnails/13.jpg)
Madrid, 26th to 28th May 2014
TEST MANAGEMENT PROCESSES
STATIC TEST
PROCESSES
ORGANIZATIONAL TEST PROCESS
DYNAMIC TEST
PROCESSES
Testing Processes – the future?
![Page 14: ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New International Software Testing Standards Stuart Reid ©Stuart Reid, 2015](https://reader034.fdocuments.in/reader034/viewer/2022052122/5aacc1327f8b9a2e088d5ef6/html5/thumbnails/14.jpg)
Madrid, 26th to 28th May 2014
ORGANIZATIONAL TEST PROCESS
TEST MANAGEMENT PROCESSES
TEST
PLANNING
TEST
MONITORING
& CONTROL
TEST
COMPLETION
ORGANIZATIONAL
TEST
DOCUMENTATION
FEEDBACK ON
ORGANIZATIONAL TEST
DOCUMENTATION
TEST PLAN UPDATES
TEST
PLAN
TEST
COMPLETION
REPORT
DYNAMIC TEST
PROCESSES
TEST
MANAGEMENT
PROCESSES
TEST PLAN,
TEST COMPLETION
REPORT,
TEST MEASURES
TEST
MEASURES
TEST PLAN,
CONTROL
DIRECTIVES
TEST PLAN,
CONTROL
DIRECTIVES
Test Management Processes
![Page 15: ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New International Software Testing Standards Stuart Reid ©Stuart Reid, 2015](https://reader034.fdocuments.in/reader034/viewer/2022052122/5aacc1327f8b9a2e088d5ef6/html5/thumbnails/15.jpg)
Madrid, 26th to 28th May 2014
Organise
Test Plan
Development
Identify &
Estimate Risks
Design Test
Strategy
Determine
Staffing and
Scheduling
Document
Test Plan
Schedule, Staffing
Profile
Test
Strategy
Analyzed
Risks
Scope
Identify Risk
Mitigation
Approaches
Gain
Consensus on
Test Plan
Approved
Test Plan
Draft
Test Plan
Test
Plan Publish
Test Plan
Understand
Context
Mitigation
Approaches
Test Planning Process
![Page 16: ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New International Software Testing Standards Stuart Reid ©Stuart Reid, 2015](https://reader034.fdocuments.in/reader034/viewer/2022052122/5aacc1327f8b9a2e088d5ef6/html5/thumbnails/16.jpg)
Madrid, 26th to 28th May 2014
Perfect Requirements?
Gra
phic
refe
rence:
htt
p:/
/777-t
eam
.org
/tm
p/p
roje
ct.
jpg
How the Customer
explained it
How the Project
Leader understood
it
How the Analyst
designed itHow the
Programmer wrote
it
How the
Consultant
described it
What the Customer
actually needed
![Page 17: ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New International Software Testing Standards Stuart Reid ©Stuart Reid, 2015](https://reader034.fdocuments.in/reader034/viewer/2022052122/5aacc1327f8b9a2e088d5ef6/html5/thumbnails/17.jpg)
Madrid, 26th to 28th May 2014
Requirements-Based Testing
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Requirements
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Tests
1 to 1
![Page 18: ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New International Software Testing Standards Stuart Reid ©Stuart Reid, 2015](https://reader034.fdocuments.in/reader034/viewer/2022052122/5aacc1327f8b9a2e088d5ef6/html5/thumbnails/18.jpg)
Madrid, 26th to 28th May 2014
Limited Risk-Based Testing
importance
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Requirements
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Tests
![Page 19: ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New International Software Testing Standards Stuart Reid ©Stuart Reid, 2015](https://reader034.fdocuments.in/reader034/viewer/2022052122/5aacc1327f8b9a2e088d5ef6/html5/thumbnails/19.jpg)
Madrid, 26th to 28th May 2014
Risk-Based Testing
1 23
45
67
89
1011
1213
1415
16
17
18
19
20
Impact/Importance
Likelihood/Complexity
Risk Level/Testing
![Page 20: ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New International Software Testing Standards Stuart Reid ©Stuart Reid, 2015](https://reader034.fdocuments.in/reader034/viewer/2022052122/5aacc1327f8b9a2e088d5ef6/html5/thumbnails/20.jpg)
Madrid, 26th to 28th May 2014
Risk-Based Panic
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
2011
13
9
6
14
4
17
5
10
12
8
2
20
18
7
15
1
16
3
19
Risk Level / Testing Prioritised Risks / Testing
![Page 21: ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New International Software Testing Standards Stuart Reid ©Stuart Reid, 2015](https://reader034.fdocuments.in/reader034/viewer/2022052122/5aacc1327f8b9a2e088d5ef6/html5/thumbnails/21.jpg)
Madrid, 26th to 28th May 2014
DYNAMIC TEST PROCESSES
TEST
SPECIFICATION[NO ISSUES
NOTICED]
TEST
ENVIRONMENT
READINESS
REPORT
TEST
ENV’T
REQUIREMENTS
INCIDENT
REPORT
TEST DESIGN &
IMPLEMENTATION
TEST
ENVIRONMENT
SET-UP
TEST
EXECUTION
TEST
INCIDENT
REPORTING
(LEVEL) TEST MANAGEMENT PROCESS
[ISSUE NOTICED or
RETEST RESULT]
TEST
RESULTS
CONTROL
DIRECTIVES TEST
MEASURES
(LEVEL)
TEST PLAN
Dynamic Test Processes
![Page 22: ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New International Software Testing Standards Stuart Reid ©Stuart Reid, 2015](https://reader034.fdocuments.in/reader034/viewer/2022052122/5aacc1327f8b9a2e088d5ef6/html5/thumbnails/22.jpg)
Madrid, 26th to 28th May 2014
Part 3 – Test Documentation
TEST DOCUMENTATION
ANNEXES - EXAMPLES
Scope, Conformance,
Normative References
AGILE &
TRADITIONAL
![Page 23: ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New International Software Testing Standards Stuart Reid ©Stuart Reid, 2015](https://reader034.fdocuments.in/reader034/viewer/2022052122/5aacc1327f8b9a2e088d5ef6/html5/thumbnails/23.jpg)
Madrid, 26th to 28th May 2014
Test Documentation Types
Organizational test documentation– Test policy
– Test strategy
Project test documentation – Project test plan
– Test project completion report
Test level documentation– Test plan
– Test specification
– Test results
– Anomaly reports
– Level test status report
– Test environment report
– Test level completion report
![Page 24: ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New International Software Testing Standards Stuart Reid ©Stuart Reid, 2015](https://reader034.fdocuments.in/reader034/viewer/2022052122/5aacc1327f8b9a2e088d5ef6/html5/thumbnails/24.jpg)
Madrid, 26th to 28th May 2014
Part 4 – Test Techniques
TEST COVERAGE MEASUREMENT
Scope, Conformance, Normative References
ANNEXE – TESTING OF QUALITY CHARACTERISTICS
ANNEXE – EXAMPLE USE OF TECHNIQUES/MEASURES
ANNEXE – TEST TECHNIQUE EFFECTIVENESS
TEST DESIGN TECHNIQUES
BLACK BOX WHITE BOX EXPERIENCE
![Page 25: ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New International Software Testing Standards Stuart Reid ©Stuart Reid, 2015](https://reader034.fdocuments.in/reader034/viewer/2022052122/5aacc1327f8b9a2e088d5ef6/html5/thumbnails/25.jpg)
Madrid, 26th to 28th May 2014
ISO 29119 –
Test case design techniques
specification-based testing
techniques
• boundary value analysis
• cause-effect graphing
• classification tree method
• combinatorial test techniques
• decision table testing
• equivalence partitioning
• random testing
• scenario testing
• state transition testing
• syntax testing
structure-based testing
techniques
• branch / decision testing
• branch condition testing
• branch condition combination
testing
• data flow testing
• modified condition decision
coverage (MCDC) testing
• statement testing
experience-based testing
technique
• error guessing
![Page 26: ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New International Software Testing Standards Stuart Reid ©Stuart Reid, 2015](https://reader034.fdocuments.in/reader034/viewer/2022052122/5aacc1327f8b9a2e088d5ef6/html5/thumbnails/26.jpg)
Madrid, 26th to 28th May 2014
Current Status
May
10
May
11
May
12
May
13…
Working Draft (WD)
Committee Draft (CD)
Draft International Standard (DIS)
Final Draft International Standard (FDIS)
Final International Standard (FIS)
Parts 1, 2 & 3
Part 4
WDCD1
DISWD
CD1
DIS
FDIS FIS!!!
CD2CD3
CD2CD3
Part 5
CD4
DIS-2
WDCD
DIS-2
FDIS
![Page 27: ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New International Software Testing Standards Stuart Reid ©Stuart Reid, 2015](https://reader034.fdocuments.in/reader034/viewer/2022052122/5aacc1327f8b9a2e088d5ef6/html5/thumbnails/27.jpg)
Madrid, 26th to 28th May 2014
What is most likely to stop
ISO 29119?Fear of change
'Not invented here' syndrome
Perceived as competition
Lack of required use
Anti-standardization – craftsmen
No evidence of efficacy
Cost of use
Complexity
Lack of responsiveness
Lack of inclusiveness…
![Page 28: ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New International Software Testing Standards Stuart Reid ©Stuart Reid, 2015](https://reader034.fdocuments.in/reader034/viewer/2022052122/5aacc1327f8b9a2e088d5ef6/html5/thumbnails/28.jpg)
Madrid, 26th to 28th May 2014
Are they applicable to you?
Intended to be a generic standards
– allow tailored conformance
For small and large organizations
Agile and Traditional
Exploratory testing
Financial, Safety-Critical, Games...
Following Regulatory Standards?
Can you afford not to?
![Page 29: ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New International Software Testing Standards Stuart Reid ©Stuart Reid, 2015](https://reader034.fdocuments.in/reader034/viewer/2022052122/5aacc1327f8b9a2e088d5ef6/html5/thumbnails/29.jpg)
Madrid, 26th to 28th May 2014
Conclusions
Parts 1, 2 & 3 were published September 2013
– My thanks to the editors and reviewers
International standards will provide practitioners
with guidelines for testing that cover all aspects of
the life cycle
– Provide a consistent set of definitions, processes,
procedures & techniques for software testing
– and accelerate take-up of risk-based testing
Will be adopted by IEEE, BSI, ISO and other
national standards bodies
![Page 30: ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New International Software Testing Standards Stuart Reid ©Stuart Reid, 2015](https://reader034.fdocuments.in/reader034/viewer/2022052122/5aacc1327f8b9a2e088d5ef6/html5/thumbnails/30.jpg)
Madrid, 26th to 28th May 2014
Do you want to be involved?
Join ISO Working Group 26
– representing your national standards body
– 6 day meetings, every 6 months
– contribute between meetings
Join a WG26 mirror group
– for your national standards body
Contribute materials
Review drafts
![Page 31: ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New International Software Testing Standards Stuart Reid ©Stuart Reid, 2015](https://reader034.fdocuments.in/reader034/viewer/2022052122/5aacc1327f8b9a2e088d5ef6/html5/thumbnails/31.jpg)
Madrid, 26th to 28th May 2014
Any Questions?
Any Volunteers?
![Page 32: ISO/IEC/IEEE 29119 - · PDF fileMadrid, 26th to 28th May 2014 ISO/IEC/IEEE 29119 The New International Software Testing Standards Stuart Reid ©Stuart Reid, 2015](https://reader034.fdocuments.in/reader034/viewer/2022052122/5aacc1327f8b9a2e088d5ef6/html5/thumbnails/32.jpg)
Madrid, 26th to 28th May 2014
Finally…
– if you have any questions on the standards
– if you are interested in trialling the standard on a project, reviewing drafts or writing examples
http://softwaretestingstandard.org/
– WG26 website
http://www.jtc1-sc7.org/
– access to official documents released by WG 26
Acknowledgement
– ISO 29119 process diagrams based on those of ISO/IEC/IEEE 29119-2, 2013