1 Staffless SQA An Effective SQA Alternative Unmesh Gundewar E-Mail: [email protected]...
-
Upload
judith-mccoy -
Category
Documents
-
view
219 -
download
1
Transcript of 1 Staffless SQA An Effective SQA Alternative Unmesh Gundewar E-Mail: [email protected]...
1
Staffless SQAAn Effective SQA Alternative
Unmesh GundewarE-Mail: [email protected] 25, 2003
EMC and the EMC logo are registered trademarks and "where information lives" is a trademark of EMC Corporation.
2December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Agenda Introduction
SQA Purpose, Goals, Roles and Responsibilities
Typical SQA Approach and Issues
EMC’s Strategic Approach for Staffless SQA SQA Coverage (Level 2 and Level3)
Challenges
Lessons Learned
Summary
3December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
To meet the needs of today’s tumultuous marketplace, delivery of products and services with superior quality is the key to corporate success and survival. As a result, to improve quality, many corporations are using CMM framework for process improvement.
“Software Quality Assurance is the least frequently satisfied Level 2 Key Process Area”
— August 2002, Process Maturity Profile published by Software Engineering Institute (SEI)
Introduction
4December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
SEI Capability Maturity Model
Level Focus Key Process Areas Result
5Optimizing
4Managed
3Defined
2Repeatable
1Initial
Continuous processimprovement
Product and processquality
Engineering process
Project management
Heroes
Defect preventionTechnology innovationProcess change management
Process measurement andanalysisQuality management
Organization process focusOrganization process definitionPeer reviewsTraining programIntergroup coordinationSoftware product engineeringIntegrated software management
Project planningProject trackingSubcontract managementQuality assuranceConfiguration managementRequirements management
Productivity& Quality
Risk
Carnegie Mellon University - Software Engineering Institute
5December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
The CMM Structure
Maturity Levels
Key Process Areas
Common Features
Key Practices
ProcessCapability
Goals
Implementation orInstitutionalization
Infrastructureor Activities
indicate
achieve
address
describe
contain
organized by
contain
6December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Expanding CMM Structure
5 CMM maturitylevels
0-7 KPAs per maturity level
5 common features per KPA with 1-n practices each
2-4 goals per KPA
= 18 total KPAs
= 52 goals
= 316 total key practices
7December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Software Quality Assurance GoalsPurpose Provide management with appropriate visibility into the
processes being used by the project, and of the products being built.
Goal 1 Quality Assurance activities are planned.
Goal 2 Adherence of products and activities to the applicable standards, procedures, and requirements is verified objectively.
Goal 3 Affected groups and individuals are informed of quality assurance activities and results.
Goal 4 Noncompliance issues that cannot be resolved within the project are addressed by senior management.
8December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Ensure that the organizational standards, practices, and procedures have been selected and followed.
Identify all deviations from the agreed upon processes as early as possible
Assist project managers in developing quality assurance Software Quality Assurance Plan
Promote best practices
Conduct audits and check all software lifecycle work products
Escalate noncompliance and non-resolvable issues
Periodic reporting to affected groups
SQA Roles and Responsibilities
9December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Approach:
Establish Group to Focus on SQA
Issues:
Low Morale
Funding Issues
Policing/Enforcement
Lack of Respect
Typical SQA Approach and Issues
10December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Program managers lead cross-functional teams and use corporate processes to drive consistency, product quality, business readiness, and speed time to market for product and service solutions.
EMC’s Strategic Approach to Staffless SQA
11December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Leads one or more cross-functional teams to drive end-to-end product and service solutions for EMC customers.
Defines scope, develops cross-functional program plans, drives program to completion, and ensures that the product is delivered on time and meets the EMC quality goals.
Leads the cross-functional team to resolve program issues. Escalates issues to the appropriate senior management team when needed.
Role of a Program Manager
EMC’s Strategic Approach to Staffless SQA
12December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Organizational Software Development (SDP) Process– Corporate Lifecycle (LC 300)
– Software Development Lifecycle
– Software Development Process and Checkpoints
– Professional Development Process and Checkpoints
Readiness Management Team (RMT) Process
Escalation Process
Compliance Report
Quality Metrics
EMC’s Strategic Approach Implementation
13December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
EMC’s Product Lifecycle
A Corporately-Accepted Process
Guideline for: Product Creation Product Testing Delivery and Release Field Support Services Replacement/End-Of-
Life
A Flexible Model for Program Management Teams (PMT’s)
Phase 0Feasibilty
Phase 1Design
Phase 2Qualification
Phase 3AvailabilityReadiness
Phase 4Availability
Phase 5End of Life
New Startor
Major Revision
Minor FeatureEnhancements
GA
14December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Software Development ProcessActivities and Checkpoints
0 2 3 4 5 7
SDS
EFS
PDS PEL
PSARelease SpecificPRD
Kic
koff
GA1
CodingPrototypeQA Tracking
Unit Testing QA TestingWaterfallExample
Int
STP Test Cases
6Time
SPPp SPPc
SRS
0 Kickoff1 Preliminary Requirements Review2 Critical Requirements Review3 Preliminary Planning Review
4 Critical Planning Review5 Functionally Complete6 Beta Ready7 Release to Operations
15December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Service Offering Development Overlay and Checkpoints
31 4Publish!
PDS
Release SpecificPRD
NPC0
RTO
Time
1 (TSG) PS Positioning & Impact Review2 Service -Dev Validation of SDR/SCR3 Final Approval to build Offering4 Field Readiness Review
PS Positioning Impact
SDRDevelop Build
Monitor EngagementsValidate
t0 t1
BetaReadyProduct
2Positioning & Bundling
PEL
65
SCR
LCR
Validate
LC300Phase 3
LC300Phase 2
LC300Phase 1
LC300Phase 0
5 Publication Review
6 Monitoring initial release of Servicet – Times will vary depending upon size of product
and type of Service
t2
16December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Functional Group Lead Deliverables CommentsProg/Prod Mgmt {Name} Program Schedule
{Name} PRD{Name} Packaging {physical, market bundles, Models etc}{Name} Positioning{Name} 3rd Party Vendors{Name} PDS
Development {Name} Product {includes Beta, Demos, all things product related}{Name} Knowledge Transfer {Includes customer docs, white papers, web casts, best practices}{Name} Train the Trainer
Marketing {Name} Marketing Collateral{Name} Pricing{Name} Reference/Testimonial
Training {Name} Training Plans {Include PS, Customer, Service, Support, SE & Sales}{Name} Course Development{Name}
Global Services {Name} SVT{Name} Co-operative Support {If needed}{Name} PS Offering
Sales {Name} Readiness {includes all things required to be ready to make this happen}{name} Channels {any new/specific channel activity}{Name} Volume {post GA tracking}
Manufacturing {Name} Supply Base Mgmt{Name} Manufacturing Plan
Readiness Management Process{Full Product Name V#} Readiness Dashboard {Date}
GAVarious Milestones Beta Starts SVT
DateDateDate Date
Program Manager: {Name} Current LC300 Phase(s): {Phase #(s)}
Steady State
Post GA
17December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Milestone
Phase 0
Tentative Dates
{date}
Phase 1
PDS Dates
{date}
Phase 2
{date}
Phase 3
{date}
Estimate of Next Phase
PRD Sign off
NPC Phase 0 Presentation
PDS Sign off
BRT Product Overview
Beta Start
NPC Go to Market Presentation
PS Offering
Marketing Collateral
Field, Help Desk, Sales trained
SVT begins
SVT With MASTER begins
RTM
GAQuality, Service, Unit Tracking begins
Readiness Management Process (Continued)
{Product Name V#} Schedule Delta {Date}
Explanation:
{What are the factors that contributed}
Impact:
{What does this mean}
{How has the plan been modified?}
18December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Readiness Management Process (Continued) {Product Name V#} Program Overview {Date}
Highlights:
{what are the items to be excited about}
{what is going very well}
{nine bullets max}
Lowlights:
{Where are we NOT performing up to our capability – past events}
{These may map directly to an ASK}
{nine bullets max}
Risks/Issues:
{Facts - Not a value judgment}
{Acknowledgement of the Risks – Future items}
{These may map directly to an ASK
{nine bullets max}
Ask:
{what does the Cross Functional Team Require from Management?}
{nine bullets max}
19December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
ProductManager Tech Pubs
G/SCustomerServices
CorporateQuality
Corp MktgMarkets &Products
PMTPMT
Corp MktgAlliance &ChannelPartners
G/SProfessional
Services
Leadership Meetings WorkflowEscalation Process
Product Management andProgram Management
RMT
ProgramManager
ProgramManager
PMT
Sr. LeadersCross-Functional
Organization
FunctionalRepresentatives
LeadersCross-Functional
Organization
Weekly Executive Report
Weekly Executive Report
Cross-Functional Representatives
20December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Late/ Escalation6%
At Risk8%
Missing6%
Future15%
On Schedule21%
Done or NA44%
Compliance MetricsSW Development Checkpoint Status – MM/DD/YY
Total: 34 Product Releases
CheckPoint Breakdown:
Exec Summary Reds:(Products Below with Arrow)
7 Programs with Missing Checkpoint Items:• 2 Program Schedules • 5 Not Scheduled for Reviews
5 Programs have a ‘Red’ Checkpoint Item: Program A•Program B• Program C Program D• Program E
Program Managers’RMT Dashboards
21December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Daily Triage for Product A GA Issues
147
86
48
12 120 00
100
200
300
5-S
ep6-
Sep
7-S
ep8-
Sep
9-S
ep10
-Sep
11-S
ep12
-Sep
13-S
ep14
-Sep
15-S
ep16
-Sep
17-S
ep18
-Sep
19-S
ep20
-Sep
21-S
ep22
-Sep
23-S
ep24
-Sep
25-S
ep26
-Sep
27-S
ep28
-Sep
29-S
ep30
-Sep
1-O
ct2-
Oct
3-O
ct4-
Oct
5-O
ct6-
Oct
7-O
ct8-
Oct
9-O
ct10
-Oct
11-O
ct12
-Oct
13-O
ct14
-Oct
15-O
ct16
-Oct
17-O
ct18
-Oct
19-O
ct20
-Oct
21-O
ct
New Resolved A.1 Open A.0 Open
22December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Product B Open Problem Tracking (OPT) and Backlog
0
100
200
300
400
500
600
700
800
900
1000
1100
1200
1300
7-Ju
n
14-J
un
24-J
un
1-Ju
l
8-Ju
l
15-J
ul
22-J
ul
29-J
ul
5-A
ug
12-A
ug
19-A
ug
26-A
ug
2-S
ep
9-S
ep
16-S
ep
23-S
ep
30-S
ep
7-O
ct
14-O
ct
21-O
ct
28-O
ct
Severity 1 Severity 2 & 3 New Sev 1 New Sev 1 (Moving average)
23December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Product A.1 Test Trends vs A.0
25%
18%
42%
22%
45%49%
44% 44%
70%
27%
63%
71%
79%
88%
98% 99% 99% 100% 100% 100%
75%
86%
81%78%
83% 83%
92% 92%89% 88%
92% 92% 93% 94% 95%93%
98% 98% 98% 99%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
7-Ju
n
14-J
un
24-J
un
1-Ju
l
8-Ju
l
15-J
ul
22-J
ul
29-J
ul
5-A
ug
12-A
ug
19-A
ug
26-A
ug
2-S
ep
9-S
ep
16-S
ep
23-S
ep
30-S
ep
7-O
ct
14-O
ct
21-O
ct
28-O
ct
Test Coverage % Tests Passed A.0 Test Coverage A.0 % Tests Passed
24December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Software Quality Assurance CoverageRequirements Management Key Process Area
KPA Verification 3 ComplianceRM
Ve 3
The activities and work product reviews and/or audits are conducted to manage requirements and results are reported.
Product requirements (PRD) are approved and signed by affected groups (Checkpoint 1).
Changes to requirements are approved by affected groups (Requirements Management Process).
Commitments are negotiated and agreed to by the affected groups.
Italic compliance is in progress
25December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Software Quality Assurance CoverageProject Planning Key Process Area
KPA Verification 3 ComplianceSPP
Ve 3
The activities and work product reviews and/or audits are conducted for project planning and results are reported.
Preliminary project plan is approved (Checkpoint 3).
Standards are used for preparing the project plan.
The content of the software development plan is reviewed.
26December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Software Quality Assurance CoverageProject Tracking and Oversight Key Process Area
KPA Verification 3 CompliancePTO The activities and work
product reviews and/or audits are conducted for project tracking and oversight and results are reported.
Revised project plan is approved [Critical Planning
Review] (Checkpoint 4).
The activities for revising the software development plan is reviewed with affected groups.
The content of the revised software development plan is reviewed.
27December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Software Quality Assurance CoverageConfiguration Management Key Process Area
KPA Verification 4 ComplianceSCM
Ve 4
The activities and work product reviews and/or audits are conducted for configuration management and results are reported.
CM responsibilities are assigned and approved. A build plan exists.
SCM standards are applied for changes to the software.
Different versions of the software exist.
28December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Software Quality Assurance coverageSoftware Quality Assurance Key Process Area
No. Commitment ComplianceCo 1 The project follows
organizational policy.Quality assurance function is in place.
Program manager, engineering group, and other affected groups have a reporting channel to senior management.
Senior management reviews the compliance activities weekly.
29December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Software Quality Assurance CoverageSoftware Quality Assurance Key Process Area
No. Ability ComplianceAB 1 SQA Group exists. Program manager owns SQA
function.AB 2 Adequate resources and
funding are provided to conduct reviews.
Reviews are an integral part of the engineering organization’s processes. Program managers are chartered to ensure that checkpoint reviews are planned and conducted.
AB 3 Individuals are trained to perform quality assurance activities.
Program managers are trained as well as coached through RMT process.
30December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Software Quality Assurance CoverageSoftware Quality Assurance Key Process Area
No. Activities Compliance
Act 1 SQA activities are planned.
Checkpoint reviews are planned in the software project plan.
Act 2 SQA activities are performed in accordance with the plan.
Checkpoints reviews are conducted and affected groups are involved for the reviews.
Act 3 Software Development Plan is reviewed.
Program manager works closely with the software development project manager.
Italic compliance is in progress
31December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Software Quality Assurance CoverageSoftware Quality Assurance Key Process Area
No. Activities Compliance
Act 4 Software Engineering activities are reviewed for compliance.
Covered in the verification of other Key Process Areas.
Act 5 Software work products verified for compliance.
Covered in the compliance metrics.
Act 6 The results of review activities are periodically reported to affected groups.
Communicated weekly in the leadership management meetings.
32December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Software Quality Assurance CoverageSoftware Quality Assurance Key Process Area
No. Measurement Compliance
Me 1 Measurements are made and used to determine the cost and schedule status of the SQA activities.
No measurements are made because the roles and responsibilities of program managers include SQA.
33December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Software Quality Assurance CoverageSoftware Quality Assurance Key Process Area
No. Verification Compliance
Ve 1 SQA activities are reviewed periodically with senior management.
Weekly leadership meetings cover compliance activities.
Ve 2 SQA activities are reviewed with the project manager.
Program manager reviews work products with the project manager.
34December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Software Quality Assurance CoverageSoftware Product Engineering Key Process Area
KPA Verification 3 Compliance
SPE
Ve 3
The activities and work product reviews and/or audits are conducted for software product engineering and results are reported.
Readiness and completion criteria for each software engineering task are satisfied.
Required testing is performed. Tests satisfy acceptance criteria, as documented in the software test plan.
35December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Software Quality Assurance CoverageSoftware Product Engineering Key Process Area
KPA Verification 3 ComplianceSPE
Ve 3
The activities and work product reviews and/or audits are conducted for software product engineering and results are reported.
Problems and defects detected are documented, tracked, and addressed.
Tracing of the allocated requirements through the software requirements, design, code, and test cases is performed.
Italic compliance is in progress
36December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Software Quality Assurance CoverageIntegrated Software Management Key Process Area
KPA Verification 3 ComplianceISM
Ve 3
The activities and work product reviews and/or audits are conducted for integrated software management and results are reported.
Software development plan is developed using organizational standards.
Organizational process database to support for planning, estimating, and tracking is not available.
Italic compliance is in progress
37December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Software Quality Assurance CoveragePeer Review Key Process Area
KPA Verification 1 CompliancePR
Ve 3
The activities and work products reviews and/or audits are conducted for Peer Reviews, and results are reported.
Peer review process is developed and peer reviews are planned.
Peer review process is being implemented.
Italic compliance is in progress
38December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Resistance to change
Need for individual group processes
Compliance of individual group processes
Tracking and oversight
Too many processes
Impact on business
Improving processes
Challenges
39December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
Program managers are key part of the solution
Executive forums helped program managers to achieve EMC goals Faster escalation, resolve issues, assign resources
Best practices shared among program managers
Quicker feedback to program management
Sponsorship and coaching
Lessons Learned
40December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.
“Using organizational and program management standards, program managers drive end-to-end product and service solutions by breaking down organizational barriers, anticipating problems before they occur, and speeding the delivery of products and service solutions.”
Summary
— Peter Conway, V.P. Software Product Management, EMC