SQA SQA -- The Implementation The Implementation of SPIof SPI
A Case Study of Software Quality A Case Study of Software Quality Assurance at USSTRATCOM/J62 Assurance at USSTRATCOM/J62 Offutt AFB, NebraskaOffutt AFB, Nebraska
Presented By: Maj Andrew Boyd, USAFPresented By: Maj Andrew Boyd, USAF
10 Nov 19993
OverviewOverview•• BackgroundBackground
•• The AcademiaThe Academia (Organizational Process Assets)(Organizational Process Assets)
•• The ImplementationThe Implementation–– Getting StartedGetting Started
–– Organizing for SQAOrganizing for SQA–– Specific Roles, Responsibilities, Assets, Plans, Specific Roles, Responsibilities, Assets, Plans,
Activities, and RelationshipsActivities, and Relationships– Metrics Collected and Reported
•• Continuing SPI Without an SEPGContinuing SPI Without an SEPG
•• Lessons Learned (Again!)Lessons Learned (Again!)
10 Nov 19994
BackgroundBackground
•• DoD Has Unique Software ChallengesDoD Has Unique Software Challenges
•• 3 Separate CBA3 Separate CBA--IPIs @ USSTRATCOMIPIs @ USSTRATCOM
•• Migration/Modernization EffortMigration/Modernization Effort
•• Consolidated 4 Complete Software Consolidated 4 Complete Software DivisionsDivisions
•• J62 Level SEPGJ62 Level SEPG
•• Lots of AssetsLots of Assets
10 Nov 19995
US Strategic Command US Strategic Command (Joint)(Joint)CJCS
J0Command Staff
J1Man/Pers
J2Intel
J3/4Ops/Logistics
J5Strategic Plans
J60Staff
J61Man/Pers
J62Intel SW Supt
J63/4C2 SW Supt
J65SWPS SW Supt
J66Program Mgmt
J67Training
J68LAN Mgmt
J6Comm/Comp
*
USSTRATCOMCINCSTRAT
****
SECDEF
10 Nov 19996
Software Support Division Software Support Division (J62)(J62)
School HouseJ67
Front Office Staff
Assessments
Training
Standards
J621Process Mgmt Branch
Test
SCM
SQA
Intel CM/QA/Test
J622Integration Branch
Command Post
...
J623C2 SW Branch
Systems
Development
...
J624Intel SW Branch
Targeting
Production
Data
...
J625SWPS SW Branch
...
J626LAN Mgmt Branch
J62Division
10 Nov 19997
Organizational Process AssetsOrganizational Process Assets
•• USSTRATCOM LevelUSSTRATCOM Level–– Administrative Instructions (E.G. SAI 709Administrative Instructions (E.G. SAI 709--1)1)
•• J6 LevelJ6 Level–– Process Definition Guide (ETVX)Process Definition Guide (ETVX)–– Lifecycle Guide (Waterfall, Evolutionary, Etc)Lifecycle Guide (Waterfall, Evolutionary, Etc)–– Process Asset Library GuideProcess Asset Library Guide–– SEPG Chartering GuideSEPG Chartering Guide
•• J62 LevelJ62 Level–– 933 Series Operating Instructions (OI)933 Series Operating Instructions (OI)
10 Nov 19998
Operating Instruction (OI)Operating Instruction (OI)
•• ReferencesReferences
•• PurposePurpose
•• Applicability (Scope)Applicability (Scope)
•• ResponsibilitiesResponsibilities
•• Resources, Training, Reviews, TermsResources, Training, Reviews, Terms
•• Area Specific Activities/productsArea Specific Activities/products
10 Nov 19999
J62 Software Process J62 Software Process StandardsStandards
OI Name Date933-01 Division Software OI Management 22 Aug 97933-02 J62 Software Development and Maintenance Policy 23 May 96933-03 J62 Standard for Software Quality Assurance 10 Jun 96933-04 J62 Standard for Software Configuration Management 27 Jun 96933-05 J62 Standard for Test 19 Jul 96933-06 J62 Enterprise Improvement Process Guide 13 May 96933-07 J62 Terms and Definitions Repository 7 Mar 97933-09 J62 Standard for Life Cycles 6 Jun 96933-10 J62 Standard for Contract Management/Monitoring 6 Aug 96933-11 J62 Standard for Software Metrics 29 Jul 96933-12 J62 Standard for Software Requirements Management 23 Aug 96933-13 J62 Standard for Project Planning 24 Oct 96933-14 J62 Standard for Risk Management 23 Jan 98933-15 J62 Standard for Peer Reviews 18 Sep 96933-16 J62 Standard for Management Reviews 12 Jun 97
10 Nov 199910
J62 Software Process J62 Software Process Standards (Cont.)Standards (Cont.)
OI Name Date933-17 J62 Standard for Assessments 12 Sep 96933-18 J62 Standard for Software Design 4 Nov 96933-19 J62 Standard for Software Tools Management 3 Mar 97933-20 J62 Standard for Software Coding 6 Nov 96933-21 J62 Standard for Resources 13 Dec 96933-22 Process Asset Library (PAL) 25 Jun 97933-23 J62 Standard for Project Documentation 6 Jul 98933-24 J62 Standard for Post Mortem 22 Sep 97933-25 J62 Software Release Review Board
Checklist updated 6 Jul 9831 Mar 98
233-01 J62 Standard for Training 5 Sep 97
10 Nov 199911
Assets (Cont.)Assets (Cont.)
•• Software Development Process FlowSoftware Development Process Flow
•• Standards (Documentation)Standards (Documentation)–– MilMil--StdStd--498 (Software Development and 498 (Software Development and
Documentation) Now JDocumentation) Now J--STDSTD--016016•• Data Item Descriptions (Some Tailored)Data Item Descriptions (Some Tailored)
–– SSS, SDP, SRS, SSDD, SDD, DBDD, IRS, STP, STD, Etc.SSS, SDP, SRS, SSDD, SDD, DBDD, IRS, STP, STD, Etc.
•• Guidebook Guidebook
–– IEEE Software Engineering CompendiumIEEE Software Engineering Compendium
•• SWSW--CMMCMM®® V 1.1V 1.1
10 Nov 199912
Assets (Cont.)Assets (Cont.)
•• Branch LevelBranch Level–– Software Development Guide (2)Software Development Guide (2)
–– SQA Guide (2)SQA Guide (2)
–– Software Configuration Mgmt Guide (2)Software Configuration Mgmt Guide (2)
–– Program Management GuideProgram Management Guide
–– Test Process GuideTest Process Guide
–– Other Stuff From Earlier Days As Separate Other Stuff From Earlier Days As Separate HouseHouse
10 Nov 199915
My View of SQAMy View of SQA
•• Seen It Done Various Ways Seen It Done Various Ways
•• Ensure ComplianceEnsure Compliance
•• Objective Vs Subjective Objective Vs Subjective
•• SWSW--CMMCMM®®
–– Process AssuranceProcess Assurance
–– Product AssuranceProduct Assurance
–– Track Deviations and Report NonTrack Deviations and Report Non--compliancecompliance
•• Senior Management’s Eyes and EarsSenior Management’s Eyes and Ears
10 Nov 199916
ProposalProposal
•• White Paper (SQA 101)White Paper (SQA 101)–– CredentialsCredentials
–– Key Objectives of QA (Used Generically)Key Objectives of QA (Used Generically)•• Reviews, AuditsReviews, Audits
•• Deviation Tracking and NonDeviation Tracking and Non--compliance Issuescompliance Issues
•• Autonomy, Level of AuthorityAutonomy, Level of Authority
•• Current SituationCurrent Situation
•• Recommendation (Org Level, Mission Alignment, Recommendation (Org Level, Mission Alignment, Responsibilities)Responsibilities)
10 Nov 199917
Initial SQA ResourcesInitial SQA Resources
•• PersonnelPersonnel–– 3 KTRs3 KTRs
–– 3 GS3 GS
–– 2 Military2 Military
•• Assets (OI 933Assets (OI 933--03, SPIP, and Work 03, SPIP, and Work Product Compliance Audit)Product Compliance Audit)
•• J62 SQA Plan (Draft)J62 SQA Plan (Draft)
10 Nov 199918
Some ChallengesSome Challenges
•• Lack of SQA TrainingLack of SQA Training
•• Product Vs Process FocusProduct Vs Process Focus
•• Management Didn’t Understand SQAManagement Didn’t Understand SQA
•• Conflict of InterestConflict of Interest–– Peer to Peer Relationship Peer to Peer Relationship
–– SQA Not a Priority for Branch ChiefSQA Not a Priority for Branch Chief
–– Personal QAPersonal QA
10 Nov 199919
More ChallengesMore Challenges
•• Disgruntled MembersDisgruntled Members
•• “We’re Being Outsourced So Who “We’re Being Outsourced So Who Cares…”Cares…”
•• Single Domain FocusedSingle Domain Focused
•• SQA Plan Too High LevelSQA Plan Too High Level
•• Most of J62 Unaware of Assets (or Told Most of J62 Unaware of Assets (or Told Not to Use)Not to Use)
10 Nov 199920
First StepsFirst Steps
•• Group OffsiteGroup Offsite
•• Individual SessionsIndividual Sessions–– Role, ExpectationsRole, Expectations
•• Tailored an SQA Course (J67)Tailored an SQA Course (J67)
•• “Borrowed” Deviation Tracking Database “Borrowed” Deviation Tracking Database From Intel FolksFrom Intel Folks
•• Provided MS Project, and Access TrainingProvided MS Project, and Access Training
10 Nov 199921
Organizing for SQAOrganizing for SQA
•• J62 After Merging 4 Software HousesJ62 After Merging 4 Software Houses
•• “Forced” a Change“Forced” a Change–– Did Audit of CM and Test SectionsDid Audit of CM and Test Sections
•• After “Fine Tuning”After “Fine Tuning”–– Report to Highest LevelReport to Highest Level
–– No Longer Conflict of InterestNo Longer Conflict of Interest
–– Lost Some Resources in ShuffleLost Some Resources in Shuffle
10 Nov 199922
Software Support Division Software Support Division (J62) After “Fine Tuning”(J62) After “Fine Tuning”
School House(J67)
Intel SW Support(J25)Front Office Staff SQA
Targeting
Production
Data
...
Lt ColSWPS Branch
Command Post
...
MajC2 Branch
...
MajLAN Branch
Test
SCM
Lt ColIntegration Branch
ColonelDivision Chief
10 Nov 199923
SQA AssetsSQA Assets
•• Vol. 1 SQA Master Audit Review ProcessVol. 1 SQA Master Audit Review Process
•• Vol. 2 SPIP ManualVol. 2 SPIP Manual
•• Vol. 3 SQA Survey ProcessVol. 3 SQA Survey Process
•• Vol. 4 SQA Deviation Tracking System Vol. 4 SQA Deviation Tracking System ManualManual
•• Vol. 5 SQA Inspection TrackingVol. 5 SQA Inspection Tracking
10 Nov 199924
SQA ACTION ITEM REPORTAudit Action Item Control Date Prepared: 04-Nov-1997 Followup Date:
ICR006 - 001 Date Notified: Suspense Date:23-Nov-1997
Initiator: Office: Phone:
Mr. McClellanTask Directed To Office: Phone:Maj Boyd J6222 294-3976
Software Project: Document: Section/CSCI: Audit/Review:
J621 Software Product J6222 Process Review
Deviation Description:How are issues identified during the review resolved? Currently, no follow up review is performed onissues/concerns raised in a previous inspection (i.e. don't see formal follow up process).
Action Required:
Resolution:It's the editor/author's responsibility to resolv issues. However, the team lead cannot exit the process ifthere are know issues. It's not part of this process to gain concurrence of the document. The appropriate review (e.g. SRR, PDR, CDR) is where this should be done
Date: Approval Authority: Office: Phone: Signature:21-Nov-97
CoordinationDate: Name of Coordinator: Office: Phone: Signature:
SQA Action Item ReportSQA Action Item Report
10 Nov 199925
SQA Assets (Cont.)SQA Assets (Cont.)
•• Vol. 6 OI Waiver ProcessVol. 6 OI Waiver Process
•• Vol. 7 SQA Acronym Definition ManualVol. 7 SQA Acronym Definition Manual
•• Vol. 8 PVol. 8 P--SQAP Building ProcessSQAP Building Process
•• Vol. 9 Peer Review ProcessVol. 9 Peer Review Process
•• Integrated Process Flow (Detailed by Integrated Process Flow (Detailed by Phase)Phase)
•• Mother of all Checklists (MOAC)Mother of all Checklists (MOAC)
10 Nov 199926
IPT Project CM QA Test Mgt Team (J62)
<STARTPROCESS>Initiate SOW
J62 OI 933-13,para 4a
Review SOWJ62OI 933-02,
para 5
Develop ProjectPlan (SDP)
J62OI 933-13,para 4c
(include riskanalysis, J62OI
933-14)
Audit SDP
No
CCBApproval
J62OI 933-02, para 5
Approve SDP
SDP
SDP
SDP Guidance/Support
SDP Guidance/Support
YES
SDP
Plans
RequirementsPhase
Baseline SDP
Develop CMP Develop SQAP
UCSRD
PlanningPlanning Phase DetailPlanning Phase Detail
10 Nov 199927
IPT Project CM QA Test Mgt Team (J62)
Peer Review,J62OI 933-12,
para 6h.
Audit SRS
Update SDP,J62OI 933-13,
para 4c1
Write SRS J62OI933-9, para 4.2.2
Approve SRS
SRS
TestStart
Monitor PeerReview
Process IAWJ62OI 933-12
para 6a
RequirementsAnalysis J62OI933-12, para 6
RequirementsClarification
Monitorprocess, J62OI933-12, para 3f
SRS
Requirements ReviewJ62OI 933-12, para 6h
Requirements
Baseline SRSSRS
SRS
DesignPhase
VerifyTestability
UCSRD/SDP
Planning
SRS (draft)
SRS
SRS
OPR
Requirement Phase DetailRequirement Phase Detail
10 Nov 199928
Mother of All ChecklistsMother of All ChecklistsJ62 OI para Task Y, N, N/A Comments
Planning Phase.933-09 LC 4.1.3 Was the customer's request documented (corrective,
adaptive, perfective or new development)? 933-09 LC 4.1.2 Was economic and technical feasibility of proposed work
determined? 933-13 3.a.1. Did the Branch Chief designate a Software Project Leader?933-13 3.a.2. Did the Branch Chief assign members to the Software Project
Team?933-14 4.a. Were technical, cost and schedule risks identified and
assessed?933-14 4.b. Are risks documented on Risk Management Form (attach A)?933-14 5. Was the probability of each identified risk determined?933-14 6. For each identified risk, was the impact estimated and then
categorized?933-14 7. Are all risks prioritized by probability and estimated impact?933-14 7. If risks are identified as priority 1 through priority 6, are they
briefed at all management reviews?933-14 9. Did all Category A risks have a Risk Abatement Plan?933-14 3.b. Did the project manager assess all risks identified and then
develop and implement a risk abatement plan, if needed?933-09 LC Was a project plan and schedule developed?933-13 3.b.1. Did the Software Project Leader negotiate on behalf of the
Software Project Team (Negotiated commitments may be made to the customer, other software groups, or senior
933-03 3.c.3. Did SQA provide guidance on the preparation/review of each project's software documents and identify applicable standards, OIs and procedures?
933-13 3.b.2. Did the Software Project Leader develop the Software Project Plan (SPP)?
10 Nov 199929
SQA PlanningSQA Planning
•• J62 LevelJ62 Level
•• J621 LevelJ621 Level–– Tactical (White Paper) Tactical (White Paper)
–– MS ProjectsMS Projects
–– Strategic (SIP) Strategic (SIP)
•• Project (PProject (P--SQAP)SQAP)–– Independent of SDPIndependent of SDP
10 Nov 199930
Project Project -- SQA Plan (PSQA Plan (P--SQAP)SQAP)
•• CoordinationCoordination•• Purpose of PPurpose of P--SQAPSQAP•• Overview of Software ProjectOverview of Software Project•• Responsibilities (W/re SQA)Responsibilities (W/re SQA)•• Documentation ProducedDocumentation Produced•• Standards (OIs, MIL, Coding, Etc)Standards (OIs, MIL, Coding, Etc)•• Processes (to be Followed)Processes (to be Followed)•• Waivers (Expected, Suggested, Granted)Waivers (Expected, Suggested, Granted)•• SQA ActivitiesSQA Activities
10 Nov 199931
PP--SQAP (Cont.)SQAP (Cont.)•• Software Product InspectionsSoftware Product Inspections
•• Management Reviews/boardsManagement Reviews/boards
•• Problem Reporting and Corrective Action Problem Reporting and Corrective Action (Procedures)(Procedures)
•• Metrics to Track and CollectMetrics to Track and Collect
•• Risk ManagementRisk Management
•• Training (W/re to SQA)Training (W/re to SQA)
•• Tools, Techniques, and MethodologiesTools, Techniques, and Methodologies
10 Nov 199932
PP--SQAP (Cont.)SQAP (Cont.)
•• Records Collection, Maintenance, and Records Collection, Maintenance, and RetentionRetention
•• Resource RequirementsResource Requirements
•• AttachmentsAttachments–– List of Software ProductsList of Software Products–– Processes and Other InformationProcesses and Other Information–– SQA Activity ScheduleSQA Activity Schedule
10 Nov 199933
Project SQA ActivitiesProject SQA Activities
•• Plan (SQA and Project)Plan (SQA and Project)
•• AuditsAudits
•• ReviewsReviews
•• Facilitate InspectionsFacilitate Inspections
•• Track and ReportTrack and Report
10 Nov 199934
AuditsAudits
•• Product OrientedProduct Oriented
•• Compare Against a StandardCompare Against a Standard
•• Can be TediousCan be Tedious
•• Contractor Products Against CDRLContractor Products Against CDRL
•• Ensure Requirements TraceabilityEnsure Requirements Traceability
•• Use Natural Work ProductsUse Natural Work Products
10 Nov 199935
ReviewsReviews
•• Process OrientedProcess Oriented
•• MOACMOAC
•• Specific Process Checklist if AvailableSpecific Process Checklist if Available
•• Sometimes Just Being There (E.g. Sometimes Just Being There (E.g. testing) Makes a Differencetesting) Makes a Difference
10 Nov 199936
Inspections (SPIP)Inspections (SPIP)
•• Tom Gilb MethodologyTom Gilb Methodology
•• Do Audit Before InspectionDo Audit Before Inspection
•• Technical Vs NonTechnical Vs Non--technical technical
•• Track SeparatelyTrack Separately
•• Statistics (Metrics) Fed Back to Team Statistics (Metrics) Fed Back to Team Leads for Planning PurposesLeads for Planning Purposes
•• Trends Only Reported to MgmtTrends Only Reported to Mgmt
10 Nov 199937
SQA Chief ActivitiesSQA Chief Activities
•• MentorMentor
•• Branch Planning/ManagementBranch Planning/Management
•• Coordinate Between MembersCoordinate Between Members
•• Independent Council for Div ChiefIndependent Council for Div Chief
•• Transition Planning, Tracking, and Transition Planning, Tracking, and OversightOversight
10 Nov 199938
Transition PlanningTransition Planning
•• Responsible for Transition Planning for Responsible for Transition Planning for OutsourcingOutsourcing
•• Required Detailed Tracking of Every Required Detailed Tracking of Every Phase of Every ProjectPhase of Every Project
•• Reported Weekly at Staff MeetingReported Weekly at Staff Meeting
10 Nov 199939
MetricsMetrics
•• DeviationsDeviations–– From Audits and ReviewsFrom Audits and Reviews
–– Open, Closed, Late, by BranchOpen, Closed, Late, by Branch
•• Inspections (Defects)Inspections (Defects)–– By Product, Author, Team, Size, Month, ...By Product, Author, Team, Size, Month, ...
•• Transition PlanTransition Plan–– By Project by Week and Delivered Vs VerifiedBy Project by Week and Delivered Vs Verified
10 Nov 199940
Continuing SPIContinuing SPI
•• Only Able to Tweak OIsOnly Able to Tweak OIs
•• Added Release Review Board (933Added Release Review Board (933--25)25)–– Direct Involvement of Senior ManagementDirect Involvement of Senior Management
•• “Fine Tuned” Some More“Fine Tuned” Some More
•• Overhaul of Testing ProcessOverhaul of Testing Process
10 Nov 199941
Lessons Learned (Again?)Lessons Learned (Again?)•• AuditsAudits
–– Natural Work Products (Be Flexible)Natural Work Products (Be Flexible)
–– Ensure Requirements TraceabilityEnsure Requirements Traceability
•• ReviewsReviews–– Hard to DoHard to Do
•• InspectionsInspections–– Expensive (~25 Day Cycle)Expensive (~25 Day Cycle)
–– Only Inspect What Will Save You DownstreamOnly Inspect What Will Save You Downstream
10 Nov 199942
Lessons Learned (Cont.)Lessons Learned (Cont.)
•• ManagementManagement–– Implement at Project LevelImplement at Project Level
–– Interface at Senior LevelInterface at Senior Level
–– Educate Them and Get Them InvolvedEducate Them and Get Them Involved
•• Those Who Played With SQA Did WellThose Who Played With SQA Did Well
•• Plan at All LevelsPlan at All Levels
•• Sometimes it Really Came Down to Sometimes it Really Came Down to People Being the ProblemPeople Being the Problem
10 Nov 199943
ClosingClosing•• It Is My Belief That SPI and SQA Go Hand in It Is My Belief That SPI and SQA Go Hand in
HandHand–– Without SPI, SQA May Be Very SubjectiveWithout SPI, SQA May Be Very Subjective
–– Without SQA, SPI Will Remain in the Academic Without SQA, SPI Will Remain in the Academic WorldWorld
•• The SEPG Had Produced Many Assets for The SEPG Had Produced Many Assets for USSTRATCOMUSSTRATCOM
•• Through the Application of SWThrough the Application of SW--CMM Style CMM Style Project Level SQA, Those Assets Were Put to Project Level SQA, Those Assets Were Put to UseUse
10 Nov 199945
Contact InfoContact Info•• [email protected]@afpc.randolph.afaf.mil.mil
•• (210) 565(210) 565--3234 Work3234 Work
•• (210) 565(210) 565--3667 Fax3667 Fax
•• [email protected] (to Post Messages) [email protected] (to Post Messages) http://www.eGroups.com/group/spi/fullinfo.htmlhttp://www.eGroups.com/group/spi/fullinfo.html
•• [email protected] (to Post Messages) [email protected] (to Post Messages) http://www.eGroups.com/group/saspin/fullinfo.htmlhttp://www.eGroups.com/group/saspin/fullinfo.html