SQA SQA -- The Implementation The Implementation of SPIof SPI

A Case Study of Software Quality A Case Study of Software Quality Assurance at USSTRATCOM/J62 Assurance at USSTRATCOM/J62 Offutt AFB, NebraskaOffutt AFB, Nebraska

Presented By: Maj Andrew Boyd, USAFPresented By: Maj Andrew Boyd, USAF

10 Nov 19992

Who Am I?Who Am I?

10 Nov 19993

OverviewOverview•• BackgroundBackground

•• The AcademiaThe Academia (Organizational Process Assets)(Organizational Process Assets)

•• The ImplementationThe Implementation–– Getting StartedGetting Started

–– Organizing for SQAOrganizing for SQA–– Specific Roles, Responsibilities, Assets, Plans, Specific Roles, Responsibilities, Assets, Plans,

Activities, and RelationshipsActivities, and Relationships– Metrics Collected and Reported

•• Continuing SPI Without an SEPGContinuing SPI Without an SEPG

•• Lessons Learned (Again!)Lessons Learned (Again!)

10 Nov 19994

BackgroundBackground

•• DoD Has Unique Software ChallengesDoD Has Unique Software Challenges

•• 3 Separate CBA3 Separate CBA--IPIs @ USSTRATCOMIPIs @ USSTRATCOM

•• Migration/Modernization EffortMigration/Modernization Effort

•• Consolidated 4 Complete Software Consolidated 4 Complete Software DivisionsDivisions

•• J62 Level SEPGJ62 Level SEPG

•• Lots of AssetsLots of Assets

10 Nov 19995

US Strategic Command US Strategic Command (Joint)(Joint)CJCS

J0Command Staff

J1Man/Pers

J2Intel

J3/4Ops/Logistics

J5Strategic Plans

J60Staff

J61Man/Pers

J62Intel SW Supt

J63/4C2 SW Supt

J65SWPS SW Supt

J66Program Mgmt

J67Training

J68LAN Mgmt

J6Comm/Comp

*

USSTRATCOMCINCSTRAT

****

SECDEF

10 Nov 19996

Software Support Division Software Support Division (J62)(J62)

School HouseJ67

Front Office Staff

Assessments

Training

Standards

J621Process Mgmt Branch

Test

SCM

SQA

Intel CM/QA/Test

J622Integration Branch

Command Post

...

J623C2 SW Branch

Systems

Development

...

J624Intel SW Branch

Targeting

Production

Data

...

J625SWPS SW Branch

...

J626LAN Mgmt Branch

J62Division

10 Nov 19997

Organizational Process AssetsOrganizational Process Assets

•• USSTRATCOM LevelUSSTRATCOM Level–– Administrative Instructions (E.G. SAI 709Administrative Instructions (E.G. SAI 709--1)1)

•• J6 LevelJ6 Level–– Process Definition Guide (ETVX)Process Definition Guide (ETVX)–– Lifecycle Guide (Waterfall, Evolutionary, Etc)Lifecycle Guide (Waterfall, Evolutionary, Etc)–– Process Asset Library GuideProcess Asset Library Guide–– SEPG Chartering GuideSEPG Chartering Guide

•• J62 LevelJ62 Level–– 933 Series Operating Instructions (OI)933 Series Operating Instructions (OI)

10 Nov 19998

Operating Instruction (OI)Operating Instruction (OI)

•• ReferencesReferences

•• PurposePurpose

•• Applicability (Scope)Applicability (Scope)

•• ResponsibilitiesResponsibilities

•• Resources, Training, Reviews, TermsResources, Training, Reviews, Terms

•• Area Specific Activities/productsArea Specific Activities/products

10 Nov 19999

J62 Software Process J62 Software Process StandardsStandards

OI Name Date933-01 Division Software OI Management 22 Aug 97933-02 J62 Software Development and Maintenance Policy 23 May 96933-03 J62 Standard for Software Quality Assurance 10 Jun 96933-04 J62 Standard for Software Configuration Management 27 Jun 96933-05 J62 Standard for Test 19 Jul 96933-06 J62 Enterprise Improvement Process Guide 13 May 96933-07 J62 Terms and Definitions Repository 7 Mar 97933-09 J62 Standard for Life Cycles 6 Jun 96933-10 J62 Standard for Contract Management/Monitoring 6 Aug 96933-11 J62 Standard for Software Metrics 29 Jul 96933-12 J62 Standard for Software Requirements Management 23 Aug 96933-13 J62 Standard for Project Planning 24 Oct 96933-14 J62 Standard for Risk Management 23 Jan 98933-15 J62 Standard for Peer Reviews 18 Sep 96933-16 J62 Standard for Management Reviews 12 Jun 97

10 Nov 199910

J62 Software Process J62 Software Process Standards (Cont.)Standards (Cont.)

OI Name Date933-17 J62 Standard for Assessments 12 Sep 96933-18 J62 Standard for Software Design 4 Nov 96933-19 J62 Standard for Software Tools Management 3 Mar 97933-20 J62 Standard for Software Coding 6 Nov 96933-21 J62 Standard for Resources 13 Dec 96933-22 Process Asset Library (PAL) 25 Jun 97933-23 J62 Standard for Project Documentation 6 Jul 98933-24 J62 Standard for Post Mortem 22 Sep 97933-25 J62 Software Release Review Board

Checklist updated 6 Jul 9831 Mar 98

233-01 J62 Standard for Training 5 Sep 97

10 Nov 199911

Assets (Cont.)Assets (Cont.)

•• Software Development Process FlowSoftware Development Process Flow

•• Standards (Documentation)Standards (Documentation)–– MilMil--StdStd--498 (Software Development and 498 (Software Development and

Documentation) Now JDocumentation) Now J--STDSTD--016016•• Data Item Descriptions (Some Tailored)Data Item Descriptions (Some Tailored)

–– SSS, SDP, SRS, SSDD, SDD, DBDD, IRS, STP, STD, Etc.SSS, SDP, SRS, SSDD, SDD, DBDD, IRS, STP, STD, Etc.

•• Guidebook Guidebook

–– IEEE Software Engineering CompendiumIEEE Software Engineering Compendium

•• SWSW--CMMCMM®® V 1.1V 1.1

10 Nov 199912

Assets (Cont.)Assets (Cont.)

•• Branch LevelBranch Level–– Software Development Guide (2)Software Development Guide (2)

–– SQA Guide (2)SQA Guide (2)

–– Software Configuration Mgmt Guide (2)Software Configuration Mgmt Guide (2)

–– Program Management GuideProgram Management Guide

–– Test Process GuideTest Process Guide

–– Other Stuff From Earlier Days As Separate Other Stuff From Earlier Days As Separate HouseHouse

10 Nov 199913

Software Development FlowSoftware Development Flow

Transition…Transition…

Questions?Questions?

10 Nov 199915

My View of SQAMy View of SQA

•• Seen It Done Various Ways Seen It Done Various Ways

•• Ensure ComplianceEnsure Compliance

•• Objective Vs Subjective Objective Vs Subjective

•• SWSW--CMMCMM®®

–– Process AssuranceProcess Assurance

–– Product AssuranceProduct Assurance

–– Track Deviations and Report NonTrack Deviations and Report Non--compliancecompliance

•• Senior Management’s Eyes and EarsSenior Management’s Eyes and Ears

10 Nov 199916

ProposalProposal

•• White Paper (SQA 101)White Paper (SQA 101)–– CredentialsCredentials

–– Key Objectives of QA (Used Generically)Key Objectives of QA (Used Generically)•• Reviews, AuditsReviews, Audits

•• Deviation Tracking and NonDeviation Tracking and Non--compliance Issuescompliance Issues

•• Autonomy, Level of AuthorityAutonomy, Level of Authority

•• Current SituationCurrent Situation

•• Recommendation (Org Level, Mission Alignment, Recommendation (Org Level, Mission Alignment, Responsibilities)Responsibilities)

10 Nov 199917

Initial SQA ResourcesInitial SQA Resources

•• PersonnelPersonnel–– 3 KTRs3 KTRs

–– 3 GS3 GS

–– 2 Military2 Military

•• Assets (OI 933Assets (OI 933--03, SPIP, and Work 03, SPIP, and Work Product Compliance Audit)Product Compliance Audit)

•• J62 SQA Plan (Draft)J62 SQA Plan (Draft)

10 Nov 199918

Some ChallengesSome Challenges

•• Lack of SQA TrainingLack of SQA Training

•• Product Vs Process FocusProduct Vs Process Focus

•• Management Didn’t Understand SQAManagement Didn’t Understand SQA

•• Conflict of InterestConflict of Interest–– Peer to Peer Relationship Peer to Peer Relationship

–– SQA Not a Priority for Branch ChiefSQA Not a Priority for Branch Chief

–– Personal QAPersonal QA

10 Nov 199919

More ChallengesMore Challenges

•• Disgruntled MembersDisgruntled Members

•• “We’re Being Outsourced So Who “We’re Being Outsourced So Who Cares…”Cares…”

•• Single Domain FocusedSingle Domain Focused

•• SQA Plan Too High LevelSQA Plan Too High Level

•• Most of J62 Unaware of Assets (or Told Most of J62 Unaware of Assets (or Told Not to Use)Not to Use)

10 Nov 199920

First StepsFirst Steps

•• Group OffsiteGroup Offsite

•• Individual SessionsIndividual Sessions–– Role, ExpectationsRole, Expectations

•• Tailored an SQA Course (J67)Tailored an SQA Course (J67)

•• “Borrowed” Deviation Tracking Database “Borrowed” Deviation Tracking Database From Intel FolksFrom Intel Folks

•• Provided MS Project, and Access TrainingProvided MS Project, and Access Training

10 Nov 199921

Organizing for SQAOrganizing for SQA

•• J62 After Merging 4 Software HousesJ62 After Merging 4 Software Houses

•• “Forced” a Change“Forced” a Change–– Did Audit of CM and Test SectionsDid Audit of CM and Test Sections

•• After “Fine Tuning”After “Fine Tuning”–– Report to Highest LevelReport to Highest Level

–– No Longer Conflict of InterestNo Longer Conflict of Interest

–– Lost Some Resources in ShuffleLost Some Resources in Shuffle

10 Nov 199922

Software Support Division Software Support Division (J62) After “Fine Tuning”(J62) After “Fine Tuning”

School House(J67)

Intel SW Support(J25)Front Office Staff SQA

Targeting

Production

Data

...

Lt ColSWPS Branch

Command Post

...

MajC2 Branch

...

MajLAN Branch

Test

SCM

Lt ColIntegration Branch

ColonelDivision Chief

10 Nov 199923

SQA AssetsSQA Assets

•• Vol. 1 SQA Master Audit Review ProcessVol. 1 SQA Master Audit Review Process

•• Vol. 2 SPIP ManualVol. 2 SPIP Manual

•• Vol. 3 SQA Survey ProcessVol. 3 SQA Survey Process

•• Vol. 4 SQA Deviation Tracking System Vol. 4 SQA Deviation Tracking System ManualManual

•• Vol. 5 SQA Inspection TrackingVol. 5 SQA Inspection Tracking

10 Nov 199924

SQA ACTION ITEM REPORTAudit Action Item Control Date Prepared: 04-Nov-1997 Followup Date:

ICR006 - 001 Date Notified: Suspense Date:23-Nov-1997

Initiator: Office: Phone:

Mr. McClellanTask Directed To Office: Phone:Maj Boyd J6222 294-3976

Software Project: Document: Section/CSCI: Audit/Review:

J621 Software Product J6222 Process Review

Deviation Description:How are issues identified during the review resolved? Currently, no follow up review is performed onissues/concerns raised in a previous inspection (i.e. don't see formal follow up process).

Action Required:

Resolution:It's the editor/author's responsibility to resolv issues. However, the team lead cannot exit the process ifthere are know issues. It's not part of this process to gain concurrence of the document. The appropriate review (e.g. SRR, PDR, CDR) is where this should be done

Date: Approval Authority: Office: Phone: Signature:21-Nov-97

CoordinationDate: Name of Coordinator: Office: Phone: Signature:

SQA Action Item ReportSQA Action Item Report

10 Nov 199925

SQA Assets (Cont.)SQA Assets (Cont.)

•• Vol. 6 OI Waiver ProcessVol. 6 OI Waiver Process

•• Vol. 7 SQA Acronym Definition ManualVol. 7 SQA Acronym Definition Manual

•• Vol. 8 PVol. 8 P--SQAP Building ProcessSQAP Building Process

•• Vol. 9 Peer Review ProcessVol. 9 Peer Review Process

•• Integrated Process Flow (Detailed by Integrated Process Flow (Detailed by Phase)Phase)

•• Mother of all Checklists (MOAC)Mother of all Checklists (MOAC)

10 Nov 199926

IPT Project CM QA Test Mgt Team (J62)

<STARTPROCESS>Initiate SOW

J62 OI 933-13,para 4a

Review SOWJ62OI 933-02,

para 5

Develop ProjectPlan (SDP)

J62OI 933-13,para 4c

(include riskanalysis, J62OI

933-14)

Audit SDP

No

CCBApproval

J62OI 933-02, para 5

Approve SDP

SDP

SDP

SDP Guidance/Support

SDP Guidance/Support

YES

SDP

Plans

RequirementsPhase

Baseline SDP

Develop CMP Develop SQAP

UCSRD

PlanningPlanning Phase DetailPlanning Phase Detail

10 Nov 199927

IPT Project CM QA Test Mgt Team (J62)

Peer Review,J62OI 933-12,

para 6h.

Audit SRS

Update SDP,J62OI 933-13,

para 4c1

Write SRS J62OI933-9, para 4.2.2

Approve SRS

SRS

TestStart

Monitor PeerReview

Process IAWJ62OI 933-12

para 6a

RequirementsAnalysis J62OI933-12, para 6

RequirementsClarification

Monitorprocess, J62OI933-12, para 3f

SRS

Requirements ReviewJ62OI 933-12, para 6h

Requirements

Baseline SRSSRS

SRS

DesignPhase

VerifyTestability

UCSRD/SDP

Planning

SRS (draft)

SRS

SRS

OPR

Requirement Phase DetailRequirement Phase Detail

10 Nov 199928

Mother of All ChecklistsMother of All ChecklistsJ62 OI para Task Y, N, N/A Comments

Planning Phase.933-09 LC 4.1.3 Was the customer's request documented (corrective,

adaptive, perfective or new development)? 933-09 LC 4.1.2 Was economic and technical feasibility of proposed work

determined? 933-13 3.a.1. Did the Branch Chief designate a Software Project Leader?933-13 3.a.2. Did the Branch Chief assign members to the Software Project

Team?933-14 4.a. Were technical, cost and schedule risks identified and

assessed?933-14 4.b. Are risks documented on Risk Management Form (attach A)?933-14 5. Was the probability of each identified risk determined?933-14 6. For each identified risk, was the impact estimated and then

categorized?933-14 7. Are all risks prioritized by probability and estimated impact?933-14 7. If risks are identified as priority 1 through priority 6, are they

briefed at all management reviews?933-14 9. Did all Category A risks have a Risk Abatement Plan?933-14 3.b. Did the project manager assess all risks identified and then

develop and implement a risk abatement plan, if needed?933-09 LC Was a project plan and schedule developed?933-13 3.b.1. Did the Software Project Leader negotiate on behalf of the

Software Project Team (Negotiated commitments may be made to the customer, other software groups, or senior

933-03 3.c.3. Did SQA provide guidance on the preparation/review of each project's software documents and identify applicable standards, OIs and procedures?

933-13 3.b.2. Did the Software Project Leader develop the Software Project Plan (SPP)?

10 Nov 199929

SQA PlanningSQA Planning

•• J62 LevelJ62 Level

•• J621 LevelJ621 Level–– Tactical (White Paper) Tactical (White Paper)

–– MS ProjectsMS Projects

–– Strategic (SIP) Strategic (SIP)

•• Project (PProject (P--SQAP)SQAP)–– Independent of SDPIndependent of SDP

10 Nov 199930

Project Project -- SQA Plan (PSQA Plan (P--SQAP)SQAP)

•• CoordinationCoordination•• Purpose of PPurpose of P--SQAPSQAP•• Overview of Software ProjectOverview of Software Project•• Responsibilities (W/re SQA)Responsibilities (W/re SQA)•• Documentation ProducedDocumentation Produced•• Standards (OIs, MIL, Coding, Etc)Standards (OIs, MIL, Coding, Etc)•• Processes (to be Followed)Processes (to be Followed)•• Waivers (Expected, Suggested, Granted)Waivers (Expected, Suggested, Granted)•• SQA ActivitiesSQA Activities

10 Nov 199931

PP--SQAP (Cont.)SQAP (Cont.)•• Software Product InspectionsSoftware Product Inspections

•• Management Reviews/boardsManagement Reviews/boards

•• Problem Reporting and Corrective Action Problem Reporting and Corrective Action (Procedures)(Procedures)

•• Metrics to Track and CollectMetrics to Track and Collect

•• Risk ManagementRisk Management

•• Training (W/re to SQA)Training (W/re to SQA)

•• Tools, Techniques, and MethodologiesTools, Techniques, and Methodologies

10 Nov 199932

PP--SQAP (Cont.)SQAP (Cont.)

•• Records Collection, Maintenance, and Records Collection, Maintenance, and RetentionRetention

•• Resource RequirementsResource Requirements

•• AttachmentsAttachments–– List of Software ProductsList of Software Products–– Processes and Other InformationProcesses and Other Information–– SQA Activity ScheduleSQA Activity Schedule

10 Nov 199933

Project SQA ActivitiesProject SQA Activities

•• Plan (SQA and Project)Plan (SQA and Project)

•• AuditsAudits

•• ReviewsReviews

•• Facilitate InspectionsFacilitate Inspections

•• Track and ReportTrack and Report

10 Nov 199934

AuditsAudits

•• Product OrientedProduct Oriented

•• Compare Against a StandardCompare Against a Standard

•• Can be TediousCan be Tedious

•• Contractor Products Against CDRLContractor Products Against CDRL

•• Ensure Requirements TraceabilityEnsure Requirements Traceability

•• Use Natural Work ProductsUse Natural Work Products

10 Nov 199935

ReviewsReviews

•• Process OrientedProcess Oriented

•• MOACMOAC

•• Specific Process Checklist if AvailableSpecific Process Checklist if Available

•• Sometimes Just Being There (E.g. Sometimes Just Being There (E.g. testing) Makes a Differencetesting) Makes a Difference

10 Nov 199936

Inspections (SPIP)Inspections (SPIP)

•• Tom Gilb MethodologyTom Gilb Methodology

•• Do Audit Before InspectionDo Audit Before Inspection

•• Technical Vs NonTechnical Vs Non--technical technical

•• Track SeparatelyTrack Separately

•• Statistics (Metrics) Fed Back to Team Statistics (Metrics) Fed Back to Team Leads for Planning PurposesLeads for Planning Purposes

•• Trends Only Reported to MgmtTrends Only Reported to Mgmt

10 Nov 199937

SQA Chief ActivitiesSQA Chief Activities

•• MentorMentor

•• Branch Planning/ManagementBranch Planning/Management

•• Coordinate Between MembersCoordinate Between Members

•• Independent Council for Div ChiefIndependent Council for Div Chief

•• Transition Planning, Tracking, and Transition Planning, Tracking, and OversightOversight

10 Nov 199938

Transition PlanningTransition Planning

•• Responsible for Transition Planning for Responsible for Transition Planning for OutsourcingOutsourcing

•• Required Detailed Tracking of Every Required Detailed Tracking of Every Phase of Every ProjectPhase of Every Project

•• Reported Weekly at Staff MeetingReported Weekly at Staff Meeting

10 Nov 199939

MetricsMetrics

•• DeviationsDeviations–– From Audits and ReviewsFrom Audits and Reviews

–– Open, Closed, Late, by BranchOpen, Closed, Late, by Branch

•• Inspections (Defects)Inspections (Defects)–– By Product, Author, Team, Size, Month, ...By Product, Author, Team, Size, Month, ...

•• Transition PlanTransition Plan–– By Project by Week and Delivered Vs VerifiedBy Project by Week and Delivered Vs Verified

10 Nov 199940

Continuing SPIContinuing SPI

•• Only Able to Tweak OIsOnly Able to Tweak OIs

•• Added Release Review Board (933Added Release Review Board (933--25)25)–– Direct Involvement of Senior ManagementDirect Involvement of Senior Management

•• “Fine Tuned” Some More“Fine Tuned” Some More

•• Overhaul of Testing ProcessOverhaul of Testing Process

10 Nov 199941

Lessons Learned (Again?)Lessons Learned (Again?)•• AuditsAudits

–– Natural Work Products (Be Flexible)Natural Work Products (Be Flexible)

–– Ensure Requirements TraceabilityEnsure Requirements Traceability

•• ReviewsReviews–– Hard to DoHard to Do

•• InspectionsInspections–– Expensive (~25 Day Cycle)Expensive (~25 Day Cycle)

–– Only Inspect What Will Save You DownstreamOnly Inspect What Will Save You Downstream

10 Nov 199942

Lessons Learned (Cont.)Lessons Learned (Cont.)

•• ManagementManagement–– Implement at Project LevelImplement at Project Level

–– Interface at Senior LevelInterface at Senior Level

–– Educate Them and Get Them InvolvedEducate Them and Get Them Involved

•• Those Who Played With SQA Did WellThose Who Played With SQA Did Well

•• Plan at All LevelsPlan at All Levels

•• Sometimes it Really Came Down to Sometimes it Really Came Down to People Being the ProblemPeople Being the Problem

10 Nov 199943

ClosingClosing•• It Is My Belief That SPI and SQA Go Hand in It Is My Belief That SPI and SQA Go Hand in

HandHand–– Without SPI, SQA May Be Very SubjectiveWithout SPI, SQA May Be Very Subjective

–– Without SQA, SPI Will Remain in the Academic Without SQA, SPI Will Remain in the Academic WorldWorld

•• The SEPG Had Produced Many Assets for The SEPG Had Produced Many Assets for USSTRATCOMUSSTRATCOM

•• Through the Application of SWThrough the Application of SW--CMM Style CMM Style Project Level SQA, Those Assets Were Put to Project Level SQA, Those Assets Were Put to UseUse

Questions?Questions?

Comments?Comments?

10 Nov 199945

Contact InfoContact Info•• Andrew.Boyd@afpc.randolph.Andrew.Boyd@afpc.randolph.afaf.mil.mil

•• (210) 565(210) 565--3234 Work3234 Work

•• (210) 565(210) 565--3667 Fax3667 Fax

•• SPI@eGroups.com (to Post Messages) SPI@eGroups.com (to Post Messages) http://www.eGroups.com/group/spi/fullinfo.htmlhttp://www.eGroups.com/group/spi/fullinfo.html

•• SASPIN@eGroups.com (to Post Messages) SASPIN@eGroups.com (to Post Messages) http://www.eGroups.com/group/saspin/fullinfo.htmlhttp://www.eGroups.com/group/saspin/fullinfo.html