Spa
cecr
aft S
oftw
are
Eng
inee
ring
Bra
nch/
ER
6
The CMMI Modelsin
Oversight of Space Flight Software Development
Liz StrassnerLead, Software Process & Process Management Group
David RetherfordSr. Software and Systems Engineer/ERC
Spacecraft Software Engineering BranchNASA/JSC
Used with permission
Spa
cecr
aft S
oftw
are
Eng
inee
ring
Bra
nch/
ER
6Agenda
Introduction Selection of CMMI Models for Spacecraft Software Engineering Branch NASA Surveillance Strategies Project Orion Surveillance Strategies Orion Software Oversight
CMMI Maturity Level 2 Rating on Oversight Who are we What we did to earn CMMI Level 2 Trials & Tribulations of Software Oversight What we are doing now
2/22/20102
Spa
cecr
aft S
oftw
are
Eng
inee
ring
Bra
nch/
ER
6Spacecraft Software Engineering & CMMI Models
We evaluated the three CMMI models (CMMI-ACQ, CMMI-SVC, & CMMI-DEV) for applicability to our work CMMI-SVC was discovered to be completely non-applicable
IT support help desk; Pizza delivery service Oversight of a contract doesn’t meet the intent of “service” in this model
CMMI-ACQ was reviewed for applicability in a SCAMPI B NASA/JSC organizational structure caused some goals to be completely out of
scope of the organization We were fully compliant with goals and practices that were in scope of the organization
The model does not allow goals to be out of scope
CMMI-DEV Software development work was obviously in scope Software oversight is also in scope, but needs to be looked at from a systems
engineering perspective Software oversight on a 30 year project is never-ending so needs to be broken
into smaller projects
2/22/20103
Spa
cecr
aft S
oftw
are
Eng
inee
ring
Bra
nch/
ER
6Selection of CMMI Model & Projects
Based on our experience with the SCAMPI B, we chose to go for a CMMI Maturity Level 2 rating under the CMMI-DEV model
Selecting projects was an interesting experience Software Development Projects Many active software development projects to select from A long history of flight software development projects from the former
Flight Software Branch that was merged into Spacecraft Software Engineering Branch
Software Oversight Role is significantly different than in the Shuttle or ISS Programs Lead appraiser insisted that we have at least one project in this area
because of the large focus in the organization on oversight Chose to create sub-projects out of milestone reviews Software Requirements Review for SCAMPI B “System PDR & Software Spiral Review" for SCAMPI A
2/22/20104
Spa
cecr
aft S
oftw
are
Eng
inee
ring
Bra
nch/
ER
6NASA Surveillance Strategies (From COTR Refresher Training 2-18-2008)
Basic strategies: insight, oversight, or hybrid Insight: relies on performance requirements & metrics. Use a
minimum set of product/process data to give adequate visibility into the integrity of the product/process. Contractor assumes more responsibility & accountability Government steps back, evaluates deliverables & existing contractor
processes—not day-to-day close monitoring Appropriate strategy for contracts with little cost risk, clearly-defined
products/services, & confidence in proven contractor performance Oversight: rely on NASA-imposed product specification’s & process
controls, MIL standards, mandatory inspection points, etc. Intrusive monitoring at contractor’s plant, & in-line NASA involvement in contract work. Appropriate strategy when NASA is assuming the liability for
performance or quality; when we determine oversight is necessary to mitigate performance risk; or when the contractor is unproven
2/22/20105
Spa
cecr
aft S
oftw
are
Eng
inee
ring
Bra
nch/
ER
6Project Orion Surveillance
The Orion Contract as a whole uses the hybrid approach to surveillance with pre-declared oversight in high risk areas Building a space vehicle is high risk so the contract is Cost Plus Award
Fee (CPAF) type Very extensive DRD requirements for insight Software was pre-declared as a major risk item by the Agency, so
oversight of software was planned and scheduled Joint NASA-Contractor Risk Board
Insight for Software All of the NPR 7150.2 chapter 5 requirements exist as individual DRDs CMMI Maturity Level 3 was levied on all software development
organizations
2/22/20106
Spa
cecr
aft S
oftw
are
Eng
inee
ring
Bra
nch/
ER
6Software Oversight of Orion
Oversight for Software NASA co-chairs all boards and panels Every Contractor functional area lead has a NASA counterpart NASA participates in ALL software working groups NASA has signature authority on all major software deliveries including
the Software Development Plan NASA has an internal software test environment NPR 7150.2 NASA Software Engineering levied on contract in its
entirety NASA Standard 7009(I) Models & Simulations Standard levied on
contract in its entirety NASA Software Assurance and Safety Standards levied on contract in a
“cut and paste” method into the contract (DRD CEV-S-001) Constellation Software documents also levied by the “cut and paste”
method or by reference to specific sections or paragraphs of the CxPdocument
2/22/20107
Spa
cecr
aft S
oftw
are
Eng
inee
ring
Bra
nch/
ER
6
Spacecraft Software Engineering Branch & CMMI Level 2
2/22/20108
Spa
cecr
aft S
oftw
are
Eng
inee
ring
Bra
nch/
ER
6Spacecraft Software Engineering Branch
Spacecraft Software Engineering Branch
Process & Process Management Group Test & Verification Group GFE Group Technology Group Systems Engineering Group
Orion Flight Software System Manager
Orion Recon System ManagerOrion Vehicle Systems Management Manager
Altair Lead
2/22/20109
Spa
cecr
aft S
oftw
are
Eng
inee
ring
Bra
nch/
ER
6Orion Software Team Interfaces
2/22/201010
Orion Software
Team
Prime Contractor
Subcontractor
Orion Project Office
Constellation
Subcontractor
Subcontractor
Subcontractor
SubcontractorSubcontractor
Customer - Provider
Stakeholder – Provider Rep
Provider - Customer
Provider - CustomerCustomer Rep -Provider
Spa
cecr
aft S
oftw
are
Eng
inee
ring
Bra
nch/
ER
6NASA Orion Software Team Organization
Orion Software Manger is part of the Orion Project Office Everyone below that level is matrixed out of Engineering organizations
at JSC, Ames, GRC, and LaRC Frank Delgado/ER6 is the Flight Software System Manager and leads
the multicenter flight software team Neil Townsend/ER6 is the Reconfiguration System Manager and leads
the multicenter software process and process management team The Spacecraft Software Engineering Branch and the Orion Software
Team are approximately the same size and have a large overlap.
2/22/201011
Orion Software Team
Spacecraft Software
Engineering Branch
Spa
cecr
aft S
oftw
are
Eng
inee
ring
Bra
nch/
ER
6Year One – Baby Steps
Organization was very small Only function was Orion oversight Responsible for Orion Vehicle System Management (Class A) Responsible for in-house flight software development environment for
all Orion software oversight (Class D & E) Interviewed all employees to find out how they did their work Wrote processes based on the “how they did their work” Assumed no higher level organizational processes affected oversight
At a conceptual level, we did not have any problem mapping the CMMI Specific Goals and Practices to processes
Late in the year, Orion Software Manager function moved here in addition to Vehicle System Manager and the branch grew
2/22/201012
Spa
cecr
aft S
oftw
are
Eng
inee
ring
Bra
nch/
ER
6Year Two – Chaos Reigns
Former Flight Software Branch merged into the branch as one of many steps JSC Engineering took to focus software Brought in several GFE Flight Software Development projects, most in
maintenance and operations phase already CMMI Level 2 compliant Consolidated all JSC Orion software oversight personnel Brought in methodology for meeting Generic Goals and Practices useful
for all projects GFE Projects must follow extensive higher level organizational
processes New Branch Chief assigned Perfect storm Policies and processes for oversight crashed into GFE flight software
policies & processes Previous branch-only view had to be extended to reflect division and
directorate infrastructure New branch chief had new goals and expectations
2/22/201013
Spa
cecr
aft S
oftw
are
Eng
inee
ring
Bra
nch/
ER
6Year Two – Coming out of Chaos
CMMI Consultation Bill Pierce provided advise on how to structure projects for oversight and
methods to apply CMMI practices to an oversight project Selected projects: 1) Class A development, 2) Class D
development/acquisition, 3) Class A-E oversight The former Intelligent Systems Branch was merged into Spacecraft
Software Engineering Branch Primarily Class E research and technology software
SCAMPI B Newly acquired projects from the recent merger were excluded Uncovered a problem with PPQA Discovered that we still had disconnects in applicability of division and
directorate policies and processes Determined that our SwRR Oversight project was mis-scoped
Appraiser direction was to do better for PDR
2/22/201014
Spa
cecr
aft S
oftw
are
Eng
inee
ring
Bra
nch/
ER
6Year 3 – Success!
Solved PPQA problem temporarily by returning to original resources Orion PDR slipped significantly, so best we could do was “plan” for
PDR without any action past that Appraiser decided to merge our SwRR Project and our start of the PDR
Project into a single “Software Oversight” project for the purpose of the SCAMPI A
Allowed us to show that we learned from SwRR and incorporated those lessons into the PDR planning activities
Since all our development projects were new, we had to pull in an old (CMMI L2 rated) project that was in maintenance to ensure end of lifecycle coverage This end of lifecycle coverage would be a problem for any new
organization
2/22/201015
Spa
cecr
aft S
oftw
are
Eng
inee
ring
Bra
nch/
ER
6
Trials & Tribulations of planningSoftware Oversight
Spa
cecr
aft S
oftw
are
Eng
inee
ring
Bra
nch/
ER
6Orion Software Spiral Review – Overview
Meet diverse set of goals & requirements Orion project SW Management Plan (CxP 72099) Need to plan component-level PDR activities for LM SW Spiral
delivery Follow general flow of Orion system PDR (CxP 72212)
Spacecraft Software Engineering Branch Demonstrate an oversight project meeting CMMI ML 2
Agency Software engineering requirements (NPR 7150.2) Systems/project requirements (NPR 7123.1)
CMMI Dev 1.2 Demonstrate that oversight project can meet ML 2 SPs & GPs
2/22/201017
Spa
cecr
aft S
oftw
are
Eng
inee
ring
Bra
nch/
ER
6Software Oversight – Changes in Attitude
Requires an orthogonal change of view of Requirements Configuration Management Evaluation & assessment vs. technical development
Requirements Technical requirements of contractor product (e.g. spacecraft) differ from
oversight As part of project tasks may influence or evaluate technical requirements Actual oversight project requirement is to review products (e.g. SRS) ,
generate comments Separation of concerns Technical requirement development vs. oversight activities
2/22/201018
Spa
cecr
aft S
oftw
are
Eng
inee
ring
Bra
nch/
ER
6SW PDR Project Plan Impact Sources
2/22/201019
Spa
cecr
aft S
oftw
are
Eng
inee
ring
Bra
nch/
ER
6Orion PDR Flow – High Level Overview
PDR Plan & Kickoff•PDR Ground Rules & POD•PDR Review Process•PDR RID Process
Subsystem Design Reviews (SSDR)•Subsystem Specifications & IRDs•Subsystem Design Data Books•Subsystem Specific Design & Requirements Review Presentations
System & Module Review•Subsystem Rollups•System Specifications and Design Review Presentations
System PDR•RID Process•RID Screening, Review, & Dispositions•PDR Pre-board & Board
Software PDR•Specification & Design Tech/Peer Review Process•SW Artifact Release and RID Process•SW RID Screening, Review, & Disposition•SW PDR Pre-board & Board
2/22/201020
Spa
cecr
aft S
oftw
are
Eng
inee
ring
Bra
nch/
ER
6Oversight Project – CMMI Model
CMMI Maturity Level 2 process areas for oversight 7 PA’s – PP, PMC, REQM, CM, PPQA, MA, SAM
CMMI Maturity Level 2 process area emphasis on managing project Planning project, process, and products Monitor & control project
2/22/201021
Spa
cecr
aft S
oftw
are
Eng
inee
ring
Bra
nch/
ER
6Spiral Review Project – CMMI Process Areas
Project Planning (PP) Develop project life cycle and
process Document in project plan
Project Monitoring and Control (PMC) Track team participation in
contractor reviews Team assessment of reviews
Requirements Management (REQM) Screen/review requests from
Project Office or Branch Management
Product & Process Quality Assurance (PPQA) Evaluate team process
compliance Evaluate team assessment
products Configuration Management
(CM) NASA ICE/Windchill area for
document storage Measurement and Analysis
(MA) Metrics for effort and effort per
review/document Supplier Agreement
Management (SAM) Contractors part of Orion SW
team Use division process for
engineering service support
2/22/201022
Spa
cecr
aft S
oftw
are
Eng
inee
ring
Bra
nch/
ER
6Basic Software Oversight Requirements
Not the same as the product technical requirements What ‘we’ must do to demonstrate Contractor development of software technical baseline Accomplished proper oversight of contractor Assure adequacy of contractor efforts and products
As an oversight team we must Review contractor technical efforts (requirements & design) Generate comments/RIDs Attend technical and peer reviews Evaluate software product quality and maturity Within known (agreed to) constraints
Basically – our requirements are: To provide a set of software engineering services to Orion project Engineering involvement and assessment
2/22/201023
Spa
cecr
aft S
oftw
are
Eng
inee
ring
Bra
nch/
ER
6Software PDR Oversight Lifecycle
Not traditional SW lifecycle – no actual SW being produced Formulate and initiate the project/process (Inception) Execute the plan/process (Execution) Finish the project & baseline (Closeout)
Product/artifact data Assessment/evaluation vs. SRS/IRD
Configuration Management Oversight project artifacts, not contractor technical artifacts
2/22/201024
Spa
cecr
aft S
oftw
are
Eng
inee
ring
Bra
nch/
ER
6Oversight Phases
InceptionDefine project lifecycle and
organizationDefine review process(es) and
boardsDefine NASA technical and
management rolesGenerate project planAnd away we go…
ExecutionMonitor contractor software
development effortsReview artifacts, generate
comments, attend reviews, attend disposition and solution meetings
CloseoutFormal review releaseRID inputsRID review/screeningRID dispositionsSW Spiral Pre-boardSW Spiral boardArtifact update/re-release
2/22/201025
Spa
cecr
aft S
oftw
are
Eng
inee
ring
Bra
nch/
ER
6Process Area Considerations
PPQA No software artifacts (e.g. SRS, SDD,
source code, etc.) Process and assessment report
focused Use outside organization – provide
independent assessment
M&ANo technical measures (e.g.
SLOCs, etc.)Effort/product based Hrs/review No. comments
Assessments Completion Quality
REQM Oversight requirements NOT the
same as technical product Activity and evaluation/assessment
based “The software shall do this …..” vs.
“The project shall participate in technical reviews.”
Control/manage between Orion SW Mgr & Av&SW Project Office
PPProject “within a project”Predetermined schedule Orion project/contractor level
2/22/201026
Spa
cecr
aft S
oftw
are
Eng
inee
ring
Bra
nch/
ER
6SW Spiral Review – CMMI Appraisal Nuggets
Double vision syndromeEasy to get cross-eyed over
apparent identical areas between LM and NASA oversight project CMMI model vs. NASA
engineering focusEx.: CM – whose CM? LM
Project Link vs. NASA ICEEx.: Requirements management
– Oversight project requirements vs. contractor project technical requirements
Project within a projectScope focus very tight for
oversight project – milestone based
Other work did not stand still
KISSKeep oversight plan, processes,
and tasks simpleSimple data/configuration
management method Only controlling documents
Reasonable/simple plan and process description results in Effective CMMI Maturity Level
2 compliance
2/22/201027
Spa
cecr
aft S
oftw
are
Eng
inee
ring
Bra
nch/
ER
6Typical Review Process
Perform Tech Review
SFM/SWEProduct Owner
Disposition Product Comments
Perform Peer Review
Product Comments
Initial attempt to use SE process to describe & define process oversightEx.: Use case diagram to represent project process for technical and peer
reviewsEach use case may breakdown and be described in project plan for each
specific processValue of approach is still being evaluated (may refine for next version &
evaluate for improvement)2/22/2010
28
Spa
cecr
aft S
oftw
are
Eng
inee
ring
Bra
nch/
ER
6
Next Steps
2/22/201029
Spa
cecr
aft S
oftw
are
Eng
inee
ring
Bra
nch/
ER
6Next Steps
New Project – Software Process Improvement Project Among other things, earn CMMI Maturity Level 3 by April 2011 Requires major culture change to implement OT, OPF, & OPD
Orion Software PDR Project Continues to evolve Creating checklists to further refine measures from reviews Expect to learn enough to provide guidance to development projects on Planning and scheduling peer reviews Planning and scheduling milestone reviews
Expect confusion in implementing PI, VAL, VER, and TS because our products are brain-power related rather than code
2/22/201030
Top Related