Facilitating NPR7150.2 Compliance and

CMMI L2/L3 audit

NASA PM Challenge 2012Michel Izygon, Scott Hetherington, Svetlana Hanson

NASA & Process

Increased Process Focus

Strong Call for Process Institutionalization

2

Process Evolution

ISO 9001CMMINPR7150.2ITILIEEE 12207Six Sigma ...

Best PracticesProcess ImprovementInstitutionalization

Resistance is Futile

You are here

COMPLIANCE IS MANDATORY

Software Productivity Consortium “The Quagmire” 2000 www.software.org

Chief Engineer increase awareness and consistency across the Agency and advance the practice of engineering …The engineering of NASA systems requires a systematic and disciplined set of processes … for the design development, operation, maintenance, and closeout of systems throughout the lifecycle of the programs and projects.

3

4

SDA Roots - Flight Software Problem

• How can we make it easier for engineers to follow the rigorous process with minimal overhead?

• Software Process Automation

• Survey of workflow tools (2002) found very limited support for dealing with unanticipated events

– Need a workflow tool to automate the flight software process but also allow for the following process exceptions:

• Go back in the process to redo a task• Go back in the process to redo an entire group of tasks• To jump ahead • To Tailor the process

• SBIR Phase I: Jan 04 – July 04• SBIR Phase II: Nov 04 – Nov 06• SBIR Phase III: Sep 07 – Mar 09 – SDA for MOD & Engineering• ER6 funded improvements: Apr 09 – Sept 11 • SARP Funded Infusion: 2009-2011

Implemented EA-WI-025

5

SDA Concepts

• Flexible Process Customization – Multiple Levels– Project, program, organization– Workflow Engine + Rules – key to FLEXIBILITY

• Framework to model/automate any software process– Currently implement NASA/JSC Engineering Flight Software and

MOD Ground Software– Can be customized to any other NASA process

• Easy to capture and share NASA “Best Practices”– Lessons learned, Process Asset Library (PAL), Best toolset

• Facilitate software process execution by supporting:– Developers: worklist, guidelines, instructions, templates,

examples• “Everything is at your fingertips”

– Teams: distribution of tasks, collaboration tools, notification – Managers: visibility, reports, metrics

6

SDA Core Features

• Flexibility– Designed to support any process

• Select existing one• Create new one

• Robust workflow engine– Process clarity– Workflow Logic

• Tool Integration– Currently: Version control, MS Project, Calendar, Message

Board, Wiki,…

• Rich Instrumentation– Real time status of projects– Dashboard– Reporting

• Mapping to any policy or regulations– NPR 7150.2A– CMMI

Evolution of SDA

Database

TieFlow Process Automation Engine V1

Web Interface

Repository

TieFlow Process Automation Engine V2

ProjectDashboard

Business Layer

Portal Framework

Reports Activity/taskProject

SummaryPortlet

…

Rules Engine V1

News Calendar

CCB Automation – 600 Users – 3 NASA SitesDCA – Budget App – Widely used at JSC

Software Process Automation (SDA)– SBIR Phase II and IIINASA Infusion Awards – SDA 2007-2012SDA used by Tietronix on most software projects

TPA 1.02003

TPA 2.02006

7

Tietronix Process Architecture 3.0

Repository Services

TieFlow Process Automation Engine

Rules EnginePortal

Framework

Project Summary Notify

Process Library

PAL Artifacts CM Rules

Business Layer

News Calendar

Project Dashboards

Reports Activity/task

3rd PartyTool

IntegrationHarness

Graphical WorkflowEditor

SOA/WebServices

MS Project

CM toolsSVN/CVS

TieTracker

TPA 3.02008

8

NASA Class A – Software Process

Example ProcessEA-WI-025 – GFE Flight SW/Firmware

230 Activities

SDA Guides/Leads team through the process 9

Iterative Process

10

Document Templates Examples & References

Processes Enactment Tool

SDA - Process Management

Task related Instructions

Individual ToDo List

Each Team Member

Any SW Process

Each Activity

11

Multiple Project Dashboard View

Task Assignments

SDA - Project Management

12

ReportsProject Status & Health

Audit Trail, etc.

SDA – Perform Work

13

NASA-JSC benefits from SDA

• Supports multiple processes and lifecycles for Mission Operations and Engineering Directorates:‒ MOD SMP processes based on the NPR 7150 class, safety criticality,

software type,…‒ Waterfall, iterative and agile lifecycles‒ EA-WI025: GFE Flight Project Software and Firmware Development

• Provides a wizard to assist team members and new project owners in selecting Process adapted to project characteristics‒ Reduces time for decision making

• Support SCAMPI Appraisals by providing a streamlined way to gather all required information‒ reduced the cost of man hours necessary in the preparation of SCAMPI B

and Appraisals significantly

14

NASA-JSC benefits from SDA (cont’d)

• Spacecraft Software Engineering Branch/ER6 (SSEB) is currently integrating all software projects into SDA

• Assist the projects in meeting all applicable project planning, monitoring, and control requirements imposed by SMPs

• Provide greater visibility into projects• Help in training novice developers• Make it easier to coordinate large, complex, geographically

dispersed project• Promote higher quality/less defects at each stage through fill-the-

blank templates• Earlier notice of issues, problems and schedule deviations

15

16

CMMI Support• Facilitate CMMI Implementation and Audit

– Provides painless enforcement of CMMI compliant process– Organization wide consistent process execution– Traceability of all work

• Reduced Effort to Achieve CMMI compliance– Accelerate the compliance schedule

• Generation & Auto capture of CMMI specific metrics• Reports – Detailing CMMI compliance• Supports Continuous Process Improvement

– Project History and Metrics stored in DB– Enable comparison with other projects– Share Best Practices with other projects/departments– Easily customize existing processes

Latest Features Additions• Iterative Processes:

– Multiple iterations and decomposition of tasks• Relationships between projects :

– Parent-Child relationships – Sharing of documents between related

projects– Parent project have read-only access to child

project – Project observers

• read-only access to the project

• External project artifacts– Documents external to SDA may be added as

project artifacts

• Multiple process standards– Processes mapped to multiple process

standards– Compliance reports available for multiple

process standards

• MS Project integration improvements– Import/export of non-SDA tasks

• Process Editor– Creation and modification of processes– Browsing of SDA processes– Publication of processes to SDA– Simulation of processes in the Process

Editor

• PAL (Process Asset Library) Improvements:– PAL portlet outside of project scope– User-requested addition and deletion of

documents to and from the PAL– PAL Administrators control PAL content– Definition of categories and association

with documents

• Usability and bug fixes

17

Latest Features Additions• Features:

– Additional PAL Capabilities– Improve Process Editor

• Process Library for Editor

– Formal document review mechanism• Document states• Document relationships• Subscription/Notification for document changes

– Additional Reporting– Backload project data for existing

projects migrating to SDA– Process Selection Wizard improvements– Editable activity instructions and

background using Wiki– Improve integration with MS-Project– Improve Project relationships

• Infrastructure:– e-Auth support for authentication– Upgrades of SDA third-party software– Section 508 Accessibility Compliance– SDA Lite

• Tools:– BugFlow improvements– Tool content integration– Relate RIDs, Bugs, and Action Items

• Processes:– WI-023, WI-035 (Engineering)– Scrum / Agile– Improve/add MOD Processes

18

SDP Lite (ER WI-035)

19

SDP Lite: simplified process for small projects

• Implement Design Elements• Conduct Formal Code Inspection• Perform Unit Testing

Completion of a single activity requires performance of multiple tasks

20

SDP1 version 2 (ER WI-025)

Instructions for 3.4.4.1 “Implement Design Elements” activity:

The Development Team implements the project’s products, based on the requirements and design. The implementation should include:

1. Implementing all of software safety design features and methods.

2. Following all applicable standards and recommendations defined in the Project Plan.

3. Following the organization polices and standards.

If the activities are broken down it is easier to assign, easier to close and easier to monitor compliance and project progress.

21

SDA Benefits for Project

• Ease of scaling from large to small and growing from small to large are the key.

• Having different templates provides flexibility to serve current needs of the organization

• Provides a jump start in your project: – it has project organization that comes with the process, – predefined set of activities, – templates, – CM and document repositories,– web based – easy to collaborate from different locations.

• The ability to keep all project artifacts in one place.

22

SDA tracks compliance

1.7.9 CMMI MappingThis process activity:- 2 Practice Areas (CM, PP)- 5 Specific Practices

23

CMMI Compliance Report

CMMIArtifacts/Objective Evidence+ embedded history + commentsFor SP 1.1 – Identify Configuration Items- six (6) artifacts containing evidence:

Direct Evidence & possibly Indirect Evidence

For Process Step 1.7.9:“Complete Project Plan”

24

SCAMPI Audit Process observationsfrom SEI Method Description Document – SCAMPI A

1.4 – Obtain & analyze initial objective evidence 1.4.1 – Prepare participants 1.4.2 – Administer Instruments 1.4.3 – Obtain initial objective evidence 1.4.4 – Inventory objective evidence 1.5 – Prepare for Collection of Evidence 1.5.1 - Perform readiness review 1.5.2 – Prepare Data Collection Plan 1.5.3 – Replan Data Collection (if needed) 2.1 – Examine Objective Evidence 2.1.1 – examine OE from instruments 2.1.2 – examine OE from Presentations 2.1.3 – examine OE from Docs 2.1.4 – examine OE from Interviews 2.2 – Verify and Validate Objective Evidence 2.2.1 – Verify Objective Evidence 2.2.2 – Characterize Implementation of Model Practices 2.2.3 – Validate Practice Implementation 2.3 – Document Objective Evidence 2.3.1 – take/review/tag notes 2.3.2 – Record presence/absence of OE 2.3.3 – Document Practice Implementation 2.3.4 – Review and Update the data collection plan2.4 – Generate Appraisal Results

Large Productivity Gain

Organizationally SimplifiedProductivity Gain

Operationally SimplifiedProductivity Gain

Simplified

Partially AutomatedProductivity Gain

SDA use

25

26

Compliance Report SampleStandard Requirements No

MappingNot

ImplementedPartially

ImplementedLargely

ImplementedFully

Implemented Details

SWE-040 Access to software products         [Compliance : 0%]

SWE-041 Open source notification         [Compliance : 0%]

SWE-042 Electronic access to Source code            

SWE-027 COTS, GOTS, MOTS, etc.   [Compliance : 100%]

SWE-034 Acceptance Criteria     [Compliance : 50%]

Legend:   Fully Implemented   Largely Implemented   Partially Implemented   Not Implemented   No Mapping   N/A

27

Process Institutionalization From SEI - CMMI

GG2: Institutionalize a Managed Process

Generic PracticesGeneric GoalsGP 1.1: Perform Base PracticesGG1: Achieve

Specific Goals

GP 2.1: Establish an Organizational PolicyGP 2.2: Plan the ProcessGP 2.3: Provide ResourcesGP 2.4: Assign ResponsibilityGP 2.5: Train PeopleGP 2.6: Manage ConfigurationsGP 2.7: Identify and Involve Relevant StakeholdersGP 2.8: Monitor and Control the ProcessGP 2.9: Objectively Evaluate AdherenceGP 2.10: Review Status with Higher Level Management

GG3: Institutionalize a Defined Process

GP 3.1: Establish a Defined ProcessGP 3.2: Collect Improvement Information

GG4: Institutionalize a Quantitatively Managed Process

GP 4.1: Establish Quantitative Objectives for the ProcessGP 4.2: Stabilize Subprocess Performance

GG5: Institutionalize an Optimizing Process

GP 5.1: Ensure Continuous Process ImprovementGP 5.2: Correct Root Causes of Problems

Institutionalization a cornerstone of CMMI

Institutionalization implies that the process is ingrained in the way the work is performed and there is commitment and consistency toperforming the process. An institutionalized process is more likely to be retained during times of stress

28

How to Achieve Institutionalization

• Organization wide Culture Adjustment– Visible, Persistent & Genuine Display of Management buy-in– Key people with strong Conceptual & Operational Process Skills– Commitment to a process-centric lifestyle– Continuous Improvement Expected – Everyone is involved– Process Mentality Woven into Organizational Fabric – Survives all

Storms

• Organization wide Support– Tools and Infrastructure supporting the methods, practices and

procedures– Strong multi-level training, references, examples, and general support– Internal Marketing – Process Models, Info/Results sharing, Community– Diligent Feedback, Monitoring and Improvement Response

• Infusion into multiple Centers/Projects– SARP funded Infusion Projects

29

SARP Infusion of SDA into the Orion Software Oversight Process

• Process workflow development– Technical Review Process in place

• Supports major review events• Compliant with NASA standards

– Orion Software Development Oversight (SDO) Process assessed• Some process elements have been identified• Rapid customization will be required by a software System Manager (SM)

• Orion Tool Chain integration assessment– Results were mixed; integration challenges remain– Complete tool suite integration will be required to ensure SDA effectiveness– All engineering information sources must be integrated

• SDA Oversight Process assessment– Evaluated SM/SFM behaviors against SDA oversight capabilities– Identified key integration concepts

30

Oversight Process Enactment• Oversight Process Flows Implemented for Tech Reviews• Based on NPR 7150.2 requirements and Oversight Review process.

31

Infusion Study Conclusions• Software Development Oversight (SDO) process enactment via SDA

will rely on a process “building block” approach rather than on the tailoring of existing processes.– Stable SDA oversight processes will exist, e.g. major milestone reviews– Majority of the insight/oversight work however is very fluid– Emphasis is on applying scarce resources & skills to pre-empt issues– SM’s will construct short-duration, high-intensity studies and sub-projects– SDA “Process Editor” capability is now available; supports agile processes

• Tool and information integration is a key to SDA institutionalization– Objective is “process transparency”

• SDA serves the needed tools and information according to the task at hand • Push the process to the background

– Web-based tools can be integrated easily using I-Frames– Integration of server-based or “installation” resident tools will be a challenge – Access to engineering data located on other internal/external networks will

require additional investigation

• 9 NASA Projects – Engineering Directorate – ARI – LIDS Software– Morpheus– JEOD– SAFER– SPIP (software process improvement project)

• 5 NASA Projects – Mission Operations Directorate (MOD)– PLATO– Timeliner – Several Bundles– RegenSafing– ConFRM– CWSyncR8toR9ConfigTest

• Non-NASA Projects – Medical Device Industry – AlfaVue Server – Micro Transponder – FDA Medical Device process– DocControl (document management system)– AlfaThermo Tablet– All Tietronix internal projects

SDA Use

33

• SDA Concept well received– Managers especially enthusiastic on automatic status capture

• Lots of feedback – Usability– Enhancement Suggestions– Eerily upbeat

• Most Believe they are more Productive – With good opportunity for improvement

• Management, Engineers – same side of fence– Unusual lack of resistance

• Everyone a Process Critic, Knowledgeable Citizen– Process awareness & understanding higher across the board– Parallels a trend in the Business Intelligence (BI) market

• Next Generation BI products allow everyone to be an analyst

SDA Observations & Results

34

SDA tool :Coordinating & recording the interactions of people, information and process execution in a project context.• guiding project/process work per SOP• on demand reports verifying compliance

Value to NASA – observations

Compliance Governance Compliance GovernanceWithout process technology using process technology - SDA

(more) Manual coordination & effort:

• Executing project processes ….. correctly• Determining & locating Objective Evidence• Handling Objective Evidence• PIID creation & maintenance• More human interviews vs. audit trail & history• Figuring who, what, when, where• Collecting metrics• Monitoring the process• Improving the process

35

SDA Roadmap & Status

• Recent Additions– Process Editor – End users

• Near Term– Multiple Standards Mappings– 3rd party upgrades– Improved Process Mgt.

• Future– Globalization – time zones, languages– Auto – Artifact Semantic Analysis– Tool Integration Harness– Requirements Level Traceability– 3rd party integrations

• Application Lifecycle Management– Other Technologies – EA, BI

• Enterprise Architecture, Business Intelligence

• Processes– SCAMPI, Process Improvement

Roadmap Status• Core Lifecycle Processes

– Waterfall, Iterative & Agile– Tailored for projects

• Process Wizard (for MOD)– User Q & A

– Lifecycle?– NPR Class – A, B, …, H?– Safety? …

– Auto generate/tailor Process• Deployments

– 19 NASA Projects– 10 Internal Tietronix Projects– 4 Medical Projects

• Infusion– SARP Infusion to Multiple NASA

Centers

36