Ortiz.james

Connect the Dots

Dr. James Ortiz, Director Independent Program Assessment Office

Seventh Annual NASA Program Management Challenge 2010

February 10, 2010Used with Permission

Introduction

• The Independent Program Assessment Office (IPAO) manages the independent review of the Agency's Programs and projects at life-cycle milestones to ensure the highest probability of mission success.

•This presentation provides an overview of the Agency’s policy context for the Agency Independent Life Cycle Review process (The “Why”, the “What”, and the “How”).

Outline

• Connecting the dots on the Agency policy context – The “Why”

» NPD 1000.0; NPD 1000.5 – The “What”

» NPR 7120.5D NID; NPR 7123.1 – The “How”

» SRB Handbook » IPAO System Operating Procedures

• Summary

Agency Policy Context for Independent Review

Policy Flow-down

NPD 1000.0NPD 1000.3NPD 1000.5

NPD 7120.4

NPR 7123.1A NPR 7120.5Dand NID

IPAO 1000IPAO MANAGEMENT

PLAN

NASA -SP 2009 -10 -015 -HQ SRB HANDBOOK

IPAO 2000STANDARD OPERATING

PROCEDURE

NPD 8700.1 NPD 8900.5A Mission Support Office

OSMA Directives OCHMO Directives Support OrganizationDirectives

21 3 4 5

LEGEND:

1: ENGINEERING REQUIREMENTS2: PROGRAM/PROJECT MANAGEMENT REQUIREMENTS3. SAFETY AND MISSIONASSURANCE REQUIREMENTS4. HEALTH AND MEDICAL REQTS5. MSO FUNCTIONAL REQTS6. MD REQUIREMENTS7. CENTER REQUIREMENTS

NASA -SP -6105SYSTEMS ENGINEERING

HANDBOOK

Center EngineeringRequirements

Mission DirectorateRequirements

7 6

DIRECT

INDIRECT

The “Why”

The Governance and Strategic Management Handbook (NPD 1000.0A)

“NASA governance and strategic management provide the discipline and rigor to enable success of NASA’s Mission- …. Mission success delivers on our commitment to be good stewards of the resources entrusted to us by the taxpayer. To enable Agency-wide accomplishments, NASA’s governance framework is founded on the following tenets: • …..NASA’s governance principles that provide this framework are:• Lean governance. • Clear roles, responsibility and decision making. • Strategic acquisition.

• Checks and balances. ……There are many process-related checks and balances built into NASA’s way of doing business. They range from peer reviews conducted at the lowest level to oversight reviews conducted by the Agency’s Program Management Council. Three checks and balances of particular importance at the program or project level are: the independent life-cycle review process, the process for tailoring a specific prescribed requirement, and the Dissenting Opinion process.”

Why Independent Review The Governance and Strategic Management

Handbook (NPD 1000.0A)

The independent life-cycle review process provides a comprehensive review of programs and projects at each life-cycle milestone by competent, independent individuals. The purpose of these reviews is to provide:

1. Agency wants P/p to receive independent assurance that they are on-track

2. NASA senior management wants: – Independent validation at key decision points of the

Program/project’s readiness to proceed into the next phase of its life-cycle

– Externally-imposed impediments to Program/project success to be removed

3. Agency needs to give external stakeholders assurance we can deliver to our commitments

4. Significant additional benefit is that preparation for the review milestone allows for a holistic examination by the Program/project and the review team

9

Policy for NASA AcquisitionNPD 1000.5

• NPD 1000.5A calling for the Joint Confidence Level (JCL) analysis became effective in January 2009

• Policy is directed to ensure appropriate level of funding is provided and maintained to increase probability of success of the Agency’s Programs and Projects

• Requirements – (a) Programs are to be baselined or rebaselined and budgeted at a confidence level

of 70% or the level approved by the decision authority of the responsible Agency-level management council. For a 70% confidence level, this is the point on the joint cost and schedule probability distribution where there is a 70% probability that the project will be completed at or lower than the estimated amount and at or before the projected schedule. The basis for a confidence level less than 70% is to be formally documented.

– (b) Projects are to be baselined or rebaselined and budgeted at a confidence level consistent with the program’s confidence level.

– The program's or project's proposed cost and schedule baseline are to be assessed by an independent review team. The program or project is to present and justify its resulting cost and schedule to the decision authority of the responsible Agency-level management council. The independent review team is to discuss with the decision authority its key concerns with the plans and baselines proposed by the program or project.

The “What”

11

NPR 7120.5D NIDIndependent Review Requirements

• Provides the basis for how NASA execute its space flight programs and projects.

• Provided a standardized life cycle review process that is built around Key Decision Points (KDP).

• Formalized the Technical Authority, Dissenting Opinion Resolution, and Waiver processes.

• Defines the Independent review policy requirements Standardizes the Standing Review Board concept across NASA Combines technical (7123) and programmatic reviews into one Boardthe Standing Review Board (SRB)

One independent review board for each program or project Conducts all 7120.5-required independent reviews for that

program or project throughout its life cycle Implemented by the IPAO for all Programs, Category 1 Projects

and Category 2 projects with a life-cycle cost > $250M

Project Life Cycle (NPR 7120.5)

NPR 7123.1 Systems Engineering Technical Review Requirements

• NPR 7123.1A – Systems Engineering Processes and Requirements Articulates and establishes the requirements on the implementing

organization for performing, supporting, and evaluating systems engineering

This systems approach is applied to all elements of a system and all hierarchical levels of a system over the complete project life cycle

• 5.2.1.3 A technical review is an evaluation of the project, or element thereof, by a knowledgeable group for the purposes of: Assessing status of and progress toward accomplishing planned activities. Validating technical tradeoffs explored and design solutions proposed. Identifying technical weaknesses or marginal design and potential

problems (risks) and recommending improvements and corrective actions. Making judgments on activities’ readiness for follow-on events, incl

additional future evaluation milestones to improve likelihood of successful outcome.

Making assessments and recommendations to project team, Center, and Agency management.

Providing historical record that can be referenced of decisions that were made during these formal reviews.

• Entrance and Success criteria for each life-cycle milestone are contained in Appendix G.

The “How”

The SRB HandbookThe How

• The SRB is written to provide guidance to the NASA Program and project (P/p) community and the Standing Review Boards (SRBs) regarding expectations, processes, products, timelines, and working interfaces with Mission Directorates, P/p, Centers, other review organizations, and the Management Councils.

• The SRB handbook was published by PA&E/IPAO with concurrence from OCE

IR-16

SRB Handbook Contents

• The SRB Handbook consist of six sections – Section 1: Context for Independent Lifecycle Review (ILCR) Process. Objectives,

intent, and governance of the Standing Review Boards (SRB).– Section 2: High level principles that govern the SRBs. Independence and issue

resolution. Points to Annex with Agency policy for “Review Board Composition, Balance and Conflict of Interest”.

– Section 3: Scope and expectations of ILCR. Provides guidelines (roadmaps) for application of review criteria in NPR 7120.5 as a function of the lifecycle stage of P/p, the types of mission (Robotics versus Human Spaceflight), and type of Program ( tightly coupled or loosely coupled).

– Section 4: SRB Initiation process. Roles and responsibilities for Chair/RM nomination, team composition and staffing, and Terms of Reference (TOR).

– Section 5: SRB Products. Technical and programmatic assessments, findings and recommendation, briefings and reports.

– Section 6: Notional review timeline. Describes the review phases for a typical review. – Appendices: Agency Independence Policy, PIR guidance

IR-17

Example of Section 3(SRB engagement roadmap)

IR-18

Operations

Pre-Phase A Phase B Phase E

SDR PDR

(PNAR) (NAR)

Key Decision Points KDP A KDP B KDP C KDP D KDP E

SRB Participation Case-by-case

Full Board Full Board Full Board Full Board Chair and Member

Subset 3

Full Board Chair and Member

Subset 3

Chair and Member

Subset 3

Chair and Member

Subset 3

Chair and Member

Subset 3

Chair and Member

Subset 3

Sample Assessments

Requirements

Preliminary Functional and Performance Baseline ; Requirements Traceability Report; SEMP

Flowdown to Functional Elements; System Requirements Document

Flowdown to Functional Elements, Descope Plans

Any Changes, Flowdown to Functional Elements, Descope Plans

Production processes , certified design

Integration Plan and Procedures

System meets acceptance criteria and has been verified and validated

Phase E, Descope plans

Flight operations are certified to proceed

Launch system and spacecraft/ payloads readiness for launch

Flight performance reporting

Technical 5

Key Technolgies,TRL, MOEs, MOPs

Preliminary Approach, TRL, Technology Devlpt. Maturity Assessment Plan (TDMAP), Software Devlpt. Plan (SDP)

Mission System Architecture, TRL, TDMAP, SEMP, SDP

Baseline: Systems Engineeering Mgt Plan, Design meets performance

Design meets performance, TRL

Design documentation, Production Plans

Previous component, subsystem and system tests have been verified to support integration

Technical data package, Certification package

All waivers and anomalies have been closed; Operational procedures and contingency planning

Hardware and software systems are configured for flight

Launch system and spacecraft/ payloads readiness for launch

Performance report and Anomaly resolution

Integrated cost and

schedule 6

ROM Cost Schedule Estimates

ROM Integrated Cost Schedule Estimates

Preliminary: BOEs IMS Cost/Budget UFE, CADRe WBS ICE

Baseline: BOEs IMS Cost/Budget UFE, CADRe JCL , Detail schedules, ICE

Performance against plans, EVM, UFE , LLIL

Production Plans

Performance against plans, EVM, UFE, JCL

Remaining liens or open items and plans for closure

Performance against plans, EVM, UFE

Performance against plans, EVM, UFE

Resources

Preliminary: Facilities Workforce



Baseline: Facilities Workforce

Facilities Workforce






Risk

Preliminary Risks

Preliminary Risks

Preliminary: Risk Mgt Plan Risk list Mitigation

Baseline: Risk Mgt Plan Risk list Mitigation

Risk List Mitigation






Management

Preliminary: Approach

Preliminary: Plans

Preliminary: Project Plans

Baseline: Project Plans

Performance against plans

Production Engineering and Planning


Performance against Plans

Phase E Mgt Plan


Reporting Venues

Project x x x x x x x x

Program 4 x x x x x x x x

CMC 4 x x x x x x x x

DPMC x x x 4 x 4 x x x 4 x

APMC Cat 1 Cat 1 Cat 1 Cat 1

6. Compliance with NPR 1000.5.

NASA Life Cycle Phases

Formulation ImplementationPre-Systems Acquisition Systems Acquisition

Phase A

SIR ORR FRR 2 LRR 2

5. Compliance with NPR 7123.1 review entrance and success criteria will be assessed.

SAR

Notes:

1. PRRs are only needed when multiple flight system copies are being developed; timing is discretionary.

Reviews MCR 1 SRR CDR PRR 1 PFAR 2

Phase C Phase D

4. The SRB chair and RM will make themselves available to support these interim briefings.

3. Chair determines which members should attend, including themselves.

2. LRRs, FRRs, PFARs are performed by the Mission Management Team (MMT); the SRB Chair (and/or Designee) attend as non-voting observers.

Example from Section 4 (SRB Formulation )

IR-19

SRB Initiation Schedule prepared monthly for the APMC

MDAANominates Chair

IPAO RMAssignment

Nominations vetted with all Approval Authorities (MDAA, DA, TA, PA&E)

Chair/RM developSRB Membership List

Chair/RM develop ToRand review with DA and TA

Proceed with SRB Reviews

* For Programs, Cat.1 and Cat 2 (LCC>$250M) Projects only

CDNominates Chair

Center RMassignment

ProgramsCat 1 &

Cat 2 Projects>$250M

Cat 3 &Cat 2 Projects

<$250M

IPAO RMAssignment

CDNominates Chair

MDAA/DA/TA/PA&E*Approval/Concurrence ?

Reiterateselection stepswith new Chair

nomination


Reiterate asnecessary to

obtain Approvals

Mission Directorate Host Center

IPAO SRB

Joint Effort

No

No

Yes

YesParticipating Organizations

Selection of the SRB Chair and RM

SRB Staffing and ToR Preparation

SRB Initiation Schedule prepared monthly for the BPR

MDAANominates Chair

IPAO RMAssignment

Nominations vetted with all Approval Authorities (MDAA, DA, TA, PA&E) and cleared for OCI/PCI.

Chair/RM developSRB Membership List

Chair/RM develop ToRand review with DA and TA

Proceed with SRB Reviews

* For Programs, Cat.1 and Cat 2 (LCC>$250M) Projects only

CDNominates Chair

Center RMassignment

ProgramsCat 1 &

Cat 2 Projects>$250M

Cat 3 &Cat 2 Projects

<$250M

IPAO RMAssignment

CDNominates Chair


Reiterateselection stepswith new Chair

nomination


Reiterate asnecessary to

obtain Approvals

Mission Directorate Host Center

IPAO SRB

Joint Effort

No

No

Yes

YesParticipating Organizations

Selection of the SRB Chair and RM

SRB Staffing and ToR Preparation

and cleared for OCI/PCI.

Nominations vetted with all Approval Authorities

Handbook Status

• Initial version of Standing Review Board (SRB) Handbook (HB) developed October 2007 by Agency team (IPAO lead and representative from each Center).

• SRB HB on hold pending review of Federal Advisory Committee Act (FACA) regulations and independence policy requirements.

• SRB HB on hold for completion of NPR 7120.5D NID (~February 2009).• NID-aligned version of the SRB Handbook released for PPMB review on Sept

25, 2009 ( 2 week review).• Baseline version of SRB HB published ( December 2009)• Next revision to be published with NPR 7120.5E

IR-20

IPAO

The NASA organization responsible for the agency independent life cycle review process

• Enables independent review of programs and projects to ensure highest probability of mission success.

• Ensures objectivity, quality, integrity and consistency of independent review process per NPRs 7120.5D and 7123.1A.

Works with SRB Chair and Convening Authorities Reviews SRB products to ensure they meet Agency

expectations• Provides Review Manager, cost, schedule and other

programmatic analysis SRB members• Provides advice and recommendations to Agency

on program and project policies Book Manager for Agency’s SRB Handbook

IPAO Standard Operating ProcedureDetail processes

• The Standard Operating Procedure (SOP) documents the Independent Program Assessment Office (IPAO) process for conducting Independent Life-Cycle Reviews (ILCRs).

• The SOP is a guide for the IPAO personnel when conducting ILCRs of the Agency’s programs and projects.

• The purpose of this document is to ensure ILCR quality and efficiency through the development of common definitions and processes.

Individual Review BudgetToR, Cost Plan, Schedule PlanChair and Team Nomination and Guidance Letters Review Schedule Contract Task Statement R&R for SRB

Monitor AgencyBaseline Program/

Project Review Determine IPAO

Review Budget Review Assignments

PLANNING PREPARING REPORTING CLOSING-OUT

Continuous 1- Month2-3 Months2- Months 1-3 Months

ICE/ICA/ISA SRB Findings RFA Recommendations

Receive CADRe Inputs SRB Kick-Off Meetings Attend Reviews Attend Site Visit Develop ICE/ICA/ISA

Verbal Pre-Briefings to P/p Write one-page summary Write the Report Prepare the Summary

Briefings Present the Briefings:

- Program/ project - CMC- MD PMC- APMC (if required)

RM/CA/SA Assignments One Page Summary SRB Report RRD ReportSRB Briefings

Review Summary Contracts Close-out by COTR Review Records RRD Report Process improvements

REVIEWING

Review Closeout Customer

Feedback Develop Lessons Learned Process Review and Improvement Administrative Close-out Team Recognition

P/p Documentation Cost & Schedule

Documentation Program/Project Briefings

Agency Review Schedule Review Chair Nomination

Minutes/Decisions/Actions Agency/Directorate/CentersDocumentation & Requirements

Initiate Contact and Determine Review Budget Identify and obtain approval of SRB Chair/RM/SRB members Determine funding mechanism Develop the TOR Develop Schedule/Logistics

of the Review Develop the Cost Plan Develop the Schedule Plan

P R O D U C T S

I N P U T S

SRB Briefings to Program/Project

Individual Review BudgetToR, Cost Plan, Schedule PlanChair and Team Nomination and Guidance Letters Review Schedule Contract Task Statement R&R for SRB

Monitor AgencyBaseline Program/

Project Review Determine IPAO

Review Budget Review Assignments

PLANNING PREPARING REPORTING CLOSING-OUT

Continuous 1- Month2-3 Months2- Months 1-3 Months

ICE/ICA/ISA SRB Findings RFA Recommendations

Receive CADRe Inputs SRB Kick-Off Meetings Attend Reviews Attend Site Visit Develop ICE/ICA/ISA

Verbal Pre-Briefings to P/p Write one-page summary Write the Report Prepare the Summary

Briefings Present the Briefings:

- Program/ project - CMC- MD PMC- APMC (if required)

RM/CA/SA Assignments One Page Summary SRB Report RRD ReportSRB Briefings

Review Summary Contracts Close-out by COTR Review Records RRD Report Process improvements

REVIEWING

Review Closeout Customer

Feedback Develop Lessons Learned Process Review and Improvement Administrative Close-out Team Recognition

P/p Documentation Cost & Schedule

Documentation Program/Project Briefings

Agency Review Schedule Review Chair Nomination

Minutes/Decisions/Actions Agency/Directorate/CentersDocumentation & Requirements

Initiate Contact and Determine Review Budget Identify and obtain approval of SRB Chair/RM/SRB members Determine funding mechanism Develop the TOR Develop Schedule/Logistics

of the Review Develop the Cost Plan Develop the Schedule Plan

P R O D U C T S

I N P U T S

SRB Briefings to Program/Project

How is it done? (IPAO Process)

SOP:Sample SRB Engagement Timeline

PDR: 2/11/10 Site Visit: 5/11/10 - 5/21/10 KDP: 6/11/10 - 6/21/10SRB KO: 10/14/09

PDR KO:12/1/09

24

• The independent life cycle review process is an integral part of the Agency’s check and balances built into the NASA governance structure and complements the programmatic and technical lines of command and authority.

• The independent lifecycle review process is encoded as part of NASA policy direction; its requirements are stipulated in policy requirements; and guidance to reviews team and implementing personnel is provided in handbooks and operating procedures.

• Its processes are continuously assessed for improvement by IPAO and its stakeholders.

• The independent lifecycle review process helps ensure the highest probability of success of the Agency’s program and projects.

Summary

Ortiz.james

Technology

Transcript of Ortiz.james