NASA Baseline Performance Review

(BPR)Steven Noneman, NASA HQ

Robert Woods, Perot SystemsOffice of Program Analysis and

EvaluationOffice of the Chief Engineer

Program Management Challenge, Galveston, TXFebruary 2010

Used with Permission

Presentation Purpose

• Describe NASA’s Baseline Performance Review (BPR) to the greater NASA community at PM Challenge 2010. Characterize the BPR objectives, scope, process, and

products Summarize NASA’s technique for monitoring Agency

performance against its baseline plans

2

Contents

• BPR History• BPR Objectives• BPR Scope, Participants• BPR Process• Sample BPR Meeting Agenda• Sample BPR Analysis Content• Summary

3

BPR History and Context

• In the spring of 2006, the Office of the Chief Engineer (OCE) was given the action by the Associate Administrator to develop a State of the Agency (SoA) process that provides the Agency Project Management Council (APMC) an independent top level view of the status of the Agency’s programs and projects.

• The first SoA presentation, to the Agency PMC, was by the Science Mission Directorate (SMD) in August of 2006. The process has matured and continued to meet the objectives outlined by the Associate Administrator, the Associate Deputy Administrator, the Chief Engineer (CE) and the various stakeholders. The SoA evolved to a distinct monthly meeting in December 2007 and was officially entitled the Baseline Performance Review (BPR).

• The BPR instituted a modified “balanced score card stop-light chart” process based on metrics encompassed in a core set of functional area questions and supporting raw data.

• Participants from the projects, programs, Centers, Mission Directorates (MDs) and Mission Support Offices respond monthly to each core area question based on provided thresholds and key metrics for use in internal and external reporting.

• All Mission Directorates are reporting at the BPR meeting the findings of the independent process.

BPR Objectives

• Provide NASA Senior Leadership comprehensive, integrated, and objective information that describes the “performance-to-plan” of the Agency’s programs, projects, and institutional capabilities. A single place to see the full execution context of the Agency

• Assure open cross-functional communications among NASA’s organizations to enhance Agency performance.

• Identify and analyze performance trends and cross-cutting or systemic issues and risks

• Serve the Agency’s decision making Councils as their ”execution arm” Within the NASA Governance model, the BPR is distinct from

the decision-making Strategic Management, Program Management, and Operations Management Councils

BPR is “action-oriented” to improve performance and inform Agency decision authorities of issues needing attention.

5

BPR Scope and Participants

• The BPR is co-chaired by the NASA Associate Administrator and the Associate Deputy Administrator.

• Membership includes the heads of NASA Staff Offices, Mission Directorates, Mission Support Offices, and its ten Field Centers.

• Mission Support functions (e.g., finance, workforce, acquisition, infrastructure, IT, etc.) report monthly and Small Business and Diversity and Equal opportunity report quarterly.

• Mission Directorates (covering all program areas) report monthly with one (and sometimes 2) “highlighted” in greater detail.

• The Office of the Chief Engineer (OCE) leads a team of “independent assessors” from OCE, Program Analysis and Evaluation (PA&E), and Office of Safety Mission Assurance that analyze programs’ and projects’ performance. “Green/Yellow/Red” criteria were established to assess of technical, cost,

schedule, and programmatic performance are provided monthly for all Mission Directorates.

• PA&E provides the BPR Coordinator who manages the review by setting agendas, tracking actions, and assisting the co-chairs, OCE, and BPR participants.

6

MDs

MD issue charts

OCE sends out notifications to users who answer questions

Mission Directorates

Mission Support Offices

NASA Centers

Projects

Programs

1

2

3

4

5

OCE consolidates input from stakeholders1- 5

(OCE, OSMA, and PA & E) Assessor Group

OCE reps. for

MDs

PA & E reps. for

MDs

OCE generates BPR package

GYR rankings for projects and programs from the Assessor Group

TechnicalX-Cutting, PCA Status

Non-TechnicalX-Cutting

Package briefing to CE for approval

BPR Package is available for Read-only viewing from site

Package presented at the BPR

OCE feedback and Action items on BPR process

OCE Provides

Exe. Sum.

BPR Detailed Process Flow

3

3

Example Outcomes/Results

• Cross-cutting issues Across missions, faulty material was discovered resulting in a mitigation

process for the materials use saving time and resources.• Organizational communication issues

Turned a conflict between projects on thermal protection systems into a collaboration between the major development projects who previously were competing for shared resources.

• Mission support issues Workforce planning resolution was expedited when a project was

confronted with changes resulting in 500 people at a Center needing reassignment because the project was delayed 2 years.

• Senior leaders as an “Execution Board of Directors” Leadership asked and Centers provided additional workforce during

transition to the NASA Support and Services Center because invoices were not getting paid on time, which had been costing projects late fees.

• Mission specific issues Timely communication resulted in a Center director de-conflicting a

vendors schedule between 2 NASA projects from different Centers.

BPR Project Questions

QUESTIONS THRESHOLDSGUIDANCE FOR R&T

PROGRAMS/ PROJECTS

Office of Procurement

Proj

ect

Prog

ram

Mis

sion

Dire

ctor

ate

COMMENTSG R

Are your top level requirements(such as KPP's) or objectives stable and

established/baselined?

G=Yes or <10%, 10%<Y<=20%

requirements/objectives not stable or established/

baselined, R>20% requirements not stable or

established/ baselined

* What is the status and level of maturity of the research

objectives?* Have you baselined the

state of the art/knowledge and are moving off this

accordingly toward those objectives? X Y X

Unknowns and lack of clarity with Lunar requirements specific to technology development.

Are you on track to meet your top level requirements/ objectives?

G=Yes, Y<=20% requirements/ objectives not

on track, R>20% requirements/ objectives not on track X G X

Does the project/program have plans that meet the needs/ objectives of the

community customer?G=Yes, Y=minor weakness

identified, R=major weakness identified

Has requirement feasabilitybeen established and have requirements flowed down

with traceability from customer needs? X G X

Are the results of research/ technology being widely disseminated?

G=Yes, Y=minor weakness identified, R=major weakness

identified X G X

What is the readiness to support the next Review?

G= On track to complete all deliverables with no delay, R= Not on track for deliverables or

delay necessary X NA X

What is the status of actions from the last review?

G= All actions complete, Y>= 50% of actions complete, R <=

50% of actions complete X NA X

9

BPR Project QuestionsWhat is your overall risk status/ posture (i.e. risk plan/ 5

x 5 matrix)?

G= All "green" or equivalent, Y= Any "yellow" but no "red" or equivalent, R= Any "red" or

equivalent

* Is the risk posture or your research portfolio balanced and have you

identified the top risks to the success of your research objectives? X R X

Currently, more than 1 risk is red.

Are risks manageable and appropriate ( i.e. risk plan/ 5 x 5 matrix)?

G= Can be resolved within available resources, Y= Can't be resolved by project but can with

other resources, R= Only resolveable by Agency resources X G X

Are technologies sufficiently mature (i.e. TRL or equivalent) at this point, or expected to be mature in

time, per program/ project plans?G=yes, Y=one is no, R=more than

one no X G X

Will all the externally reported performance goals (Annual Performance Goals)be met?

G=all req met, Y= 1 not met, R= more than 1 not met X G X

Have the externally reported performance goals changed within the current year? G=no, R=yes X G X

Are the externally reported performance goals per your plan? G=yes, R=no X G X

Has the life-cycle budget (LCB)/ top line budget changed since last reported?

G= no, Y= has changed once, R= changed more than one time in same phase or last 6 months X G X

Has the current year funding changed since last reported?

G= no, Y= has changed once, R= changed more than one time in same phase or last 6 months X G X

Are costs over/ under running your phasing plan? Overrun 0<=G<5% LCB, 5%LCB<=Y<10%LCB,

10%LCB<=R X G XIs there a significant need to reallocate resources between Centers in order to accomplish technical

tasks? G=no, R=yes X G XIs the current year funding per the approved baseline

plan?90<=G<100%, 80%<=Y<90%,

R<80% X G XWhat is the difference between your current estimate at

completion and budget at completion?Difference of 0<=G<5%, 5%<=Y<10%, 10%<=R

* Are the cost estimates for obtaining your research objectives still valid? X G X

Are mechanisms in place and functioning properly to ensure that NASA receives external funding and sends

out funding to others in proper time? G=no, R=yes X G XFor projects/ programs over $250M over 5 years, is the

project/ program cost per baseline plan and beneath congressional limits (15% of Baseline plan)? 0<=G<5%, 5%<=Y<10%, 10%<=R

NAX G X

10

BPR Project Questions

Are major milestones (per Integrated Master Schedule or equivalent) being met?

G=Next major milestone slippage less or equal to 45 days, Y=Greater than 45 but less than 90, R= Greater than 90

* Are the schedule estimates for obtaining your major research

objectives milestones still valid? X G X

Are all GPRA milestones on schedule?G=Yes, Y= 30-60 days behind, R= greater than 60 days behind

* For those Outcomes/APGs that are also key milestones are they still on track for completion per the annual

performance plan? X G X

Has the schedule been significantly impacted since last reported?

G=No change, Y=Change but no schedule slippage, R=Change but with

schedule slippage X G X

For projects/ programs over $250M over 5 years, is the project/ program under congressional limits for key

milestone slippage (less than 6 months)?G=Next major milestone slippage less or equal to 45 days, Y=Greater than 45 but less than 90, R= Greater than 90

NA

X G X

Are all external agreements in place and executing per plan? G = yes , R = no X G X

Are key partners delivering their commitments on time? G = yes , R = no X G XHave key partners changed their commitments since

last reported? G = no , R = yes X G XAre critical contracts in place and executing per plan? G = yes , R = no X G XAre undefinitized contract actions within guidelines? G = yes , R = no X X G X

Are existing facilities available (or do results of required infrastructure assessment/business case

analysis show existing facilities are available to support the activities)?

G=Yes, Y=No - scheduling issue. R=No - need to refurb or build X G X

Are all critical staff and skills identified and available per plan?

90% Critical staff<=G<100%Critical staff, 80%<=Y<90%, R<80% critical

staff X G X

Are all support staff and skills identified and available per plan?

90% Critical staff<=G<100%Critical staff, 80%<=Y<90%, R<80% critical

staff X G X

Are required documents per Agency and Center policy complete (i.e, FAD, PCA, Program/Project Plan)?

G=Y, Y= 1 required document not complete, R> required document not

complete X G X

11

Sample BPR Agenda

10:00 Opening Remarks, Actions Associate Administrator, Coordinator10:05 Recovery Implementation Status Recovery Act Implementation Team10:15 Mission Support Reports

- Finance Office of the Chief Financial Officer- Workforce Office of Human Capital Management- Acquisition Office of Procurement- Infrastructure Office of Infrastructure

11:30 Highlighted Mission Support: Small Business Programs* Office of Small Business Programs12:00 Lunch12:15 Agency Aggregate Assessments

OSMA Safety Assessment: NASA Mishap Posture Office of Safety and Mission AssuranceCross Cutting Non-Technical Issues Program Analysis and Evaluation- Special Topic: 2nd Quarter OMB Cost/Schedule ReportCross Cutting Technical Issues Office of the Chief Engineer- Special Topic: Flight S/W Complexity Jet Propulsion LabDocuments Compliance Office of the Chief EngineerLife Cycle Milestone Schedule Office of the Chief EngineerCenter Summaries (Programs/Projects and Institutional) Center Representatives

2:45 Roll-up of MD Programs/Projects Office of the Chief Engineer- JWST Project On-going Status/Recovery Plans Science Mission Directorate

3:30 Highlighted Mission Directorate Office of the Chief EngineerSpace Ops Mission Directorate (SOMD) Assessment* Space Operations Mission Directorate Launch Services Program Status/Recovery Plan Space Operations Mission Directorate

4:45 Action Summary Coordinator5:00 Adjourn Associate Administrator*Specific topic is rotated

12

Samples of BPR Analysis Content

• Finance• Workforce• Acquisition• Small Business• Infrastructure• Safety and Mishaps• Crosscutting Issues• Field Center Summaries• Independent Assessments of Programs and Projects

(“Stoplight” ratings)

13

14For NASA Internal Use Only

Workforce Metrics

15

0

2

4

6

8

10

12

14

16

Center

1

Center

2

Center

3

Center

4

Center

5

Center

6

Center

7

Center

8

Center

9

Center

10

Num

ber o

f UC

As

1-180 Days 180-360 Days >360 Days

2 1

9

4

41

8

Undefinitized Contract Action (UCA) StatusFebruary 27 – March 27, 2009

1

1 $35M

$749K

$836M

$27M

$1.3B

1$700K$3.5M

1

$0 $5,000 $10,000 $15,000 $20,000 $25,000

In Million $

Department of Energy

Health and Human Services

National Aeronautics & SpaceAdministration

Department of VeteransAffairs

Department of HomelandSecurity

Department of Justice

General ServicesAdministration

Total Dollars Small Business Dollars

FY08 Federal Agency Prime Small Business Dollars Overview

Total Dollars ObligatedSB Dollars

Top Seven Agencies w/o DOD

BPR: I&A Functional AreasARC DFRC GRC GSFC JPL JSC KSC LaRC MSFC SSC Remarks

Design & ConstructionG G G G G Y G G G G

JSC damage from Ike could impact JWST Project.. Chamber A MOU and Management Plan not complete. Concern about communication between Eng Div and Facil Div at JSC

Facility MaintenanceY Y Y Y Y R Y Y Y Y

Facility repair backlog is 8.6% of facility value with increasing trend.JSC damage from Ike could impact SOMD Operations

Planning/Real EstateG Y G Y G Y Y G Y Y

DFRC, KSC, and SSC were changed to yellow since they have not submitted revised Master Plans. GSFC presented new Master Plan, but has not submitted document for formal approval.

Supply

G G Y G G G G Y G Y

Delay in replacement of critical civil service personnel at GRC and KSC. Also at KSC, a concern existed as retiring civil servants positions were not backfilled. Situation improved by increased use of Support Services Contractors. Sinlge point failure for multiple logistics functions at LaRC and SSC. Anticipated hire at SSC currently on hold.

Equipment

G G Y G NA G G Y G Y

Delay in replacement of critical civil service personnel at GRC and KSC. GRC expects to fill vacancy by end of 2008. At KSC, a concern existed as retiring civil servants positions were not backfilled. Situation improved by increased use of Support Services Contractors. Single point failure for multiple log functions at LaRC and SSC. Anticipated hire at SSC currently on hold. GSFC filled HQ SEMO position.

Contract Property

Y Y Y Y Y Y Y Y Y Y

CFO management control findings related to property management are being addressed, however center IPO staffing concerns to support transition workload and FAR update present an on-going staffing issue. JSC in process of filling Contract Property Administrator position. JSC also submitted request to fill the following: 2 positions for Property Disposal, 1 position for Shuttle Clerical (Property) and 1 position for Packing/Shipping in Transportation.

Transportation

G G G Y G G G Y Y Y

Agency risk for not meeting EO 13423 (alternate fuel): discussions with MSFC to resolve center contribution ongoing. Single point failure for multiple log functions at SSC, anticipated hire currently on hold. GSFC loss of CNG capability for approx. 13 vehicles. KSC added additional contractor to assist with transportation functions. At LaRC, TO/SEMO is one person, he will be out in Jan. (medical leave) no replacement appointed.

Property DisposalG G G Y G Y Y G Y Y

Increase in excess property workload due to Shuttle and Hubble disposition at GSFC, MSFC, KSC, SSC & JSC. Single point failure for multiple log functions at SSC.

Logistics

Facilities

NASA and Contractor Safety StatisticsDashboards

Jan 2008 – Mar 2008 Jan 2009 - Mar 2009 Jan 2008 – Mar 2008 Jan 2009 - Mar 2009 Jan 2008 - Mar 2008 Jan 2009 - Mar 2009

March Mishaps:

• No Type A Mishaps

• One Type B Mishap: DRC- (Deluge System Release in Shuttle Hangar - 3/13/09)

• One Potential Type B Mishap: DRC- (Global Hawk Electronics Failure 3/12/09) – currently a Type D

Data does not include 4/2/09 Type B Mishap: Water Drill mishap resulting in multiple amputations

Close Call Incidents Show An Increase From Previous Reporting Period (Jan – Mar 2008).

February Mishap OCO

Mission Directorate Impact

Status as of: Date x, 200xFalse Material Certification • Issue Description: Company xyz, manufacturer of a material used in flight hardware, was indicted on federal

charges of issuing false material certifications for supplies they sold. Material of uncertain pedigree may have been used by NASA projects.

MD 1 MD 2 MD 3 MD 4• Suspect material found in

Kepler spacecraft in a critical area wherein failure could be catastrophic.

• Suspect material also found in the AIA instrument on SDO.

• Other projects are investigating usage.

• Suspect material found on the Taurus and Delta IV vehicles used by OCO, Glory, and NRO L-26.

• NRO mission cleared.• SSP & ISS cleared.• LSP - All vehicles now

cleared.

POC: Ken LedbetterInformation on Agency Actions and Status (Estimated Completion)

• Top Level Description/ Status: Material found to have uncertain pedigree due to false certifications. Extent of usage in NASA projects and resulting impact, are under investigation.

• The Kepler project procured material from Company xyz who obtained some of the material from Company xxx Inc, including the material for the Telescope Spider Hub Assembly. Company xyz had 46 different material procurements. The Kepler project performed tests on residual material at xyz and confirmed that it was sufficient to satisfy their mission application.

• GIDEP indicates earliest occurrence was no later than 2002 with the latest false certification in 7/03 for a purchase order in 11/04 by Company xyz.

• SDO project also found suspect material in the AIA instrument. AIA has now been cleared. • Centers responding with evaluations on a project-by-project basis.

• No use of Suspect material determined as yet

• Supplier traceability investigation still underway

• Cleared: LRO, LCROSS, Ares I-X, J2-X, Orion PA-1, HRP ISS Experiments & ETDP

• No other project assessments required

Highlight indicates change from previous report

SMD

ESMD

ARMD

SOMD

MD

Astrophysics

Physics of the Cosmos

Cosmic Origins

Planetary Science

Lunar Quest

Constellation (in development; expect approval at KDP-II)

Formulation Implementation

Astrophysics

SOFIA (7/26/07)

Hubble Space Telescope (2/27/09)

Fermi (formerly GLAST) (8/18/08)

James Webb Space Telescope (3/17/09)

ExoPlanet Exploration (4/14/09)

Earth Science

Earth Systematic Missions (4/14/08)

ESSP (4/25/08)

Lunar Precursor Robotic (Waived requirement)

Fundamental Aeronautics (5/21/07)

Aeronautics Test (5/21/07)

SSP (1/26/07)

SCaN (10/16/08)

TDRS K/L (4/18/07)

Planetary Science

Discovery (8/19/08)

Mars Exploration (1/26/07)

New Frontiers (3/18/08)

Heliophysics

Explorer (2/2/09)

Living with a Star (8/25/08)

New Millennium (10/6/08)

Solar Terrestrial Probes (12/21/07)

Human Research (5/21/07) Rev A in Rev

Technology Development (10/21/07)

Airspace Systems (9/14/2007)

Aviation Safety (9/17/2007)

ISS (8/01/08)

LSP (1/28/08)

RPT (1/23/09)

CHSP (waived requirement)

Agency Programs: PCA Status

PCA Status Legend Dates are last signed PCA

= Signed PCA with no update required = MD has identified for update = No signed PCA BLACK = NA

G

G

G

G

G

G

G

G

G Y R

G

G

G

G

G

G

G

G

G

G

G

GG

G

G

Y

Y

Y

21February 2010

Agency Review Schedule

For NASA Internal Use OnlyFebruary 2010

NASA Monthly Baseline Performance Review (BPR)

GSFC Center Summary InputHighlights from 11/17/2009

Technical Authority/Mission Support/InstitutionalInstitutional Risk to Mission

Budget/Finance:Grants reconciliation and transition: The ORR for the Phase 2 transition

to NSSC is scheduled for November 30th; will complete GSFC transition activities.

Workforce:Early Career Hiring: FY09 - 66 on-board; FY10 – 10 with an additional

10 offers accepted and negotiating dates.

Acquisition:Aerospace Corp. Support: Consolidated Agency contracting approach

being pursued. GSFC had restricted authorization to use the existing GSFC contract for critical tasks ending in December. “Authorization” to seek services on the Airforce contract granted but is not expected to save resources and requires unnecessary actions.

SCNS contract protest activities continue. NENS contract has been extended through April 2010. The delayed transition from NENS to SCNS continues to impact the contractor’s ability to retain experienced employees to perform the work.• Agency-level protests denied on August 11th and the Corrective Action Plan (CAP) implementation was initiated on August 17th. • A protest was submitted on August 21st to GAO with regard to the CAP. CAP implementation permitted to continue. GAO decision due NLT November 30th. • Congressional request for information with regard to the CAP was submitted October 22nd.

Programs/ProjectsSummary Performance

ESMD:ESPLRO – Supported LCROSS Lunar Impact, data being analyzed.

Transition to SSMO will occur in December.SMD:

HST- SM4 SMOV activities complete.– Working possible options to increase FY10 reserves. OTE

CDR complete.GLORY- Successful CMC and ESD rebaseline reviews conducted. LV

actions being worked prior to DPMC.GPM – Re-planned road to KDP-C. Working toward Mission CDR

planned for December 14-17th.LDCMNPP - Systemic issues with systems engineering and mission

assurance escapes on IPO-provided instruments reduce confidence in Instrument life expectancy to be less than the 5-year design life.

MMS – S Band antenna height change complete. Swedish Univ, KTH ability to deliver Spin Plane Probe Electric Field Instruments.

RBSP–Schedule delays are being worked.SDO – LRD is 2/3/2010. AIA Hopa Circuit anomaly. SAM – Power Distribution Box is integrated onto the flight suite.GOES-R – ITT performance on ABI.NOAA Climate Sensors

SOMD:Ground Network Space Network- See Acquisition, SCNS.TDRS – Continue to Work with BSS regarding requirements flow-down

to subcontractors.SGSS – RFP development activities continue.ELC – Successful SMSR October 23rd. Decks #1 and #2 were

Launched November 16th.

23February 2010

SMD OverviewNovember 2009

Status:

Programmatic

TechnicalPlanetary: MSL actuators. MRO safemode implications on developing missions using heritage avionics (e.g. Juno, GRAIL, MAVEN)All: Continued difficulties developing instruments; Implications from usage of Western Titanium.

ScheduleAstrophysics:Earth Sci: Glory Taurus XL return to flight schedule. Potential further NPP launch delays from instrument rework.

CostAll: Access to Space continues to be an issue: Delta II replacement; Taurus XL failure, & impact on Glory. Very crowded launch manifest in fall 2011.

Planetary: Funding of MSL slip severely impacts divisionAstrophysics: Overall division budget issues, esp JWST.

Watch list Items:

– 2 project rated Red overall: NPP, SOFIA– 4 projects rated Yellow overall: JWST, MSL, Juno, Glory– SOFIA changed from Yellow to Red due to eroding schedule, cost issues & delay in rebaseline APMC.– Juno progress degraded from Green to Yellow; Glory returned to Yellow pending replan DPMC.– Replan DPMCs pending on Glory, Aquarius, & MSL.– MRO in safemode due to computer resets; Restarts unsuccessful for Last IceSat laser failure

Uncertainty of SMAP launch date for project planning

24February 2010 For NASA Internal Use Only

2009 Month COMMENTS

Ares I-X

Y Y G Successful Launch!! October 28, 2009 at 11:30 am ET

Post-Flight Data Review working group will meet twice a week to review progress of data collection and reduction

The first stage solid rocket motor recovered; hardware disposition is in progress; data recorder removed and shipped to Lockheed Martin. Lockheed Martin who will provide best and final set of all flight data from a combination of data from telemetry and the on-board recorder

Lunar Surface Systems

Insufficient funds in PMR09 budget to meet HLR in 2020. Constrained to “study only” after FY2010.

A B C D

ESMD Projects Status Roll Up Continued

S O N S O N S O N S O N

25February 2010

PSD Budget Reporting Matrix (11/16/09)

InternalMPAR

(Baseline) IBPD (FY10)*1

OMB Qtrly (4QFY09) PCA/PRLA/other*2

Development $968.6 $1,631.0 $1,719.9 $1657 ₃LCC $1,642.2 $2,299.3 $2,394.3 $2,318.5LRD Sep 2009 Oct 2011 Nov-11 Oct-11

Development $427.0 $427.0 $427.0 $407.7LCC $496.2 $465.6M* $496.2 $468.2LRD Sep 2011 Sep 2011 Sep 2011 Sep 2011

Development $742.3 $742.3 $742.3 $695.3LCC $1,107.0 $1,091.9M* $1,107.0 $1,050.0LRD Aug 2011 Aug 2011 Aug 2011 Aug-11

*1 - IBPD is updated through N2 automatically and as of 4-15-9 N2 has not been updated to reflect the FY09 Op Plan Changes nor the GRAIL and Juno KDP-C decisions.*2 - The PCA (Program Commitment Agreement) is located in Science Works under Requirements Management system, Planetary Science, Program, Key Document PCA, and KPD *3 - MSL project assumes an approved $13.4M adjustment from Phase E to Phase C/D, which is NOT included in the $1657M internal development number.

Project

MSL

GRAIL

JUNO

External

26

Status: In Phase C

Technical:

Cost:

Schedule:

Programmatic:

JWST YY

R

Y G

• WATCH: Low near-term reserves pose a threat to LRD.

• NIRCam off-axis image quality• NIRSpec microshutter separation - decision to change coating

• FY'10 reserves negative & FY'11 reserves only 0.9%• Project requested $20M addition to FY'10 reserves. HQ told project to continue to fund JSC facility mod, with decision on extra $20M in 4-6 weeks.

Oct Nov

Y

G

Summary

• NASA Senior Management monitors Agency-Wide performance monthly through its baseline Performance Review (BPR) Process Each Program area is assessed quarterly on a rotating basis

• The automated tool is coming online at MSFC• BPR continues to evolve and improve

Backup Charts

29

BPR Actions and Status

7.23.2009 BPR Meeting• 2009-07-026: PA&E – Revisit the FY09 Q3 Contract reporting chart

to clarify the way the information is being conveyed. (E.g. Contract base value apparent misalignment with LCC.) Due: October 2009 BPR

Status: Propose to CLOSE. PA&E has met with ESMD to identify how to add clarity to this reporting. ESMD will provide additional information that clarifies the contract values. Interim information was presented in September BPR by PA&E. ESMD has provided data from the Ares I and Orion. Language has been added to the contract chart to explain what contracts are and are not included in contract reporting to OMB.

Assessment Color Criteria

Green Criteria

A Category for a Project is Green if:There are minimal programmatic issues and adequate margin (budget reserve,

schedule slack, technical margin), or capability (for operational missions), to complete the Project on plan.

A Project is Green overall if:All 4 categories are Green, orOne or more are Yellow and the issues are collectively not a threat/impact to the

project by virtue of their being solvable within the project's planned resources. (Margin in one category may provide capability to resolve a lack of margin in another category.)

A Program is Green if:The constituent projects are Green overall, or are Yellow with realistic, defined plans

to return to green and all other program elements are on budget and schedule.

Yellow Criteria

A Category for a Project is Yellow if:The remaining margin within the category appears inadequate to meet Project requirements or external commitments, however, mitigation plans are in place or in development to recover margin and achieve commitments within the available resources of the Project. (For operational projects in the technical category "margin" may be interpreted as operational capabilities.)

A Project is Yellow overall if:Any one category is Red, the overall is at least Yellow, orOne or more categories are Yellow and the issues are collectively a threat to the project's ability to meet their commitments within the project's planned resources.

A Program is Yellow if:Any of its constituent projects are Red overall, or if one or more are Yellow and the issues are collectively a threat to the Program's ability to meet its commitments within planned resources.

Red Criteria

A Category for a Project is Red if:There is minimal margin or negative margin within the category such that the Project is estimated not to meet its requirements or commitments without significant impact to other categories. There is high risk that the Project will require external resources or relief from performance or programmatic requirements.

A Project is Red overall if:One or more categories are Red, and it is estimated that the Project will not have sufficient resources to meet its commitments.

A Program is Red if:Most of its constituent projects are Red overall, or if one or more are Red and the issues are collectively a threat to the Program's ability to meet its commitments.

Assessment Rubric for Space Flight Projects and Multi-Project Programs

Cost Schedule Technical Programmatic

Cost estimates remain consistent with cost commitments:-- Formulation: Current estimated formulation costs are consistent with the program FAD. -- Development: Current estimated LCC at completion is at or below the LCC commitment. For multi-project programs, the current cost estimate by year for the program’s projects, including reserve, is within the 5 year cost run-out in the program plan. The program is consistently executing its projects within their cost cap or commitment.-- Operations: Annual operational costs are at or below planned costs for these operationsand there are minimal issues and adequate cost margin (budget reserve) to complete the Program or Project on plan: -Lifecycle cost reserves are consistent with plan and are being drawn down at a rate consistent with plan and emergent cost risks/threats. For multi-project programs, reserves are being managed consistent with the annual reserves planned for each project. -- The project/program will not require additional funds to complete the current fiscal year with required carryover. -- Phasing of cost reserves are consistent with emergent risks or threats. -- Current year budget level and timing of funds distribution are consistent with plan or commitments are not threatened as a result of any or delay in current year funding. -- External partners are providing key resources as planned.

Schedule progress remains consistent with schedule commitments: -- Formulation: The current estimated authority-to-proceed to implementation date is at or before the FAD.-- Development: Current estimated operational readiness date (ORD) is at or before the ORD commitment. For multi-project programs, the current estimated schedule for planned start dates for future projects is consistent with the program plan.-- Operations: Planned operations, including maintenance, are proceeding on schedule. and there are minimal issues and adequate schedule margin to complete the Program or Project on plan:--Funded schedule reserves & phasing are consistent with plan and are being drawn down at a rate consistent with plan and emergent schedule risks/threats. For multi-project programs, any faster-than-planned project draw-down of schedule reserve is not impinging on the ability of the program to meet its overall schedule commitments. --Key interim milestones (KDP as well as WBS milestones) being met. -- For multi-project programs, any delay in project schedule is not jeopardizing inputs required for another project or resulting in an increased risk to the program’s schedule commitments.-- External partners are providing key deliverables on schedule.

Technical progress remains consistent with technical commitments: -- Formulation: The project/program is making planned progress in developing technical design, reliability, safety, and risk mitigation requirements necessary to proceed to implementation as planned. -- Development: Technical requirements are stable and progress and risk management is consistent with plan while maintaining safety. For multi-project programs, any lack of planned technical progress at the project level is not resulting in an increased risk to the program’s technical commitments and objectives.-- Operations: Operational systems are delivering the planned level of services and reliability, with a general trend towards improved safety, and efficiency. and there are minimal issues and adequate technical margin to complete the Program or Project on plan:-- Technical reserves are being drawn down are consistent with plan; technical difficulties/risks encountered are typical for this type of project/program and are being mitigated or controlled as required to make planned technical progress.-- Are consistent with the requirements of safety, reliability & quality assurance, and incorporate good systems engineering & integration practice.-- External partners are making planned technical progress and external events (e.g., import approvals) are not altering technical requirements. -- For operational projects, the ground and in-flight hardware is operating satisfactorily on prime or backup units and mission requirements are being met.

Programmatic performance remains consistent with program/project requirements: -- Formulation: An approved FAD is in place and planned progress is being made in completing programmatic arrangements necessary to enter development as planned. -- Development: Top-level objectives and programmatic requirements in the plan are stable and programmatic capabilities are being developed/ managed as planned. Plans are consistent with 7120.5 or necessary waivers are in place. A multi-project program is effectively managing its projects, with new projects beginning per the Program Plan.-- Operations. Operational requirements are stable consistent with plan, and operations are being managed as planned. Retirement/ transition plans are being developed/ implemented consistent with the timing for these events. and there are minimal issues and adequate programmatic flexibility to complete the Program or Project on plan:--Key contract actions are being undertaken and completed as planned. The multi-project program’s acquisition strategy (both contracted and in-house) is resulting in the timely start of projects and delivery of project elements.--Key critical workforce, access to key facilities, EVM, and management systems are in place consistent with plan. --Key partnerships are in place and operating as planned; external requirements are stable; and there are no other factors external to the project/program (e.g., new Agency requirements) which are significantly disrupting planned project/program management.

Assessment Rubric for Space Flight Projects and Multi-Project Programs (continued)

Cost Schedule Technical Programmatic

In general, the program or project follows the green assessment guidelines, but cost commitments are now threatened.

Remaining cost margin appears inadequate to meet requirements due to one or more of the following deviations from plan, however, mitigation plans are in place or in development to recover margin and achieve commitments within the planned resources of the Program or Project:

-- Lifecycle cost reserves (including those held at the program level) are being drawn down faster than planned or required for emergent cost risks/threats.

-- Phasing of cost reserves are inconsistent with emergent risks or threats.

-- Current year cost growth is exceeding planned current year cost reserves and will require a re-phasing of the annual budget in order to complete the fiscal year with required carryover.

-- A reduction in current year budget or slow- down in funds distribution will require the project/program to rephrase future year annual costs.

-- External partner changes or factors external to the project/program are placing additional pressure on project/program costs.

-- For operating missions, in flight anomalies are draining cost reserves.

In general, the program or project follows the green assessment guidelines, but schedule commitments are now threatened.

Remaining schedule margin appears

inadequate to meet requirements due to one or more of the following deviations from plan, however, mitigation plans are in place or in development to recover margin and achieve commitments within the planned resources:

-- Schedule reserves are being drawn down faster than planned or as required for emergent schedule risks/threats.

-- Key interim milestones (KDP and/or WBS milestones) have slipped.

-- For multi-project programs, delay in project schedules is resulting in an increased risk to the program’s schedule commitments.

-- External partners are delayed in providing key deliverables or external factors are delaying completion of a key milestone.

In general, the program or project follows the green assessment guidelines, but technical commitments are now threatened:

Remaining technical margins appear

inadequate to meet requirementsdue to one or more of the following technical deviations from plan however, mitigation plans are in place or in development to recover margin and achieve commitments within the planned resources:

-- Technical reserves are being drawn down at a rate inconsistent with plan; or technical difficulties/risks encountered exceed those which are typical for this type of project/program or are not being mitigated or controlled as required to make planned technical progress.

-- Are not consistent in some key areas with the requirements of safety, reliability & quality assurance, and the incorporation of good systems engineering & integration practice.

-- External partners are not making planned technical progress or external events (e.g., import approvals) are altering technical requirements.

-- For operational projects, ground or flight capabilities have been curtailed, but plans are in work to recover the capability or develop a workaround such that mission requirements will still be met.

In general, programmatic performance remains consistent with program/project requirements for a green assessment, but remaining programmatic flexibility may be

inadequate to meet requirements due to one or more of the following programmatic deviations from plan however, mitigation plans are in place or in development to recover margin and achieve commitments within the planned resources:

-- Key contract actions are not being undertaken or completed as planned.

-- A multi-project program’s acquisition strategy (both contracted and in-house) is not resulting in the timely start of projects and delivery of project elements.

-- Key critical workforce, access to key facilities, EVM, and management systems are not consistent with plan..

-- Key partnerships are not in place or operating as planned; external requirements are not stable; or there are other factors external to the project/program (e.g., new Agency requirements) which are significantly disrupting planned project/program management.

Assessment Rubric for Space Flight Projects and Multi-Project Programs (continued)

Cost Schedule Technical ProgrammaticThe Program or Project is estimated not to

meet its cost commitments without significant impact to other categories.

Depending on phase, the project's formulation costs, or development costs, or annual operating costs are exceeding the planned budget beyond all planned reserves.

For multi-project programs, total annual costs for the program’s projects exceed the program’s 5 year planned run-out, including reserves.

There is minimal or negative cost margin such that there is high risk that the Program or Project will require additional funding or relief from performance or programmatic requirements due to one or more of the following deviations from plan for a realistic recovery strategy has not been identified:

-- Lifecycle cost reserves (including those held at the program level) are being drawn down at an unsustainable rate and remaining reserves are insufficient given emergent risks/threats.

-- Phasing of cost reserves are inconsistent with emergent risks or threats.

-- Current year cost growth exceeds planned costs and cannot be accommodated by rephrasing annual costs.

-- A reduction or delay in current year funding is inconsistent with maintaining the overall LCC commitment.

-- A loss of key external partners or resources is inconsistent with keeping the project or program within planned costs.

-- For operational projects, in-flight anomalies have drained all cost reserves and the project will need more funds to complete the prime mission.

The Program or Project is estimated not to meet its schedule commitments without significant impact to other categories. The key end-of-phase KDP milestones are unlikely to be met.

For multi-project programs, the planned start-up of future projects is delayed beyond program commitments or objectives.

For operations missions, the current operations schedule is insufficient to meet project commitments or the mission failed prior to the planned end-of-operations date. For multi-project programs: The current operations schedule is insufficient to meet program commitments or the failure of one or missions prior to the planned end date has resulted in the project not being able to meet its commitments or objectives.

There is minimal or negative schedule margin is such that there is high risk that the Program or Project will require additional schedule or relief from performance or programmatic requirements due to one or more of the following deviations from plan for which a realistic strategy for recovery has not been identified:

-- Schedule reserves are being drawn down at an unsustainable rate and remaining reserves are insufficient given emergent risks/threats.

-- A slip of one or more key interim milestones (KDP and/or WBS milestones) is inconsistent with the planned ORD. For programs, delay in project schedules is inconsistent with program’s schedule commitments.

-- An external partner delays or delays caused by other external factors are inconsistent with the planned ORD. For multi-project programs, delays caused by external factors are inconsistent with the program’s schedule commitments.

The Program or Project is estimated not to meet its technical requirements without significant impact to other categories:.

A project in formulation is not making the technical progress required to enter development.

For a project in development, technical failures or delays preclude the project from being able to meet its technical requirements and commitments.

There is minimal or negative technical margin such that there is high risk that the Program or Project will require additional technical resources or relief from performance or programmatic requirements due to one or more of the following technical deviations from plan for which the project or program has not identified a realistic strategy for recovery:

-- Technical reserves are being drawn down at an unsustainable rate or emergent technical difficulties, risks, or threats are not being resolved as required to meet the technical commitments.

-- Are not consistent with the requirements of safety, reliability & quality assurance, and the incorporation of good systems engineering & integration practice.

-- Lack of technical progress by external partners or external events (e.g., import approvals) are inconsistent with meeting the project/program technical requirements, commitments, or objectives, or additional/different external cannot be met within plan.

-- For operational missions, in-flight anomalies have used up redundancy and made it unlikely that the mission will achieve its mission requirements.

The Program or Project is estimated not to be able to meet its requirements due to programmatic issues affecting the other categories.

A formulation project has failed to make the programmatic progress required to be confirmed for development.

A development project lacks an approved plan or has failed to implement that plan consistent with meeting its commitments. The multi-project program lacks an approved PCA or project plan, or has failed to implement the projects require to meet its commitments.

There is limited or no programmatic flexibility such that there is high risk that the Program or Project will require relief from performance or programmatic requirements due to one or more of the following deviations from plan for which the project/ program has not identified a realistic strategy for recovery:

--Key contract actions are not being undertaken or completed as required. The multi-project program’s acquisition strategy (both contracted and in-house) is not resulting in the timely start of projects and delivery of project elements.

--Key critical workforce, availability of key facilities, EVM, and management systems are missing or inconsistent with meeting the project’s commitments.

--Key planned partnerships did not materialize, quit or not to delivering consistent with project/program commitments; unstable or newly emergent external requirements are not implementable; or other external developments are inconsistent with the project/program commitments.

Associate AdministratorAnd Associate Deputy

Administrator

Division Chief for Engineering and

Program & Project Management

ChiefEngineer

BPRTeam Lead

BPR Process Organizational Structure

As of June 3, 2008

38February 2010 For NASA Internal Use Only

PA&ETeam Lead

TechnicalCross-Cutting

Team

OSMATeam Lead

Center POC’s

BPR Process Organizational Structure

As of June 3, 2008

ExecutiveSecretariat

I&AITOCFOOHCMOPIIPROC

Non-Technical

Cross Cutting

Special Topic / Root Cause

Analysis Lead

BPRTeam Lead

Institutional Assessment Team Lead

(*) One representative from each

OCE AdminSupport

PA&E Assessor

OSMA Assessor

MSO Representatives

(One representative from each MD)

Program&

Projects

MissionDirectorate

(*)Centers

MissionSupportOffices

OCE MD P&PAssessment

Lead

39For NASA Internal Use OnlyFebruary 2010

Primary BPR Deliverables

• Monthly meeting BPR presentation package• Executive summary and pre-brief• Meeting Summary Notes and Post-brief• Executive Summary for the Administrator and Deputy Administrator• Meeting Actions List and tracking

40