SCAMPI-B for Contract Monitoring - Carnegie Mellon … · SCAMPI-B for Contract ... Sponsored by...
Transcript of SCAMPI-B for Contract Monitoring - Carnegie Mellon … · SCAMPI-B for Contract ... Sponsored by...
© 2006 Carnegie Mellon University
SCAMPI-B for Contract Monitoring—A Case Study of the Mission Planning Enterprise Contractors
Sponsored by the U.S. Department of Defense
Lorraine Adams, SEIKathy Bastien, BlueForce LLCMarch 2007SM SCAMPI is a service mark of Carnegie Mellon University®CMMI and Capability Maturity Model are registered in the US Patent and Trade Office© 2006 BlueForce LLC. All rights reserved
2
SCAMPI-B for Contract MonitoringLorraine Adams and Kathy BastienMarch 2007© 2006 Carnegie Mellon University
Contents / Agenda
Background
What is SCAMPI-B for Contract Monitoring
SCAMPI-B for Contract Monitoring at MPEC
Program Office Changes Resulting from SCAMPI-B for Contract Monitoring
Lessons Learned
Way Ahead
Summary
3
SCAMPI-B for Contract MonitoringLorraine Adams and Kathy BastienMarch 2007© 2006 Carnegie Mellon University
DoD Weapon Systems Acquisition Still a High Risk Area
While DOD’s acquisition process has produced the best weapons in the world, it also yields undesirable consequences in weapon system programs – cost increases, schedule delays, and performance shortfalls
Problems occur because weapon programs do not capture early on the requisite knowledge that is needed to effectively manage risks
Programs move forward with unrealistic cost and schedules estimates, lack clearly defined and stable requirements, use immature technologies, and fail to solidify design and manufacturing processes at appropriate junctures in development
As a result, programs require more resources than planned, the buying power of the defense dollar is reduced, and funds are not available for other competing needs
Paul Francis, Director Acquisition and Sourcing ManagementU.S. Government Accountability OfficeMay 18, 2004
4
SCAMPI-B for Contract MonitoringLorraine Adams and Kathy BastienMarch 2007© 2006 Carnegie Mellon University
Problem Statement
• A statement of organizational process maturity or capability level does not guarantee performance to that same level of proficiency on an individual project
• Most DoD contractors claim high maturity/capability levels, yet from the perspective of the acquirer, systems engineering and program management practices are severely lacking
• Teaming arrangements further cloud the issue of process execution and proficiency
• Associated problems may not be evident until significant cost, schedule, or performance objectives have been missed at a late point in the program where corrective actions are very costly
How can the acquirer gain the necessary insight into process execution and proficiency as well as reinforce desired behaviors?
5
SCAMPI-B for Contract MonitoringLorraine Adams and Kathy BastienMarch 2007© 2006 Carnegie Mellon University
The Mission Planning Solution and What was Learned
A “simple” solution - solve the problem through contract requirements
• All contractors must be at least CMM maturity level 3 to bid on IDIQ software development contracts
• All winning contractors must achieve CMMI maturity level 3 NLT 24 months after contract award
Monitor the contracts to assure that CMMI requirement is met by deadline
Conduct a round of CMMI baseline SCAMPI-Bs for Contract Monitoring to assist process improvement toward meeting that goal
The process of conducting 5 baseline SCAMPI-Bs for Contract Monitoring was an “eye opening” experience
6
SCAMPI-B for Contract MonitoringLorraine Adams and Kathy BastienMarch 2007© 2006 Carnegie Mellon University
What is SCAMPI-B for Contract Monitoring
7
SCAMPI-B for Contract MonitoringLorraine Adams and Kathy BastienMarch 2007© 2006 Carnegie Mellon University
• Maturity Levels are indicators of organizational potential performance
• They describe how the next project may perform based on a sampling of existing projects
• Maturity Levels reside at the organizational level and are not an indication of how an individual project is performing
XYZ Corp.
DivisionA
Project 1
Project 2
Project 3
Division B
Division N
Project 5
Project 6
Project 7
Project x
Project y
Project z
Project 4
MLs apply HEREbased uponappraisals ofTHESE …
… but yourproject isHERE or HERE
High Maturity High MaturityOrganizations Projects
8
SCAMPI-B for Contract MonitoringLorraine Adams and Kathy BastienMarch 2007© 2006 Carnegie Mellon University
Standard CMMI Appraisal Method for Process Improvement (SCAMPI)
Con
fiden
ce/A
ccur
acy
Cost/Duration
Class C MethodsClass C Methods
Class B MethodsClass B Methods
Class A Method
Class A Method
No ratings
No ratings
Maturity or Capability ratings
Benchmarking and Baselining
Quick Look“Intentions” for execution
Tailorable Assessment“Mini” Appraisal
as executed
9
SCAMPI-B for Contract MonitoringLorraine Adams and Kathy BastienMarch 2007© 2006 Carnegie Mellon University
SCAMPI-B for Contract Monitoring Ground Rules - 1
Use the process model – CMMI
• Interview questions based on model
Appraisal of process performance and adherence
Focus on risk assessment – risks associated with process performance, adherence, and capability
10
SCAMPI-B for Contract MonitoringLorraine Adams and Kathy BastienMarch 2007© 2006 Carnegie Mellon University
SCAMPI-B for Contract Monitoring Ground Rules - 2
Observe strict confidentiality
• Results not attributable to individuals or interview groups
Approach SCAMPI collaboratively
Results in actionable findings by Program Office and/or Contractor
11
SCAMPI-B for Contract MonitoringLorraine Adams and Kathy BastienMarch 2007© 2006 Carnegie Mellon University
SCAMPI-B for Contract Monitoring for MPEC
12
SCAMPI-B for Contract MonitoringLorraine Adams and Kathy BastienMarch 2007© 2006 Carnegie Mellon University
Mission Planning Contracting Structure
Contractor#4
Contractor#1
Contractor#2
Contractor#3
Contractor#5
IDIQ
Delivery Orders
DO 1
DO 4
DO 3
DO 2
DO 7
DO 5
DO 6
DO 9
DO 8
DO 10
13
SCAMPI-B for Contract MonitoringLorraine Adams and Kathy BastienMarch 2007© 2006 Carnegie Mellon University
How SCAMPI-B for Contract Monitoring was Employed for MPEC - 1
Used appraisals to baseline compliance with CMMI requirement
• Demonstrate capability or
• Develop findings which yield improvement opportunities
One project would be selected from each contractor as a ‘representative’of all Mission Planning projects for that contractor
14
SCAMPI-B for Contract MonitoringLorraine Adams and Kathy BastienMarch 2007© 2006 Carnegie Mellon University
How SCAMPI-B for Contract Monitoring was Employed for MPEC - 2
Appraisal findings would be used to assess progress toward meeting this the 24 month CMMI requirement
• Results to be factored into future DO competitions
Appraisal finding to be used for contract monitoring to identifyareas of risk in project execution
Appraisal findings resulted in request for Process Improvement Plans from the contractors
• Way ahead to fix findings in representative project
• Way ahead to fix similar known deficiencies in all Mission Planning work
15
SCAMPI-B for Contract MonitoringLorraine Adams and Kathy BastienMarch 2007© 2006 Carnegie Mellon University
Program Office Changes Resulting from SCAMPI-B for Contract Monitoring
16
SCAMPI-B for Contract MonitoringLorraine Adams and Kathy BastienMarch 2007© 2006 Carnegie Mellon University
Results Changed the Way the Program Office Does Business - 1
• Contractor Process Improvement Plans and status are tied to contractor award fee
• RFP language has been modified to better reflect the program office desire for CMMI compliance across development teams – including subcontractors
• Based on input from appraisals, the program office is reviewing/modifying the standard CDRL list to get best ROI
• Business and technical rules for MP developers have been modified
17
SCAMPI-B for Contract MonitoringLorraine Adams and Kathy BastienMarch 2007© 2006 Carnegie Mellon University
Results Changed the Way the Program Office Does Business - 2
• Program Office is modifying internal processes to better take advantage of information provided through contractor’s standard processes (e.g. metrics)
“In light of significant personnel cuts, we need to provide a smart way to provide a laser-like focus to our key issues. SCAMPI-B for Contract Monitoring
reviews and EVM analysis provide that focus for us.”Steven A. Cote, GS-15, DAF
Deputy Director, 951st Electronic Systems Group
18
SCAMPI-B for Contract MonitoringLorraine Adams and Kathy BastienMarch 2007© 2006 Carnegie Mellon University
Lessons Learned
19
SCAMPI-B for Contract MonitoringLorraine Adams and Kathy BastienMarch 2007© 2006 Carnegie Mellon University
Positive Lessons / Benefits - 1
Overcoming common misperceptions program offices have about CMMI
• What does a maturity level mean – how is it attained
• Need to understand the scope of previous appraisals and certifications
— Organizational unit appraised
20
SCAMPI-B for Contract MonitoringLorraine Adams and Kathy BastienMarch 2007© 2006 Carnegie Mellon University
Positive Lessons / Benefits - 2
More savvy consumers
• Monitoring is more necessary than people think
• CMMI compliance and appraisal finding resolution should be tied into program office business activities
— Award Fee Criteria (powerful motivator)
— Business and Technical Rules
— Proposal Evaluation Criteria
21
SCAMPI-B for Contract MonitoringLorraine Adams and Kathy BastienMarch 2007© 2006 Carnegie Mellon University
Positive Lessons / Benefits - 3
Appraisal results can be used to assist program management
• PMRs can focus on areas of concern based on findings
Uncovering program risks is more important than specific maturity levels
Real-time contract monitoring is key
22
SCAMPI-B for Contract MonitoringLorraine Adams and Kathy BastienMarch 2007© 2006 Carnegie Mellon University
Hard Lessons / Ongoing Challenges - 1
Spirit vs. letter of the model
• Assure intent of the model is met versus using “checklist” mentality
• Determine consistent approach to dealing with corporate “process lawyers”
• Shouldn’t judge “goodness,” but you should judge “reasonableness”
23
SCAMPI-B for Contract MonitoringLorraine Adams and Kathy BastienMarch 2007© 2006 Carnegie Mellon University
Hard Lessons / Ongoing Challenges - 2
Variability in the process itself, within the allowable scope of the method, can itself modify results
• Teams take on personalities based on membership (team members and lead appraisers)
• Different personalities can provide inconsistent results
• Need to ensure that decision criteria are consistently applied
• Having a core group as a subset of each appraisal team provided a consistent methodology and interpretation of evidence
24
SCAMPI-B for Contract MonitoringLorraine Adams and Kathy BastienMarch 2007© 2006 Carnegie Mellon University
Future Directions
25
SCAMPI-B for Contract MonitoringLorraine Adams and Kathy BastienMarch 2007© 2006 Carnegie Mellon University
Way Ahead -1
Mission Planning is reviewing options for verification of the November 2006 CMMI Maturity Level 3 requirement
• Accept results of contractor SCAMPI As?
— Review of detailed appraisal results
• Conduct Program Office sponsored SCAMPI As?
— Requires core team to re-appraise contractors
• Verify completion of all process improvement activities outlined in approved Process Improvement Plan
26
SCAMPI-B for Contract MonitoringLorraine Adams and Kathy BastienMarch 2007© 2006 Carnegie Mellon University
Way Ahead - 2
How do we “maintain” the process improvement/process maturity focus to ensure consistent levels of performance/execution?
• The Program Office is working to develop a multi-year plan for continual assessment/monitoring
— Additional appraisal for all contractors (cost/benefit)
— “Spot check” areas of known deficiency
— Appraisals for projects with performance issues
— Appraisals for projects that fall within another organizational unit than “representative” project
27
SCAMPI-B for Contract MonitoringLorraine Adams and Kathy BastienMarch 2007© 2006 Carnegie Mellon University
Way Ahead - 3
Beginning to analyze SCAMPI-B for Contract Monitoring results against product quality to determine if there are correlations
• Could impact risk management/corrective actions in the future
28
SCAMPI-B for Contract MonitoringLorraine Adams and Kathy BastienMarch 2007© 2006 Carnegie Mellon University
Summary
29
SCAMPI-B for Contract MonitoringLorraine Adams and Kathy BastienMarch 2007© 2006 Carnegie Mellon University
Summary - 1
SCAMPI-Bs for Contract Monitoring were very positive for Mission Planning
• Provided insight into contractor processes and potential program risks
Consistency is key
Appraisal results have become a useful source of data to assist in program management activities
• PMs can focus attention on areas of the project that have highest risk due to inconsistent (or non-existent) standard processes
30
SCAMPI-B for Contract MonitoringLorraine Adams and Kathy BastienMarch 2007© 2006 Carnegie Mellon University
Summary - 2
SCAMPI-Bs for Contract Monitoring started as a way to do a “quick check” of contractor CMMI requirement compliance – and have now become a powerful management tool
Savvy consumers can utilize SCAMPI-Bs for Contract Monitoring as part of their management “tool box”
32
SCAMPI-B for Contract MonitoringLorraine Adams and Kathy BastienMarch 2007© 2006 Carnegie Mellon University
Contact Information
Lorraine AdamsMember of the Technical Staff
Software Engineering Institute
4500 5th Avenue
Pittsburgh, PA 15213
412-268-7777
Kathy BastienSenior Process Manager
BlueForce LLC
1919 Commerce Drive, Suite 400
Hampton, VA 23666
603-548-5936