DoD Acquisition Best Practices Clearinghouse Content Advisory Group (CAG) 12 April 2006

DoD Acquisition Best Practices Clearinghouse

Content Advisory Group (CAG)

12 April 2006

2

Meeting Purpose

Tasking from 14 Sep 05 OSD Briefing: Complete and demonstrate the practice and evidence information handling process such that there is clear data to indicate the “validation and vetting” system can be executed with available and affordable resources in DAU,

OSD, Services, Agencies and Industry.

• Finalize the CAG charter

• Review planned content evolution schedule– Current content activities– Planned content activities– Existing content status

• Identify pilot working groups– Establish goals– Define the approach

3

Agenda• Introduction and BPCh Program Update [Mike Lambert]

– CAG Meeting Purpose/Agenda – CAG Charter and Comments– Other Program Activities

• BPCh Content Update [Forrest Shull]– Content Schedule– Risk Management– Earned Value Management– Other Content Areas– Lessons Learned

• BPCh Information Handling Process [Michele Shaw / Mike Lambert]– Current Roles and Activities– Scenarios List– Recommended Subgroups– Scenarios to Pilot

• CAG Wrap Up [Mike Lambert]– Next Steps / Action Items / Next Meeting

4

CAG Charter Summary

• A small group of people familiar with – SE/SA practices that are useful / recommended– Needs / “pain points” of likely users

• …that can help improve BPCh:– Define, manage, and track content evolution

activities to ensure that they are consistent across the BPCh program goals.

– Lead efforts to establish efficient and effective information handling processes for populating content within the BPCh.

– Establish and build relationships to support the growth and continued evolution of BPCh content.

– Provide an action-oriented collaborative forum for improving content gathering, processing with the overall goal of promoting the increased use of BPCh.

5

Round Table

• What have you done for BPCh since we last met?– Suggested an evidence source– Presented BPCh to an interested party– Identified a potential practice owner– Participated on working group– Sent the BPCh link to a potential user– Added content (lead, war story, evidence,

practice summary etc)– Submitted BPCh process improvements

BPCh Program Update

7

Since we last met…

• Recent/Current Activities– $75K each received from DS, ARA & OSD(NII),

added to $20K from DAU I.T. brings total BPCh FY06 budget to $245K

– Bug tracking system established for current BPCh prototype and issues are being reported/solved (per schedule)

– Work on risk mitigation plan to begin in earnest in May (per schedule)

– Met w/ DAU taxonomy PM to discuss integration of developing taxonomy into BPCh

– “Bigger picture”: working w/ DAU KM team on how best to integrate BPCh & other KM systems into an Acq Knowledge Management System (AKMS), which is a foundational part of DAU’s evolving Engaged Learner Architecture (ELA)

8

Central Repository for AT&L Policy and Reference Materials

Collaborative Tool for the AT&L Community Where the Workforce Contributes Knowledge and Interacts to Share “Know-How”

Enterprise Search SystemDoD & Industry Best Practices

Also serves as the home for knowledge gateways like:• Defense Acquisition Guidebook• AT&L Integrated Framework Chart (IFC)• Ask A Professor

Provides a nest of collaborative tools:• Communities of Practice/Interest• Special Interest Areas• Limited Access Workspaces• DAU Course Spaces• Workflow Learning Tools• IFC Templates

• Stand-alone search and discovery for AT&L workforce• Integrated search for AKSS• Searches open areas of ACC• Integrated search for DAU Homepage• Integrated search for DAU Intranet

Integrated Systemof Systems

• Will stand alone as a best practices resource• Will also provide content for CoPs to allow for additional collaboration/input on best practices• Will be included in the enterprise search index/results

Elements of the AT&L Knowledge Management System (AKMS)

9

Program Concerns or Issues

• Planned FY06 activities were based on committed funds of $435K; confirmed and received FY06 funding stands at $245K

• Curtailed activities (or postponed to FY07) may include:– Pursuit of/formalizing PPN partnerships– Reduced risk mitigation efforts– Prototype testing & evaluation master plan

• Engaging DAU faculty/staff in the practice/evidence information handling processes and to become practice “owners”

10

The BPCh Practice Providers Network

Responsible Content Sources - “Trusted Sites”Offices/individuals in OSD, Services, Agencies, Industry that are

responsible for capturing “local” best practices and lessons learned for their workforce:

• OSD– Defense Acquisition Policy Working Group (DAPWG) – Skip Hawthorne– AT&L Functional Advisors for workforce career management – Various OSD

Leadership– DAU Curriculum Development Support Center – Bill Erie/Dr. Ainsley/CDs/PDs

• Services– Air Force

• Acquisition Center of Excellence (ACE) – Terry Balvin• Air Force CIO/Air Force Knowledge Now (AFKN) – Randy Adkins/Harry Pape

– Army• Acquisition Lessons Learned Management System (ALLMS) – Major Ellison• Army CIO/Chief Knowledge Officer – Gary Winkler

– Marine Corps• MARCORPSYSCOM Tiger Portal – Barry Dillon/Dick Bates

– Navy • DON Acquisition One Source – Christine Stelloh-Garner/Page Glennie• DON CIO/Chief Knowledge Officer – Jim Knox

• Agencies– DCAA, DCMA, DLA, MDA

• Industry – Lenn Vincent, Radm USN, Retired, DAU Industry Chair– Aerospace Industries Association (AIA) – John Douglas

• Boeing – Alan Brown• LMCO – Paul Robitaille• Northrop/Grumman – Curt Newill• Raytheon – Mike Sharpe

– Data and Analysis Center for Software (DACS)– IEEE (Technical Council for Software Engineering)– International Council on Systems Engineering (INCOSE) – Paul Robitaille/Dr. John

Snoderly– National Defense Industrial Association and Related Special Interest Groups – Bob

Rassa– Software Engineering Institute (SEI) of Carnegie Mellon– Systems and Software Consortium (SSCI) – Dr. Richard Turner

BPCh Content Update

12

When we last met…

• Reviewed the System Engineering Plan• Agreed that the SEP would drive the

capture of content• Brainstormed practices / categories• Sticky votes on listed practices /

categories• Identified “Top 4” Practices / Categories

– Risk Management (5)– Earned Value Management (5)– Software Assurance (4)– Spiral Development (4)

• Suggested Practice / Category Owners

13

Definitions

• Lead– Unprocessed data coming into the BPCh – Experience– Lessons Learned– Stories– Practice– Not from the Preferred Provider Network

• War Story– Lead in narrative form possibly associated with a practice

• Evidence– External data (lead or PPN input) that has been

processed and entered into the BPCh– One-to-many relationship to practices

• Practice– Repeatable– 0 to many evidences

• Trusted Published Source– A source or site identified by the content manager from

which content (practices, evidence, stories) can be expedited

14

BPCh User or Information Provider

BPCh

Practice Processor or Information Handler

Practice Owner (Expert)

Practice Provider Network

Start

1.1 Enter Leads

1.3 Enter Evidence

1.2 Process Leads

1.5 Process Stories

1.4 Review Evidence

1.3 Enter Evidence

Leads andStories Evidence

Info Handler Process Pilot

Since February: Piloted Info Handler activities

•Developing guidelines & templates

Next phase: Pilot mechanisms for increasing leads and stories from live user community

15

Content Evolution Schedule

• Since February CAG: EVM & Risk Management– Surveyed the literature– Mined ACC / COP

AprMarFebCAG kickoffmtg

CAG statusmtg

CAG members “market” BPCh: Find experts & content sources

Content developers (FCMD) add evidence from pubs, interviews

Existing sites are linked and/or mined for content Content owners review material

Content developers (FCMD & others) add evidence from pubs, interviews, trust sites


CAG statusmtg

EVM &

RM

May Jun

Content developers (FCMD) add evidence from pubs, interviews

Existing sites are linked and/or mined for content

IA &

Spir

16

Res

pons

ible

; Dav

e B

achm

an (

?)

EVM Practices: summary vettedevidence

leads

resources

published evidence

war stories

interviews

trusted sites

• EVM (C/SCSC-compliant) 10 5 0 3 0 (3?

)

• Performance-Based EV 3 1 0 1 0 0

• Additional Content 3 0 0 0 0 0

Content Growth Overview

17

Res

pons

ible

; Bil

l Bah

nmai

er (

?)

Risk ManagementPractices: summary vettedevidence

leads

resources

published evidence

war stories

interviews

trusted sites

• SEI: ContinuousRisk Management

2 1 0 0 0 (1)

• SEI: Team Risk Management 0 0 0 0 0 0

• RiskIT 6 1 0 0 0 0• (NASA Risk

Management) 5 2 0 0 0 (1)

• Additional Content 48 14 0 0 0 0

Content Growth Overview

18

Content Concerns or Issues

• Identified a need for tool support, esp. related to work flows– Working this with CSC development team

• When to involve experts/practice owners?

• When do we have enough evidence?– Sufficient attribute or context coverage

driven initially by the CAG and ultimately by the users

• How to get more leads/stories/ideas from users?

• Topic owners vs. practice owners

19

Next Steps

– EVM & Risk Mgmt:– Pilot interviews– Meet with Practice Owners & establish

responsibilities– Info assurance & Spiral Development:

– Begin with content mining – Kickoff meetings with POs

Information HandlingProcess Roles and Activities

21


BPCh




Start

1.1 Enter Leads

1.3 Enter Evidence

1.2 Process Leads

1.5 Process Stories

1.4 Review Evidence

1.3 Enter Evidence


Info Handler: Processing Evidence

Practice Processors:

• Enter leads on behalf of end users (elicit the relevant information from e.g. an email, maybe follow-up if info too light)

• Turn selected leads into evidence profiles with BPCh

• Enter evidence profiles from the literature or other sources

22

Content Manager & CAG


BPCh




Trusted Published Site

Feedback Practices

2.5 Enter Approved Practices

Start

Start

Start

2.4 Identify Practice Owner

2.3 Identify New Practices

2.1 Enter Feedback

2.7 Process Trusted

Information

2.2 Evaluate Feedback

2.6 Enter Candidate Practices


Info Handler: Identify Practices

Practice Processors:

• Mine information from trusted sites and create records in BPCh

• May enter feedback on behalf of users

• Evaluate feedback to mine ideas for new practices, pass good candidates along to CAG

23


BPCh




Start

1.1 Enter Leads

1.3 Enter Evidence

1.2 Process Leads

1.5 Process Stories

1.4 Review Evidence

1.3 Enter Evidence


PPN: Process Evidence

Practice Provider Network:

• Trusted members of the community who can enter evidence directly into the database (e.g. KPOs extracting knowledge from CoPs)

24



BPCh





Feedback Practices


Start

Start

Start



2.1 Enter Feedback

2.7 Process Trusted

Information




PPN: Identify Practices

Practice Provider Network:

• Maintain their own repositories of helpful knowledge and may provide some level of assistance for BPCh to process & link to them

• Can submit ideas for practices BPCh ought to include

25


BPCh




Start

1.1 Enter Leads

1.3 Enter Evidence

1.2 Process Leads

1.5 Process Stories

1.4 Review Evidence

1.3 Enter Evidence


Practice Owner: Process Evidence

Practice Owners:

• Review (sanity-check) evidence entered on their practice

• Review relevant “stories” to see which contribute useful info and should be turned into evidence

•Suggest practices, evidence, sources

26



BPCh





Feedback Practices


Start

Start

Start



2.1 Enter Feedback

2.7 Process Trusted

Information




Practice Owner: Identify Practices

Practice Owner:

• Assigned by CAG when a new practice record is created

• Responsible for entering links to useful resources, summary (when required), and other information in the practice record

• Enter practices without review by the CAG

27



BPCh





Feedback Practices


Start

Start

Start



2.1 Enter Feedback

2.7 Process Trusted

Information




CAG: Identify Practices

Content Manager & CAG:

• Determine new practices to go after

• Identify and support owners of new practices

Information HandlingOperationalScenarios

29

Scenarios

• Process Evidence– Lead from a BPCh User that is Evidence– Lead from a BPCh User that is a Weak War

Story– Lead from a BPCh User that is Strong War

Story– Evidence or War Story from PPN (including

trusted published site)• Process Feedback and Identify Practices

– Candidate Practices from BPCh User– Candidate Practices from PPN – New Practices from Practice Owner – New Practices from CAG– Feedback from a BPCh User

• Prepare and Validate Summaries

Information Handling Scenario Pilot Approach

31

BPCh FY06 Activity (Identified Funding)

• Minimum analytical analysis to prepare for MS B. Basic Analysis of Alternatives & Risk Reduction w/ Prototyping– Update and complete existing AoA & Risk Management

Plan to include the existing BPCh platform and the following possible options:• Integration of existing BPCh database & information handling

system with new ACC enterprise upgrade (ECCO)• Integration of BPCh with Microsoft SharePoint portal system

– Update and complete Concept of Operations (ConOps)– Test and evaluate a BPCh prototype leading to Milestone B– Review and firm up BPCh information handling process– Formalize Best Practice Providers Network partnerships– Leverage DAU student and faculty resources for risk

reduction efforts and development of CDD– Develop joint service acquisition reference “taxonomy”– Support Workforce Management BP Clearinghouse

initiative

32

Scenarios

• Process Evidence– Lead from a BPCh User that is Evidence– Lead from a BPCh User that is a Weak War

Story– Lead from a BPCh User that is Strong War

Story– Evidence or War Story from PPN (including

trusted published site)• Process Feedback and Identify Practices

– Candidate Practices from BPCh User– Candidate Practices from PPN – New Practices from Practice Owner – New Practices from CAG– Feedback from a BPCh User

• Prepare and Validate Summaries

Working groups?

33

Proposed Pilot Approach

• Form CAG Working Groups for each Key Scenario• Identify Working Group Leader• Develop working group plan/schedule and

coordinate with project manager• Review existing scenario process flows to validate• Do pilot

– Implement scenario process flow with content– Coordinate content to use with Content Manager– Collect data

– Recommended process flow changes– Time spent on key process flow steps– Develop guidelines to support execution of the process

– Provide status to BPCh Project Manager• Prepare and analyze process implementation

results• Report results to CAG

34

Process Concerns or Issues

– Are sufficient resources available to complete the working group activities?

– Are the working groups goals aligned with needs for the upcoming milestone?

– What’s the timeframe for completing the working group results?

CAG Wrap-Up

36

Exit Criteria / Next Steps

• Are the targeted content categories and practices still the correct priority?

• Are there other content resources for risk management and EVM that we should go after?

• Do we have a list of scenarios that we will pilot?

• Do we know the approach for piloting the scenarios?

• Have we set a date for next meeting to review the content?

Backup Slides

38

CAG Charter Comments

39

•EVM (C/SCSC-compliant)-Evidence: !A-12 (McDonnell Douglas/Gen.Dynamics), AV-8B

(NAVAIR), !B-2 Stealth Bomber (NG)

-Poss. Interviews: ?USAF SSG, ?DynPort (vaccine acq prgm), ?BAE

•PBEV-Evidence: B-2 Stealth Bomber (NG)

•general comments

Content Growth Overview:EVM Resources

Show resources on the BPCh practice page (with links)

??

40

•SEI: Continuous Risk Management- Contact: Kathleen Dangle / Patricia Costa

•RiskIT-Contact: Jyrki Kontio

•(NASA Risk Management)- Risk management plan for project “Zeus”- Contact for NASA: Myrna Regardie / Sally Godfrey

•general comments- Barry Boehm / Charles W Bosler jr. / Robert N. Charette / David A. Cook / Brian Hagen / David T. Hulett / Tim Lister / Don O’Neill / Carol Smidts / Peter G. Anderson / Tim Best / Steve Botzum / Alexis Flippen / Al Gallo

Content Growth Overview:Risk Management Resources

41

March 30 - Dry Run Agenda

• Review Proposed CAG Material

• Time boxing 9:00 – 1:00

• BPCh Front Page Enhancements

• ANSI Presentation

• Planning Meetings (Configuration Control Board)

• Funding Status

• Demonstrate BPCh Backend

• Other Topics

42

BPCh Program Current Status

• OSD principle officers were briefed on 6 January and the following decisions were established:

– Move forward with the BPCh development plan as briefed, using identified FY 06 funding ($435K), to accomplish the following objectives:

• All FY06 funding activities highlighted on previous slide, including the assessment of “Managing the Force” Best Practices Clearinghouse initiative for possible integration with the AT&L BPCh

– Develop a business case and document requirements on the way to meeting other Milestone B requirements by the end of FY 06.

– Designate Mr. Anderson as the Acquisition Executive (AE) who becomes the Milestone Decision Authority for BPCh, until or unless the newly established Business Transformation Agency’s (BTA) AE, MG Carlos “Butch” Pair is identified as AE.

– A Memoranda of Agreement (MOA) outlining governance structure and signatories among BPCh sponsors is NOT now required.

Mike to keep or update

43

BPCh Program Current Status

• Final tasking from OSD principle officers briefing on 6 January:

– Compose Memorandum for Record documenting group discussion/decisions; MFR written in mid-January and is currently being coordinated by Col. Anderson for approval/signature among group principles.

• BPCh FY06 funding from NII and DS ($75K each) in process of being transferred to DAU; still working receipt of committed funds from ARA and crosscutting study.

• Working on draft partnership language and tailored program overview presentation for potential Practice Providers Network participants and POCs

• Working w/ DAU Taxonomy PM to determine how we will incorporate taxonomy metadata into BPCh design

• Biweekly status meetings to re-commence on 28 Feb

Mike to keep or update

44

“Alpha” prototype version developed and demonstrated

“Beta” prototype version evolved and demonstrated

AoA & Risk Mitigation ongoing; content generation, security and user testing at DAU

SSTC 21-25 April 2005

NDIA SE Conference

Oct 2005

AoA / Risk ReductionJul 2006

Primary Lead: CSCSupporting: FC-MD

Primary Lead: FC-MDSupporting: N/G-IT

DAU Dev/Testing

Oct 2006

Version 1.0FSD begins

Oct 2007

Functional Updates (every six months)

BPCh Development Schedule

SSTCApr 2006

Primary Lead: CSCSupporting: DAU-IT & FC-MD

Primary Lead: CSCSupporting: DAU-IT & FC-MD

MOU to be Signedbefore or at MS-B

Apr 2008 Oct 2008

Start Turnover of PM responsibility to DAU

MS-B

Industry focusPM & Log focus

Industry focusComplete all AT&L Functions

AcceptancePublic accessSE & SAM focusStart O&S

IOCApr 2007

MS-C(V1.0) (V2.0)

45

BPCh demo: http://bpch.dau.mil

• Latest front-end, integrated with DAU KS assets

46

Repository Forms/Data Structures

Evidence 1

SourceContextResults

Evidence 4


Evidence 2


Evidence 3


Practice A

47


Evidence 1


Evidence 4


Evidence 2


Evidence 3


Practice A

Summary of sourcesSummary of results in context

48


Evidence 1


Evidence 4


Evidence 3


Evidence 2


Practice A

Summary of sourcesSummary of results in contextVetted as of March 2005

49


Evidence 1


Evidence 4


Evidence 3


Evidence 2


Practice A

Summary of sourcesSummary of results in contextVetted as of March 2005

Evidence 5


50

BPCh demo: http://bpch.dau.mil

• Latest back-end, to manage practice information

51

CAG Role in Content Evolution

• Bi-monthly CAG meetings start off content development cycles by prioritizing leads & “hot” practices, then review progress at cycle end

• Validation meetings can be run after each content cycle, on whichever practices exhibited the most content growth

• Outputs: 1) Content populated into the BPCh2) Content generation processes exercised 3) List of practices to target, continually edited and re-

prioritized in Content Management Plan

Month 3Month 2Month 1CAG kickoffmtg (long)

CAG statusmtg (short)

Validationmtg #1


Content developers (FCMD and others) add evidence from publications, interviews

Existing sites are linked and/or mined for content

Content owners review material as it comes in

52

“Practice” Operational Definition

• Operational Definition:– A documented activity that is described in an

actionable, repeatable way;– A description of how to do something, not a

general goal of what to do– Usable by targeted acquisition end users– About which we can collect empirical data or

experiences.• Distinguished from:

– A best practice area • …a type of activity the user can’t neglect,

without specific advice on how to do it. E.g. risk mgmt.

– A lesson learned• …good advice, drawn from experience, without

enough detail to be clearly repeatable. E.g. Don’t overestimate cost savings from using COTS components.

53

Browse Perspectives for Practices

Evidence 1


Evidence 4


Evidence 2


Evidence 3


Willoughby Templates(Summary)

Systems Eng. Plan section outline• …• Acq. Strategy• Risk Mgmt• Integrated Master Plan• Earned Value Mgmt• …

Evidence 1


Evidence 4


Evidence 2


Evidence 3


SEI Taxonomy-Based Risk ID(Summary)

DAG section outline• …• 4.2.3.3 Tech. Assessment• 4.2.3.4 Reqts Mgmt• 4.2.3.5 Risk Mgmt• 4.2.3.6 Config Mgmt• …

1-to-many

1-to-many

54

Browse Perspectives for Practices

Evidence 1


Evidence 4


Evidence 2


Evidence 3


COCOMO II Cost Model(Summary)


Evidence 1


Evidence 4


Evidence 2


Evidence 3


Earned Value Mgmt(Summary)

CMMI-AM process areas• …• 2.9 Project Planning• 2.10 Reqts Dev.• 2.11 Reqts Mgmt• 2.12 Risk Mgmt• 2.13 Solicitation & Contract Monitoring• …

1-to-1

1-to-many

55

Content for practices

• Users will find content at various levels of completeness:

Evidence 1


Evidence 2


Evidence 3


Practice A(Name, links, and evidence)

“Practice A”(Name only, possibly links)

Evidence 1


Evidence 4


Evidence 2


Evidence 3


Practice A(Summary, links, and evidence)

56

Initial User Experience

Evidence 3


Evidence 1


Evidence 2


COTS Usage Risk Eval.(Evidence only)


Evidence 1


Evidence 4


Evidence 2


Evidence 3


SEI Taxonomy-Based Risk ID (Summary)

1-to-many

Evidence 3


Evidence 1


Evidence 2


Willoughby Templates(Evidence only)

Probability Consequence Software(Name & links only)

57

Brainstorming

• For the existing / new practices:– Which ones are the top priorities?

• “Low-hanging fruit”• Most important practices that users should find• What activities do users need most guidance

on?• What practices have the most (mis)information

already available?• What questions are students asking in DAU

classes?• (Other criteria?)

– Can anyone suggest evidence sources?• Existing best practice lists, publications, PMs to

interview, …– Can anyone suggest resources?

• Courses, templates, tools, how-to books, …– Who can be the practice owners?

58

Current content: System Capabilities, Reqts, and Design Considerations

• Capabilities to be Achieved– Architectural Reviews [5]– Needs Elicitation

[3]– SEF Procedure for Reqts Analysis

[0]• Key Performance Parameters

– (none yet)• Statutory and Regulatory Reqts

– (none yet)• Certification Reqts

– (none yet)• Design Considerations

– (none yet)

59

Current content: SE Organizational Integration

• Organization of IPTs– (none yet)

• Organizational Responsibilities– (none yet)

• Integration of SE into Program IPTs– (none yet)

• Technical Staffing and Hiring Plan– (none yet)

60

Current content: SE Process

• Process Selection– Lean Management [5]– Modeling and Simulation [0]– Six Sigma [0]– Total Quality Management (TQM) [0]

• Process Improvement– (none yet)

• Tools and Resources– Inspections [11]

• Approach for Trades– Trade Studies [8]– Utility Curve Methodology [0]

61

Current content: Technical Mgmt and Control

• Technical Baseline Mgmt & Control (Strategy and Approach)– Technical Performance Measurement

[4]• Technical Review Plan

– Technical Reviews[1]

62

Current content: Integration with Other Program Mgmt and Control Efforts

• Acquisition Strategy– Evolutionary Acquisition Strategy [3]

• Risk Management– COTS Usage Risk Evaluation (CURE) [0]– Willoughby Templates [0]– SEI’s Taxonomy-based Risk Identification [4]– Probability Consequence Software [0]

• Integrated Master Plan– DAG’s Lifecycle Cost Estimation Model [0]– COCOMO II Cost Estimation Model [0]– Work Breakdown Structure [2]

• Earned Value Management– EVM [0]

• Contract Management– Trade Studies [0]– Quarterly Mgmt Reviews [0]

DoD Acquisition Best Practices Clearinghouse Content Advisory Group (CAG) 12 April 2006

Documents

Transcript of DoD Acquisition Best Practices Clearinghouse Content Advisory Group (CAG) 12 April 2006