Acquisition Modeling & Simulation Working Group (AMSWG ...

ITEA LVC Conference

24-27 Jan 2011 Page-1

Defense Modeling & Simulation

Verification, Validation & Accreditation

Campaign Plan

John Diem, Associate Director (Services)

OSD/AT&L Modeling & Simulation Coordination Office

ITEA LVC Conference: January 24 – 27 , 2011

DISTRIBUTION STATEMENT A -- Cleared for public release by OSR (case numbers 10-S-3029 & 10-S-3232 apply).

ITEA LVC Conference

24 - 27 Jan 2011 Page-2

VV&A Campaign Plan Summary

• Established May 2010 to be broadly focused – policy,

standards, guides, tools, methodologies

• Driven largely by work led by Acquisition M&S Working

Group

• On-going effort – work in 2011 includes several efforts to

gain feedback from users and sponsors

• Feedback will influence re-look of key policy and standards

components in 2012, including:

– DOD 5000.61 (“DoD M&S VV&A”)

– MIL-STD-3022 (“Documentation of VV&A for M&S”)

– DOD VV&A Guidance (“The VV&A Recommended Practices Guide (RPG)”)

ITEA LVC Conference

24 - 27 Jan 2011 Page-3

Key VV&A Documents

• DoD VV&A Policy: DoDI 5000.61 ―DoD M&S VV&A‖

• DoD VV&A Guidance: The VV&A Recommended

Practices Guide (RPG)

• MIL-STD-3022 ―Documentation of VV&A for M&S‖

• IEEE 1516.4-2007 ―IEEE Recommended Practice

for VV&A of a HLA Federation‖

• IEEE 1278.4-1997(R2002) ―IEEE Recommended

Practice for Distributed Interactive Simulation (DIS)

VV&A)‖

3

ITEA LVC Conference

24 - 27 Jan 2011 Page-4

• Implements policy, assigns responsibilities, prescribes

procedures, and establishes common terminology

• Current version dated December 9, 2009

• Modified to reflect updated format requirements as defined

in DoDI 5025.01 (Oct 2007)

• Streamlined document that establishes responsibilities,

procedures, information requirements, and key definitions

• Incorporated reference to MIL-STD 3022

• Scheduled for review in 2014 – work will begin in 2012

DoD Instruction 5000.61

―DoD M&S VV&A‖

DoDI 5000.61 dated 9 December 2009

available at http://www.dtic.mil/whs/directives/corres/pdf/500061p.pdf

ITEA LVC Conference

24 - 27 Jan 2011 Page-5

5

DoD VV&A Recommended Practices Guide

• RPG promotes the effective application of VV&A

methodology

– Defines philosophy, principles and methodologies

– Developed by subject matter experts

– Built to address diverse audiences, situation and

applications

• Web-based format intended to:

– Provide the foundation for multi-tiered, multi-

dimensional information at various levels of depth

and detail

– Be extensible to allow for the addition of

information over time

– Allow the reader to build guidance tailored to a

situation or application

Available at

vva.msco.mil

ITEA LVC Conference

24 - 27 Jan 2011 Page-6

6

MIL-STD-3022

• 2005-2006: DoD-sponsored Tri-Service VV&A

Templates Tiger Team developed templates for

four core VV&A documents

– Accreditation Plan, V&V Plan

– V&V Report, Accreditation Report

• 2007: Processed through DoD M&S

management system and Defense

Standardization Program

• 2008: Approved as a DoD Standard Practice

• May be cited as a solicitation requirement and

Data Item Descriptions (DIDs) included on

Contract Data Requirements List (CDRL)

– DI-MSSM-81750 DoD M&S Accreditation

Plan

– DI-MSSM-81751 DoD M&S V&V Plan

– DI-MSSM-81752 DoD M&S V&V Report

– DI-MSSM-81753 DoD M&S Accreditation

Report

• 2009: DoD Instruction 5000.61, DoD M&S

VV&A, references MIL-STD-3022 Free: Available from the Acquisition Streamlining

and Standardization Information System (ASSIST)

ITEA LVC Conference

24 - 27 Jan 2011 Page-7

Information Shared across the Templates

Accreditation Plan V&V Plan V&V Report Accreditation Report

Executive Summary Executive Summary Executive Summary Executive Summary

1 Problem Statement 1 Problem Statement 1 Problem Statement 1 Problem Statement

2 M&S Requirements and Acceptability

Criteria

2 M&S Requirements and

Acceptability Criteria





3 M&S Assumptions, Capabilities,

Limitations & Risks/Impacts

3 M&S Assumptions,

Capabilities, Limitations &

Risks/Impacts





4 Accreditation Methodology 4 V&V Methodology 4 V&V Task Analysis 4 Accreditation Assessment

5 Accreditation Issues 5 V&V Issues 5 V&V Recommendations 5 Accreditation Recommendations

6 Key Participants 6 Key Participants 6 Key Participants 6 Key Participants

7 Planned Accreditation Resources 7 Planned V&V Resources 7 Actual V&V Resources

Expended

7 Actual Accreditation Resources

Expended

8 V&V Lessons Learned 8 Accreditation Lessons Learned

Suggested Appendices

A M&S Description

B M&S Requirements Traceability

Matrix

C Basis of Comparison

D References

E Acronyms

F Glossary

G Accreditation Programmatics

H Distribution List


A M&S Description

B M&S Requirements

Traceability Matrix


D References

E Acronyms

F Glossary

G V&V Programmatics

H Distribution List

I Accreditation Plan


A M&S Description


Matrix


D References

E Acronyms

F Glossary

G V&V Programmatics

H Distribution List

I V&V Plan

J Test Information


A M&S Description


Matrix


D References

E Acronyms

F Glossary

G Accreditation Programmatics

H Distribution List

I Accreditation Plan

J V&V Report

Information that is common and shared across the documents appears in italicized font in the table above

ITEA LVC Conference

24 - 27 Jan 2011 Page-8

DoD VV&A Documentation Tool

http://msco.mil

Implements MIL-STD 3022 and Data Item Descriptions

DI-MSSM-81750 - DI-MSSM-81751 - DI-MSSM-81752 - DI-MSSM-81753

ITEA LVC Conference

24 - 27 Jan 2011 Page-9

DoD VV&A Documentation tool

Producers register here or

log in if already registered.

There is online help for

using the tool, as well as

help for VV&A in general.

The DVDT automates the

MIL-STD-3022

documentation templates.

A benefit of using the tool

is easier management of

VV&A information. The

tool is designed to walk

the Producer through

VV&A planning,

implementation, and

reporting.

Register, log in, and create a project to download a set of XML-

enabled document templates for local use. Compliant with DoD M&S

Community of Interest Discovery Metadata Specification (MSC-DMS).

https://dvdt.nmso.navy.mil

ITEA LVC Conference

24 - 27 Jan 2011 Page-10

IEEE Std 1516.4

• Defines processes and

procedures to implement VV&A

for federations being developed

using the HLA Federation

Development and Execution

Process (FEDEP) (IEEE Std

1516.3™-2003)

• Focuses on unique aspects of

VV&A of federations as a higher-

level framework into which

VV&A practices can be

integrated and tailored for

specific uses.

• Provides implementation-level

guidance to VV&A practitioners.

ITEA LVC Conference

24 - 27 Jan 2011 Page-11

IEEE Std. 1278.4

• Guidelines established for the VV&A

of distributed interactive simulation

(DIS) exercises.

• Provides ―how-to‖ procedures for

planning and conducting DIS

exercise VV&A

• Intended for use in conjunction with

IEEE Recommended Practice for DIS-

Exercise Management and Feedback

(IEEE Std 1278.3-1996)

• IEEE-SA approved reaffirmation of

IEEE Std 1278.3-1996 and IEEE Std

1278.4-1997 in June 2010

• Presents data flow and connectivity

for all proposed verification and

validation activities and provides

rationale and justification for each

step.

• Provides VV&A guidance to exercise

users/sponsors and developers.

ITEA LVC Conference

24 - 27 Jan 2011 Page-12

Where to Obtain VV&A Standards

• MIL-STD-3022, Documentation of Verification, Validation, and

Accreditation for Models and Simulations, 28 Jan 2008

– https://assist.daps.dla.mil/quicksearch/

• IEEE Std 1516.4™-2007, IEEE Recommended Practice for

Verification, Validation, and Accreditation of a Federation—An

Overlay to the High Level Architecture Federation Development

and Execution Process, 20 December 2007

– https://sbwsweb.ieee.org/

• IEEE Std 1278.4-1997, IEEE Trial-Use Recommended Practice for

Distributed Interactive Simulation - Verification, Validation, and

Accreditation, 20 July 1998 (reaffirmed June 2010)

– https://sbwsweb.ieee.org/

IEEE Standards are free to SISO Members

ITEA LVC Conference

24 - 27 Jan 2011 Page-13

Acquisition Community Led VV&A Efforts - key inputs to Campaign Plan

VV&A

Strategy

ITEA LVC Conference

24 - 27 Jan 2011 Page-14

Acquisition Community-led

VV&A High Level Task (HLT) Summaries

V-AQ-2: ―Risk Based Methodology for Verification, Validation and

Accreditation (VV&A)‖ The degree of VV&A required is explicitly tied to both

M&S use and the user risk incurred if the M&S does not provide accurate results.

A methodology that tailors VV&A planning and implementation based on known

risk factors will provide a framework in which VV&A implementation trade-offs

can be made, information/fidelity requirements can be assessed, and a VV&A

cost model can be developed.

V-C-2: ―Improving VV&A Implementation‖ Increase VV&A implementation

and enhance M&S credibility by transforming VV&A practices from current

subjective methods into objective examples or use cases. Explore emerging

technologies, standards, and applicable methods that could be applied to reduce

costs, schedule, and improve reuse.

ITEA LVC Conference

24 - 27 Jan 2011 Page-15

Risk-Based VV&A (M&S PE Funded V-AQ-2)

The purpose of this effort is to:

Improve Department of Defense

(DoD) Modeling and Simulation

(M&S) Verification, Validation,

Accreditation (VV&A) by

establishing an effective risk-based

methodology for VV&A.

Since the cost of verifying and

validating simulations is often high,

V&V investments should be

weighed against the risk of making

a bad decision because of

unreliable M&S results.

Risk Resources

VV&A

Balancing the cost of knowing

against the risk of assuming.

ITEA LVC Conference

24 - 27 Jan 2011 Page-16

Systems Engineering Research Center VV&A Topic

(M&S PE Funded V-C-2)

Proposed research topic provided SERC:

Explore & promote a deeper

understanding of emerging & current

approaches, technologies, or practices

underway within the academic

community & gain a broader perspective

on their potential applicability to the M&S

community V&V needs

The 19 SERC university partners

generated six responses, with selection

of two proposals for implementation

lasting 6 to 12 months

Two performers:

Georgia Institute of Technology

University of Alabama Huntsville

ITEA LVC Conference

24 - 27 Jan 2011 Page-17

Georgia Institute of Technology VV&A Study


The purpose of this SERC effort :

Demonstrate how to address many of

the gaps identified in this VV&A

Research Topic by applying SysML and

MBE/MBSE technology

Engage practitioners in the DoD

ecosystem to prioritize needs and to

test resulting concepts

Apply concepts in one or more key

DoD test beds

ITEA LVC Conference

24 - 27 Jan 2011 Page-18

University of Alabama at Huntsville VV&A Study


The purpose of this SERC effort :

Focus on verification & validation

(V&V) of complex systems

A survey & evaluations of useful tools

Leverages experience working the

government customers in the Army Aviation

Directorate, Software Engineering

Directorate, Redstone Test Center, and PEO-

Aviation.

Will benefit from insights resulting from

Systems Architecture Virtual Integration

(SAVI), an international industry consortium

involving Boeing, Airbus, Lockheed Martin,

BAE, FAA, SEI, etc

ITEA LVC Conference

24 - 27 Jan 2011 Page-19

ITEA LVC Conference

24 - 27 Jan 2011 Page-20

Domain-relevant “Best of Breed” M&S Tool Suite VV&A NAVAIR Use Case


Multiple communities within the

Department of Defense are interested in

accelerating schedules, improving

credibility, & reducing costs of technical

execution of M&S support to decision-

makers through tailored accreditation

support

The scope of this effort was to:

Bring knowledge of existing VV&A

―best practices‖ to funded

implementation of credibility

assessments of high-priority DoD

M&S applications;

Examine the practicality of

leveraging Service-specific VV&A for

domain-relevant Department-wide

tool suite applications; and

Identify Service-specific needs &

estimate both cost-avoidance &

incremental costs for plan to address.

ITEA LVC Conference

24 - 27 Jan 2011 Page-21

VV&A Program Management & Integration


The VV&A Subcommittee of the

Acquisition M&S Working Group

(AMSWG) is a coalition of the willing.

Help evolve the practice of VV&A by:

Reviewing and commenting on

VV&A studies

Providing input to the review and

update of VV&A policy and

standards

Supported by:

User feedback

―Gray Hair‖ Support to Key

Activities

ITEA LVC Conference

24 - 27 Jan 2011 Page-22

• Expand leverage of DoD, government

agency and Industry standards for M&S

VV&A

• Mature existing products with documented

and archived lessons learned

• Emerging concepts (e.g., Risk-based VV&A)

will result in a standard/guideline

downstream

DoD M&S VV&A Campaign Plan Way Forward

ITEA LVC Conference

24 - 27 Jan 2011 Page-23

Key M&S VV&A Contact Information

DoD Modeling & Simulation Coordination Office John Diem, Associate Director (Services)

(703) 681-6631, [email protected]

Acquisition Modeling & Simulation Working Group

Chair Phil Zimmerman, DDRE/SE/SA

(703) 681-6544; [email protected]

Support Team Ralph Gibson, BAH Support to M&S CO (Services – VV&A)


Crash Konwin, BAH Support to DDRE/SE/SA


mailto:[email protected]

mailto:[email protected]

ITEA LVC Conference

24 - 27 Jan 2011 Page-24

Back-up slides

ITEA LVC Conference

24 - 27 Jan 2011 Page-25

Acquisition Community Led VV&A Efforts - key inputs to Campaign Plan

V-C-2

Improving VV&A Implementation

Johns Hopkins APL

Improving VV&A

Implementation

Systems Engineering

Research Center

VV&A Shortfalls & Needs Complex Systems Modeling

Pilot Implementations

(SIMAF & NAVAIR)

VV&A/Distributed Joint

VV&A/Vehicle Vulnerability

Practitioners Forum

AMSWG VV&A

Sub-Committee

ITEA LVC Sub-committee

“Gray Hairs” Study Support

V-AQ-2

Risk Based Accreditation

FY: 09-10 FY: 10-11

ITEA LVC Conference

24 - 27 Jan 2011 Page-26

Risk-Based VV&A (FY09/10 M&S PE Funded V-AQ-2)

Since the cost of verifying and

validating simulations is often high,

V&V investments should be

weighed against the risk of making

a bad decision because of

unreliable M&S results.

The purpose of this work effort is to

improve Department of Defense

(DoD) Modeling and Simulation

(M&S) Verification, Validation,

Accreditation (VV&A) by

establishing an effective risk-based

methodology for VV&A.

ITEA LVC Conference

24 - 27 Jan 2011 Page-27

JHU/APL VV&A Implementation: Defining & Addressing Gaps


The purpose of the effort is to:

Support the identification of VV&A gap

areas that inhibit effective

implementation of VV&A processes;

Develop supplemental capability

required to implement the Risk-based

Accreditation (RBA) methodology; and

Evolve technology, in the form of

automated tools and metadata

specifications, that increase VV&A

implementation efficiencies.

ITEA LVC Conference

24 - 27 Jan 2011 Page-28

Standard Intended Use & Tailoring

• Intended to support VV&A of DoD models, simulations,

federations, and other types of distributed simulations by

providing a common framework for documenting

information produced during the VV&A processes.

• Specifies procedures on documenting information obtained

through implementing the VV&A processes when the

outputs will be used to supplement decisionmaking in the

DoD.

• These types of information, at least some of the time, are

obtained via contract from commercial firms.

• Tailoring guidance for contractual applications. Invitation

for bids, request for proposals, and contractual statements

should tailor Appendices A through D in accordance with

the scope of the contractual effort.

ITEA LVC Conference

24 - 27 Jan 2011 Page-29

29

RPG Assessment Overview

• Purpose of the Recommended Practices Guide (RPG)

Assessment

– Assess the viability of the web-based RPG

– Identify immediate problems that inhibit the RPG’s effective use

– Identify potential improvements that could enhance the RPG’s use

– Identify advanced technology features to increase the RPG’s

reach

• Assess Primary RPG characteristics

– Content: The information contained in the website

– Presentation and Structure: Organization and navigation

– Technology: Web-based features

• RPG Update Status

– Phase I updates completed (e.g., fixing hyperlinks, updating

special topics documents)

– Elements of Phase II updates in process (e.g., developing new

special topics, updating examples)

ITEA LVC Conference

24 - 27 Jan 2011 Page-30

IEEE 1516.4 Top-Level Process View

ITEA LVC Conference

24 - 27 Jan 2011 Page-31

IEEE-1516.4 Activities

ITEA LVC Conference

24 - 27 Jan 2011 Page-32

DIS Exercise VV&A Process

ITEA LVC Conference

24 - 27 Jan 2011 Page-33

DIS Exercise Life Cycle and VV&A

Acquisition Modeling & Simulation Working Group (AMSWG ...

Documents

Transcript of Acquisition Modeling & Simulation Working Group (AMSWG ...