I/ITSEC Tutorial #1019 The Basics of the M&S VV&A
ProcessSimone Youngblood
The Johns Hopkins University Applied Physics [email protected]
Agenda
•Definitions•The VV&A Processes•VV&A as Applied to the Simulation
Categories•The Key Players•Conceptual Model•Documentation•Conclusion
Definitions
What is Verification?
• The process of determining that a model or simulation implementation and its associated data accurately represents the developer's conceptual description and specifications.
• Addresses the structural integrity and the stability of the model
• Goal: Did I build the model or simulation RIGHT?
What is Validation?
• The process of determining the degree to which a model or simulation and its associated data are an accurate representation of the real world from the perspective of the intended uses of the model.
• Addresses the representational aspects of the model or simulation
• Goal: Did I build the RIGHT model or simulation?
What is Accreditation?
• The official certification that a model or simulation and its associated data are acceptable for use for a specific purpose.
• The accreditation decision is based on an accreditation assessment of the V&V evidence.
• Goal: Are the results of the M&S FIT FOR PURPOSE and USEFUL for making decisions?
The Balancing Act
Risk Resources
VV&AVV&A
Balancing the cost of knowing against the risk of assuming.
The VV&A Processes
VV&A and the Problem Solving Process
From the DOD RPG
Tailoring Guidance for VV&A : Stay Focused on the Problem
• Maintain a problem orientation• Understand the relationships between
– The problem being solved– The role of M&S in the problem– The role of VV&A in establishing M&S credibility
• If the problem is too complex, decompose it into workable units
– Ensure that you are still trying to solve the same problem
• Focus on accreditation• Ensure that the V&V evidence is clearly applicable to
the intended purpose - solving ‘this’ problem
Tailoring Guidance for VV&A : Understand the Requirements
• Define the explicit problem requirements• What specific questions are to be answered? • How will M&S be used? • What does the M&S need to do? • How good does the M&S need to be?• Define what question(s) need to be answered...• Assess the completeness, correctness, and consistency of
the M&S requirements• Define the acceptability criteria needed to assess M&S
attributes• Identify the M&S options, the VV&A credibility needs, and the
level of resources necessary to meet defined requirements
An explicit statement of the requirements is the most important aspect of cost-effective VV&A and the hardest to come by
Tailoring Guidance for VV&A : Know the Level of Credibility Required
• Credibility is Defined by the Customer - but it has a cost• Ensure that the User determines the level of credibility needed• Ensure the level of credibility is explicitly defined by the
Acceptability Criteria
• Credibility and Risk Assessment - know the limits & tolerances
• How will incorrect simulation results affect the ‘end user’ decision
• What are the costs in time and resources
• Tailor Risk Analysis to Level of Credibility Requirements• Determine how much evidence will be needed to accredit M&S• Determine the impact of NOT doing (or failing) V&V • Provide mechanisms to facilitate development of workaround
strategies and assessment of these solutions
Tailoring Guidance for VV&A: Determine How to Optimize Resources
• Not all V&V supports Accreditation to the same extent• Understand what each type of V&V evidence ‘buys’ you
relative to the cost of producing it• Determine which techniques to apply to derive required level
of credibility
• Try to avoid (re)doing it all yourself • Capitalize on existing VV&A information whenever possible• Know how to extract useful data from software V&V results
• Cooperative application of resources : use common VV&A data elements and report structure
• The VV&A Templates provides a common structure for documenting VV&A information
• Each of the Services has adopted a common database structure for their M&S Resource Repositories
VV&A as Applied to the Simulation Categories
Simulation CategoriesNew M&S Development
Any stand-alone model or simulation under development being built to address the intended purpose or purposes of the User.
Legacy Any model or simulation that was developed either in the past or for a different purpose.
FederationA system of interacting models and/or simulations, with supporting infrastructure, based on a common understanding of the objects portrayed in the system.
VV&A and New M&S Development
`
ACCREDITATION PROCESS
PROBLEM SOLVING PROCESS
EstablishObjectives
Define Problem
Accept & RecordSolution
Analyze Results
Non-M&S Methods
M&S USE PROCESS
SelectApproaches
SelectApproaches
ApplyResultsApply
Results
Repository
Executeand
Prepare Results
yes
noMakeAccreditation
Decision
Develop New M&SDefineM&S
Rqmts
PlanApproach
M&S Method M&S DEVELOPMENT/PREPARATION PROCESS
Refine M&S
Rqmts
Refine M&S
Rqmts
Plan M&S
Develop-ment
Plan M&S
Develop-ment
Develop Con-
ceptual Model
Develop Con-
ceptual Model
Imple-ment &
Test
Imple-ment &
Test
Develop Design
Develop Design
Collect and Evaluate Accreditation InformationCollect and Evaluate Accreditation Information
PerformAccreditationAssessment
PerformAccreditationAssessment
Develop Accreditation
Plan
Develop Accreditation
Plan
V&V PROCESS
Verify DesignVerify Design
Validate Concep
-tual Model
Validate Concep
-tual Model
Verify Imple-ment-ation
Verify Imple-ment-ation
Validate Results
Validate Results
Verify Rqmts
Verify Rqmts
Develop V&V Plan
Develop V&V Plan
Prepare M&S for
Use
Prepare M&S for
Use
Challenge Issues that Impact VV&A Implementation
• The lack of a well defined referent• Determining how to use and when to
use Subject Matter Experts• The lack of insight into the M&S• The lack of M&S stability• Starting the V&V/A process “late”• Keeping the second “V” in V&V
VV&A and Legacy M&S
Is There Sufficient Info to Assess the
M&S?
NO
Refine RequirementsAnd Verify
Obtain Needed Information and Verify
NO
YES
NO
Plan/Conduct Accreditation Assessment
Plan M&S Modification
Modify and Verify
Use As-IsUse As- Is or Modify?
Reject M&S
Are the Requirements
Adequate?
Validate
Prepare M&Sfor Use
Issue Accreditation
Decision
AA
UserAA
Define Requirements
UserAA
Select Legacy M&S
AA
AA AAVV
User
User
UserProp
UserAA, VV
VVUser
Prop. UserDev, VV
PropDev
Dev, VVProp
Is the Infoon M&S Capability
Adequate?
Can the M&S be Used?
Modify
V&V as Needed
VV
NO
YES
Prop, DevAA, VV
[ AA ] Accreditation Agent[ User ] User[ Dev ] Developer [ VV ] V&V Agent[ Prop ] M&S Proponent
[ AA ] Accreditation Agent [ User ] User[ Dev ] Developer [ VV ] V&V Agent[ Prop ] M&S Proponent
VVYES YES
Advantages of Using Legacy M&S• Starting with a legacy simulation may be more expedient, requiring
less time and fewer resources than a new development
• The experience of previous developers and users may be leveraged to establish a knowledge base that can provide guidance and support training
• Testing and VV&A histories can provide evidence of simulation capabilities and limitations allowing the current effort to focus on high risk areas
• A legacy simulation with a good reputation for providing credible results in similar applications can increase the simulation’s credibility (i.e., the User’s confidence in the simulation’s ability to address the current situation) due to
– the demonstrated ability to address similar requirements
– the demonstrated capabilities of the existing simulation
– the direct experiences of previous users who successfully applied the simulation to related problems
Disadvantages of Using Legacy M&S
• The simulation was developed to address different requirements
• Unexpected and undesirable side effects may occur during execution of the code
• The simulation comes with specific hardware and software, which may be obsolete
• The simulation comes with specific data needs, however the data to fulfill those needs may no longer be authoritative or available
• Information about the simulation and its previous usage may be incomplete or inconsistent
• Participants (e.g., Accreditation Agent, V&V Agent) may need time up front to acquire knowledge about the simulation
What Can Help Reduce Legacy V&V Cost
• Documentation that can directly support the accreditation assessment can reduce the burden of an extensive discovery effort
• Documentation that completely and accurately describes simulation capabilities and use history can reduce the discovery and training costs
• A simulation under configuration control with an established user community that can provide necessary information can reduce the support and discovery costs
• Participants familiar with the simulation can reduce the need for extensive training
Scope of the Distributed M&S Problem
Distributed M&S:Technical Interoperability
From “HLA and Beyond: Interoperability Challenges” by Dahmann, Salisbury, Barry, Turrell, and Blemberg (99F-SIW-073)
• Characterized by the ability of federates to physically connect and exchange data
• Involves the use of common standards, compatible interfaces and coordinated data structures
• Elements of technical interoperability– Hardware compatibility– Standards compatibility– Time management coordination– Coordinated use of Runtime Infrastructure
(RTI) services– Security issues
Distributed M&S: Substantive Interoperability Issues
Simulation Interoperability focuses on the capability of federates, when connected, to provide adequate, accurate & consistent simulated representations that adhere to the principles of “fair fight” & address the mission objectives.
•Elements of Substantive Interoperability• Logical interaction between entities modeled in
different federates• Temporal resolution• Spatial resolution • Coherent relationships between the
components of the physical environment
Top Level View of the VV&A Overlay to the FEDEP (IEEE 1516.4)
FEDEP
VV&A Overlay to the FEDEP
FEDEP INFORMATION & PRODUCTS
Plan,Integrate
& TestFederation
Plan,Integrate
& TestFederation
Step 5
Validate &Accept
Federation
Validate &Accept
Federation
Phase 5
•Validate the integrated federation results against the Federation Acceptability Criteria and against the Federation Referent.
•Formulate federation acceptance/ accreditation recommen-dations.
ExecuteFederation& PrepareOutputs
ExecuteFederation& PrepareOutputs
Step 6
V&VFederation
Output
V&VFederation
Output
Phase 6
•Verify the federation outputs internally.
•Validate federation output against the Federation Acceptability Criteria and against the Federation Referent.
DefineFederationObjectives
DefineFederationObjectives
Verify Federation Objectives
Verify Federation Objectives
Phase 1
•Verify the Federation Objectives with the User /Sponsor.
•Define the Federation Acceptability Criteria.
•Assemble the Federation Referent.
•Plan the VV&A activities.
Step 1
PerformConceptual
Analysis
PerformConceptual
Analysis
V&VFederationConceptual
Model
V&VFederationConceptual
Model
Phase 2
•Verify the Federation Conceptual Model internally.
•Validate the Federation Conceptual Model against the Federation Referent and against the Federation Acceptability Criteria.
Step 2
DesignFederation
DesignFederation
VerifyFederation
Design
VerifyFederation
Design
Phase 3
•Verify the Federation Design internally.
•Verify the Federation Design against the Federation Conceptual Model.
Step 3
DevelopFederation
DevelopFederation
VerifyFederation
DevelopmentProducts
VerifyFederation
DevelopmentProducts
Phase 4
•Verify the federation development products internally.
•Verify the federation development products against the Federation Conceptual Model.
Step 4
AnalyzeData &
EvaluateResults
AnalyzeData &
EvaluateResults
ConsolidateFederation
VV&AProducts
ConsolidateFederation
VV&AProducts
Phase 7
•Collect all VV&A products and lessons learned.
•Assemble them into a VV&A package to support federation reuse.
Step 7
Tasks Associated with each VV&A Overlay Phase
Verify Federation
Design
Verify Federation
Design
3.0
Support Selecting Federates
Support Preparing Federation
Design
Contribute to Verifying
Federation Design
Support Preparing Federation
Development & Execution
Plan
Verify Federation Objectives
Verify Federation Objectives
1.0
Support Identifying
User/Sponsor Needs
Plan Accreditation
Activities
Contribute to Verifying
Federation Objectives
Support Developing Federation Objectives
Assemble Federation Referent
Define Federation
Acceptability Criteria
Plan V&V Activities
V&V Federation Conceptual
Model
V&V Federation Conceptual
Model2.0
Support Developing Federation Scenarios
Contribute to Verifying
Federation Scenarios
Contribute to Verifying
Federation Requirements
Support Developing Federation
Requirements
Support Developing Federation Conceptual
ModelContribute to
Verifying Federation Conceptual
ModelValidate
Federation Conceptual
Model
Validate & Accept
Federation
Validate & Accept
Federation
5.0
Support Planning
Federation Execution
Support Integrating Federation
Contribute to Verifying
Integrated Federation
Support Testing
Federation
Perform Acceptance Assessment
Validate Integrated Federation
Results
V&V Federation
Output
V&V Federation
Output
6.0
Support Executing Federation
Contribute to Verifying Raw Execution and
Derived Output
Validate Federation
Outputs
Consolidate Federation
VV&A Products
Consolidate Federation
VV&A Products
7.0
Support Analyzing
Data
Prepare Federation
VV&A Products for
Reuse
V&V Federation Data Sets
Verify Federation
Development Products
Verify Federation
Development Products
4.0
Support Developing
FOM
Contribute to Verifying FOM
Support Establishing Federation
Agreements
Contribute to Verifying
Federation Agreements
Contribute to Verifying
Federation Infrastructure
Support Implementing
Federate DesignsSupport
Implementing Federation
Infrastructure
The Key Players
User
• The User wants to use the simulation to provide an answer or solve a problem; he is concerned that the simulation can provide credible answers.
• The User defines the objectives and the requirements, scenarios and use cases, and generally serves as the subject matter expert (SME) for issues in the problem and user domains.
• Ultimately the User is the one who decides whether the simulation is credible enough to use.
Accreditation Agent
• The Accreditation Agent prepares for and conducts a cost-effective accreditation assessment that results in a logical, sufficient, and fully justified accreditation recommendation to the User.
• The Accreditation Agent assists the User with activities that help establish the scope of the problem to be addressed.
• The Accreditation Agent serves as the User’s advocate throughout the M&S development process.
Accreditation Agent Challenges
• Ensuring the definition of the intended use
• Using existing V&V documentation• Coping with configuration
management difficulties• Locating appropriate Subject Matter
Experts• Overcoming delay in appointment of
the accreditation agent• Obtaining needed resources• Communicating the results of the
accreditation assessment
V&V Agent
• The V&V Agent plans and performs the verification and validation (V&V) activities
• The V&V Agent ensures that sufficient evidence is gathered to demonstrate that the simulation meets the stated requirements in terms of their associated measures and acceptability criteria.
V&V Agent Challenges• Obtaining well-defined accreditation
information needs• Dealing with missing documentation• Establishing V&V support system and
infrastructure• Finding adequate resources• Selecting the right people• Managing the V&V effort• Leveraging configuration management
resources• Tracking and reporting V&V effort progress
Conceptual Model
Definition
“The conceptual model is a simulation developer’s way of translating modeling requirements into a detailed design framework, from which the software, hardware, networks, and systems/equipment that will make up the simulation can be built, modified, or assembled.”
from SIW Paper 00F-SIW-019, “Simulation Conceptual Model
Development Issues and Implications for Reuse of SimulationComponents” by Dr. Dale Pace
Why Conceptual Models?
“The biggest indication that Trespasser had game design problems was the fact that it never had a proper design spec…. Trespasser started and finished weak in the game design, and this affected every other part of the project.” – Richard Wyckoff,
“Postmortem: DreamWorks Interactive’s Trespasser” Gamasutra, May 14, 1999.
From presentation 03F-SIW-080.ppt by Gustavson, Turrell, Zimmerman at Fall 2003 SIW
Why Conceptual Models?
“We never had a proper design document, which meant that we generated a lot of code and art that we later had to scrap. What’s worse, because we didn’t have a detailed outline of what we were trying to build, we had no way to measure our progress (or lack thereof) accurately. We only realized that we were in trouble when it became glaringly obvious. If we’d been about the design rigorous up front, we would have known that we were slipping much sooner.”
– Brian Upton, “Postmortem: Red Storm's Rainbow Six” Gamasutra, January 21, 2000.
From presentation 03F-SIW-080.ppt by Gustavson, Turrell, Zimmerman at Fall 2003 SIW
Conceptual Model Components
(1) Simulation Context
Authoritative Information re: relevant entities/processes, data,
algorithms, assumptions, behaviors, etc.
Sets constraints/bounds on the Simulation Concept
Simulation Concept
(2) Mission Space -- Representation Aspect
Simulation Elements
Entities/processes represented (tasks, actions, behaviors, etc.) by
assumptions, algorithms, data, and relationships (architecture)
(3) Simulation Space
Operational/functional capabilities
RequirementsRequirements
Constraints
Conceptual Model
SpecificationsSpecifications
Benefits of a Conceptual Model
• A good conceptual model can improve requirements
• Good requirements are important because
– Faulty requirements cause half of software errors
– Late error detection/correction is much more expensive than early error detection/correction
• A good conceptual model allows early
– Estimation of simulation fidelity
– Development of validation sampling methodology
– Assessment of simulation validity for the intended use
– Determination of compatibility with other M&S/tools
Conceptual Model Documentation• Conceptual Model Portion Identification• Principal Simulation Developer POCs• Requirements and Purpose• Overview• General Assumptions• Identification of Possible States, Tasks, Actions,
and Behaviors, Relationships and Interactions, Events, and Parameters and Factors for Entities and Processes being described
• Identification of Algorithms• Simulation Development Plans• Summary and Synopsis
Documentation
Documentation Philosophy
• Organize information according to a standard outline– Rationale
• Eases the comparison of data with accreditation requirements
• Facilitates V&V data accumulation for future use• Reduces costs of multiple accreditations over time
– Specific outlines vary with type of data element
• Structure V&V results to facilitate comparison with M&S requirements
– Include description of how the results impact m&s use– State newly discovered discrepancies as assumptions,
limitations on model use, or as errors– Identify the range of applicability for all results (particularly
regarding any limitations or errors)– Ensure results are linked to the applicable M&S version
• Configuration Management
MIL-STD 3022: VV&A Documentation Templates
Accreditation Plan V&V Plan V&V Report Accreditation Report
Executive Summary Executive Summary Executive Summary Executive Summary
1 Problem Statement 1 Problem Statement 1 Problem Statement 1 Problem Statement
2 M&S Requirements and Acceptability Criteria
2 M&S Requirements and Acceptability Criteria
2 M&S Requirements and Acceptability Criteria
2 M&S Requirements and Acceptability Criteria
3 M&S Assumptions, Capabilities, Limitations & Risks/Impacts
3 M&S Assumptions, Capabilities, Limitations & Risks/Impacts
3 M&S Assumptions, Capabilities, Limitations & Risks/Impacts
3 M&S Assumptions, Capabilities, Limitations & Risks/Impacts
4 Accreditation Methodology 4 V&V Methodology 4 V&V Task Analysis 4 Accreditation Assessment
5 Accreditation Issues 5 V&V Issues 5 V&V Recommendations 5 Accreditation Recommendations
6 Key Participants 6 Key Participants 6 Key Participants 6 Key Participants
7 Planned Accreditation Resources
7 Planned V&V Resources 7 Actual V&V Resources 7 Actual Accreditation Resources
8 V&V Lessons Learned 8 Accreditation Lessons Learned
Suggested Appendices:A M&S DescriptionB M&S Requirements Traceability MatrixC Basis of ComparisonD ReferencesE Acronyms F GlossaryG Accreditation ProgrammaticsH Distribution List
Suggested Appendices:A M&S DescriptionB M&S Requirements Traceability MatrixC Basis of Comparison D ReferencesE AcronymsF GlossaryG V&V ProgrammaticsH Distribution ListI Accreditation Plan
Suggested Appendices:A M&S DescriptionB M&S Requirements Traceability MatrixC Basis of ComparisonD ReferencesE AcronymsF GlossaryG V&V ProgrammaticsH Distribution ListI V&V PlanJ Test Information
Suggested Appendices:A M&S DescriptionB M&S Requirements Traceability MatrixC Basis of Comparison D ReferencesE AcronymsF GlossaryG Accreditation ProgrammaticsH Distribution ListI Accreditation PlanJ V&V Report
VV&A Data Archiving• VV&A Documents
– Should be archived in central location: • Defense Department level or• Program level
– Indexed in a electronic database– Possibly available on-line
• Electronic Database– Should contain summarized,
unclassified data from VV&A reports– Should be sufficient for model
comparisons and selection
VV&A data are only useful if they are documented, and the documentation is readily available to model
users.
VV&A data are only useful if they are documented, and the documentation is readily available to model
users.
Conclusion
Summary of VV&A• Cost-effective accreditation requires
– Complete problem definition– Clear application description – Explicit statements of M&S requirements and
acceptability criteria– A well-planned, tailored and focused accreditation
assessment effort • Efficient V&V to support accreditation requires
– A clearly defined application context to permit focused V&V efforts
– The use of risk to focus and tailor the V&V effort– A clear understanding of how each V&V activity
contributes to M&S credibility– A means of extracting key accreditation support
information from previous V&V (and other M&S) efforts
– An efficient means to produce, document, and archive all V&V information (from software and M&S V&V)
Simone [email protected]
(240) 228-7958
Top Related