LLNL ASC V&V Strategy · LL-DNT-U-2007-012184.1 UNCLASSIFIED UNCLASSIFIED LLNL ASC V&V Strategy...
Transcript of LLNL ASC V&V Strategy · LL-DNT-U-2007-012184.1 UNCLASSIFIED UNCLASSIFIED LLNL ASC V&V Strategy...
This work was performed under the auspices of the U.S. Department of Energy by the University of California Lawrence Livermore National Laboratory under Contract No. W-7405-Eng-48.
LL-DNT-U-2007-012184.1
UNCLASSIFIED
UNCLASSIFIED
LLNL ASC V&V Strategy
Associate Program LeaderDefense and Nuclear TechnologiesLawrence Livermore National Laboratory
Salishan ConferenceApril 22, 2007
Dr. Joseph A. Sefcik
UCRL-PRES-229957
LL-DNT-U-2007-012184.2
UNCLASSIFIED
UNCLASSIFIED
QMU is best understood from the perspective of the overall Stockpile Stewardship enterprise
Original DesignNuclear testsDesign codes
HydrosDatabasesExperience
Manufacturingdrawings andsurveillance
requirements
Manufacturingplants
Surveillance
Today’s AssessmentCapability
Advanced ASC codesUGT databases
Property databasesFacilities: CFF,
DAHRT, Subcrits,JASPER, Z, NIF
50 years experience
Addressing stockpileissues requiresintegrating input
from surveillance,experimental facilities,
nuclear test data,and simulations
Quantification of Marginsand Uncertainties
LLNL/Los Alamoscommon approach
Warheads
Peer and Red Team Reviews
Informed Decisions:NNSA, DoD, Policy
Annual Certification Letters
LL-DNT-U-2007-012184.3
UNCLASSIFIED
UNCLASSIFIED
Assessments of warhead performance “at target” involves both reliability and QMU
• Standard industrial reliability techniques are applied to the myriad components of the weapon system to determine the probability that the system will arrive at target and fire the device
• The performance of the device is assessed based on fundamental physics data taken from experimental measurements
LL-DNT-U-2007-012184.4
UNCLASSIFIED
UNCLASSIFIED
The V&V Program supports QMU and the ASC roadmap to predictive capability
• Apply Software Quality Engineering as an integral component of increasing the confidence in ASC codes
• Identify and assess adjustable code parameters with PDFs provided by ASC and Campaigns
• Apply sensitivity analysis to guide resource investments• Develop methodologies to quantify uncertainties (competing methods
will be fostered) and use them to assess weapon performance• Assure these methodologies meet the criteria for numerical
convergence
V&V will provide the discipline and the structure for the UQ in QMU
LL-DNT-U-2007-012184.5
UNCLASSIFIED
UNCLASSIFIED
V&V is an essential part of the ASC product cycle, and an important interface to Science Campaigns
Focused ExperimentsLiterature searchTechnical assessment
New Capability(new models) Software Quality
Assurance SQA(SQE)
Version releaseRegression testingChange controlCode coverageCode A´= Code A
UQ for QMUAssessment of codePhysics and numericsCode A´= Code AMethod A = Method B
Verification TestsAnalytical test problemsNumerical convergenceCode coverageRegression testingCode A´= Code A
Validation TestsComparison with experimentsIntegral test problemsRegression testingCode coverage
LL-DNT-U-2007-012184.6
UNCLASSIFIED
UNCLASSIFIED
Necessary (and probably sufficient) conditions for achieving predictive capability
• Codes with assured software quality• Problems run with demonstrated numerical convergence• Complete identification of all parameters that can be “adjusted”
(free parameters)• Development of distribution functions for the “free parameters”• An iterative combinatorial methodology that samples the hyperspace
of free parameters and uses the code as an “operator” to project a quantity of interest
• Capacity computing sufficiently large to accommodate all of the above in 3D
• Program to elucidate underlying physical phenomena via experiment, theory, and simulation
• An ability to explain the anomalies in the experimental database
LL-DNT-U-2007-012184.7
UNCLASSIFIED
UNCLASSIFIED
CODES
The codes have hundreds of adjustable parameters; sensitivity analysis will down-select the important ones; UQ techniques can create the final output distributions
We will quantify probability distributions in key adjustable code parameters and their impact on weapon performance
Outputquantity
Compareto data
Databases Supportcodes
Proven numerical convergence
Iterative combinatorial methodology
Software QA
k1
Input knobs Distributionof outputquantity
k2
k3
kn-2
kn-1
kn
LL-DNT-U-2007-012184.8
UNCLASSIFIED
UNCLASSIFIED
We are moving the full suite of ASC codes and tools to compliance with DOE Orders and Policies
DOE/NNSA
CFR-830Order 414.1C QC-1 Rev. 10
LLNL ISQAP
DNT QAP
ASC Software Requirements
SQAPASC3
SQAPASC2
SQAPASC1
SQAPASC4
SQAPASC5
SQAPASC6
SQAPASC7
SQAPData2
SQAPInterpreter1
SQAPDataManipulator1
SQAPData1
SQAPSolver
OrdersLLNL InstitutionalDNT LevelProgram LevelPrincipal Code TeamsFeeders/Libraries/Support Codes
LL-DNT-U-2007-012184.9
UNCLASSIFIED
UNCLASSIFIED
We are working to demonstrate QC-1 (DOE Weapons Quality Policy) compliance in FY08
• Issue tracking (SQAP)• Automated regression testing (STP)
— Function/Statement coverage using commercial tools
• Configuration management (SCMP)• Disaster recovery (SDRP)• Coding standards• Library standards• ASC V&V SQE embedded staff• Continuous process improvement
— Commercial error checking software
We have over 30 code packages and several million lines of code to address
LL-DNT-U-2007-012184.10
UNCLASSIFIED
UNCLASSIFIED
We know where the current “free parameters” are—they are in the current codes
• If a designer can input a parameter change into the deck that changes an output, then that parameter is a free parameter
• Numerical parameters (e.g., mesh size, rezoning settings, mesh “stiffeners”, etc.) can play in the game, but can be “studied” to convergence
• All free parameters have a distribution function (point values or ranges with shapes, means, moments, etc.)
• The multiple free parameters and their distributions form a hyperspace of inputs and the code (as operator) projects that hyperspace onto an axis of interest
Fortunately, the physics of our devices does not exhibit chaotic behavior (barring obvious errors in Monte Carlo statistics)
LL-DNT-U-2007-012184.11
UNCLASSIFIED
UNCLASSIFIED
Complex computational simulations generally consist of aleatory and epistemic input variables
• Aleatory (Latin “alea”—throwing of dice)• Epistemic (Greek “Episteme”—knowledge)• An aelatory input distribution must be sampled based on a
probabilistic outcome and is random (the dice must be rolled at each input step)
• An epistemic distribution is based on scientific assumptions regarding the range of the variable and its distribution
• Input parameters for the physics package are purely epistemic (currently)
This makes the interval of the input variable the most important thing to know when beginning the iterative combinatorial methodology
LL-DNT-U-2007-012184.12
UNCLASSIFIED
UNCLASSIFIED
Aleatory and epistemic distributions generate different kinds of “error bars”
Aleatory
Epistemic
Probabilistic distribution function with some variance based on sampling data
Uncertainty in physics of an adjustable parameter, but a known range beyond which values would be inconsistent with physical reality
LL-DNT-U-2007-012184.13
UNCLASSIFIED
UNCLASSIFIED
Margin is a well-understood concept in engineering
Selected operating point
Point of “unacceptable performance”
Margin(M)
Ope
ratin
g pa
ram
eter
(pre
ssur
e, v
eloc
ity, R
PM, s
tres
s, e
tc.)
LL-DNT-U-2007-012184.14
UNCLASSIFIED
UNCLASSIFIED
Uncertainty is folded into the picture to reflect confidence, or level of risk, in our decision-making process
Selected operating point
Point of “unacceptable performance”
Uncertainty that we are atthe selected operating point
Uncertainty in our assessmentof “unacceptable performance”
Margin(M)
Ope
ratin
g pa
ram
eter
(pre
ssur
e, v
eloc
ity, R
PM, s
tres
s, e
tc.)
LL-DNT-U-2007-012184.15
UNCLASSIFIED
UNCLASSIFIED
We use M/U to give us an estimate of our risk
Margin(M)
Ope
ratin
g pa
ram
eter
(pre
ssur
e, v
eloc
ity, R
PM, s
tres
s, e
tc.)
Uncertainty in the “unacceptable performance”minimum set point that reduces margin (UminB)
MU
MarginUOB + UminB
=
Operating point uncertaintythat reduces margin (UOB)
LL-DNT-U-2007-012184.16
UNCLASSIFIED
UNCLASSIFIED
Error bars should not be applied to single calculations, but uncertainties can be assessed for an ensemble of calculations for a model
Cod
e ou
tput
val
ue
The “error” associated with one computer simulation is
essentially the round-off error of the computer hardware
Cod
e ou
tput
val
ueAn ensemble of calculations where
the input parameters span the available space define the
uncertainty associated with a particular model
LL-DNT-U-2007-012184.17
UNCLASSIFIED
UNCLASSIFIED
The potential interactions among free parameters produce non-linearities that could prove disastrous to performance
LL-DNT-U-2007-012184.18
UNCLASSIFIED
UNCLASSIFIED
We would like to robustly locate the wormholes in our hyperspace of variables
Each sample represents a computation in 1D, 2D, or 3D with numerical convergence
Valu
e
P
Number of sample points = M
Epistemic variable Number of free parameters = N
The Curse of DimensionalityTotal number of samples required = (M)N
3 samples 2 parameters (3)2
10 samples 10 parameters (10)10
100 samples 600 parameters (10)1200
LL-DNT-U-2007-012184.19
UNCLASSIFIED
UNCLASSIFIED
Since calibrated models are used, a single experimental point maps to a volume in the model parameter space
Observable space Model parameter space
Observable 2 Parameter value
Obs
erva
ble
1
k1
kn
k2
Mapping
Boundary of“physical” region
LL-DNT-U-2007-012184.20
UNCLASSIFIED
UNCLASSIFIED
Our 10-year objective is to expand our models and reduce the standard deviation of the set
This will require development of
models that move to 3D on Sequoia
Goal: circa 2010
Goal: circa 2015
LL-DNT-U-2007-012184.21
UNCLASSIFIED
UNCLASSIFIED
What does predictive capability (sufficiently numerically converged) and in 3D mean for capacity machines?
• 3D design calculations probably require fine zones• A full device 3D numerical convergence study cannot be run today on
purple (100 teraflops)• A full model set (50–100 data points) in 3D (fully converged) with fine
zoning to do parameter surveys will require 10 petaflops of capacity to leave room for others
• Full vetting of the adjustable parameters to match data using stochastic bootstrapping (or MOAT, MC, LHS, etc.) and sampling all of the distribution functions could consume 10–100 petaflops of capacity
We can use sensitivity analysis to cut down on the number of sample free parameters
LL-DNT-U-2007-012184.22
UNCLASSIFIED
UNCLASSIFIED
There are unique “capabilities” that a “capacity” machine can provide to the program
• There is nothing wrong with performing sensitivity analysis in 1D on one processor
• A good 1-D run can be accomplished in 30 minutes• A 120,000 processor machine can run one billion calculations in 6
months• This does not solve the “Curse of Dimensionality” (MN), but it helps us
narrow the playing field
Our greatest hope is that we can get our models sufficiently refined so that we live in “flatland”
LL-DNT-U-2007-012184.23
UNCLASSIFIED
UNCLASSIFIED
We don’t have all the answers yet,but we have a path forward
• Continue to apply industry best practices and commercial experience to software quality
• Codify our assessments of numerical convergence• Expand our understanding of the possible domains of our free
parameters• Expand sensitivity studies, screening efforts, and sampling techniques
to span the hyperspace of input parameter combinations• Develop new physics models that reduce our standard deviation for
predicting multiple experiments
We will use margin to compensate for uncertainties as we make continuous progress