Acquisition Insight DaysScience of Test
Brent Russell, ctr.SOT Course Director
AFIT School of Systems and Logistics
Scientia Prudentia et Valor
STAT T&E COE: Scientia Prudentia et Valor
What is STAT?
• Scientific Test and Analysis Techniques (STAT) are the scientific and statistical methods and processes used to enable the development of efficient, rigorous test strategies that will yield defensible results.
• STAT encompasses such techniques as • Design of experiments (DOE),
• Data regression,
• Basic statistical analysis,
• Reliability growth,
• Survey design,
• Combinatorics
2DOE is the workhorse of STAT
STAT T&E COE: Scientia Prudentia et Valor
What is Design of Experiments (DOE)?
• Sequential, progressive test method
• Supports comprehensive analysis; main effects, interactions, and empirical modeling
• Based on proven statistical techniques and mathematical models
• Designed, randomized experiments establish correlation and causation
• Mathematically quantifiable as fewest points required for a given level of confidence
• Experimental noise is considered in the design and estimated through the analysis.
• Deals with test constraints while minimizing variance, maintaining statistical integrity.
• Empirical modeling of responses allows precise estimation of future responses.
3
“Scientific research is a process of guided learning. The object of statistical methods is to make that process as efficient as possible.” (Box, Hunter, Hunter 1978)
STAT T&E COE: Scientia Prudentia et Valor
DOE in Defense
INPUTS
(Factors)
OUTPUTS
(Responses)
PROCESS:
Air-to-Ground
Munitions
weather, training, TLE, launch conditions
Noise
Altitude
Weapon type
Impact Velocity
Delivery Mode
Impact Angle Delta
Impact AngleImpact Velocity Delta
Miss Distance
4
STAT T&E COE: Scientia Prudentia et Valor
Requirements with Ivy Hooks
• Day in the life (“Operational Concepts”)
– NOT design concepts
• “The system shall…”
• Necessary, verifiable, unambiguous
6
STAT T&E COE: Scientia Prudentia et Valor
7
Mismatch of Requirementsand Evaluation
• Evaluation of systems against specific requirements versus performance across the operational envelope– Often requirements are narrowly-focused, don’t cover the envelope
– Static in time and do not keep pace with evolving threat
– Test scope is often limited to thesystem under test while the systemwill be operated as asystem-of-systems in a jointenvironment
– DOT&E EmphasisD
iffic
ulty
of t
he E
nviro
nmen
t
Difficulty of the Target
Tests designed to requirements alonecould limit examination of system performance
Operational Envelope
Requirements Definition
STAT T&E COE: Scientia Prudentia et Valor
Other STAT Considerations
• What accuracy is required and is it the same across all conditions?
• What conditions matter?
• What if conditions change?
• How should the failing X% be evaluated?
• Are there critical conditions where failing cannot be allowed?
• How is failing performance evaluated?
• How rapidly does performance degrade once it goes below threshold?
• What analysis is required for the decision-maker?– Must the system demonstrate passing performance across the whole operational space?
– How will the space outside the specified region be evaluated?
– Is performance aggregated into a single performance distribution (across all conditions)?
– How can demos be proven with some level of assurance if they are one-off events?
– Which points will inform the average?
9
STAT T&E COE: Scientia Prudentia et Valor
DoDI 5000.02 Acquisition
“use scientific test and analysis techniques to design an effective and
efficient test program that will produce the required data to characterize system behavior across an appropriately selected set of factors and conditions” (DT&E)
“defensible statistical measures of merit (power and confidence)
associated with the coverage of the factors in a quantification of test risk. Specifically, the TEMP must discuss and display the calculations done to derive the content of testing and to develop the associated resource estimates” (OT&E)
11
STAT T&E COE: Scientia Prudentia et Valor
Confidence
Seeing this small difference from the original mean is not
a rare event
Seeing this large difference from the original mean is a
rare event
s
12
Original mean & distribution
Next data point
s
Original mean & distribution
Next data point
5% a level 5% a level
The selected confidence level will determine if the observed difference is “significant.”
In DOE, this is used to assess factor significance.
STAT T&E COE: Scientia Prudentia et Valor
Power
b is the risk we incorrectly conclude there is no
difference
13
Power is the probability to detect an effect that is present. The signal to noise ratio (d/s) is used to size the test design.
s
Original mean & distribution
Next data point &
distribution
5% a
b
Larger d and/or smaller s*will decrease b
and increase power
s*
5% a
b
d
STAT T&E COE: Scientia Prudentia et Valor
Decomposition for Test Planning
Decomposition helps define DT scope and allowsassignment of mission related requirements for OT
15
Process Principles• Decompose system components
– Defines low level testable elements
– Screen factors early in DT
– Develop models in DT (not just point check)
• Decompose mission segments– Assign mission-related (derived) requirements
– Include significant DT factors
• Tailor test space for OT– Critical operational regions or DT risk areas
– OT events can verify DT model predictions
STAT T&E COE: Scientia Prudentia et Valor
•Mission Relevant Factors List
•Predictive Model
•Operational Performance Expectations
Nu
mb
er
of
Fact
ors
Co
ntr
olle
d
Time
Venue•Open Air•Range•InstalledFactor Emphasis•Environmental•MissionOutput•Validate interfaces•Validate predictions•Assess environmentSTAT Benefit•Increasing Realism
Venue•Op Assessment•Mission Scenarios•ExercisesFactor Emphasis•Remaining factorsOutput•Evaluate Sys of Sys•Validate predictions•Assess OT readinessSTAT Benefit•Most Realism•Focused design
Collecting Data/Predicting Performance On Shared Measures Throughout
Increasing Realism
Venue•M&S•Lab•ChamberFactor Emphasis•TechnicalOutput•Validate design•Explore performance•Reduce riskSTAT Benefit•High Factor Control•Lower Cost
Extensive Prediction and
Modeling
More Realistic Testing
ValidatesPredictions
Leveraging Across DT and OT
Leverage DT for Information!
16
STAT T&E COE: Scientia Prudentia et Valor
A Design in 28 Runs…
18
4 Continuous Factors
Response
All data is simulated
28 runs
STAT T&E COE: Scientia Prudentia et Valor
Really Basic Analysis
19
If threshold was “zero”• 4/28 points < 0• 85.7% “passed”• But what contributed to failures?
But I am interested in much more information than this!
We can plot a histogram• Fit a curve (normal happens to fit)• We compute 17.5% actually < 0• Only 82.5% “pass”
STAT T&E COE: Scientia Prudentia et Valor
Plot The Response vs. Factors
20
How do I interpret this data?
This is only 2 of 4 factors? How do I assess all the information?
What does this spread mean?
Is this a trend?
How does this trend relate to the other one?
STAT T&E COE: Scientia Prudentia et Valor
What I Want To Know Is…
• What factors influence the response?
• What interactions influence the response?
• I want to
– Easily plot and interpret my data
– Analyze data against my requirement
– Predict other points in the space
– Simulate more data points
21
Can DOE Improve My Analytical Output?
STAT T&E COE: Scientia Prudentia et Valor
DOE…Significant Factors!
22
Wow! I have 10 significant effects!• Based on a confidence level (95%)• 3 Main Effects (ME)• 3 2-Factor Interactions (2FI)• 4 Quadratic Effects (Q)
ME
ME
ME
2FI
2FI2FI
Q
QQQ
I have an estimate on the magnitude of the effects
DOE analysis has created an empirical model of my data
using these terms
STAT T&E COE: Scientia Prudentia et Valor
What Can I Do With The Empirical Model?
23
See how well it explains my data
Get an estimate of noise in the data
Simultaneously explore the response throughout the factors space
The model can be used to better estimate how much of the space “passes”… and why
Where do I fail?By how much?
Predict the response at points not tested
STAT T&E COE: Scientia Prudentia et Valor
DOE Delivers Powerful Analysis!
• DOE goes well beyond basic analysis
• Set analysis goals… build design to meet them
• Empirical modeling provides superior insight
– Significant factors: ME, 2FI, higher order
– Easy to plot and interpret
– Permits prediction in the factor space
– Assess performance and causal conditions
– Simulate points for additional data
24
Starting at the end shows you the benefits of DOE before you learn the “whys”
Top Related