Post on 13-Jul-2020
Mark Kiemele
Air Academy Associates
1650 Telstar Drive, Ste 110
Colorado Springs, CO 80920
Phone: 719-531-0777
email: aaa@airacad.com
All rights reserved. Do not reproduce. www.airacad.com
Accelerating the Analysis of Test
Data Using Effective and Efficient
Experimentation
ITEA TIW
Las Vegas, NV
May 14, 2019
Copyright © 2018
19-ITEATIWDOE-5A
© 2018
1|
Introductions
1
• Name
• Position and Company/Organization
• Experience in experimentation and T&E
• Expectations
© 2018
2|
Agenda and Objectives
• Prerequisites for successful experimentation
• Remove excessive variation from the system.
• Ensure the measurement system is capable.
• The Four Pillars of Design of Experiments
• Demonstrate each of these pillars using the Statapult
• Screening
• Modeling
• Performance Verification and Validation
2
© 2018
3|
Statapult ® Catapult
Catapulting Power into Knowledge Gain and T&E
3
© 2018
4|
Simulating a Rapid Improvement Event (RIE)
• Statapult = Delivery Process
• Ball = Service Provided
• Setup = Job prep
• Measurement = Outcome of the service
Using the statapult as a metaphor
4
© 2018
5|
Statapult® Exercise #1(Baselining a Process)
1. Insure all pins are at position #3
2. Pull the arm to 177° and launch the rubber ball
3. Have someone measure your distance
4. Disconnect rubber band between shots
5. Your standard is no more than 15 seconds between shots
6. Record distances; Calculate Range
#1 #2 #3 #4 #5
Range = Longest - Shortest =
Each team member will shoot the Statapult® 3 times using the following steps:
Shot #1
Shot #2
Shot #3
Team Member:
5
#6 #7 #8
© 2018
6|
C = Constants Standard Operating Procedures (SOPs)
N = Noise
X = Experimental
Output(s) and
Specs
(1) PROCESS FLOW (PF) OR PROCESS MAP
(2) CUSTOMER DRIVEN CAUSE AND EFFECT (CE)
How What Who___________
___________
___________
... removes waste, reduces variation, and decreases cycle time
PF/CE/CNX/SOPA Rapid Improvement Event and a Management Tool that
6
© 2018
7|
X = Experimental
• These are key variables that can be controlled and held constant at different
levels or settings for the purpose of determining the effect of this variable on
the CTC.
C = Controlled
• To hold a variable as constant as possible requires controlling the variable via
Mistake Proofing and SOPs to eliminate errors and reduce variation.
• Controlling a variable or holding it as constant as possible doesn't just
happen. It must be “engineered” into the process.
• Mistake Proofing: The process of eliminating conditions that lead to variation
in the CTCs and ultimately cause errors.
N = Noise
• Noise variables are those that are not being controlled or held as constant as
possible
• Mistake Proofing is needed to change an "N" variable to a "C" variable.
(3) Partitioning the Variables into CNX
7
© 2018
8|
(4) Standard Operating Procedures (SOPs)
• Define the interaction of people and their environment when processing
a product or service
• Detail the action and work sequence of the operator
• Provide a routine to achieve consistency of an operation
• Specify the best process we currently know and understand for
controlling variation and eliminating waste
• Provide a basis for future improvements
• Validate mistake proofing in the process
• Strongly impact compliance to QMS such as ISO 9000, CMM, Sarbanes-
Oxley, etc.
8
© 2018
9|
1. Process flow "Shooting the Statapult®"
2. Complete Cause-and-Effect Diagram
3. Label inputs as C or N
4. Use SOPs with mistake proofing to change N’s into C’s
5. Re-shoot Statapult® using the first example instructions (all pins at #3; pull angle = 177°; 15 sec. between shots; etc.)
6. Record data taken after PF/CE/CNX/SOPs and evaluate
7. If improved, develop control plans
Statapult® Exercise #2: Demonstrating Improvement
9
#1 #2 #3 #4 #5
Range = Longest - Shortest =
Shot #1
Shot #2
Shot #3
#6 #7 #8
Team Member:
© 2018
10|
Process Flow
10
Start/Set Up
Place ball in cup
Pull back the arm
177o
?
Spot the Ball/Record Retrieve the Ball Reset the Rubber Band
Final Shot of
Operator ?
Change Operators
No
No
Yes
Yes Final Operator?
Yes
No
Stop
Release the arm
© 2018
11|
Cause-and-Effect Diagram Worksheet
11
Distance
Machine Materials Methods
Manpower Mother Nature Measurement
© 2018
12|
Measurement System Analysis
12
Statapult ® Catapult
© 2018
13|
Data Collection Template #1
13
© 2018
14|
Measurement is a Process
14
Measure-
ment
Process
My Process
Y (true)
Y (recorded)
Where DOES the variation that we see in Y come from? Is it from the process itself or the
measurement system? Which one of those two variations is stronger?
Is the measurement system –
Accurate?
Precise?
Stable?
© 2018
15|
Partitioning the Measurement System Variability
15
• MSA identifies and quantifies the different sources of variation that
affect a measurement system.
• Variation in measurements can be attributed to variation in the
product/service itself or to variation in the measurement system.
• The variation in the measurement system itself is measurement error.
Product / Service
Variability
Measurement
Variability
Total (recorded) Variability is broken into two major pieces:
This piece of the pie
will be further divided
into two smaller
pieces called
Repeatability and
Reproducibility.
© 2018
16|
Measurement Variability Broken Down Further
16
Purpose:
To assess how much variation is associated with the measurement system and
to compare it to the total process variation or tolerances.
Repeatability:
Variation obtained by the same person using the same procedure on the same
product, transaction or service for repeated measurements (variability within
operator).
Reproducibility:
Variation obtained due to differences in people who are taking the
measurements (variability between operators).
total product measurement= +2 2 2σ σ σ
repeatability reproducibility+2 2σ σ
© 2018
17|
More on Repeatability and Reproducibility
17
For the scenario below, look at the data and indicate which of the following
you think is true:
a. Repeatability appears to be more of a problem than reproducibility
b. Reproducibility appears to be more of a problem than repeatability
c. Repeatability and Reproducibility appear to be about the same
Operator 1 Operator 2
Rep 1 Rep 2 Rep 1 Rep 2
21 23 26 28
19 18 24 24
20 23 27 24
19 22 21 20
▪ MSA will help quantify, more exactly, the capability of the measurement
system and answer questions about repeatability, reproducibility, and
capability with respect to the customer specs.
▪ Rule of Thumb for MSA Sample Size:
- Variables (Continuous) Data: (# operators)*(number of parts) ≥ 20
- Attribute (Binary) Data: (# operators)*(number of parts) ≥ 60
Part 1
Part 2
Part 3
Part 4
© 2018
18|
Measurement System Diagnostics
18
1. Precision-to-Tolerance Ratio (P/TOL)
P/TOL = (Specification Limits are needed)
ROT: If P/TOL .10 : Very Good Measurement System
P/TOL .30 : Unacceptable Measurement System
2. Precision-to-Total Ratio (P/TOT)
P/TOT =
ROT: If P/TOT .10 : Very Good Measurement System
P/TOT .30 : Unacceptable Measurement System
3. Discrimination or Resolution
(# of truly distinct measurements that can be obtained by the
measurement system) ROT: Resolution ≥ 5
total
meas
LSLUSL
6 meas
−
product
meas
.
=
1 41
© 2018
19|
Analysis of Variance (ANOVA) Results
19
• P/TOL and P/TOT are too high.
• Resolution is unacceptable.
• Reproducibility is significantly larger than Repeatability and
appears to be the biggest problem with this measurement
process.
© 2018
20|
Graphical View of Variance Components
20
ReproducibilityRepeatability
Part-to-Part
0
1
2
3
4
5
6
7
8
Z Axis
X Axis Category
Interaction
Operator
© 2018
21|
Graphical View of Operator by Part Interaction
21
Operator By Part
0
5
10
15
20
25
30
1 2 3 4
Part #
Meas
ure
men
t
Operator 1
Operator 2
© 2018
22|
Data Collection Template #2
22
© 2018
The Foundations of Design of Experiments (DOE)
Ort
ho
gon
alit
y
Re
plic
atio
n
Ran
do
miz
atio
n
Blo
ckin
g
Creating a Culture of Effective and Efficient
Experimentation
The Four Pillars of DOE
23
© 2018
Famous Quote
“All experiments (tests) are designed;
some are poorly designed,
some are well designed.”
George Box (1919-2013), Professor of Statistics, DOE Guru
24
© 2018
Design of Experiments (DOEs):
A Subset of All Possible Test Design Methodologies
27
The Set of All Possible Test Design
Methodologies (Combinatorial Tests)
Orthogonal or
Nearly
Orthogonal
Test Designs
(DOEs)
One
Factor
At a
Time
(OFAT)
Best Guess
(Oracle)
Boundary Value Analysis
(BVA)
Equivalence Partitioning (EP)
© 2018
OFAT (One Factor at a Time) Testing
4. One factor at a time results
versus optimal results
3. One factor at a time
results
Chemical
Process
Yield (gr.)X1 = time
Y
80
70
60 90 120 150 180X1
1. Hold X2 constant and vary X1
Find the “best setting” for X1Y
80
70
210 220 230 240 250X2
2. Hold X1 constant at “best setting” and vary X2.
Find the “best setting” for X2.
20060 90 120 150 180
X1
210
220
230
240
250
X2
9080
6070
X2
X1
60 90 120 150 180
• • • • •
•
•
•
•
X2 = temp
200
210
220
230
240
250
25
© 2018
The Good and Bad about OFAT
• Good News
• Simple
• Intuitive
• The way we were originally taught
• Bad News
• Will not be able estimate variable interaction effects
• Will not be able to generate prediction models and thus
not be able to optimize performance or assess risk
26
© 2018
What is a DOE ?
Purposeful and systematic changes of the inputs (factors) in order
to observe corresponding changes in the output (response).
Run
1
2
3
.
.
X1 X2 X3 X4 Y1 Y2 . . . . . . Y SY
Inputs
A = X1
B = X2
D = X4
C = X3
Y1
Outputs
.
.
.
.
.
.
PROCESS Y2
28
© 2018
Statistically Designed Experiments (DOE):
Orthogonal or Nearly Orthogonal Designs
• FULL FACTORIALS (for small numbers of factors)
• FRACTIONAL FACTORIALS
• PLACKETT - BURMAN
• LATIN SQUARES Taguchi Designs
• HADAMARD MATRICES
• BOX - BEHNKEN DESIGNS
• CENTRAL COMPOSITE DESIGNS
• HIGH THROUGHPUT TESTING (ALL PAIRS)
• NEARLY ORTHOGONAL LATIN HYPERCUBE DESIGNS
SIMPLE DEFINITION OF A TWO-LEVEL
ORTHOGONAL DESIGN
Run Actual Settings Coded Matrix Interactions
1
2
3
4
5
6
7
8
(5, 10) (70, 90) (100, 200)
A: Time B: Temp C: Press
(A) (B) (C)
Time Temp Press
5
5
5
5
10
10
10
10
70
70
90
90
70
70
90
90
100
200
100
200
100
200
100
200
-1 -1 -1
-1 -1 +1
-1 +1 -1
-1 +1 +1
+1 -1 -1
+1 -1 +1
+1 +1 -1
+1 +1 +1
Yield
Example:
Chemical
Process
A: Time (5,10)
B: Temp (70,90)
C: Press (100,200)
(AB) (AB)
Uncoded Coded
350
350
450
450
700
700
900
900
+1
+1
-1
-1
-1
-1
+1
+1
Response Surface
Designs
29
© 2018
Correlations Using Actual vs Coded Units
31
Correlation MatrixA B C AB AC BC ABC
A 1.0 0.0 0.0 0.929981 0.688247 0.0 0.661581
B 1.0 0.0 0.348743 0.0 0.348743 0.248093
C 1.0 0.0 0.688247 0.929981 0.661581
AB 1.0 0.640057 0.121622 0.711392
AC 1.0 0.640057 0.961256
BC 1.0 0.711392
ABC 1.0
Summary
Mean 7.5 80.0 150.0 600.0 1125.0 12000.0 90000.0
StDev 2.6726 10.69 53.452 229.907 582.482 4598.14 48476.8
Count 8 8 8 8 8 8 8
A B C AB AC BC ABC
5 70 100 350 500 7000 35000
5 70 200 350 1000 14000 70000
5 90 100 450 500 9000 45000
5 90 200 450 1000 18000 90000
10 70 100 700 1000 7000 70000
10 70 200 700 2000 14000 140000
10 90 100 900 1000 9000 90000
10 90 200 900 2000 18000 180000
A B C AB AC BC ABC
-1 -1 -1 1 1 1 -1
-1 -1 1 1 -1 -1 1
-1 1 -1 -1 1 -1 1
-1 1 1 -1 -1 1 -1
1 -1 -1 -1 -1 1 1
1 -1 1 -1 1 -1 -1
1 1 -1 1 -1 -1 -1
1 1 1 1 1 1 1
Correlation MatrixA B C AB AC BC ABC
A 1.0 0.0 0.0 0.0 0.0 0.0 0.0
B 1.0 0.0 0.0 0.0 0.0 0.0
C 1.0 0.0 0.0 0.0 0.0
AB 1.0 0.0 0.0 0.0
AC 1.0 0.0 0.0
BC 1.0 0.0
ABC 1.0
Summary
Mean 0.0 0.0 0.0 0.0 0.0 0.0 0.0
StDev 1.069 1.069 1.069 1.069 1.069 1.069 1.069
Count 8 8 8 8 8 8 8
© 2018
The Beauty of Orthogonality
(Vertical and Horizontal Balance)
A Full Factorial Design for 3 Factors A, B, and C, each at 2 levels:
30
© 2018
Takata Airbag Defect Findings:
Independent Testing Coalition (ITC) makes key findings
ITC says exposure to heat and humidity, and the use of ammonium nitrateare all required to produce what the commission and the National Highway Traffic
Safety Administration (NHTSA) call an “energetic disassembly.”
“You can’t have the energetic disassembly without all three factors," David Kelly,
leader of the ITC and former chief of the NHTSA told Automotive News Europe.
"You have to have all three.”
In DOE, this is called a significant 3-way interaction effect.
31
© 2018
Design of Experiments (DOE)
▪ An optimal data collection methodology
▪ “Interrogates” the process or product
▪ Used to identify important relationships between inputs (factors)
and outputs (response variables)
▪ Identifies important interactions between input variables
▪ Can be used to characterize and optimize a process
▪ Can be used to asses risk
▪ Changes “I think” to “I know”
▪ Is the science of test and the key connector between test and
evaluation
32
© 2018
Three Major Types of DOEs
• Modeling Designs
- For building functions that can be used to predict outcomes,
assess risk, and optimize performance. These include
the ability to evaluate interaction and higher order effects.
• Performance Verification and Validation Testing
- For confirming that a system performs in accordance with
its specifications/requirements.
• Screening Designs
- For testing many factors in order to separate the vital few
critical factors from the trivial many.
33
© 2018
WHAT IS
THE
OBJECTIVE?
SCREENING VALIDATION or CONFIRMATION
Rules of Thumb for Design Selection
MODELING
Notes:
1. “Mixed” factors means a combination of quantitative and qualitative (categorical)
2. “Mixed” levels means that not all factors have the same number of levels (settings)
3. “K” = Number of Factors and “L” = Number of Levels
4. HTT = High Throughput Testing
5. DSD = Definitive Screening Design
6. “OA” stands for Orthogonal Array; “PAVO” = Pairwise Value Ordering
7. Software such as HD ToolsTM, rdExpertTM Lite, Pro-TestTM and Quantum XLTM
generate some or all of these designs
Mixed Factor/Mixed
Level Designs:
HTT (OA or PAVO)
3-Level Designs:
Full Factorial (K ≤ 3)
CCD, BB, D-optimal
2-Level Designs:
Full Factorial (K ≤ 4)
Fractional Factorial
(K = 5)
D-Optimal
Fixed Number
of Samples:
Descriptive
Sample (DS)*
Latin
Hypercube
Sample (LHS)*
Not a Fixed Number
of Samples:
HTT (all pairs, OA,
PAVO)
* DS and LHS are sampling techniques to generate representative samples
according to a specified distribution and a specified sample size
* Representative samples do not give orthogonal designs. They are often
used for getting test coverage, validating performance/ determining
capability, or creating noise combinations for test
DOE ProTM software is copyright Air Academy Associates, LLC and Digital Computations, Inc.HD ToolsTM is a trademark of Air Academy Associates, LLC and software is copyright SigmaXL.rdExpertTM Lite software is copyright Phadke Associates, Inc.Pro-Test TM software is copyright Digital Computations, Inc.Quantum XLTM software is copyright SigmaZone.com.
2-Level
Designs:
L12 (6 ≤ K ≤ 11)
Fractional
Factorial (res IV)
3-Level
Designs:
L18 (4 ≤ K ≤ 8)
DSD
High Factor/High Level
Designs: (K ≥ 9 and L ≥ 5)
Nearly Orthogonal Latin
Hypercube Designs
(NOLHDs) with K*L runs(NOLHDs require all factors have the
same number of levels)
Mixed Factor/Mixed
Level Designs:
Full Factorial
HTT (OA or PAVO; with
only select interactions)
D-optimal
34
© 2018
36|
Screening DOE on the Statapult
36
• Conduct an L12 Screening Design using the following factors:
Settings
Low (-) High (+)
A: Pull Back Angle 160° 180°
B: Stop Angle 2 3
C: Pin Height 2 3
D: Cup Height 4 5
E: Rubber Band Position 2 3
F: Ball Type 1 2
G: Operator 1 2
• Perform the data collection phase of the experiment and then conduct the analysis as
demonstrated on the subsequent pages of this notebook.
© 2018
37|
DOE PRO Software to Select the Design
37
DOE PRO > Create Design > Non-Computer Aided
© 2018
38|
Naming the Factors, Settings, Responses and Replicates
38
Default setting is for 95% confidence
and power in the standard deviation
assessment.
© 2018
39|
Example Data
39
Factor A B C D E F G
Row #Pull Back
Angle
Stop
Angle
Pin
Height
Cup
Height
Ball
TypeOperator Y1 Y2 Y3 Y4
Rubber Band
Position
1
12
11
10
9
8
7
6
5
4
3
2
160
180
180
180
180
180
180
160
160
160
160
160
2
3
3
3
2
2
2
3
3
3
2
2
2
2
2
3
2
3
3
3
3
2
3
2
4
4
5
4
5
4
5
5
4
5
5
4
2
3
2
2
3
3
2
2
3
3
3
2
1
1
2
1
2
2
1
2
2
1
1
2
1
2
1
1
1
2
2
2
1
2
1
2
31
99.5
106.75
108
101.5
118
112.5
93
89.5
77.5
68.5
33.5
34
99
103
106
103
117
112.5
92.5
90.5
77
63.5
34.75
30.5
100.5
107
110
107
114.5
114.5
93.5
92
78.5
62
35
32.5
100.5
109.5
112
103
117
111.5
92
86.5
76.5
70
33
© 2018
40|
Software Commands to Analyze the Data
40
Analyze the design matrix by selecting
DOE PRO > Analyze Design > Marginal Means Plot…
Analyze the design matrix by selecting
DOE PRO > Analyze Design > Multiple Response Regression
© 2018
41|
Regression and Marginal Means Plots for Standard Deviation (s)
41
© 2018
42|
42
Regression and Marginal Means Plots for the Mean (Y bar)
© 2018
43|
0
5
10
15
20
25
Ab
so
lute
Co
eff
icie
nt
pull back
angle (A)
pin height (C) stop angle
(B)
cup height
(D)
rubber band
position (E)
ball type (F) operator (G)
Effect Name
Y-hat Pareto of Coeffs -
pull back angle (A)
pin height (C)
stop angle (B)
cup height (D)
rubber band position (E)
ball type (F)
operator (G)
Pareto of Coefficients for y-hat and s-hat
43
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Ab
so
lute
Co
eff
icie
nt
operator (G) pin height (C) cup height
(D)
rubber band
position (E)
stop angle
(B)
pull back
angle (A)
ball type (F)
Effect Name
S Pareto of Coeffs -
operator (G)
pin height (C)
cup height (D)
rubber band position (E)
stop angle (B)
pull back angle (A)
ball type (F)
© 2018
44|
Results of the Screening Design
44
• What factors, if any, had a significant effect on the average
distance launched?
• What factors, if any, had a significant effect on the process
standard deviation?
• Any lessons learned?
© 2018
45|
Modeling the Statapult
45
Statapult ® Catapult
© 2018
46|
The Theoretical Approach
46
© 2018
47|
A Theoretical Model
47
)θsinθ(sin)rmgrMg(θdφcosθins)θF(rθI2
10BGF
2
0−+−=
θsin)rmgrMg(φcosθsin)θ(FrθIBGF0
+−= ,cosrd
sinrDtan
F
F
+
−=
).θsinθ(sin)rmgrMg(θdφcosθsin)θF(rθI2
101BGF
2
10−+−=
1B1Bθcosr
2
1tθ
2
πcosvx −
−= .gt
2
1tθ
2
πsinvθsinry 2
1B1B−
−+=
.0
2cos
)cosrR(
V2
g
2tan)cosrR(sinr
2
12
1B2B
11B1B =
−
+−
−
++
−
++
−
+
11B1B12
21B
B
0
2tan)cosrR(sinr
2cos
)cosrR(
r4
gI
θ
θ0
1
0
θ
θ
1
0
θ
θ
).sin(sin)rmgrMg(dcossin)F(r 01BGF −+−=
..
.
.
© 2018
Statapult®
DOE
(The Empirical Approach)
Factor A B C Distance
Row # Pull Back Angle Pin Height Stop Pin Y1 Y2 Y3 Y4 Y bar S
1 160 2 2 33.5 33.75 34 34 33.8125 0.239357
2 160 2 3 54.25 54 55.5 54 54.4375 0.71807
3 160 3 2 44 44 41.5 41.5 42.75 1.443376
4 160 3 3 67 66 67.5 65.5 66.5 0.912871
5 180 2 2 55.5 56 55.5 55 55.5 0.408248
6 180 2 3 73 72 71 71.25 71.8125 0.898494
7 180 3 2 74 73.75 74.5 75 74.3125 0.554339
8 180 3 3 88.5 88.5 90 90 89.25 0.866025
Random Order = 2 ,8 ,4 ,7 ,3 ,5 ,6 ,1
47
© 2018
Regression Analysis on Statapult DOE Data
Y-hat Model
Distance
Factor Name Coeff P(2 Tail) Tol Act
ive
Const 61.047 0.0000
A Pull Back Angle 11.672 0.0000 1 X
B Pin Height 7.156 0.0000 1 X
C Stop Pin 9.453 0.0000 1 X
AB 1.906 0.0000 1 X
AC -1.641 0.0000 1 X
BC 0.21875 0.1494 1
ABC -0.56250 0.0008 1 X
R20.9982
Adj R2 0.9976
Std Error 0.8307
F 1877.9499
Sig F 0.0000
FLOF NA
Sig FLOF NA
Source SS df MS
Regression 9071.9 7 1296.0
Error 16.6 24 0.7
ErrorPure 16.6 24 0.7
ErrorLOF 0.0 0 NA
Total 9088.4 31 48
© 2018
Statapult®
DOE Demo
(The Empirical Approach)
Run
1
2
3
4
A B A B AB Y1 Y2 Y S
Actual
FactorsCoded Factors Response Values
144 2
144 3
160 2
160 3
-1 -1 +1
-1 +1 -1
+1 -1 -1
+1 +1 +1
Percent Confidence
that a term identified
as significant, truly
does belong in [ ]
95% (a = .05)
95% (a = .05)
95% (a = .05)
95% (a = .05)
95% (a = .05)
Percent chance of
finding a significant
variance [average]
shifting term if one
actually exists
40% (b = .60)
75% (b = .25)
90% (b = .10)
95% (b = .05)
99% (b = .01)
Number of Runs in 2 Level Portion of the Design
2
Sample Size per Experimental Condition
4 8 12 16
5 [3]
9 [5]
13 [7]
17 [9]
21 [11]
3 [2]
5 [3]
7 [4]
9 [5]
11 [6]
2 [1]
3 [2]
4 [2]
5 [3]
6 [3]
N/A
2 [1]
3 [2]
4* [2]
5* [3]
N/A
2 [1]
N/A
3 [2]
4* [2]
Simplified Table for Determining Sample Size Based on Confidence and Power
s y
49
© 2018
Value Delivery: Reducing Time to Market for New Technologies
• Total # of Combinations = 35 = 243
• Central Composite Design: n = 30
Modeling Flight
Characteristics
of New 3-Wing
Aircraft
Pitch )
Roll )
W1F )
W2F )
W3F )
INPUT OUTPUT
(-15, 0, 15)
(-15, 0, 15)
(-15, 0, 15)
(0, 15, 30)
(0, 15, 30)
Six Aero-
Characteristics
Patent Holder: Dr. Bert Silich
50
© 2018
Aircraft Equations (predictive models)
CL = .233 + .008(P)2 + .255(P) + .012(R) - .043(WD1) - .117(WD2) + .185(WD3) + .010(P)(WD3) -
.042(R)(WD1) + .035(R)(WD2) + .016(R)(WD3) + .010(P)(R) - .003(WD1)(WD2) -
.006(WD1)(WD3)
CD = .058 + .016(P)2 + .028(P) - .004(WD1) - .013(WD2) + .013(WD3) + .002(P)(R) - .004(P)(WD1)
- .009(P)(WD2) + .016(P)(WD3) - .004(R)(WD1) + .003(R)(WD2) + .020(WD1)2 + .017(WD2)2
+ .021(WD3)2
CY = -.006(P) - .006(R) + .169(WD1) - .121(WD2) - .063(WD3) - .004(P)(R) + .008(P)(WD1) -
.006(P)(WD2) - .008(P)(WD3) - .012(R)(WD1) - .029(R)(WD2) + .048(R)(WD3) - .008(WD1)2
CM = .023 - .008(P)2 + .004(P) - .007(R) + .024(WD1) + .066(WD2) - .099(WD3) - .006(P)(R) +
.002(P)(WD2) - .005(P)(WD3) + .023(R)(WD1) - .019(R)(WD2) - .007(R)(WD3) + .007(WD1)2
- .008(WD2)2 + .002(WD1)(WD2) + .002(WD1)(WD3)
CYM= .001(P) + .001(R) - .050(WD1) + .029(WD2) + .012(WD3) + .001(P)(R) - .005(P)(WD1) -
.004(P)(WD2) - .004(P)(WD3) + .003(R)(WD1) + .008(R)(WD2) - .013(R)(WD3) + .004(WD1)2
+ .003(WD2)2 - .005(WD3)2
Ce = .003(P) + .035(WD1) + .048(WD2) + .051(WD3) - .003(R)(WD3) + .003(P)(R) - .005(P)(WD1)
+ .005(P)(WD2) + .006(P)(WD3) + .002(R)(WD1)
51
© 2018
Cyber Penetration Testing Example*
Putting these factor names and their levels, along with the test constraints, into Pro-Test software yields these 14 optimal test configurations:
* Courtesy of Raytheon
Cyber
Penetration
Testing
Operating System
IS Type
y (did/did not penetrate)
Port
52
© 2018
Penetration Vulnerability Test Design Matrix*
* Courtesy of Raytheon 53
© 2018
Command and Control Test Design Example(15 factors each at various levels)Total Combinations: 20,155,392
Variable or Factor Levels (# of levels)
Mission Snapshots Entry, Operations, Consolidation (3)
Network Size 10 Nodes, 50 Nodes, 100 Nodes (3)
Network Loading Nominal, 2X, 4X (3)
Movement Posture ATH, OTM1, OTM2 (3)
SATCOM Band Ku, Ka, Combo (3)
SATCOM Look Angle 0, 45, 75 (3)
Link Degradation 0%, 5%, 10%, 20% (4)
Node Degradation 0%, 5%, 10%, 20% (4)
EW None, Terrestrial, GPS (3)
Interoperability Joint Services, NATO (2)
IA None, Spoofing, Hacking, Flooding (4)
Security NIPR, SIPIR (2)
Message Type Data, Voice, Video (3)
Message Size Small, Medium, Large, Mega (4)
Distance Between Nodes Short, Average, Long (3)
54
© 2018
Command and Control Test Example (cont.)All Pairs Testing from Pro-Test generates 26 test cases
55
© 2018
INPUTS OUTPUTS
CONSTRAINTS
Number of Targets (1,2)
Operator Type (Tier 2 -
male/female)
Time of Activity day/night
Ancillaries
7 configurations
Soldier Combat
Ensemble (i.e. Body
Armour
EF88 and Baseline
(SA2) Ancillary
Testing
Probability-of-hit (Target Hit
Ratio)
Time-to-Engage Target
NOISE
Weather
Target Distance
100, 200, 300, 400, 600
Type of Engagement (deliberate,
snap)
Firing Position –
prone/kneeling/standing
* The description and results of this study appear in the June 2016 ITEA Journal, under the title of “Australia’s First Official Use of Design of Experiments in T&E: User Trials to Select Rifle Enhancements” by Joiner, Kiemele, and McAuliffe.
Using DOE to Select Rifle Enhancements*(trials conducted by Australian Defence Test & Evaluation Office)
56
© 2018
Submarine Threat Detection Test ExampleAll Pairs Testing from Pro-Test generated 15 test cases
Suppose we want to perform a verification test with the following 7 input factors (with their respective settings):
Submarine Type (S1, S2, S3)Ocean Depth (Shallow, Deep, Very Deep)Sonar Type (Active, Passive)Target Depth (Surface, Shallow, Deep, Very Deep)Sea Bottom (Rock, Sand, Mud)Control Mode (Autonomous, Manual)Ocean Current (Strong, Moderate, Minimal)
All possible combinations would involve how many runs in the test?
If we were interested in testing all pairs only, how many runs would be in the test? Pro-Test generated the following test matrix:
1296
57
© 2018
The Value of Predictive Models
58
© 2018
The Big Picture: What can DOE do for us?
1. Infuses power into Systems Engineering (see next page)
2. Provides systematic coverage of the test region
3. Improves the quality of testing, and gives us
▪ Faster detection of problems
▪ Higher probability of detecting faults
▪ Cost and time efficiencies
▪ Ability to quantify risks inherent to any test program
Better testing done faster
at lower cost with less risk
Improved Customer Value and Organizational Success
59
© 2018
DOE Empowers Systems Engineering
FL
OW
ING
RE
QU
IRE
ME
NT
S D
OW
N
Customer Needs
System
Requirements
Sub-system
Requirements
Module
Requirements
Parts
Requirements
Parts
Performance
Module
Performance
Sub-system
Performance
System
Performance
Customer Acceptance
FL
OW
ING
CA
PA
BIL
ITY
UP
Design & Development
60
© 2018
On Leading Change and Driving Business Results
61
© 2018
Best Practices for “Operationalizing” DOE(i.e., changing the culture to one of habitually using it)
1. Training plus coaching on projects is an absolute must.
2. A Keep-It-Simple-Statistically (KISS) approach with easy-to-comprehend
materials and easy-to-use software.
3. Gaining and propagating quick-hitting successes.
4. Getting leadership on board and continuously re-invigorating them is
necessary.
5. Developing a culture of continuously generating transfer functions (predictive
models) for the purpose of optimization, prediction, and risk assessment.
62