Robust Design - Experiments For Better Products - Taguchi Techniques
How to do better experiments in SE
19
-
Upload
cs-ncstate -
Category
Technology
-
view
135 -
download
2
Transcript of How to do better experiments in SE
FROM TURKISH TOASTERS TO NASA SPACE SHIPS
2
Turhan, ESEj’09
Q: WHY IS THISAN URGENT QUESTION?
4
WHAT’S AT STAKE?
• “Transfer” is a core scientific issue
• Lack of transfer of causal effects is the scandal of SE
• Replication is Empirical SE is rare
• Conclusion instability• It all depends.
• The full stop syndrome
• The result?
• A funding crisis
5
THE BAD NEWS
7
WAR STORIES (DEFECT PREDICTION)
Menzies:TSE’13
MANUAL TRANSFER (WAR STORIES)
• Kitchenham, Mendes et al, TSE 2007: for = against
• Zimmermann FSE, 2009: cross works in 4/600 times
8
THE GOOD NEWS
BETWEEN TURKISH TOASTERS AND NASA SPACE SHIPS
10
Turhan, ESEj’09
Q: HOW TO TRANSFER LESSONS LEARNED?
Ignore most of the data
• relevancy filtering: Turhan ESEj’09; Peters TSE’13• variance filtering: Kocaguneli TSE’12,TSE’13• performance similarities: He ESEM’13
Contort the data
• spectral learning (working in PCA space or some other rotation) Menzies, TSE’13; Nam, ICSE’13
Buildi a bickering committee
• Ensembles Minku, PROMISE’12
11
BTW, SOMETIMES, TRANFER BETTER THAN LOCAL
12/1/2011
12
Minku:PROMISE’12 Nam:ICSE’13
Peters:TSE’13
THERE IS HOPE
• We’ve been looking in the wrong direction
• SE project data = surface features of an underlying effect• Go beneath the surface
14
REFLECT LESS ON RAW DIMENSIONS
12/1/2011
15
WHAT’S CHANGED?Mark of the old novice:
• Mostly manual analysis
• Obsesses on all the raw data
• Shares “the” model (the only, the single)
• E.g. “Depth of inheritanceis “the” most important predictor for defects.”
Mark of the new expert:
• Manual and automatic analysis• Combinations of Human + AI: • Each offering input and insights to the other
• Filters most of the data, transforms the rest• Shares analysis methods
• Cost effective methods for generating local lessons
12/1/2011
16
Most probably
wrong
NOT EXTERNAL VALIDITYBUT “META-EXTERNAL VALIDITY”
No pair programming, CMM5, agile programming, etc etc
But conclusion stability, generality
10/11/2013
17
With new data mining technologies, true picture emerges, where we can see what is going on
12/1/2011
18
SO THERE IS HOPE
19
