Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006...
Transcript of Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006...
![Page 1: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/1.jpg)
1
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
byGiovanni CantoneExperimental Informatics, ESE
Dept. of Informatics, Systems and Production Eng., DISPUniversity of Rome at Tor Vergata Campus, UNIRM2
Experimental Software Engineering:
Role and Impact of Measurement Models on
Empirical Processes
![Page 2: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/2.jpg)
2
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
Outline
• ESE (E2SE)
•EPM (Typical)• Case 1: Traditional & Simple• IMPACT OF MM ON EPM• Case 2: New & Complex
![Page 3: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/3.jpg)
3
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
ESE & Modeling
1- Development and continual improvement of empirical-evidence-based
software models.
2- Capitalization organization wide of the results.
![Page 4: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/4.jpg)
4
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
Basic components
• GQM
• QIP
• Applied Statistics
• MMLC
EPM
• EXPERIENCE FACTORY
![Page 5: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/5.jpg)
5
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
Empirical strategies
SurveyCase studyExperiment
See for instance[C. Wohlin et others.: Experimentation in SE,See for instance[C. Wohlin et others.: Experimentation in SE, KluwerKluwer AP] AP]
![Page 6: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/6.jpg)
6
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
Types of experiment context
Blocked subjects and objects study
Single object multi-test studyMany
Variation study on multiple objects
Single object study
One
ManyOne
Objects
Team |Subjects
Replication
![Page 7: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/7.jpg)
7
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
The Experiment
RRII EE
![Page 8: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/8.jpg)
8
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
Dep
ende
nt v
aria
bles
Dep
ende
nt v
aria
bles
FactorsFactors
TreatmentsTreatments
EOEO
Inde
pend
ent v
aria
bles
Inde
pend
ent v
aria
bles
EDED
[Wohlin][Wohlin]
The Experiment
![Page 9: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/9.jpg)
9
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
The Experiment
RRII
DefinitionDefinition
Planning & designPlanning & design
OperationOperation
InterpretationInterpretation
Package & Package & DiffusionDiffusion
![Page 10: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/10.jpg)
10
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
The Experiment Process
DefinitionDefinition
Planning & designPlanning & design
OperationOperation
InterpretationInterpretation
Package & Package & diffusiondiffusion
II
RR[Wohlin][Wohlin]
![Page 11: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/11.jpg)
11
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
The Experiment ProcessExperiment Experiment definitiondefinition
Experiment Experiment planning & designplanning & design
Experiment Experiment operationoperation
Experiment Experiment interpretationinterpretation
Package & Package & diffusiondiffusion
RR
II
![Page 12: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/12.jpg)
12
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
The Experiment Process
RR
II
EE MM
![Page 13: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/13.jpg)
13
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
CASE I
SW Quality : Benefits and Costs of CCM
Inspection of HL CCM SW:Effectiveness and Efficacy
![Page 14: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/14.jpg)
14
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
CASE ISW Quality : Benefits and Costs of CCM Inspection of CCM SW
Inspection of Analysis and Design SW CCM Documents:
Effectiveness and Efficacy
CCMCCMSWSW
DocumentsDocuments
!CCM!CCMSWSW
DocumentsDocuments
![Page 15: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/15.jpg)
15
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
CASE ISW Quality : Benefits and Costs of CCM Inspection of CCM SW
Defects of Inspected SW Entering Testing
CCMCCMSWSW
!CCM!CCMSWSW
![Page 16: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/16.jpg)
16
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
CASE ISW Quality : Benefits and Costs of CCM Inspection of CCM SW
Finding Seeded Defects:Efficiency & Efficacy
Code readingCode reading TestingTesting
![Page 17: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/17.jpg)
17
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
Experiment ProcessDefinitionObject Strategies of Code-reading
and of Functional testing.Purpose To evaluate performanceQuality focus Efficiency & Efficacy in
finding seeded defects.Perspective End-user.Context URM2.DISP.ESEG + X.Z.com
![Page 18: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/18.jpg)
18
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
MeasurementsCounting measures
• Number of faults found
• Duration of the session
Simple Indirect measurements
• Fault Classification
![Page 19: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/19.jpg)
19
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
Experiment ProcessExperiment planning
• Context selection
• Hypothesis formulation
• Variable selection
• Independent variables•Treatments (Code reading, F. Testing)•Factors: (Categorized faults)
•Dependent Variables
![Page 20: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/20.jpg)
20
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
Experimental ProcessFault Categories
Initialization: e.g. wrong initialization of attributesComputing: e.g. wrong computations of variablesControl: e.g. wrong definitions of logic variables.Building and using complex structures of data: e.g. inserting
elements unrelated to the logic and the structure of data.Graphical interface: e.g. wrong settings of interface windows.Functionality: e.g. wrong realization of functionality.Events managing: e.g. wrong management of event.Exceptions handling: e.g. unforeseen produced exception
![Page 21: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/21.jpg)
21
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
Experimental Process
• Experiment design
• Validity evaluation
• Instrumentation
• Objects• Guidelines• Measurement instruments
![Page 22: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/22.jpg)
22
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
Experimental Process
• Subject training
• Context selection
• Mode of work• Types of subjects• Type of application• Type of validity
![Page 23: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/23.jpg)
23
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
Experimental ProcessOperation
• Preparation
• Execution
![Page 24: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/24.jpg)
24
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
Experimental Process
Pre-Analysis & Interpretation
![Page 25: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/25.jpg)
25
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
Code reading Testing & debugging
Group numbers
Number of faults detected
Average time relatedPer minute
Groups numbers
Number of faults
detected
Average time
relatedPer
minute
1 4 86 1 2 972 4 111 2 3 603 2 118 3 5 374 8 49 4 3 905 6 115 5 3 946 5 111 6 3 687 1 115 7 8 1088 3 108 8 1 819 3 94 9 4 9710 6 83 10 0 011 6 68 11 0 012 5 42 12 0 013 5 102 13 0 0•D
ata
colle
cted
by
subj
ect g
roup
s
![Page 26: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/26.jpg)
26
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
Code reading Testing & debugging
Average of total faults
detected
Average time
relatedPer
minute
Average of total faults
detected
Average time related
Per minute
4 92 min 2 56
min
• Results
![Page 27: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/27.jpg)
27
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
• Data Reduction
0
2
4
6
8
10
0 5 10 15
code readin
testing &debugging
Lost observations
![Page 28: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/28.jpg)
28
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
• Enhanced results
Code reading Testing & debugging
Average of total faults
detected
Average time
relatedPer
minute
Average of total faults
detected
Average time related
Per minute
4 92 min 4 79
min
![Page 29: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/29.jpg)
29
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
• Faults detection factor
Subjects CR T&D
Number of faults detected 58 32
Average number of
faults detected4 4
Average time for detected
faults92 79
![Page 30: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/30.jpg)
30
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
• Classification of collected data
Faults detecting using code testing & debugging technique
0
50
100
150
0 5 10
Faultsdetecting usingcode testing &debugging
Faults detecting using code reading technique
0
50
100
150
0 5 10
Faultsdetecting usingcode readingtechnique
Data shown in circle are faults detected after 70-80 of the experiments impact
![Page 31: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/31.jpg)
31
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
• Reclassification process of collected data
Classification type
Faults detected
using Code reading
Percentage
%Percentage out of 38 seeded faults
Faults detected using
Testing & debugging
Percentage
%Percentage out of 38
seeded faults
0 10 17 0.01 7 22 0.01
1 10 17 0.01 8 25 0.01
2 38 66 0.03 17 53 0.01
Total 58 100 32 100
![Page 32: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/32.jpg)
32
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
• Analysis of statistical approaches
• One sample analysis
Type 2 Faults. T & D
• Distribution using testing and debugging and their mean
![Page 33: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/33.jpg)
33
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
• Analysis of statistical approaches
• One sample analysis
Data distributions and mean
• Difference between CR and T & D techniques
![Page 34: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/34.jpg)
34
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
• Analysis of statistical approaches
• One sample analysis
t-test for the hypothesis for Type2 faults only
• t-Test: Two-Sample Assuming Equal Variances
Testing & debugging (TS) Code reading (CR)
Mean 2.428571429 3.454545455
Variance 2.952380952 3.472727273
Observations 7 11
T calculated -1.172108115t Critical two-tail 2.119904821p(T<=t) one-tail 0.129155674
![Page 35: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/35.jpg)
35
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
• Analysis of statistical approaches
• One sample analysis: Code reading
P-values (ANalysis Of VAriance table) < 0.05 => Statistically significantrelationship between Fault and Time at the 95% confidence level.R Square statistics: Variability in Fault 37.0629% ; Correlation coefficient = -0.608793 => Moderately strong relationship
Total fault 4 4 2 8 6 5 1 3 3 6 6 5 5
Average time 86 111 118 49 115 111 115 108 94 83 68 42 102
Time = 129.075 - 8.20641*Fault
• The equations of regression
x on y Faults = 8.63742 - 0.0451634*Timey on x
![Page 36: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/36.jpg)
36
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
• Analysis of statistical approaches
• One sample analysis: Testing & Debugging
P-values (ANOVA table) < 0.10 => Statistically ! significant relationship between Fault and Time at the 90% confidence level. R-Squared statistic: Variability in Time =1.58026% ; Correlation coefficient = 0.125708 => Week relationship
Time = 76.331 + 1.4069*Faults
• The equations of regression
x on y Faults = 2.642 + 0.0112322*Timey on x
Total faults 2 3 5 3 3 3 8 1 4
Average time 97 60 37 90 94 68 108 81 97
![Page 37: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/37.jpg)
37
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
CASE IITechnology Transfer:
Evaluation of Competing Software Technologies
What kind of technology?
Workflow Automation, WA, development suites & engines
![Page 38: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/38.jpg)
38
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
GoalObject WA Technologies.Purpose To identify the “best” tech.
Quality focus Organization HL goals.
Perspective ResearchContext URM2.DISP.ESEG &&
Org/Dept/A.Italy.Admin
![Page 39: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/39.jpg)
39
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
Case study: GoalObject WA Technologies.Purpose To identify the “best” tech.
(for certain stated HL goals).Quality focus Goal-driven &
Characteristic-processes-based comparison.
Perspective ResearchContext URM2.DISP.ESEG &&
Org/Dept/A.Italy.Admin
![Page 40: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/40.jpg)
40
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
Goal breakdown, entities, their attributes and properties.
Perception of attributes
Affecting entities
StakeholdersGoals
AttributeProperties
Lower Level Goals
Breakdown
Processes
CharacteristicProcesses
![Page 41: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/41.jpg)
41
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
Case study: GoalObject WA Technologies.Purpose To identify the “best” tech. Quality focus Adequacy to Reference
organization Goals and Characteristic-processes.
Perspective ResearchContext URM2.DISP.ESEG &&
Org/Dept/A.Italy.Admin
![Page 42: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/42.jpg)
42
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
1- Goals+
2- Questions+
3- Metrics+
Goal breakdown, entities, their attributes and properties.
![Page 43: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/43.jpg)
43
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
Stakeholders of a Measurement Model
Entity Model
Attribute Properties
Mapping Function
Measurement Model
relevantcharacteristics
constraints
measure
perception of the attribute
measured value
Entity to measure
abstraction
stakeholders
![Page 44: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/44.jpg)
44
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
The Experiment Process
RR
GG
EE MMAA
![Page 45: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/45.jpg)
45
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
MMIdentification
MM Creation &validation
MMAcceptance
MMAccreditation
MMApplication
organizationalgoals
draft MM valid MM
acceptedMM
feedbackon MM utility
non-accredited MM
non-accepted MM
MMs repository
new, reused, improved MMs
ESEG-UNIROMA2-MMLC
A Measurement Model Life Cycle for software
![Page 46: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/46.jpg)
46
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
MMIdentification
MM Creation &validation
MMAcceptance
MMAccreditation
MMApplication
organizationalgoals
draft MM valid MM
acceptedMM
feedbackon MM utility
non-accredited MM
non-accepted MM
MMs repository
new, reused, improved MMs
ESEG-UNIROMA2-MMLC
Acceptance is experiment too
![Page 47: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/47.jpg)
47
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
1. Identifying and characterizing quantitatively High Level Goals, HLG.
From goals to measurement models
![Page 48: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/48.jpg)
48
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
1. Identifying and characterizing quantitatively High Level Goals, HLG.
2. Analyzing HLG, deriving and tracing Technical Analysis Goals, TAG.
From goals to measurement models
![Page 49: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/49.jpg)
49
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
From goals to measurement models1. Identifying and characterizing
quantitatively High Level Goals, HLG.
2. Analyzing HLG, deriving and tracing Technical Analysis Goals, TAG.
3. Deriving further Constrains, HLC, from the organization’s vision.
![Page 50: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/50.jpg)
50
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
From goals to measurement models1. Identifying and characterizing
quantitatively High Level Goals, HLG.
2. Analyzing HLG, deriving and tracing Technical Analysis Goals, TAG.
3. Deriving further Constrains, HLC, from the organization’s vision.
4. Deriving Design Goals, DTG, from HLC, HLG e ATG.
![Page 51: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/51.jpg)
51
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
2. Analyzing HLG, deriving and tracing Technical Analysis Goals, TAG.
3. Deriving further Constrains, HLC, from the organization’s vision.
4. Deriving Design Technology Goals, DTG, from HLC, HLG e ATG.
5. Giving DTG a structure.
From goals to measurement models
![Page 52: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/52.jpg)
52
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
3. Deriving further Constrains, HLC, from the organization’s vision.
4. Deriving Design Technology Goals, DTG, from HLC, HLG e ATG.
5. Giving DTG a structure.
6. Involving stakeholders, deriving metric attribute from SDTG and weighting such attributes level after level.
From goals to measurement models
![Page 53: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/53.jpg)
53
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
5. Giving DTG a structure.
6. Involving stakeholders and weighting DTG.
7. Developing measurement models, TTMM, to filter, evaluate, and eventually select the control & treatment technologies.
From goals to measurement models
![Page 54: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/54.jpg)
54
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
Using measurement models, TTMM, to filter, evaluate, and eventually select control & treatment technologies.
From goals to measurement models
MModel Specialization
General models (Costs, Benefits, etc.)
Screening models
AAWWAARREE TTEECCHHNNOOLLOOGGYY SSEETT
EEVVOOKKEEDD TTEECCHHNNOOLLOOGGYY SSEETT
CCOONNTTRROOLL && TTRREEAATTMMEENNTTSS
GGooaallss
IImmppoorrtteeddKKnnoowwlleeddggee
IInnffrraa--CCoorrppoorraattiioonnKKnnoowwlleeddggeeEExxcchhaannggee
MeasurementModel-BaseExperiment Design
EEmmppiirriiccaallIInnvveessttiiggaattiioonn
MMooddeellss
ExperimentationKnowledge Base
![Page 55: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/55.jpg)
55
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
5. Giving DTG a structure.
6. Involving stakeholders and weighting DTG.
7. [Developing, and] using measurement models, TTMM, to filter and evaluate the technology Awareness set.
8. Verifying and Accepting TTMMs&& Redesigning original TT case study or experiment in order to reuse the incoming new empirical evidence.
From goals to measurement models
![Page 56: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/56.jpg)
56
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
A Process Model for Experimenting with WAT
URM2URM2--MSSMSS• Multiple developments of EUO’s Synthetic
Processes
• Single development of a “Laboratory Project”, i.e., a scaled−down realistic case study: less than a pilot project but much more than a toy project.
• Single development of a Pilot Project that is an observed field project.
NB: One more MM, EP (at least!)
![Page 57: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/57.jpg)
57
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
Selected Data from an ESEG DISP URM2 Experiment
![Page 58: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/58.jpg)
58
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
The TT Experiment Process: Design Allocation of Groups for Parallel
Development of Synthetic Processes
Phase Step ControlGroup
Activity TreatmentGroup
Activity SyntheticProcess
0 0.1 A, B, C 0.1.1 A, B, C 0.1.2Training
0.2 A, B 0.2.1 A, B 0.2.2 SPT
Training1 1.1 B 1.1.1 A 1.1.2 SP1
2 2.1 A 2.1.1 B 2.1.2 SP2
3 3.1 A 3.1.1 - - SP3
- - B 3.1.2 SP4
![Page 59: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/59.jpg)
59
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
Phase Step ControlGroup
Activity TreatmentGroup
Activity SyntheticProcess
0 0.1 A, B, C 0.1.1 A, B, C 0.1.2 Common Training
0.2 A, B 0.2.1 A, B 0.2.2 SPT
C 0.2.3 C 0.2.4 Training1 1.1 B 1.1.1 A 1.1.2 SP1
C 1.1.3.n C 1.1.4.n SP3
2 2.1 A 2.1.1 B 2.1.2 SP2
C 2.1.3.n C 2.1.4.n SP4
3 3.1 A 3.1.1 - - SP3
- - B 3.1.2 SP4
C 3.1.3.n C 3.1.4.n SP2
4 4.1 C 4.1.1.n C 4.1.2.n SP1
The TT Experiment Process: Design Allocation of a further group for Virtual Parallel Development of
Synthetic Processes
![Page 60: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/60.jpg)
60
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
The Experiment Process: Characterization of the lab. projects,
and the synthetic processesModeling Entity No. before WA No. after WA
SYSTEMS 1 1SUB-SYSTEMS 10 10ACTORS 25 27• People 21 22
• Primary 8 8
• Secondary 13 14• Systems 4 5CLASSES 65 70ASSOCIATIONS 92 100• Inheritance 9 10• Others 83 90USE−CASES (full courses) 24 40• Pilot Project 24 40• Synthetic Processes 5 5
![Page 61: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/61.jpg)
61
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
The Experiment Process: Involved Roles & Effort
Role Effort(Man−months)
Project Manager 0.8Business Process Analyst 3.5Customer organization 0.4Reference organization 0.3WA Developer and Measurer 5.6WA Application Verifier 1.4Inspector 0.9Observer & Editor 1.4
![Page 62: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/62.jpg)
62
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
The Experimental Process: a Synthetic Process
![Page 63: Cantone ICEIS 2002 · Title: Microsoft PowerPoint - Cantone ICEIS 2002.ppt Created Date: 10/27/2006 6:40:14 PM](https://reader036.fdocuments.in/reader036/viewer/2022062607/605330d11e33ff5d4615d823/html5/thumbnails/63.jpg)
63
ICEIS 2002, Ciudad Real, April 4th © UniRoma2 – DISP – ESEG - Giovanni Cantone
ConclusionsCase study and experiment processes may lead the experimenter to start nested experiment processes.
This occurred when we were involved with testing hypothesis that related to high impact SW technology, which was still broadly modeled.