Guidelines for the application of Data Envelopment Analysis

to assess evolving software

Alexander Chatzigeorgiou

University of Macedonia

Thessaloniki, Greece

6th International Workshop on Foundations and Techniques for OSS Certification

Efficiency

1,000,000 €

25,000 €

Efficiency =profit

#employees

Analogy between companies and software

v1.0

v1.2

v2.0

vx.y

. . .

size properties

size properties

size properties

size properties

metrics

metrics

metrics

metrics

Software Context

CBO

LCOM

WMC

DIT

. . .

Software Context (2)

version i

version i+1

Complexity = 5

Complexity = 5

Inputs

The ideal input variable would be a measure of functionality (e.g. FPs)

S o f t w a r e A n a l y s i s a n d D e s i g n

AnalysisPreliminary

DesignDesign...

Functional Requirements ...

software metrics...

(most)

method/attributes

Size metrics can be used as a substitute

How DEA works…

v1

v2

v3

v4

v5

v6

v7

v8

v.1 v.2 v.3 v.4 v.5 v.6 v.7 v.8

NOO (x) 130 150 200 230 300 320 400 450

fan-in (y1) 10.6 11.2 13 13.6 15 15.1 15.8 14

cohesion

(y2)9 12.5 10.5 11 12 10 9 13

v.1 v.2 v.3 v.4 v.5 v.6 v.7 v.8

NOO (x) 1 1 1 1 1 1 1 1

fan-in (y1) 0.08 0.07 0.07 0.06 0.05 0.05 0.04 0.03

cohesion

(y2)0.07 0.08 0.05 0.05 0.04 0.03 0.02 0.03

How DEA works…(2)

0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10

fan-in coupling / NOO

coh

esio

n /

NO

O

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0.10

0

v.2

v.1

v.3v.4

v.5

v.6v.8

v.7

Efficient Frontier

A’

A

Efficiency =0A

0A’v.4

Added value of DEA

01002003004005006007008009001000

8

9

10

11

12

13

14

15

16

0.9.

00.

9.1

0.9.

20.

9.3

0.9.

40.

9.5

0.9.

60.

9.7

0.9.

80.

9.9

0.9.

100.

9.11

0.9.

120.

9.13

0.9.

140.

9.15

0.9.

160.

9.17

0.9.

180.

9.19

0.9.

200.

9.21

fan-in fan-out cohesion NOC

NO

C

fan

-in

, fan

-ou

t, c

oh

esio

n

Versions

improving qualitydegrading quality

improving quality

Added value of DEA (2)

01002003004005006007008009001000

8

9

10

11

12

13

14

15

16

0.9.

00.

9.1

0.9.

20.

9.3

0.9.

40.

9.5

0.9.

60.

9.7

0.9.

80.

9.9

0.9.

100.

9.11

0.9.

120.

9.13

0.9.

140.

9.15

0.9.

160.

9.17

0.9.

180.

9.19

0.9.

200.

9.21

fan-in fan-out cohesion NOC

NO

C

fan

-in

, fan

-ou

t, c

oh

esio

n

Versions

DEA offers a unified view by combining selected design metrics and by normalizing over inputs

DEA is particularly useful when the trends of individual metrics are highly convoluted

Case Study 2

Project: JDeodorant (20 revisions, 4 years of development)Evaluation of: changeability (CBO, LCOM*, DIT **)

** Samoladas I, Gousios G, Spinellis D, Stamelos I. The SQO-OSS quality model: Measurement based open source software evaluation. 4th International Conference on Open Source Systems (OSS'2008).

0

10

20

30

40

50

60

70

80

0

200

400

600

800

1000

1200

1400

1600

1800

2000

1 54 84 152 194 208 223 227 232 244 250 261 266 275 285 295 325 340 343 360

CBO

(10x

), LC

OM

*, D

IT (

10x)

NO

O, N

OA

CBO (10x)

NOO

LCOM*

NOADIT (10x)

Revisions

Case Study 2

is the evolution of DEA scores depicting the truth ?

• If a metrics combination is successful in capturing several aspects associated with the same property (e.g. changeability),

• then it should be demonstrated that the combined value is highly correlated to the desired property.

• However, quality sub-characteristics are not directly quantifiable; that is why a number of metrics are required to assess them.

• To confront this problem, we employed secondary measures, which are related to some extent to the property of interest.

Case Study 2 - JDeodorant

is the evolution of DEA scores depicting the truth ?

0

5

10

15

20

25

30

35

40

Add

ed/M

odifi

ed LO

C to

fix

a si

ngle

bug

Revisions5 14 26 4250

53

51

59 79 94

96

98

99

103 129

133

140

142

163

165

170 190

195

204

209

210 235 248 272259 302 326 369

366

364

Case Study 3

Project: Industrial Telecom System (12 versions, 30 years of development)Evaluation of: quality in terms of the number of faults

0

10

20

30

40

50

60

70

80

90

100

0

50

100

150

200

250

300

1 2 3 4 5 6 7 8 9 10 11 12

Faul

ts

Inte

rfac

es

Interfaces

Programming Faults

Design Faults

Versions

Case Study 3

is the evolution of DEA scores depicting the truth ?

• there is no single property that can be related to the number of design and programming faults,

• the engineers suggested to assess the validity of the extracted scores by means of the number of faults having a priority beyond a certain level (trouble reports with priority A or B).

Case Study 3 - Telecom

is the evolution of DEA scores depicting the truth ?

0

2

4

6

8

10

12

14

1 2 3 4 5 6 7 8 9 10 11 12Faul

ts w

ith

high

est p

rior

ity

leve

l

Versions

Super-efficiency

0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10

fan-in coupling / NOO

coh

esio

n /

NO

O

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0.10

0

v.2

v.1

v.3v.4

v.5

v.6v.8

v.7

Efficient Frontier

A’

A

Efficiencyv.2 > 1

Super-efficiency (2)

0

0.2

0.4

0.6

0.8

1

1.2

1 2 3 4 5 6 7 8 9 10

DEA

Effi

cien

cy S

core

Versions

Telecom

When the basic DEA model leads to multiple versions with score equal to one, the super-efficiency DEA model should be used instead.

Selection of outputs

DE

A E

ffic

ien

cy S

core

versions

fan-in, fan-out, cohesion

Output variables can have varying impact on the extracted DEA scores. When not sure about the necessity to consider a metric, experimentation could highlight whether its impact is significant or not.

No Inputs

Inputs can be neglected by providing a constant input for all DMUs

0

0.2

0.4

0.6

0.8

1

1.20.

9.0

0.9.

10.

9.2

0.9.

30.

9.4

0.9.

50.

9.6

0.9.

70.

9.8

0.9.

90.

9.10

0.9.

110.

9.12

0.9.

130.

9.14

0.9.

150.

9.16

0.9.

170.

9.18

0.9.

190.

9.20

0.9.

21

Effici

ency

Sco

re

Versions

considering input

excluding input

Conclusions

• Main benefits from the application of DEA:

• the ability to provide a unified view of several metrics

• the normalization of the derived scores over size properties of the examined systems

• Eventually, DEA leads to an efficiency score for each examined project enabling the ranking of various versions and the analysis of the evolution.

Thank you for your attention

6th International Workshop on Foundations and Techniques for OSS Certification

How DEA works (math…)

The efficiency score of each version (DMU), is given by maximizing the weighted ratio of all outputs over all inputs, subject to the constraints that the efficiency scores for each DMU must be less than or equal to unity:

4.4.

4.24.24.14.1

,,max

4.4.24.1 vv

vvvv

vuu xv

yuyu

vvv

subject to

14.

24.214.1

iv

iviv

xv

yuyu}8.,...,1.{ vvi

0,, 4.4.24.1 vvv vuu