• Overview of MCDA– General definition– MCDM process– MCDA methods

• Evaluation of WBA– Quality Attribute Relationships– Aggregation by Choquet Integral– Implementation– Case study and results

Outlines

• Aims to give the decision-maker some tools in order to enable him to advance in solving a decision problem where several – often contradictory-points of view must be taken into account.

What is MCDA?

• Highly structured, disciplined and formal approach to decision making

• evaluating the alternatives in the given set A against the set C of criteria

• Aggregating the individual evaluations to produce global evaluation

• Could be used for selection the best possible alternatives or for ranking the alternatives

What is MCDM?

MCDM Process

Set of Alternatives Set of Criteria

Weights wi /Importance of

Criteria

Overall worth of an alternative Ai

Aggregation Measure

C1, C2,………Cn

A1 x11……..………x1n

A2 x21……..………x2n

.

.

Am xn1……..………xmn

Evaluation of MCDA methods

• Criteria – interdependence, completeness, non-linear preferences

• Weights – transparency of process, type of weights, meaning

• Solution finding procedure – ranking, option

• Project constraints – cost, time

Evaluation of MCDA methods

• Structure of problem solving process – stakeholder participant, tool for learning transparency, actors communication

• Data Situation – Type of data - qualitative or quantitative– Risk/uncertainties – probabilities, thresholds, fuzzy

numbers, sensitive analysis– Data processing amount– Non-substitutability

Evaluation of WBA• Quality of web site is hard to evaluate

– Consists of multiple criteria to be measured• Simple weighted average cannot be used to

summaries the various quality measurements into a single score.

• Inability to account for dependency among the quality criterion.

• Tend to construct independent criteria, or criteria that are supposed to be so– Causing some bias effect in evaluation

Single criteria

• usability aspects(Collins, 1996; Stefani & Xenos, 2001; Hassan & Li, 2005),

• content and structure (Bauer & Scharl, 2000).

• accessibility (Vigo et al., 2007)

WBA Evaluation Approaches

Multi-criteria

• WEBQEM (Olsina et al., 1999)

• EWAM (Schubert & Selz, 1998)

• WebQual (Barnes and Vidgen, 2002)

• WAI (Miranda et al., 2006)

• FQT4Web (Davoli et al., 2005)

WBA Evaluation Approaches

Stated or implied needs

QualityRequirement

Definition

Software Developmen

t

ISO 9126 & other technical info

Quality requirement specification

Metric Selection

Rating level

definition

Assessment criteria definition

Measurement

Rating

Assessment

Products

Measured value

Rated value Result

(acceptable or unacceptable)

Requirement definitionManagerial

requirement

Preparation

Evaluation

ISO/IEC 9126 Evaluation Process

Quality Model

Indicators, scales and preferred values

QU

AL

ITY

CH

AR

AC

TE

RIS

TIC

S

APPLICATIONDOMAIN

e-commercee-learning

e-educatione-government

etc.

SU

B C

HA

RA

CT

ER

IST

ICS

Functionality Reliability Usability Efficiency PortabilityMaintainability

suitabilityaccuracyinteroperabilitysecuritytraceabilityfunctionalitycompliance

maturityfault tolerancerecoverabilityavailabilitydegradabilityreliabilitycompliance

understandabilitylearnabilityoperabilityattractivenessexplicitenesscustomisabilityclarityhelpfulnessuser-friendlinessusability compliance

time behaviourresourceutilisationefficiencycompliance

Analysabilitychangeabilitystabilitytestabilitymanageabilityreusabilitymaintainabilitycompliance

Adaptabilityinstallabilitycoexistencereplaceabilityportabilitycompliance

Quality Attributes for WBA

• Define software product qualities as a hierarchy of factors, criteria and metrics.

• Quality factor represents behavioral characteristics of the system

• Quality criterion is an attribute of a quality factor that is related to software production and design

• Quality metrics is a measure that captures some aspect of a quality criterion.

Factor A

Criteria a1,

weight 4

Criteria a2,

weight 1

Criteria a3,

weight 2

Overall Quality Score

Factor B Factor C

         

 

Factor A is split up into three criteria a1, a2, and a3.

Criteria a1 with the weight 4

is considered four times as important as criteria a2 and

twice as important as criteria a3.

Similarly, we can set different weight for each factor to indicate its importance.

Definition of Quality Attributes

Name Description

Functionality 

The capability of the Web site to provide functions and properties which meet stated and implied needs when the site is used under specified conditions

Usability 

The capability of the Web site to be understood, learned and liked by the user, when used under specified conditions

Reliability 

The capability of the Web site to maintain a specified level of performance when used under specified conditions.

Efficiency 

The capability of the site to provide appropriate performance, relative to the amount of resource used, under stated conditions

Maintainability The capability of the site to be modified. Modifications may include corrections, improvements or adaptation of the site to changes in environments, and in requirements and functional specifications

Portability 

The capability of the site to be transferred from one environment to another

Quality Attributes Relationships

Three types of relationships• Positive, i.e. a good value of one attribute result in a

good value of the other (synergistic goals).– Relationships definitions: If characteristics A is enhanced,

then characteristics B is likely to be enhanced (+)

• Negative, i.e. a good value of one attribute result in a bad value of the other (conflicting goals).– Relationships definitions: If characteristics A is enhanced,

then characteristics B is likely to be degraded (-)

• Independent, i.e. the attributes do not affect each other.– Relationships definitions: If characteristics A is enhanced,

then characteristics B is unlikely to be affected (0)

Interrelationships between quality factors (Perry, 1987)

Relationship Chart (Gillies, 1997)

Techniques to explore the relationships

Ref Attributes Purpose Techniques used  

[8, 9]

Correctness, ReliabilityIntegrity, UsabilityEfficiency, MaintainabilityTestability, FlexibilityPortability. ReusabilityInteroperability

To study the relations of different quality goals attribute in developing software

Survey -questionnaire

[10]PerformanceAdaptabilityMaintainability

To address the importance of design decision made during software development

Case Study - Interview

[11]UsabilityTime to marketReliability, UsabilityCorrectness, Portability

To increase the understanding of software quality attributes and their relations

Research Literature and Survey –structured interview

[12] Quality attributes in 3 different perspectives: management, developer and user perspective

To merge different view and discuss the relationships between the quality attributes

Discussion (meeting and offline discussion)

Quality Attributes Relationships for WBA

• method of combining several numerical values into a single one, so that the result of aggregation takes into account in a given manner all the individual values

What is Aggregation?

• use simple weighted average approach• methods are not transparent• assume independency• the choice of summarization method somehow

should depend on the certain parameters– E.g. the kind of importance parameters (weights) and

the type of dependency and interaction

• the definition of the quality factors and their relationships must be clearly specified

Aggregation issues

Common aggregation operators

• Quasi-arithmetic means (arithmetic, geometric, harmonic, etc.)– Not stable under linear transformation and

consider criteria as non interacting• Median

– Typical ordinal operator – defined the middle value of the ordered list

• Weighted minimum and maximum– Possible to increase one of the weights without

having any change in the result• Ordered weighted averaging operators

– Can express vague quantifiers

23

Properties of an aggregation operator

mathematical properties– Properties of extreme values– Idempotence– Continuity– Monotonicity– Commutativity– Decomposability– Stability under the same positive

linear transformation

behavioural properties– express the decisional behavior,

interaction between criteria, interpretability of the parameters and weights on the arguments

Properties of an aggregation operator

Aggregation by fuzzy integral

• Different methods have been developed according to– type of information to be aggregated

and – the properties have to be satisfied.

26

Fuzzy measures and integral

Definition 1: A fuzzy measure on the set X of criteria is a set function : Ƥ (X) [0,1], satisfying the following axioms

i. ()=0, (X)=1.ii. A B X implies (A) (B)

(A) represent the weight of importance of the set of criteria A.

Additive : if (AB) = (A) + (B); A B=Superadditive: if (AB) (A) + (B); A B=Subadditive if (AB) (A) + (B); A B=If a fuzzy measure is additive, then it suffices to define n

coefficients (weights) ({ I}), … ({ n})27

Choquet integral

Definition 2: Let be a fuzzy measure on X. The choquet integral of a function ƒ : (X) [0,1] with respect to is defined by

28

C (f(x1),…. f(xn)):= (f(x(i)) - f(x(i-1))) (A(i) )

ƒ ((0)) = 0

n

i = 1

•Fuzzy integral model does not need to assume independency•Fuzzy integral of ƒ with respect to gives the overall evaluation of an alternative

Importance and interaction of criteria

• Problem of evaluation of student with respect to three subjects: mathematics (M), Physics (P) and literature (L).

• By weighted sum (3 , 3, 2) result:

29

Solved by fuzzy measure and the choquet integral

1. Scientific subjects are more important than literature;

({M}) = ({P}) =0.45; ({L}) = 0.32. M and P are redundant,

({M, P}) = 0.5 < 0.45 + 0.453. Students equally good at scientific subjects and

literature, ({L, M}) = 0.9 > 0.45 + 0.3

({L, P}) = 0.9 > 0.45 + 0.34. ()=0, ({M, P, L})=1

30

Result by applying fuzzy measure:

* The initial ratio of weight (3, 3, 2) is kept unchanged

31

Complexity of the model• Number of coefficients grows exponentially with the

number of criteria to be aggregated. • 3 approaches (to reduce the number of

coefficients)1. Identification based on semantics

– Importance of criteria– Interaction between criteria– Symmetric criteria– Veto effects

2. Identification based learning data– Minimization of squared error– Constraint satisfaction

3. Combining semantics and learning 32

Proposed solution

• Apply 2-additive Choquet integral• provide the information about the

relationships among criteria (redundancy or support among criteria) and the preference among alternatives

• Derive fuzzy measures by constraint satisfaction

Explore relationships

• Techniques to explore how the different attributes are related to each other:– Experience Based Approach – Mathematical Modeling – Statistical Technique (Correlation Analysis)

• measures the strength of a linear relationship among different quality factors

• The main result of a correlation is called the correlation coefficient (r)

Correlation Result

1. Definition of the initial preferences.

2. Convert into Choquet integral form

3. Identify threshold values.

4. If solution exists, calculate the Choquet integral, Shapley value and Interaction indices

Implementation of Choquet Integral

A partial weak order A over A (ranking of the webs), A partial weak order N over N (ranking of the importance of the quality factor),

Quantitative intuitions about the relative importance of some quality factor

A partial weak order P over the set of pairs of quality factor (ranking of interactions),

Intuitions about the type and the magnitude of the interaction between some quality factor,

The behavior of some quality factor as veto or favor, Etc.

Define preference thresholds

P r e f e r e n c e s C h o q u e t I n t e g r a l 'xx A CxuCxuC ))'(())(( R a n k in g o f

a l t e r n a t iv e s '~ xx A CC xuCxuC ))'(())((

ji N Shji )()( R a n k in g o f c r i t e r ia ( w e ig h t s )

ji N~ ShSh ji )()(

klij P IklIijI )()( R a n k in g o f p a i r s o f c r i t e r ia ( in t e r a c t io n s )

klij P~ II klIijI )()(

R a n g e o f in t e r a c t io n s

bijIa )( , 1,1, ba S ig n o f s o m e in t e r a c t io n s

C o m p le m e n t a r y o r R e d u n d a n t

mijI )( o r

mijI )( ; ]1,0[m

Convert into Choquet integral form

• Three preference thresholds C, Sh & I have to be determined before the aggregation take part.

• Range of : 0 to 1• no rule to fix the , we need to compare

the solutions obtain with different value of .

• Once the solution exist, Choquet integral will be calculated

Define preference thresholds

Calculate the Choquet integral

n

i Xjijiijii xxaxaK

1 },{

)(

For 2-additive fuzzy measure, we have for any KX:

w i t h 1ix i f i K , 0ix o t h e r w i s e . W e d e d u c e t h a t ii a f o r

a l l i , a n d ijjiijjiij aaaa .

kKixk

KiK

n

kkiv

||,\

1

0

)(

!

!)!1(

n

kknk

with

Calculate the Shapley value

n = total number of criteria, k = number of elements in a sub-set

Shapley index can be interpreted as a kind of average value of the contribution of element i, individual criteria, alone in all coalitions.

Summation of these Shapley values for a given set of elements would represent the importance of the complete set

Calculate the Interaction Index

kKjixk

KjKiKijK

n

kkijI

||},,\{

2

0

)(

With

)1(21

)!1(

!)!2(

n

k

nn

kknk

The interaction index Iij can be interpreted as a kind of average value of

the added value given by putting i and j together, all coalitions being considered. When Iij is positive (resp. negative), then the interaction is

said to be positive (resp. negative).

Case Study

• Perform on 3 types of WBA– Academic– E-commerce– Museum

• Four quality factor were evaluated– Usability,Functionality, Reliability, Efficiency

• Each has different preference, importance and interaction

Table 1. Academic websites data

Univ Websites Usability (U) Functionality(F) Reliability(R) Efficiency(E)

A1 76.18 61.84 60.40 69.09

A2 51.01 50.39 87.62 52.03

A3 80.08 48.49 90.86 76.11

A4 57.71 38.99 88.70 53.12

A5 71.93 82.04 83.05 85.99

A6 60.94 71.12 63.61 69.47

Result for academic website

Table 2. Relative importances for each quality attribute

U F R E Relative importance

(Weight) 0.3 0.3 0.2 0.2

Table 3. Interaction between quality attribute

U F R

U

F -

R + +

E - - +

T a b l e 4 . I n t e r a c t i o n p r e o r d e r c o n s t r a i n t m a t r i x

U F R

U

F IUFI )(1

R 1)( URII 1)( FRII

E IUEI )(1 IFEI )(1 1)( REII

E a c h r o w c o r r e s p o n d i n g t o a c o n s t r a i n t o f f o r m bijIa )( , 1,1, ba .

Table 5. The Möbius values

LP MV {} 0.000000 0.000000 {U} 0.366704 0.321697 {F} 0.326805 0.285376 {R} 0.000001 0.094367 {E} 0.277765 0.298559 {U, F} -0.226804 -0.100000 {U, R} 0.137865 0.100000 {U, E} -0.100000 -0.100000 {F, R} 0.217664 0.100000 {F, E} -0.100000 -0.100000 {R, E} 0.100000 0.100000

Threshold C= 1, Sh = 0.1, I =0.1,

Table 7. The interaction indices for LP

U F R

U

F -0.2268038

R 0.1378647 0.2176637

E -0.1000000 -0.1000000 0.1000000

Table 8. The interaction indices for MV

U F R U F -0.1000000 R 0.1000000 0.1000000 E -0.1000000 -0.1000000 0.1000000

Table 6. The Shapley values

U F R E LP 0.2722348 0.2722348 0.2277652 0.2277652

MV 0.2716970 0.2353761 0.2443674 0.2485595

Table 9. Overall Scores

Global evaluation obtained by

Univ Websites WA LP MV

A1 67.304 67.73138 67.32480

A2 58.350 51.26041 54.75639

A3 71.965 72.10125 74.05597

A4 57.374 52.36041 55.79382

A5 79.999 81.44097 81.17426

A6 66.234 66.63138 66.32480

LP MV Möbius values Very extreme value

and does not necessarily lead to a unique solution

Lead to unique solution

Shapley value Sometimes incompatible with the initial relative importance

Mostly compatible with the initial relative importance

Interaction indices Satisfy the constraints Satisfy the constraints

Global evaluation Leads to more dispersed values

Closer to the simple weighted arithmetic mean

Summary(1)

Aggregation approaches

Arithmetic Mean (AM)

Weighted Average

(WA)

Logic Scoring

Preference (LSP)

Choquet Integral (CI)

Importance / / / Veto / /

Favour / / Interaction /

Additive / / / / Non-additive / /

Summary(2)

Rank Score Rank Score Rank Score Rank Score Rank Score Rank

A1 3 66.88 3 67.3 3 66.91 3 67.32 3 67.08 3

A2 6 60.26 5 58.35 5 56.55 5 54.76 6 60.16 5

A3 2 73.89 2 71.97 2 69.61 2 74.06 2 73.18 2

A4 5 59.63 6 57.37 6 54.46 6 55.8 5 59.16 6

A5 1 80.75 1 80 1 79.76 1 81.17 1 80.66 1

A6 4 66.29 4 66.23 4 66.05 4 66.32 4 66.09 4

MIN-MAX

STD DEV

Univ Websites

Global evaluation obtained by

AM WA LSPChoquet Integral

(Interaction)Choquet Integral (No Interaction)

59.63-80.75

8.141

57.37-80

8.507

54.46-79.76

9.214

54.76-81.17

10.250

59.16-80.66

8.133

Comparison with other approaches

Conclusion• Aggregation by Choquet integral can be

alternated if there is interaction exist between quality factors.

• The proposed approach can be applied for non-interactive criteria as well. If there is no interaction between the criteria, then the fuzzy measure will be additive measures.

• Results show that the global evaluation obtained is compatible with the weighted average method.

Future works

• The evaluation of WBA which cater the dynamic changes of the quality factors.– Behavior (Preferences,

importance,interaction, etc.) can be change continuously.

• Investigate more than 2 quality attribute interactions