! Macbeth Ijitdm 2012

MACBETH

CARLOS A. BANA E COSTA

Centre for Management Studies of Instituto Superior T�ecnicoTechnical University of Lisbon, Av. Rovisco Pais

1049-001 Lisbon, Portugal

[email protected]

JEAN-MARIE DE CORTE* and JEAN-CLAUDE VANSNICK†

Centre de Recherche Warocqu�e

Universit�e de Mons, Place du Parc, 207000 Mons, Belgium

*[email protected]†[email protected]

This paper presents an up-to-date comprehensive overview of the MACBETH approach to

multicriteria decision-aid. It requires only qualitative judgements about di®erences of attrac-tiveness to help a decision maker, or a decision-advisory group, quantify the relative value of

options. The approach, based on the additive value model, aims to support interactive learning

about evaluation problems and the elaboration of recommendations to prioritize and select

options in individual or group decision making processes. A case study based on a real-worldapplication of MACBETH for multicriteria value measurement of IT solutions is presented. It

shows how the M-MACBETH decision support system can be used in practice to construct an

additive evaluation model. The paper addresses key issues related to structuring the model,

building value scales, weighting criteria and sensitivity and robustness analyzes. Reference isalso made to applications of MACBETH reported in the scienti¯c literature.

Keywords: MACBETH; qualitative value judgements; multicriteria analysis.

1. Introduction

MACBETH (Measuring Attractiveness by a Category-Based Evaluation Tech-

nique)13 is a decision-aid approach to multicriteria value measurement.63,64,90 The

goal behind its conceptualization is to allow measurement of the attractiveness or

value of options through a non-numerical pairwise comparison questioning mode,

which is based on seven qualitative categories of di®erence in attractiveness: is there

no di®erence (indi®erence), or is the di®erence very weak, weak, moderate, strong,

very strong, or extreme? The key distinction from numerical value-measurement

procedures, such as the simple multi-attribute rating technique, or SMART

approach,47,48 is that MACBETH uses only such qualitative judgements of di®erence

in attractiveness in order to generate, by mathematical programming, value scores

International Journal of Information Technology & Decision Making

Vol. 11, No. 2 (2012) 359�387°c World Scienti¯c Publishing CompanyDOI: 10.1142/S0219622012400068

359

http://dx.doi.org/10.1142/S0219622012400068

for options and weights for criteria. Since the early 1990s, the initial mathematical

formulation was revised and a ¯rst software application implementing MACBETH

was developed.27 This software was later replaced by the current M-MACBETH

decision support system (see Ref. 2 and www.m-macbeth.com) that respects the

theoretical foundations already established,12 allows modeling challenges arising

from practice to be addressed (for instance, it allows for hesitation in choosing

between two or more consecutive categories, except indi®erence) and supports the

\on-the-spot" creation of a computer-based additive value model and sensitivity and

robustness analyzes of the model outputs.

The current paper presents an up-to-date comprehensive overview of MACBETH

that is consistent with the original ideas presented by Bana e Costa and Vansnick

(see Ref. 28). As they observed, decision making in private and public organizations

is above all a human activity in which value judgements of managers and other

actors about the desirability or attractiveness of organizational decision opportu-

nities and alternative courses of action play a crucial role. An important research

challenge is therefore the integration of information technology and human decisions

by means of the design and use of decision-support techniques and systems to answer

the key question of how to elicit and numerically represent value judgements. It is

also an opportunity to avoid the risk of decision science becoming simply a closed

branch of pure mathematics. The existence of a theoretical base is a desirable but not

su±cient condition for the legitimization of a decision-aid theory; it must also be

subjected to practical validation. Moreover, visual appeal and user-friendly \black-

box" software,71 often based on theoretically weak technical procedures, are not the

right answer to the above question: on the contrary, they are a trap for managers and

decision makers in general. Decision-aiding tools need to be simultaneously seman-

tically meaningful, practically operational (user-friendly) and theoretically well

founded.

The MACBETH decision-support approach and software make a contribution in

this direction. Indeed, the M-MACBETH decision support system was designed to be

used by a consultant (facilitator or decision analyst) at di®erent multicriteria

modeling stages (Fig. 1), following the constructivist principles of process consul-

tation.88 This is a socio-technical process that combines the technical elements of

MACBETH with the social aspects of decision conferencing.79,80

Section 2 is devoted to the presentation of the MACBETH questioning and

technical procedure for value elicitation. Section 3 addresses a case study based

on a real-world application of MACBETH for multicriteria value measurement of

IT solutions. It shows how M-MACBETH can be used to construct an additive

evaluation model based on qualitative value judgements of di®erence in attractive-

ness. Key issues of interactive value modeling are discussed. First, how to de¯ne

and operationalize evaluation criteria (the structuring phase, see Sec. 3.2). Second,

how to construct interval value scales which enable one to score the options

on each criterion (the intra-criteria modeling component of the evaluation

phase, see Sec. 3.4). Third, and before or after scoring, how to weight the criteria

360 C. A. Bana e Costa, J. M. De Corte & J.-C. Vansnick

(the inter-criteria modeling component). Weights can be derived by applying the

MACBETH procedure to a set of reference, hypothetical, options (see Sec. 3.3).

Scoring the options on an interval scale within each criterion is important because it

permits one to meaningfully take a weighted average of each option's scores on the

criteria. These overall scores provide answers to the question of how to measure the

relative attractiveness of the options across all the criteria (the synthesis modeling

component). Finally, how to validate the model in the face of uncertainty by using

the software to perform, throughout the modeling process, sensitivity and robustness

analyzes of the model outputs (see Sec. 3.5). As observed by Phillips and Bana e

Costa (see Ref. 80, p. 55), \extensive sensitivity analyzes show that many dis-

agreements or uncertainties in the data make no di®erence to the overall results, and

gradually a sense of common purpose emerges". The paper concludes with some

re°ections in Sec. 4 and includes references to selected real-world applications of

MACBETH reported in the scienti¯c literature.

2. The MACBETH Procedure

2.1. Ordinal and cardinal value information

Let X be a ¯nite set of elements �� di®erent options or performance levels under

evaluation. Ordinal measurement of the attractiveness (or desirability) of the

elements x of X consists in associating each x with a numerical score �� a real

number vðxÞ �� that satis¯es the ordinal measurement conditions (1) (the condition

of strict preference) and (2) (the condition of indi®erence):

8 x; y 2 X : ½x is more attractive than y ðxPyÞ , vðxÞ > vðyÞ�; ð1Þ8 x; y 2 X : ½x is as attractive as y ðxIyÞ , vðxÞ ¼ vðyÞ�: ð2Þ

Such a numerical scale v : X ! < : x ! vðxÞ can be constructed by asking for

ordinal value information from an individual or group evaluator (DM). That is, the

DM is asked for a ranking of the elements of X in order of decreasing attractiveness

Fig. 1. Phases of the MACBETH decision-aiding process.

MACBETH 361

(with the possibility of indi®erence). If this is done for each one of several criteria,

Arrow's impossibility theorem1 shows that, unfortunately, aggregating individual

rankings into an overall ranking always implies some form of arbitrariness.

Cardinal measurement of the attractiveness of the elements x of X consists in

associating each x with a numerical score �� a real number vðxÞ �� that satis¯es not

only the ordinal conditions (1) and (2) but also the additional condition (3):

8w; x; y; z 2 X with x more attractive than y and w more attractive than z :

the ratio ½vðxÞ � vðyÞ�=½vðwÞ � vðzÞ� measures the difference in attractiveness

between x and y when the difference in attractiveness between w and z is

taken as the measurement unit: ð3ÞSuch a numerical scale v : X ! < : x ! vðxÞ can be constructed by positioning

the elements of X on a vertical axis so that:

(1) 8 x; y 2 X : x is positioned above y if and only if x is more attractive than y

(ordinal value information)

(2) the relative distances between the elements de X on the vertical axis re°ect the

relative di®erences in attractiveness between these elements (cardinal value

information).

A scale v that satis¯es the measurement conditions (1)�(3) is an interval scale of

measurement, that is, v is a numerical scale unique up to a positive linear (or a±ne)

transformation.

Several elicitation procedures can be conceived in order to obtain cardinal value

information. One could ask the DM to provide, 8w; x; y; z 2 X with x more attractive

than y and w more attractive than z, a direct numerical estimation of the ratio of the

di®erences in attractiveness between x and y on the one hand and between w and z

on the other hand. This mode of questioning is far from straightforward,90 however.

Moreover, the number of questions dramatically increases with the number of

elements of X . One could reduce the number of questions by taking the di®erence

between two arbitrarily chosen elements of X as a reference unit of measurement and

ask the DM to estimate, 8 x; y 2 X , the number of times Eðx; yÞ the di®erence

between x and y is greater or smaller (if not equal) to the reference di®erence. It

would, however, be surprising if the judgements made were so perfectly consistent

that they determined an interval value scale. In the procedure proposed by Kirkwood

(see Ref. 64) the smallest of the di®erences between consecutive elements in a ranking

is identi¯ed by the DM that subsequently numerically rates the number of times each

one of the remaining consecutive di®erences is greater (if not equal) to the smallest

one. Alternatively, the direct rating technique, used in SMART, requires three main

tasks to be performed: (1) selection of two reference elements for the rating scale;

(2) assignment of numerical values to these reference elements, usually 100 and zero,

respectively; and (3) asking the DM to assign to each one of the remaining elements a

numerical value that denotes its relative attractiveness with respect to the two


references. The consistency of the rating scale is tested in such a way that di®erences

between numerical ratings should re°ect (i.e., measure) di®erences of attractiveness

for the DM.

With the deliberate intent of o®ering a way to avoid the eventual di±culty90 and

cognitive uneasiness50 experienced by evaluators when trying to express their pre-

ference judgements numerically, in MACBETH the transition from ordinal to car-

dinal information is facilitated by a non-numerical pairwise comparison questioning

mode, according to which qualitative judgements, instead of quantitative ones, are

elicited from the DM and are the basis on which a interval value scale will be

constructed, progressively and interactively with the DM, as explainsed in detail in

Sec. 2.2.

2.2. The MACBETH procedure for obtaining cardinal information

2.2.1. Obtaining ordinal information

One can start by asking the DM to rank the elements of X by decreasing attrac-

tiveness. Alternatively, when this is found di±cult, one can ask the DM to compare

the elements two at a time: is one of the two elements more attractive than the other

and if yes, which one? For each paired comparison, M-MACBETH tests the com-

patibility of the ordinal information provided by the DM with the existence of a

ranking and, if an incompatibility is detected, it displays the source of inconsistence

graphically and makes suggestions to overcome it. Figure 2 shows an example of an

inconsistent set of ordinal judgments. A \P" in an entry in the matrix of judgments

means that the DM judges the element in the row to be more attractive than (i.e.,

strictly preferable to) the element in the column, whereas an \I" in an entry means

that the two elements are judged to be equally attractive (indi®erent). The three

judgements bId, dIe and bPe, represented in the graph on the right in Fig. 2, are not

consistent and three alternative suggestions are presented in the respective cells in

the matrix: switching one of the two indi®erences to strict preference (indicated by

an arrow up icon) or switching the strict preference to indi®erence (indicated by an

arrow down icon).

Fig. 2. Example of ordinal inconsistency and suggestions for obtaining a ranking.

MACBETH 363

2.2.2. Obtaining pre-cardinal information

When x is more attractive than y ðxPyÞ, the DM is asked for a qualitative judgement

about the di®erence of attractiveness between x and y, by presenting the DM with six

categories: very weak di®erence (category C1Þ, weak (C2Þ, moderate (C3Þ, strong(C4Þ, very strong (C5Þ and extreme (C6Þ. Judgemental disagreement or hesitation

between two or more consecutive categories, except indi®erence, is also allowed. The

questions may be asked in any sequence and can be stopped at any moment: if xPy

and no MACBETH judgement of di®erence in attractiveness was elicited for the

ordered pair ðx; yÞ, M-MACBETH assigns it to the union of all six categories

(a \positive" di®erence of attractiveness, meaning that, for that pair, the information

available is merely ordinal). That is, for a set X of n elements, it is not necessary to

perform all of the nðn � 1Þ=2 paired comparisons and populate the upper triangular

part of the MACBETH matrix completely (see Fig. 3). Indeed, the minimal number

of judgements required is n � 1, e.g., comparing one element with the remaining ones

or comparing the elements rank-ordered consecutively (the same number of paired

comparisons as in the above-mentioned Kirkwood's numerical procedure). As

emphasized in Ref. 17, however, it is good practice to ask for additional judgements

to perform \a number of consistency checks" (Ref. 90, p. 228), e.g., by ¯lling in the

two ¯rst diagonals of the matrix, as suggested by Belton and Stewart (Ref. 31, p.173)

and exempli¯ed in Fig. 3. This implies a total of 2n � 3 judgements, whereas ¯lling

the borders of the upper triangular matrix, as done in Ref. 8, requires 3n � 6 paired

comparisons. Each time that a qualitative judgement is elicited, M-MACBETH tests

the consistency of all the judgements made by the DM, that is, their compatibility

with cardinal information, verifying whether it is possible to associate with each

element x of X a number vðxÞ satisfying conditions (4)�(6):8 x; y 2 X : xIy ) vðxÞ ¼ vðyÞ; ð4Þ8 x; y 2 X : xPy ) vðxÞ > vðyÞ; ð5Þ8 x; y 2 Ci [ � � � [ Cs

and 8w; z 2 Ci 0 [ � � � [Cs 0 with i; s; i 0; s 0 2 f1; 2; 3; 4; 5; 6g;i � s and i 0 � s 0 : i > s 0 ) vðxÞ � vðyÞ > vðwÞ � vðzÞ: ð6Þ

That is, the judgements of the DM are such that the elements of X can be

positioned in a vertical axis in such a way that:

. 8 x; y 2 X : (the DM judged x and y equally attractive)) (x and y are coincident)

. 8 x; y 2 X : (the DM judged x more attractive than y)) (x is positioned above y).

. 8 x; y;w; z 2 X with x more attractive than y and w more attractive than z, if it

results from the judgements of di®erence in attractiveness given by the DM that

the di®erence of attractiveness between x and y is greater than the di®erence of

attractiveness between w and z, then the distance between x and y is greater than

the distance between w and z.

When such a scale exists, the information assessed is called pre-cardinal value

information. Conversely, if inconsistency is detected, i.e., if conditions (4)�(6)


cannot be simultaneously veri¯ed by the set of judgements elicited, suggestions are

o®ered to resolve it and render the information pre-cardinal. Figure 3 shows an

example with ¯ve elements, a; b; c; d, and e. Suppose that the DM started by ranking

the elements as follows: aPbPcPdIe. Then the DM pairwise compared the elements

consecutive in this ranking by assigning each pair to a category of di®erence in

attractiveness, i.e., ða; bÞ 2 C1 (very weak di®erence), ðb; cÞ 2 C2 [ C3 (hesitation

between weak and moderate di®erence �� the \weak-mod" entry in the matrix of

Fig. 3), ðc; dÞ 2 C2 (weak) (as d and e were judged as indi®erent, the software

indicates there is \no" di®erence of attractiveness between them). Next, element \a"

was judged strongly more attractive than \c". All these judgements are consistent,

but the software pointed out an inconsistence once the moderate judgement between

\b" and \d" was entered in the matrix. Indeed, these judgements imply, given

condition (6), that vðaÞ � vðbÞ > vðcÞ � vðdÞ and vðbÞ � vðdÞ > vðaÞ � vðcÞ, whichare incompatible inequalities because, by summation, vðaÞ � vðdÞ > vðaÞ � vðdÞ.Four alternative modi¯cations of only one category are presented in the respective

cells in the matrix: to move ða; bÞ or ðb; dÞ one category up (indicated by an arrow up

icon) or to move ða; cÞ or ðc; dÞ one category down (indicated by an arrow down

icon), each one su±cient, if accepted, to make the set of judgements verify conditions

(4)�(6) (all other possible ways that resolve the inconsistency can also be displayed

in M-MACBETH). We suppose hereafter that the DM decided to move ða; cÞ downto the category \moderate", while keeping all the remaining judgements unchanged,

which gives rise to the MACBETH matrix of consistent qualitative judgements shown

in Fig. 4.

Fig. 4. Consistent MACBETH judgements.

Fig. 3. Example of inconsistent MACBETH judgements and suggestions for obtaining pre-cardinalinformation.

MACBETH 365

2.2.3. Pre-cardinal and MACBETH scales

From pre-cardinal information it is always possible to determine a pre-cardinal scale,

that is, a numerical scale that satis¯es conditions (4)�(6). A particular pre-cardinal

scale v, the \basic MACBETH scale", can be derived by M-MACBETH, at any

moment in the interactive modeling process, from the pre-cardinal information eli-

cited. This scale is obtained by solving the following linear programming problem

(LP-MACBETH), where vðxÞ is the score assigned to element x of X , xþ and x� are

two elements of X such that xþ is at least as attractive as any other element of X and

x� is at most as attractive as any other element of X .

LP-MACBETH:

Min½vðxþÞ � vðx�Þ�s:t:

ðc1Þ vðx�Þ ¼ 0 ðarbitrary assignmentÞ:ðc2Þ vðxÞ � vðyÞ ¼ 0; 8 x; y 2 C0:

ðc3Þ vðxÞ � vðyÞ � i; 8 x; y 2 Ci [ � � � [ Cs

with i; s 2 f1; 2; 3; 4; 5; 6g and i � s:

ðc4Þ vðxÞ � vðyÞ � vðwÞ � vðzÞ þ i � s 0; 8 x; y 2 Ci [ � � � [ Cs

and 8w; z 2 Ci 0 [ � � � [ Cs 0 with i; s; i 0; s 0 2 f1; 2; 3; 4; 5; 6g;i � s; i 0 � s 0 and i > s 0:

Conditions c2 to c4 are \conditions of order preservation" (COP): conditions c2 and

c3 ensure the preservation of the order in the DM ranking of the elements and

condition c4 ensures the preservation of the order between di®erences of attrac-

tiveness implicit in the qualitative judgements of di®erent categories expressed by

the DM (note that no condition is imposed for judgements of a same category).

The optimal solution of LP-MACBETH is not necessarily unique and, therefore, to

ensure mathematically the uniqueness of the basic MACBETH scale, supplementary

technical linear programs are used (see Bana e Costa et al.12 for technical details). The

basic MACBETH scale for the set of judgements in Fig. 4 is vðaÞ ¼ 5; vðcÞ ¼ 4;

vðbÞ ¼ 2, and vðdÞ ¼ vðeÞ ¼ 0. Any pre-cardinal scale obtained by a positive a±ne

transformation of the basic MACBETH is an \anchored MACBETH scale". In par-

ticular, the scale anchored on vðxþÞ ¼ 100 ð¼ vðaÞÞ and vðx�Þ ¼ 0 ð¼ vðdÞ ¼ vðeÞÞ isautomatically displayed by M-MACBETH on a vertical axis, as shown in Fig. 5.

2.2.4. From pre-cardinal to cardinal information

The DM is invited to observe the MACBETH scale axis and compare value intervals.

This discussion is an essential ¯nal step in the learning path for obtaining cardinal

information, i.e., of measuring attractiveness in an interval value scale. The quali-

tative information thereto elicited, however, is not su±cient for this purpose and it is

worth noting that many other possible scales would equally well respect the con-

ditions of order preservation c2, c3 and c4. When an element is selected in the axis, an


interval is displayed, within which the DM can adjust the position of that element

without violating those conditions. For instance, the DM may want to decrease the

numerical value of element c (within the range shown in Fig. 6) so that the scale

interval associated with the qualitative di®erence in attractiveness between b and c

(where the DM hesitated between weak or moderate) becomes greater than the scale

interval associated with the judgement \weak" between c and d. The value of c

cannot be decreased to 20 or less, however, because this would make the interval

between c and d smaller than the interval between a and b that numerically rep-

resents a very weak di®erence of attractiveness. Adjustments in position can be made

for any element, each time keeping the positions of the remaining ones ¯xed, to make

the intervals between the elements re°ect the relative di®erences of attractiveness

between the elements, for the DM (note: should the DM want to position an element

outside its interval, this would only be possible if the position of at least one other

element were changed and conditions c2 to c4 remained inviolate �� for instance, to

set the value of c below 20 would require a previous reduction of the size of the

interval between a and b by moving up b on the axis). At the end of the discussion of

the MACBETH scale, the ratio between any two positive di®erences of value should

represent the proportion between the respective di®erences in attractiveness. For

example, the ¯nal value of element c ¯xed as 30 in Fig. 6 re°ects that the di®erence

between b and c is half the di®erence between a and d.

Fig. 5. Basic and anchored MACBETH scales.

MACBETH 367

2.3. Determining by hand the basic MACBETH scale

In the case of small consistent sets of judgements, the basic MACBETH scale can be

determined by hand. This is very useful for a facilitator or decision analysis during a

value elicitation session as it allows them to show to the DM and other participants a

clear way to derive a quantitative representation of the DM's qualitative judgements

of di®erence in attractiveness. In this section, X denotes a set of n elements x1; x2;

. . . ; xn previously ranked from left to right by decreasing relative attractiveness, that

is, for r; p 2 f1; 2; . . . ; ng; r < p implies xrðP [ I Þxp, and vðxrÞ denotes the numerical

value of xr . Using these notations, the LP-MACBETH program (see Sec. 2.2.3) can

be rewritten as follows (with Cij ¼ Ci [ . . . [ Cj):

min½vðx1Þ � vðxnÞ�s:t:

ðc1Þ vðxnÞ ¼ 0:

ðc2Þ vðxpÞ � vðxrÞ ¼ 0; 8 ðxp; xrÞ 2 C0 with p < r:

ðc3Þ vðxpÞ � vðxrÞ � i; 8 i; j 2 f1; 2; . . . ; 6g with i � j; 8 ðxp; xrÞ 2 Cij :

ðc4Þ vðxpÞ � vðxrÞ � vðxkÞ � vðxmÞ þ i � j 0; 8 i; j; i 0; j 0 2 f1; 2; . . . ; 6gwith i � j; i 0 � j 0 and i > j 0; 8 ðxp; xrÞ 2 Cij ; 8 ðxk ; xmÞ 2 Ci 0j 0 :

If none of the judgements of the DM are expressed by more than one category (i.e.,

there is no hesitation between two categories for any of the judgements elicited),

Fig. 6. Adjusting the value of an element within the respective range of variation.


constraints c3 and c4 can be written simply as:

ðc5Þ vðxpÞ � vðxrÞ � vðxkÞ � vðxmÞ þ i � i 0; 8 ðxp; xrÞ 2 Ci

and 8 ðxk ; xmÞ 2 Ci 0 with 0 � i 0 < i � 6:

We will show how to determine by hand the basic MACBETH scale in the case of

no hesitation. Since, from c1, vðxnÞ ¼ 0, one only needs to determine the n � 1

elementary di®erences vðx1Þ � vðx2Þ; vðx2Þ � vðx3Þ; . . . ; vðxn�1Þ � vðxnÞ. The way to

determine these elementary di®erences is to proceed as follows:

. Initialization:

8 ðxp; xrÞ 2 C0; vðxpÞ � vðxrÞ ¼ 0;

S : \empty" system of constraints that will be built progressively.

. Start with � ¼ 1:

˚ 8 ðxp; xpþ1Þ 2 C� with p 2 f1; . . . ; n � 1g : vðxpÞ � vðxpþ1Þ ¼ n� þ k�;pwhere: n� ¼ maxfvðxiÞ � vðxjÞji < j and ðxi; xjÞ 2 C��1g þ 1 and

k�;p ¼ \minimal" positive correction with respect to n� to satisfy S.

¸ 8 i; j 2 f1; . . . ; ng with i < j and j � i � 2, calculate vðxiÞ � vðxjÞ whenever it ispossible.

� Verify c5:

� if c5 is not respected,

add to S the condition(s) that must be satis¯ed by the elementary

di®erences so that c5 is satis¯ed

! � ¼ 1

! ˚

� if c5 is respected,

if � ¼ 6, end!

if � < 6:

if 8 p 2 f1; . . . ; n � 1g; vðxpÞ � vðxpþ1Þ calculated, endif not, set � �þ 1, then go to ˚.

We illustrate this procedure in the case of n ¼ 6 on the basis of the consistent

matrix of judgements in Fig. 7.

Initialization: 8 ðxp; xrÞ 2 C0; vðxpÞ � vðxrÞ ¼ 0; S ¼ �

(1) � ¼ 1:

˚ vðx1Þ � vðx2Þ ¼ vðx4Þ � vðx5Þ ¼ vðx5Þ � vðx6Þ ¼ 1 ðk1;1 ¼ k1;4 ¼ k1;5 ¼ 0

because S is \empty").

¸ vðx4Þ � vðx6Þ ¼ ½vðx4Þ � vðx5Þ� þ ½vðx5Þ � vðx6Þ� ¼ 2.

� � c5 is respected� � 2.

MACBETH 369

(2) � ¼ 2:

˚ n2 ¼ 3; vðx2Þ � vðx3Þ ¼ 3ðk2;2 ¼ 0 because S is \empty").

¸ vðx1Þ � vðx3Þ ¼ ½vðx1Þ � vðx2Þ� þ ½vðx2Þ � vðx3Þ� ¼ 4.


(3) � ¼ 3:

˚ n3 ¼ 5; vðx2Þ � vðx3Þ ¼ 5ðk3;3 ¼ 0 because S is \empty").

¸ vðx1Þ � vðx3Þ ¼ ½vðx1Þ � vðx2Þ� þ ½vðx2Þ � vðx3Þ� ¼ 4;

vðx1Þ � vðx4Þ ¼ ½vðx1Þ � vðx2Þ� þ ½vðx2Þ � vðx3Þ� þ ½vðx3Þ � vðx4Þ� ¼ 9;

vðx1Þ � vðx5Þ ¼ ½vðx1Þ � vðx2Þ� þ ½vðx2Þ � vðx3Þ� þ ½vðx3Þ � vðx4Þ�þ ½vðx4Þ�vðx 5Þ� ¼ 10;

vðx1Þ � vðx6Þ ¼ ½vðx1Þ � vðx2Þ� þ ½vðx 2Þ � vðx 3Þ� þ ½vðx3Þ � vðx4Þ�þ½vðx4Þ � vðx5Þ� þ ½vðx5Þ � vðx6Þ� ¼ 11;

vðx2Þ � vðx4Þ ¼ ½vðx2Þ � vðx3Þ� þ ½vðx3Þ � vðx4Þ� ¼ 8;

vðx2Þ � vðx5Þ ¼ ½vðx2Þ � vðx3Þ� þ ½vðx3Þ � vðx4Þ� þ ½vðx4Þ � vðx5Þ� ¼ 9,

vðx2Þ � vðx6Þ ¼ ½vðx2Þ � vðx3Þ� þ ½vðx3Þ � vðx4Þ� þ ½vðx4Þ � vðx5Þ�þ ½vðx5Þ � vðx 6Þ� ¼ 10;



vðx4Þ � vðx6Þ ¼ ½vðx4Þ � vðx5Þ� þ ½vðx5Þ � vðx6Þ� ¼ 2 (see Fig. 8).

� c5 is not respected: as ðx1; x4Þ 2 C4 and ðx2; x5Þ 2 C3, one must have

vðx1Þ � vðx4Þ � vðx2Þ � vðx5Þ þ 1, that is, in terms of the elementary di®er-

ences:

½vðx1Þ � vðx2Þ� þ ½vðx2Þ � vðx3Þ� þ ½vðx3Þ � vðx4Þ�� ½vðx2Þ � vðx3Þ� þ ½vðx3Þ � vðx4Þ� þ ½vðx4Þ � vðx5Þ� þ 1,

which implies:

vðx1Þ � vðx2Þ � vðx4Þ � vðx5Þ þ 1

S ¼ fvðx1Þ � vðx2Þ � vðx4Þ � vðx5Þ þ 1g thus k1;1 ¼ 1

� ¼ 1:

Fig. 7. Consistent matrix of MACBETH qualitative judgements with no hesitation.


(4) � ¼ 1:

˚ vðx4Þ � vðx5Þ ¼ vðx5Þ � vðx6Þ ¼ 1 and vðx1Þ � vðx2Þ ¼ 2 ðk1;4 ¼ k1;5 ¼ 0;

k11 ¼ 1Þ:¸ vðx4Þ � vðx6Þ ¼ ½vðx4Þ � vðx5Þ� þ ½vðx5Þ � �vðx6Þ� ¼ 2:

� � c5 is respected� � 2:

(5) � ¼ 2:

˚ n2 ¼ 3; vðx2Þ � vðx3Þ ¼ 3 ðk2;2 ¼ 0Þ.¸ vðx1Þ � vðx3Þ ¼ ½vðx1Þ � vðx2Þ� þ ½vðx2Þ � vðx3Þ� ¼ 5.


(6) � ¼ 3:

˚ n3 ¼ 6; vðx3Þ � vðx4Þ ¼ 5 ðk3;3 ¼ 0Þ:¸ vðx1Þ � vðx3Þ ¼ ½vðx1Þ � vðx2Þ� þ ½vðx2Þ � vðx3Þ� ¼ 5;


vðx1Þ � vðx5Þ ¼ ½vðx1Þ � vðx2Þ� þ ½vðx2Þ � vðx3Þ� þ ½vðx3Þ � vðx4Þ�þ ½vðx4Þ � vðx5Þ� ¼ 12;

vðx1Þ � vðx6Þ ¼ ½vðx1Þ � vðx2Þ� þ ½vðx2Þ � vðx3Þ� þ ½vðx3Þ � vðx4Þ�þ ½vðx4Þ � vðx5Þ� þ ½vðx5Þ � vðx6Þ� ¼ 13;



vðx2Þ � vðx6Þ ¼ ½vðx2Þ � vðx3Þ� þ ½vðx3Þ � vðx4Þ� þ ½vðx4Þ � vðx5Þ�þ ½vðx5Þ � vðx6Þ� ¼ 11;

vðx3Þ � vðx5Þ ¼ ½vðx3Þ � vðx4Þ� þ ½vðx4Þ � vðx5Þ� ¼ 7,


vðx4Þ � vðx6Þ ¼ ½vðx4Þ � vðx5Þ� þ ½vðx5Þ � vðx6Þ� ¼ 2:

� � c5 is respected� 8 p 2 f1; 2; 3; 4g; vðxpÞ � vðxpþ1Þ calculated� end.

Fig. 8. Applying the \hand procedure".

MACBETH 371

As all the elementary di®erences are determined (see Fig. 9), the basic MAC-

BETH scale can be obtained:

vðx6Þ ¼ 0;

vðx5Þ ¼ vðx6Þ þ ½vðx5Þ � vðx6Þ� ¼ 1;




vðx1Þ ¼ vðx2Þ þ ½vðx1Þ � vðx2Þ� ¼ 13:

3. Multicriteria Evaluation with MACBETH: The SOA Case Study

3.1. The context

ANACOM (\Autoridade Nacional de Comunicações"; www.anacom.pt) is the

national regulatory authority for electronic communications and postal services in

Portugal. It also provides advice on public policy to the government and is the

international representative of the Portuguese communications sector. In 2007,

ANACOM adopted the MACBETH approach and software to support option

evaluation and decision making in \beauty contests",37 when awarding telecom

licences, and public procurement processes across the entire corporation, covering

the provision of contracts for the execution or concession of public services and

works, and the acquisition of goods and services. The case study developed in this

section is inspired by the evaluation process of the tenders presented in response to

the international call for tenders for the \acquisition of BPMS-DMS-SOA solution"

(nr. 2/2008) launched by ANACOM and conducted by an internal technical jury

(the DM) in the Information Systems Division. In brief, a BPMS-DMS-SOA solution

includes the delivery of a business process management system (BPMS) and a

document management system (DMS) integrated in a service-oriented architecture

(SOA). It should be a process-centric information technology solution that is able to

integrate people, systems and data41 and make available business functionality or

Fig. 9. Applying the \hand procedure" (continued).


application logic to system users, or consumers, as shared reusable services.68 Three

IT companies answered the BPMS-GD-SOA call for tenders. In this case study,

performances of their tenders (labeled A, B, and C) and DM judgemental infor-

mation were disguised for con¯dentiality reasons and to simplify or improve the focus

of the presentation.

3.2. Structuring issues

The three criteria de¯ned to evaluate the tenders and announced in the call for

tenders were: the extent to which the solution proposed in each tender was techni-

cally and functionally suitable for achieving ANACOM objectives (criterion Cr1:

\Suitability"), the extent to which each tender presented an adequate execution plan

(criterion Cr2: \Execution") and the total price proposed in each tender including

licensing, technical support and document management services (criterion Cr3:

\Price").

One recommended practice when structuring and operationalizing criteria fol-

lowing the MACBETH approach in a public call for tenders is to make clear to the

potential tenderers what constitutes a good performance (goodjÞ on each criterion

Crj (see Refs. 9 and 26), e.g., good3 ¼ 400 thousand , a price below which a tender

would be assessed as very attractive. The description of references (goals, aspiration

levels, standards) of good performance may increase the chances of receiving good

proposals from potential tenderers. Of course, not all of them, if any, will be able to

achieve good performances in all criteria. Indeed, quality versus cost or risk are

crucial trade-o®s, as it may be necessary to privilege some goal(s) to the detriment of

others. A complementary structuring recommendation of MACBETH is to de¯ne a

reference of neutral performance (neutraljÞ on each criterion Crj , e.g., neutralj ¼ 600

thousand , a price above which a tender will be assessed as unattractive. Once the

two reference performances are entered into the MACBETH model, the scores of 100

and 0 are assigned to them by default and the criterion value scale automatically

displayed by M-MACBETH is anchored on vjðgoodjÞ ¼ 100 and vjðneutraljÞ ¼ 0.

In general, performance on quality and risk criteria depends on a signi¯cant

number of interrelated indicators or characteristics. These are often combined by

means of point systems57 that su®er from the drawback of ignoring that elementary

value dimensions are often not additive-independent.64 As remarked by Edwards and

Barron (Ref. 48, p. 315), \violations of conditional monotonicity usually easy to

detect judgementally mean that additive models should not be used" (to aggregate

interrelated dimensions). Alternatively, multidimensional performance scales (see

Refs. 5, 17 and 60) can be constructed by applying noncompensatory aggregation

procedures such as the \determinants technique" (see Ref. 9):

(1) Establish two references, \satisfactory" and \neutral", in each of the dimensions.

(2) Classify each dimension as \determinant" (D), \important" (I) or \secondary"

(S). A dimension is determinant if a performance that is negative (worse than

neutral) in that dimension is a necessary and su±cient condition for a proposal to

MACBETH 373

be considered negative (worse than neutral) in the set of all dimensions (this

means that a determinant dimension has a noncompensatory nature).

(3) De¯ne a reference of good performance on the set of dimensions as a reference

pro¯le such that all determinant dimensions are satisfactory and a majority of

the important dimensions are satisfactory; and a reference of neutral perform-

ance on the set of dimensions as a reference pro¯le such that a majority of the

determinant and important dimensions are neutral, without any dimension being

negative.

(4) Similar rules can be used to de¯ne other reference performance pro¯les, e.g.,

\better than good" as all determinant and important dimensions are satisfactory

and no secondary characteristic is negative, or \worse than neutral" as at least

one determinant characteristic is negative.

In the SOA case, for example, there were six elementary dimensions related to

ease of implementation of the solution. As indicated in Table 1, only one was con-

sidered determinant, two were viewed as important and the remaining three were

deemed secondary.

3.3. Issues in inter-criteria modeling: MACBETH weighting

It was set in the SOA call for tenders that the overall value score V ðxÞ of each tender

x would be calculated by the simple additive value model (7)

V ðxÞ ¼X3j¼1

kjvjðxÞ; withvjðgoodjÞ ¼ 100

vjðneutraljÞ ¼ 0;

�kj > 1 ðj ¼ 1; 2; 3Þ and

X3j¼1

kj ¼ 1;

ð7Þ

where the parameters kj ðj ¼ 1; 2; 3Þ are scaling constants �� usually called

\weights"�� enabling one to convert the single-criterion value scores vjðxÞ into unitsof overall value. In this section, we will illustrate how these relative weights can be

determined with the MACBETH procedure. It is worth noting that the numerical

weights assigned to the SOA criteria were set before the tenders were known and

Table 1. Elementary dimensions of ease of implementation.

Elementary Dimensions Type Satisfactory Levels Neutral Levels

Time needed to install the platform in the ANACOM

infrastructure

D 4 h 8 h

Independence of DBA (Database Administrator)

privileges

S Yes No

Disk space required S 5GB 50GB

RAM required S 3GB 6GBSize of the technical team I 2 technicians 3 technicians

Duration of the concept test I 2 days 4 days


were disclosed in the call for tenders, in compliance with national and European

Union regulations.

Let [Crj �; j ¼ 1; 2; 3, denote three hypothetical tenders, each with performance on

criterion Crj equal to the reference goodj and performance on each one of the two

remaining criteria equal to the respective neutral reference performance; and let

[neutral] denote the hypothetical tender with performance on all criteria equal to the

respective neutral reference performances:

[Cr1] ¼ [good1, neutral2, neutral3],

[Cr2] ¼ [neutral1, good2, neutral3],

[Cr3] ¼ [neutral1, neutral2, good3],

[neutral] ¼ [neutral1, neutral2, neutral3].

It is interesting to observe that for each i ¼ 1; 2 or 3:

V ð½Crj �Þ � V ð½neutral�Þ ¼ kj ½vjðgoodjÞ � vjðneutraljÞ� ¼ 100kj : ð8ÞTherefore, in order to set the weights it is su±cient to elicit cardinal information

concerning the (overall) attractiveness of the four hypothetical tenders [Cr1], [Cr2],

[Cr3], and [neutral]. This can be done by paired comparisons, following the MAC-

BETH procedure described in Sec. 2. Suppose that the DM made the qualitative

judgements shown in Fig. 10, for which M-MACBETH suggested a ¯rst weighting

scale, shown in percentages at the right of the matrix, subsequently adjusted to the

weights shown in percentages in the bar chart: k1 ¼ 0:55; k2 ¼ 0:25 and k3 ¼ 0:20.

This MACBETH weighting procedure is a qualitative alternative to the quanti-

tative \swing weighting procedure" used in SMART, which, as described in Ref. 90

(see also Ref. 48), would consist in asking the DM to rank the hypothetical tenders

(this can be done by paired comparisons, as in Ref. 60, p. 290), assign a \swing

weight" of 100 to the preferred improvement and swing-weight the other improve-

ments as percentages of this.

In practice, a straightforward MACBETH question�answer process to elicit

weighting judgements could develop as follows between the facilitator (F) and

the DM:

F: Suppose there is one tender with neutral performances in all criteria ([neutral]). If

its performance on one criterion could be improved to good, on which criterion would

the improvement from neutral to good be the most important? (note that this

Fig. 10. MACBETH weighting.

MACBETH 375

question is asking on which criterion the di®erence in attractiveness between good

and neutral is the greatest).

DM: The most important improvement from neutral to good would be on suitability.

F: How important would the improvement be from neutral to good suitability?

DM: The improvement from neutral to good suitability has very strong importance.

F: Which would be the next most important improvement from neutral to good, on

execution or on price?

DM: Execution.

F: How important would be the improvement from neutral to good on execution?

DM: Moderately.

F: How much more important would be the improvement from neutral to good on

suitability than the improvement from neutral to good on execution?

DM: The improvement on suitability would be strongly more important than the

improvement on execution.

F: How important is the improvement from neutral to good price?

DM: Moderately.

F: Would the improvements from neutral to good on execution and on price be

equally important?

DM: No, the improvement on delivery would be more important.

F: How much more important?

DM: Very weakly.

F: One last question. Would the improvement from neutral to good on suitability be

strongly or very strongly more important than the same improvement on price?

DM: Strongly.

Note that the facilitator's questions in this dialogue would be misleading if they were

asked simply in terms of the importance of the criteria rather than in terms of the

importance of the improvements from neutral to good on the criteria. That would be

a mistake (unfortunately, \the most common critical mistake" (see Ref. 60, p. 147

and p. 279).

3.4. Issues in intra-criteria evaluation: MACBETH scoring

At the level of each criterion, a MACBETH model can be built following one of two

basic paths: direct evaluation, in which options are compared with one another and a

value score is assigned to each one; or indirect evaluation, in which a value function is

constructed upon a previously de¯ned descriptor of performances and then the value

function is used to transform each option performance in a value score. The pros and

cons of these two paths are discussed by Bana e Costa and colleagues.22

In the SOA case, the tenders were directly compared for Suitability and Execution

(criteria Cr1 and Cr2Þ, whereas for the Price criterion (Cr3Þ a value function was

a priori de¯ned and announced in the procurement documents. The value function is

often assumed to be linear within the Price range, but this does not prevent very high

prices. On the contrary, a set of MACBETH judgements as shown in Fig. 11, with a


weak di®erence of attractiveness between 400 and 600 thousand (the good and

neutral prices) and an extreme di®erence of attractiveness between 600 and 700

thousand (i.e., prices higher than neutral), giving rise to a two-piecewise linear

value function, reveals the DM's intention to discourage the proposal of high prices

by potential tenderers.

As regards the Execution criterion (Cr2Þ, ¯rst the DM appraised the intrinsic

attractiveness of each tender, by comparing its performance with the good and

neutral reference performances. Then, the tenders and the two references were

pairwise compared, qualitatively, with the MACBETH procedure. Figure 12 shows

an illustrative example of the MACBETH judgements on Execution: an initial

inconsistent matrix, with the software indicating two possible modi¯cations of one

category, and the ¯nal consistent matrix, together with the value scores resulting

from the discussion by the DM. A similar interactive process was followed to score

the tenders on the Suitability criterion (Cr1Þ (see Fig. 13).

The synthesis of all the intra-criteria and inter-criteria cardinal information

so far assessed (see Table 2) was then performed by applying the additive model

V ðxÞ ¼ 0:55v1ðxÞ þ 0:25v2ðxÞ þ 0:20v3ðxÞ to calculate the overall value of each

tender: V ðAÞ ¼ 50:35;V ðBÞ ¼ 53:45 and V ðC Þ ¼ 92:90 (the prices of tenders A, B,

and C were, respectively, 378, 397, and 546 thousand ).

3.5. Issues of sensitivity and robustness analysis

Decision-making processes often involve imprecise data and uncertain information.

Managers are often concerned with the stability of the outputs of the additive value

model. For example, they may be interested in analyzing if the best option would

Fig. 11. Building a value function on Price (Cr3Þ.

Fig. 12. MACBETH intra-criterion evaluation process on Execution (Cr2Þ.

MACBETH 377

change in the face of small variations of the weight of one criterion, e.g., Price. The

sensitivity analysis graph in Fig. 14 shows that C remains the best tender even when

the weight of Price is increased from 20% to nearly 47%. This is highlighted in Fig. 14

by the band of variation of the weight, compatible with the DM judgements indi-

cated in Fig. 10, whose upper bound is around 25%.

Fig. 13. MACBETH judgements and value scale on Suitability (Cr1Þ.

Table 2. Table of scores.

Tenders Value Scores

Suitability Execution Price Overall

C 100 130 27 92.90

B 33 60 101.5 53.45

A 33 40 111 50.35

Weights 0.55 0.25 0.20

Fig. 14. Sensitivity analysis of the weight of Price.


It may also be interesting to analyze what conclusions can be drawn in the face of

only ordinal or/and pre-cardinal intra-criteria and inter-criteria information.

M-MACBETH provides another tool for this purpose, the \robustness analysis"

function. Let x and y be any two options. One would say that \x is globally more

attractive than y in the face of speci¯ed information" when k1v1ðxÞ þ k2v2ðxÞþk3v3ðxÞ > k1v1ðyÞ þ k2v2ðyÞ þ k3v3ðyÞ, for all single-value scales v1; v2; v3 and all

weights k1; k2; k3 compatible with the speci¯ed information (ordinal, pre-cardinal, or

cardinal). Such comparisons can be made by calculating, for each pair ðx; yÞ ofoptions, the minimal mðx; yÞ and maximum Mðx; yÞ values of the di®erence between½k1v1ðxÞ þ k2v2ðxÞ þ k3v3ðxÞ� and ½k1v1ðyÞ þ k2v2ðyÞ þ k3v3ðyÞ� for the speci¯ed

information. Three situations can arise: if mðx; yÞ � 0 and Mðx; yÞ > 0, then x is

globally more attractive than y (for the speci¯ed information); if mðx; yÞ < 0 and

M ðx; yÞ � 0, then y is globally more attractive than x; and, if mðx; yÞ < 0 and

M ðx; yÞ > 0, then neither of the two options is more attractive than the other. The

¯rst two are \additive dominance" situations and the third is \incomparability", as

de¯ned by Bana e Costa and Vincke (see Ref. 30).

Let us analyze whether the most attractive tender of the SOA case (tender C)

remains the same if one takes into account, simultaneously: (a) only the pre-cardinal

information elicited in Cr1 and Cr2 (i.e., the respective consistent sets of MACBETH

judgements of di®erence in attractiveness), (b) the cardinal information on Cr3 (i.e.,

the value function on Price) that lead to the ¯xed value scores v3ðAÞ ¼ 111; v3ðBÞ ¼101:5, and v3ðC Þ ¼ 27, and (c) the cardinal weighting information (i.e., the con-

sistent set of weighting judgements) that lead to the ¯xed weights k1 ¼ 0:55;

k2 ¼ 0:25, and k3 ¼ 0:20. The robustness analysis table (Table 3) shows that when

the value information of (a), (b), and (c) is selected simultaneously, tender C always

remains globally more attractive than tenders A and B (C additively dominates A

and B), although neither of the two tenders A and B can be robustly considered more

attractive than the other (A and B are incomparable).

The meaning of the contents of Table 3 in Fig. 12 is:

. C additively dominates A and B because, for all value scales v1 and v2 compatible

with the pre-cardinal information on criteria Cr1 and Cr2, respectively,

0:55v1ðCÞ þ 0:25v2ðCÞ þ 0:20� 27 > 0:55v1ðAÞ þ 0:25v2ðAÞ þ 0:20� 111 and

0:55v1ðCÞ þ 0:25v2ðCÞ þ 0:20� 27 > 0:55v1ðBÞ þ 0:25v2ðBÞ þ 0:20� 101:5;

Table 3. Table of robustness analysis

ð� denotes additive dominance and ?denotes incomparability).

! C B A

C ¼ � �B ¼ ?

A ¼

MACBETH 379

. A and B are incomparable because there exist at least two value scales v1 and u1and two value scales v2 and u2 compatible with the pre-cardinal information on Cr1and Cr2, respectively, for which,

0:55v1ðAÞ þ 0:25v2ðAÞ þ 0:20� 111 < 0:55v1ðBÞ þ 0:25v2ðBÞ þ 0:20� 101:5 and

0:55u1ðAÞ þ 0:25u2ðAÞ þ 0:20� 111 > 0:55u1ðBÞ þ 0:25u2ðBÞ þ 0:20� 101:5:

For example, if v1ðAÞ ¼ 33; v1ðBÞ ¼ 33; v2ðAÞ ¼ 40 and v2ðBÞ ¼ 60, we obtain

0:55v1ðAÞ þ 0:25v2ðAÞ þ 0:20� 111 ¼ 50:35 and

0:55v1ðBÞ þ 0:25v2ðBÞ þ 0:20� 101:5 ¼ 53:45:

If, however, u1ðAÞ ¼ 33; u1ðBÞ ¼ 33; u2ðAÞ ¼ 40; and u2ðBÞ ¼ 45, we obtain

0:55u1ðAÞ þ 0:25u2ðAÞ þ 0:20� 111 ¼ 50:35 and

0:55u1ðBÞ þ 0:25u2ðBÞ þ 0:20� 101:5 ¼ 49:70:

4. Some Concluding Re°ections

The MACBETH non-numerical approach to cardinal value measurement is based on

sound theory, like the traditional numerical approaches, but has the additional

advantage of o®ering a practical method of interactive veri¯cation of the reliability of

the preference information elicited. The MACBETH value-elicitation procedure is

composed of an input stage, aiming at eliciting, from an individual or group eva-

luator, a consistent set of non-numerical pairwise comparison judgements of quali-

tative di®erence in attractiveness, and an output stage, aiming at constructing from

the sets of judgements a multicriteria evaluation model that numerically measures

the relative attractiveness of options for the evaluator who made the judgements.

Respect for these judgements is the foundation stone of MACBETH, as modeled by

the condition of order preservation (COP). The nonconformity with COP-type

conditions of the eigenvalue procedure used to derive priorities from (ratio) pairwise

comparisons is viewed, from our constructive perspective of decision-aiding, as a

fundamental criticism (see Ref. 29) to the AHP method proposed by Saaty,84

although we recognize the in°uence of Saaty's original ideas (see, for example,

Ref. 85).

There are many well-known mistakes in making value trade-o®s (see

Refs. 60�62). The direct comparison of the criteria58 in terms of the intuitive notion

of importance can give rise to arbitrary recommendations from misleading average

sums of scores produced to aggregate value scores of options on the criteria. The

additive value model continues to be \by far the easiest to use and most familiar

model for such aggregations" (Ref. 48, p. 314) and it is therefore essential to avoid

the use of elicitation procedures that do not conform to the principles of additive

value measurement (for details, see the section title \the interpretation of criteria

weights" in Ref. 31, p. 147). For example, Edwards and Barron48 recognized the

weakness of the weighting procedure of SMART47 �� which \ignores the fact that

range as well as importance must be re°ected in any weight" (Ref. 48, p. 316) �� and


they replaced it with the correct swing-weighting technique. It is however a

numerical elicitation of weights and we were therefore motivated to propose the non-

numerical MACBETH weighting that can be viewed, appropriately, as a qualitative

swing-weighting technique.

A wide variety of real-world cases using the MACBETH approach and software,

both in the private and public sectors, are reported in the scienti¯c literature (see

Table 4; see also Ref. 12). In some of these multicriteria evaluation processes, we have

ourselves acted as decision-analysts and facilitators helping people with di®erent

tasks: managers, experts, civil servants, o±cers, politicians, etc. We observed that

they particularly liked three distinctive features:

. MACBETH's consistency checking at the input stage, especially the suggestions of

what to change if needed.

. The ability to put more than one qualitative rating in each cell of the matrix of

judgements, particularly useful when there are many disagreements within a

group; yet when the members of the group see the scale generated by MACBETH,

they usually agree that it is acceptable.

. The display of the range of possible scores on the scale so the ¯nal scale can be

adjusted.

As observed by Phillips (Ref. 78, p. 89): \Words are essential, more essential than

numbers, but a blending of the two can enable individuals and groups to achieve new

Table 4. Selected applications of MACBETH.

Agriculture, Manufacturing & Services:

Finance: Refs. 4, 18, 24 and 53

Performance measurement: Refs. 32�35, 42, 65�67 and 69

Production & service planning: Refs. 14, 73, 81, 82 and 86Quality management: Refs. 7, 39 and 40

Risk management: Refs. 43 and 54

Strategy & resource allocation: Refs. 10 and 15Supply chain management: Ref. 77

Energy:

Project prioritization and selection: Ref. 17Technology choice: Refs. 38, 52 and 72

Environment:

Landscape management: Ref. 89

Risk management: Ref. 22, 44 and 59Water resource management: Ref. 6

Medical: Refs. 45, 46, 74 and 75

Public Sector:

Con°ict analysis and management: Ref. 3, 23, 36, 51 and 83

Procurement: Refs. 9, 17 and 77

Project prioritization & resource allocation: Refs. 16, 21, 26, 27, 70, 76 and 87Strategic planning & development: Ref. 11

Human resource management & job selection: Refs. 8, 19, 20, 49, 55 and 56

MACBETH 381

depths of understanding which would not have been possible using either words or

numbers alone."

Concerning the extent to which the MACBETH approach has been helpful to

ANACOM, we reproduce here a message from Prof. Eduardo Cardadeiro, a member

of the Board of Directors, dated 2 November 2011: \In fact, some years ago ANA-

COM has adopted the use of Macbeth in its internal procedures for procurement of

goods and services over 75 thousand euros. Based on our experience this approach

has been very useful in several phases of those processes: (1) it helps to design

consistent valuation/selection criteria, ex-ante; (2) it makes the valuation of

alternatives much more objective and robust; and (3) the robustness analysis is

always performed before taking the ¯nal decision and, ex-post, helps to support the

decision, in case of litigation."

Acknowledgments

The authors would like to thank ANACOM for the opportunity to develop the SOA

case study and acknowledge the support provided by the Fundação para a Ciencia e aTecnologia.

References

1. K. Arrow, Social Choice and Individual Values (Wiley, New York, 1951).2. Bana Consulting, M-MACBETH Version 1.1: User Manual (Bana Consulting, Lisbon,

2005), http://www.m-macbeth.com/help/pdf/M-MACBETH%20User's%20Guide.pdf.3. C. A. Bana e Costa, The use of multi-criteria decision analysis to support the search for

less con°icting policy options in a multi-actor context: Case study, Journal of Multi-Criteria Decision Analysis 10(2) (2001) 111�125.

4. C. A. Bana e Costa, L. A. Barroso and J. O. Soares, Qualitative modeling of creditscoring: A case study in banking, European Research Studies Journal 5(1�2) (2002)37�51.

5. C. A. Bana e Costa and E. Beinat, Model-structuring in public decision-aiding (LSEOR05.79) (London School of Economics and Political Science, London, 2005), http://eprints.lse.ac.uk/22716/1/05079.pdf.

6. C. A. Bana e Costa, P. A. da Silva and F. N. Correia, Multicriteria evaluation of °oodcontrol measures: The case of Ribeira do Livramento, Water Resources Management18(3) (2004) 263�283.

7. C. A. Bana e Costa, M. Carnero and M. D. Oliveira, A multi-criteria model for auditing aPredictive Maintenance Programme, European Journal of Operational Research 217(2)(2012) 381�393.

8. C. A. Bana e Costa and M. P. Chagas, A career choice problem: An example of how to useMACBETH to build a quantitative value model based on qualitative value judgments,European Journal of Operational Research 153(2) (2004) 323�331.

9. C. A. Bana e Costa, E. C. Correa, J. M. De Corte and J.-C. Vansnick, Facilitating bidevaluation in public call for tenders: A socio-technical approach, Omega-InternationalJournal of Management Science 30(3) (2002) 227�242.

10. C. A. Bana e Costa, E. Correa, L. Ensslin and J.-C. Vansnick, Decision support systems inaction: Integrated application in a multicriteria decision aid process, European Journal ofOperational Research 113(2) (1999) 315�335.


11. C. A. Bana e Costa, M. L. Costa-Lobo, I. A. J. Ramos and J.-C. Vansnick, Multicriteriaapproach for strategic town planning: The case of Barcelos, in Aiding Decisions withMultiple Criteria: Essays in Honor of Bernard Roy, eds. D. Bouyssou, E. Jacquet-Lagr�eze, P. Perny et al. (Kluwer Academic Publishers, Boston, 2002), pp. 429�456.

12. C. A. Bana e Costa, J. M. De Corte and J. C. Vansnick, On the mathematical foundationsof MACBETH, in Multiple Criteria Decision Analysis: The State of the Art Surveys, eds.J. Figueira, S. Greco and M. Ehrgott (Springer, New York, 2005), pp. 409�442.

13. C. A. Bana e Costa, J. M. De Corte and J. C. Vansnick, MACBETH (MeasuringAttractiveness by a Categorical Based Evaluation Technique), in Wiley Encyclopedia inOperational Research and Management Science, ed. J. J. Cochran 4 (Wiley, New York,2011), pp. 2945�2950.

14. C. A. Bana e Costa, L. Ensslin and A. P. Costa, Structuring the process of choosing ricevarieties at the south of brazil, multicriteria analysis for land-use management, inEnvironment and Management, eds. E. Beinat and P. Nijkamp 9 (Kluwer AcademicPublishers, Dordrecht, 1998), pp. 33�45.

15. C. A. Bana e Costa, L. Ensslin and I. J. Zanella, A real-world MCDA application incellular telephony systems, in Trends in Multicriteria Decision Making, eds. T. J. Stewartand R. C. van den Honert (Springer, Berlin, 1998), pp. 412�423.

16. C. A. Bana e Costa, T. G. Fernandes and P. V. D. Correia (2006), Prioritisation of publicinvestments in social infrastructures using multicriteria value analysis and decision con-ferencing: A case study, International Transactions in Operational Research 13(4) (2006)279�297.

17. C. A. Bana e Costa, J. C. Lourenço, M. P. Chagas and J. C. Bana e Costa, Developmentof reusable bid evaluation models for the Portuguese Electric Transmission Company,Decision Analysis 5(1) (2008) 22�42.

18. C. A. Bana e Costa, J. C. Lourenço and J. O. Soares, An interval weighting assignmentmodel for credit analysis, Journal of Financial Decision Making 3(2) (2007) 1�9.

19. C. A. Bana e Costa, P. A. F. Martins, M. D. Oliveira, A. Sernadas and C. A. Mota Soares,Faculty evaluation using multicriteria value measurement, Advances in Mathematicaland Computational Methods (WSEAS Press, 2010), pp. 287�290.

20. C. A. Bana e Costa and M. D. Oliveira, A multicriteria decision analysis model for facultyevaluation, Omega-International Journal of Management Science 40(4) (2012) 424�436.

21. C. A. Bana e Costa and R. C. Oliveira, Assigning priorities for maintenance, repair andrefurbishment in managing a municipal housing stock, European Journal of OperationalResearch 138(2) (2002) 380�391.

22. C. A. Bana e Costa, C. S. Oliveira and V. Vieira, Prioritization of bridges and tunnels inearthquake risk mitigation using multicriteria decision analysis: Application to Lisbon,Omega-International Journal of Management Science 36(3) (2008) 442�450.

23. C. A. Bana e Costa, F. N. Silva and J.-C. Vansnick, Con°ict dissolution in the publicsector: A case-study, European Journal of Operational Research 130(2) (2001) 388�401.

24. C. A. Bana e Costa and J. O. Soares, A multicriteria model for portfolio management,European Journal of Finance 10(3) (2004) 198�211.

25. C. A. Bana e Costa and J.-C. Vansnick, MACBETH �� An interactive path towards theconstruction of cardinal value functions, International Transactions in OperationalResearch 1(4) (1994) 489�500.

26. C. A. Bana e Costa and J.-C. Vansnick, Applications of the MACBETH approach in theframework of an additive aggregation model, Journal of Multi-Criteria Decision Analysis6(2) (1997) 107�114.

27. C. A. Bana e Costa and J.-C. Vansnick, The Macbeth approach: Basic ideas, software,and an application, in Advances in Decision Analysis, eds. N. Meskens and M. Roubens(Springer, Dordrecht, 1999), pp. 131�157.

MACBETH 383

28. C. A. Bana e Costa and J.-C. Vansnick, Cardinal value measurement with MACBETH,in Decision Making: Recent Developments and Worldwide Applications, eds. S. H.Zanakis, G. Doukidis and C. Zapounidis (Kluwer Academic Publishers, Dordrecht, 2000),pp. 317�329.

29. C. A. Bana e Costa and J. C. Vansnick, A critical analysis of the eigenvalue method usedto derive priorities in AHP, European Journal of Operational Research 187(3) (2008)1422�1428.

30. C. A. Bana e Costa and P. Vincke, Measuring credibility of compensatory preferencestatements when trade-o®s are interval determined, Theory and Decision 39(2) (1995)127�155.

31. V. Belton and T. J. Stewart, Multiple Criteria Decision Analysis: An IntegratedApproach (Springer, Berlin, 2002).

32. L. Berrah and V. Clivill�e, Towards an aggregation performance measurement systemmodel in a supply chain context, Computers in Industry 58(7) (2007) 709�719.

33. L. Berrah, G. Mauris and J. Montmain, Monitoring the improvement of an overallindustrial performance based on a Choquet integral aggregation, Omega-InternationalJournal of Management Science 36 (2008) 340�351.

34. L. Berrah, G. Mauris, J. Montmain and V. Cliville, E±cacy and e±ciency indexes for amulti-criteria industrial performance synthesized by Choquet integral aggregation,International Journal of Computer Integrated Manufacturing 21(4) (2008) 415�425.

35. L. Berrah, G. Mauris and F. Vernadat, Industrial performance measurement: Anapproach based on the aggregation of unipolar or bipolar expressions, InternationalJournal of Production Research 44(18�19) (2006) 4145�4158.

36. D. Bollinger and J. Pictet, Potential use of e-democracy in MCDA processes. Analysis onthe Basis of a Swiss case, Journal of Multi-Criteria Decision Analysis 12 (2003) 65�76.

37. T. B€orgers and C. Dustman, Awarding telecom licenses: The recent European experience,Economic Policy 36 (2003) 215�268.

38. J. Burton and K. Hubacek, Is small beautiful? A multicriteria assessment of small-scaleenergy technology applications in local governments, Energy Policy 35 (2007)6402�6412.

39. M. C. Carnero, Evaluating a maintenance department in a service company, Inter-national Journal of Mathematical Models and Methods in Applied Sciences 3(3) (2009)230�237.

40. M. C. Carnero, Maintenance audit by means of an additive multicriteria model, inAdvances in Marketing, Management and Finances, eds. S. Hashemi and C. Vobach(WSEAS Press, 2009), pp. 147�152.

41. J. F. Chang, Business Process Management Systems: Strategy and Implementation(Auerbach Publications, Boca Raton, 2006).

42. V. Cliville, L. Berrah and G. Mauris, Quantitative expression and aggregation of per-formance measurements based on the MACBETH multi-criteria method, InternationalJournal of Production Economics 105(1) (2007) 171�189.

43. H. R. Costa, M. D. O. Barros and G. Travassos, Evaluating software project portfoliorisks, Journal of Systems & Software 80 (2007) 16�31.

44. F. Dall'Osso, M. Gonella, G. Gabbianelli, G. Withycombe and D. Dominey-Howes, Arevised (PTVA) model for assessing the vulnerability of buildings to tsunami damage,Natural Hazards and Earth System Sciences 9(5) (2009) 1557�1565.

45. A. K. A. de Castro, P. R. Pinheiro and M. Pinheiro, Towards the neuropsychologicaldiagnosis of Alzheimer's disease: A hybrid model in decision making, in Best Practices forthe Knowledge Society: Knowledge, Learning, Development and Technology for All, eds.M. D. Lytras et al. (Springer, Berlin, 2009), pp. 522�531.


46. A. K. A. de Castro, P. R. Pinheiro, M. Pinheiro and I. Tamanini, Towards the appliedhybrid model in decision making: A neuropsychological diagnosis of Alzheimer's diseasestudy case, International Journal of Computational Intelligence Systems 4(1) (2011)89�99.

47. W. Edwards, How to use multiattribute utility. measurement for social decisionmaking,IEEE Transactions on Systems, Man and Cybernetics 7 (1977) 326�340.

48. W. Edwards and F. H. Barron, SMARTS and SMARTER: Improved simple methods formultiattribute utility measurement, Organizational Behavior and Human DecisionProcesses 60 (1994) 306�325.

49. L. Ensslin, A. Dutra and S. R. Ensslin, MCDA: A constructivist approach to the man-agement of human resources at a governmental agency, International Transactions onOperational Research 7(1) (2000) 79�100.

50. B. Fasolo and C. A. Bana e Costa, Tailoring value elicitation to decision makers'numeracy and °uency: Expressing value judgements in numbers or words (London Schoolof Economics and Political Science, London, 2011).

51. P. Eklund, A. Rusinowska and H. D. Swart, A consensus model of political decision-making, Annals of Operations Research 158 (2008) 5�20.

52. M. B. Fernandes, M. C. Almeida and A. G. Henriques, Assessing desalination as a sus-tainable alternative using a multiple criteria decision support model, Water Practice &Technology 3(3), doi:10.2166/wpt.2008.078

53. F. A. F. Ferreira, S. P. Santos and P. M. M. Rodrigues, Adding value to bank branchperformance evaluation using cognitive maps and MCDA: A case study, Journal of theOperational Research Society 62(7) (2010) 1320�1333.

54. M. S. M. Figueiredo and M. D. Oliveira, Prioritizing risks based on multicriteria decisionaid methodology: Development of methods applied to ALSTOM Power, IEEE Int.Conf. Industrial Engineering and Engineering Management (Hong Kong, China, 2009),pp. 1568�1572.

55. A. B. Filho, A. S. Marçal, G. Costa, P. R. Pinheiro and R. F. Pinheiro, A novel approachbased on sta® scheduling optimization in information technology projects, InternationalJournal of Computer Science and Network Security 9(9) (2009) 277�286.

56. T. Gurbuz, Multiple criteria human performance evaluation using Choquet integral,International Journal of Computational Intelligence Systems 3(3) (2010) 290�300.

57. Z. Hatush and M. Skitmore, Criteria for contractor selection, Construction ManagementEconomics 15(1) (1997) 19�38.

58. D. P. Henriksen and S. W. Palocsay, An excel-based decision support system for scoringand ranking proposed R&D projects, International Journal of Information Technology &Decision Making 7(3) (2009) 529�546.

59. F. Joerin, G. Cool, M. J. Rodriguez, M. Gignac and C. Bouchard, Using multi-criteriadecision analysis to assess the vulnerability of drinking water utilities, EnvironmentalMonitoring and Assessment 166(1�4) (2010) 313�330.

60. R. L. Keeney, Value-Focused Thinking: A Path to Creative Decisionmaking (HarvardUniversity Press, Cambridge, MA, 1992).

61. R. L. Keeney, Common mistakes in making value trade-o®s, Operations Research 50(6)(2002) 935�945.

62. R. L. Keeney, Eliciting knowledge about values for public policy decisions, InternationalJournal of Information Technology & Decision Making 5(4) (2006) 739�749.

63. R. L. Keeney and H. Rai®a, Decision with Multiple Objectives: Preferences and ValueTradeo®s (Cambridge University Press, New York, 1993).

64. C. W. Kirkwood, Strategic Decision Making: Multiobjective Decision Analysis withSpreadsheets (Duxbury Press, Belmont, CA, 1997).

MACBETH 385

65. R. T. O. Lacerda, L. Ensslin and S. R. Ensslin, A performance measurement framework inportfolio management: A constructivist case, Management Decision 49(4) (2011)648�668.

66. R. T. O. Lacerda, L. Ensslin and S. R. Ensslin, A performance measurement view of ITproject management, International Journal of Productivity and Performance Manage-ment 60(2) (2011) 132�151

67. M. Lauras, G. Marques and D. Gourc, Towards a multi-dimensional project PerformanceMeasurement System, Decision Support Systems 48 (2010) 342�353.

68. E. A. Marks and M. Bell, Executive's Guide to Service-Oriented Architecture (JohnWiley& Sons, Hoboken, 2006).

69. G. Marques, D. Gourc and M. Lauras, Multi-criteria performance analysis for decisionmaking in project management, International Journal of Production Management 29(8)(2010) 1057�1069.

70. R. Mateus, J. A. Ferreira and J. Carreira, Multicriteria decision analysis (MCDA):Central Porto high-speed railway station, European Journal of Operational Research187(1) (2008) 1�18.

71. D. L. McLain and R. J. Aldag, Complexity and familiarity with computer assistance whenmaking ill-structured business decisions, International Journal of Productivity and Per-formance Management 8(3) (2009) 407�426.

72. F. Montignac, I. Noirot and S. Chaurdoune, Multi-criteria evaluation of on-boardhydrogen storage technologies using the MACBETH approach, International Journal ofHydrogen Energy 34 (2009) 4561�4568.

73. J. Montmain, C. Sanchez and M. Vinches, Multi criteria analyzes for managing motorwaycompany facilities: The decision support system SINERGIE, Advanced EngineeringInformatics 23(3) (2009) 265�287.

74. L. Moraes, R. Garcia, L. Ensslin and M. J. Conceição, The multicriteria analysis forconstruction of benchmarkers to support the clinical engineering in the healthcaretechnology management, European Journal of Operational Research 200(3) (2010)607�615.

75. L. C. Nunes, P. R. Pinheiro and T. C. Pequeno, Toward an application to psychologicaldisorders diagnosis, Software Tools and Algorithms for Biological Systems 696 (2011)573�580.

76. M. D. Oliveira, T. C. Rodrigues, C. A. Bana e Costa and A. B. de S�a, Prioritizing healthcare interventions: A multicriteria resource allocation model to inform the choice ofcommunity care programmes, in Advanced Decision Making Methods Applied to HealthCare, eds. E. Tanfani and A. Testi (Springer, forthcoming), pp. 139�152.

77. R. C. Oliveira and J. C. Lourenço, A multicriteria model for assigning new orders toservice suppliers, European Journal of Operational Research 139 (2002) 390�399.

78. L. D. Phillips, Decision Analysis in the 1990s, in Tutorial Papers in Operational Research,eds. A. Shahani and R. Stainton (The Operational Research Society, Birmingham, 1989),pp. 73�90.

79. L. D. Phillips, Decision conferencing, in Advances in Decision Analysis: From Foun-dations to Applications, eds. W. Edwards, R. F. Miles and D. vonWinterfeldt (CambridgeUniversity Press, New York, 2007), pp. 375�399.

80. L. D. Phillips and C. A. Bana e Costa (2007), Transparent prioritisation, budgeting andresource allocation with multi-criteria decision analysis and decision conferencing, Annalsof Operations Research 154(1) 51�68.

81. P. R. Pinheiro, A. K. A. de Castro and M. C. D. Pinheiro, Application multicriteriadecision analysis on TV digital, in Advances in Computer and Information Sciences andEngineering, ed. T. Sobh (Springer, USA, 2008), pp. 39�44.


82. A. Rodrigues, P. R. Pinheiro, M. M. Rodrigues, A. B. Albuquerque and F. M. Gonçalves,Towards the selection of testable use cases and a real experience, in Best Practices for theKnowledge Society: Knowledge, Learning, Development and Technology for All, ed. M. T.Lytras (Springer, Berlin, 2009), pp. 513�521.

83. M. Roubens, A. Rusinowska and H. de Swart, Using MACBETH to determine utilities ofgovernments to parties in coalition formation, European Journal of Operational Research172(2) (2006) 588�603.

84. T. L. Saaty, A scaling method for priorities in hierarchical structures, Journal of Math-ematical Psychology 15(3) (1997) 234�281.

85. T. L. Saaty and S. Mujgan, Extending the measurement of tangibles to intangibles,International Journal of Information Technology & Decision Making 8(1) (2009) (7�27).

86. C. Sanchez, J. Montmain, M. Vinches and B. Mahieu, Planning of maintenance oper-ations for a motorway operator based upon multicriteria evaluations over a ¯nite scaleand sensitivity analyzes, in Informatics in Control, Automation and Robotics, eds. A. J.Cetto, J. L. Ferrier, J. M. C. D. and J. Filipe (Springer, Berlin, 2008), pp. 23�35.

87. R. Sanchez-Lopez, C. A. Bana e Costa and B. De Baets, The MACBETH approach formulti-criteria evaluation of development projects on cross-cutting issues, Annals ofOperations Research (2011), doi:10.1007/s10479-011-0877-4.

88. E. H. Schein, Process Consultation Revisited: Building the Helping Relationship (AddisonWesley Longman, MA, 1998).

89. N. Soguel, M.-J. Martin and A. Tangerini, The impact of housing market segmentationbetween tourists and residents on the hedonic price for landscape quality, Swiss Journal ofEconomics and Statistics 144(4) (2008) 655�678.

90. D. von Winterfeldt and W. Edwards, Decision Analysis and Behavioral Research(Cambridge University Press, New York, 1986).

MACBETH 387

! Macbeth Ijitdm 2012

Documents

Transcript of ! Macbeth Ijitdm 2012