CIO Measurement

8/6/2019 CIO Measurement

1/21

DIPSITD i p a r t i m e n t o d i S t u d i p e rl I m p r e s a e i l T e r r i t o r i o

Objective and subjective measures of CEOs

performance: can economic and managerialliteratures be reconciled?

Anna [email protected]

Working paper n. 5, Ottobre 2005


2/21

2

Abstract

Research on compensation practices of firms has traditionally focused on theobjective criteria of performance evaluation, mainly because of data availability.

Subjective criteria have received minor attention. Nonetheless, many workers areevaluated on the basis of subjective criteria, and the academic work on this topichas provided important insights into how incentives operate, how they translateinto contracts, and how this relates to the overall firms performance. Both

economic and managerial literatures have contributed to the topic, but findings

are sometimes distant or difficult to reconcile. This paper offers a review ofmain, recent contributions in both areas and a guide to their reconciliation.


3/21

3

ANNA MENOZZIFacolt di Economia di Novara dellUniversit del Piemonte Orientale

OBJECTIVE AND SUBJECTIVE MEASURES OF CEOS PERFORMANCE:

CAN ECONOMIC AND MANAGERIAL LITERATURES BE RECONCILED?

SUMMARY: 1. Introduction - 2. Measures of CEOs performance - 3. Objective and subjective criteria in performance evaluation - 4. Are objective and subjective measures of performance substitutes orcomplements? - 5. Conclusions.

1. IntroductionThis study investigate economic and managerial literatures point of view about

objective and subjective measures of performance in reward systems.

Although there are multiple ways to partition performance measures, the most

popular has been between objective and subjective measures. Objective measures are

defined as direct measures of countable behaviors or outcomes, such as productivity,

profit, return on assets, typically taken from externally recorded and audited accounts.

Subjective measures consist of supervisor evaluations of employee performance or

judgments on employees personal characteristics. Although these categories are

somewhat arbitrary, they provide a useful distinction by which research may be

organized and interpreted.There are good reasons why subjective measures of company performance have been

employed (and they might continue to be). Economic and managerial literature agrees

on the fact that subjective measures are cost effective, if compared with the objective

ones: in principle, they can be collected through questionnaire or interview surveys, so

that great numbers of individuals can be contacted at a time. In reality, social scientists

know how difficult is to gather information from people or organizations: the database

construction requires a patient dedication and a continuous follow up. In this respect,

the definition of objective measures based on financial statements may be less costly

and the preference toward subjective criteria is no more justified.

Sometimes subjective measures are the only evaluation criterion. For many publicservice and voluntary sector organizations, or for small enterprises, there are no

appropriate financial records. In other cases, subjective evaluations are the main influent

factor, even when objective criteria are ready to use. In example, when reward takes the

form of promotion, it is difficult to justify it simply on the basis of goals achieved or

numbers obtained. There is no rule (no contract) specifying that to the attainment of a

given result promotion automatically follows. Even in companies where the career path


4/21

4

is quite regularly scheduled over time (think in example in the early stages growthassured by the top consultancy companies, at least up to some years ago), subjective

evaluations play a major role in the decision on who is eligible for promotion and when.

If objective and subjective measures of performance are considered to both matter in

the contractual terms definition, differences among contracts should reflect a diverse

combination of objective and subjective evaluations of individuals.

Reality tells a different history. Managers contracts are compressed into a few

categories and less variable than actual performance. The implementation of a

continuum of different contracts faces evident problems of cost feasibility. Even

disregarding this aspect, there is another reason why contracts are smoother than

individuals performance. Performance evaluations are unable to capture personal

differences among managers, so that performances are actually different, but this

distinction is lost in the appraisal process: managers appear to be more similar than

what they really are when their contracts are signed or revised on the basis of their

performance evaluations.

The economic literature, in particular, is devoted to this aspect. The limited

variability of performance evaluations is a constraint to the optimal contract definition,

which is a common concern for labor and industrial economists, behavioral researchers

and game theorists, among the others. On the other hand, the effectiveness of incentive

contracting in organizations depends on a large set of social, psychological, and

economic factors, only a few of which have been explored by economists.

The managerial literature has concentrated on the measurement of company performance and the relationship between performance and compensation practices,

strategic choices, total quality management, and so on. Managerial, psychological,

sociological literatures have the merit of having introduced in the debate the non-

exclusively economic factors that matter in the incentives mechanism.

The basic motivation of the paper can be resumed in the following questions:

1. How do objective and subjective performance measures affect performance

evaluation?

2. Are objective and subjective measures interchangeable?

3. Can economic and managerial literature be reconciled on the subject matter?

The agenda of the paper is as follows. Section II defines performance measurescontents. Section III reviews some economic and managerial articles illustrating

objective and subjective measures of performance, then answering question 1. Section

III addresses the question whether objective and subjective measures of performance are

substitutes (question 2). The section IV drives the conclusions and answers question 3.


5/21

5

2. Measures of CEOs performance

The measures, objectives and targets used in the evaluation process are strictly

connected to the set of performance dimensions to be evaluated. There are three generic

classes of CEO performance: bottom-line impact, operational impact, and leadership

effectiveness.

An underlying assumption of almost all CEO evaluation and pay-for-performance

plans is that the CEO has a direct and significant impact on corporate performance, i.e.

on firms bottom line. Figure 1 shows the types of bottom-line metrics used to evaluate

the effectiveness of one CEO.

FIGURE 1. BOTTOM-LINE IMPACT DIMESIONS USED

IN EVALUATION OF CEO

These metrics are objective measures of CEO performance: they are derived from

firms financial statements, internal reports, stock market.While CEOs know that it is critical to keep focused on corporate financial success,

these bottom-line measures have evident severe deficiencies as sole indicators of CEO

performance. As the person at the top, CEO recognizes that their ability to affect the

organizations bottom line is not exactly direct and not always overwhelming.

Operational impact refers to the CEOs effect on the companys operational and

organizational effectiveness. Operational impact measures include indicators of

organizational functioning (e.g., retention rates, employee satisfaction, scores),

Capital

Expenditures

Share Price

Earnings pershareNet Income

Net Operating

Revenues

Operating

Cash Flow


6/21

6

operational effectiveness (e.g., quality, rating of products, time to market), and strategicimplementation (e.g. number of acquisitions, total headcount reduction). The

Conference Boards survey of corporate directors (2001) identifies the operational

impact dimensions as the most important in evaluating CEO performance. Figure 2

shows these dimensions.

FIGURE 2. MOST IMPORTANT DIMENSIONS OF

OPERATIONAL IMPACT ACCORDING TO SURVEY OF

CORPORATE DIRECTORS

While still subject to considerable external and internal forces outside of the CEOs

immediate control, this type of performance is more closely related to the CEOs

actions. Similarly to the bottom-line metrics, these indexes may be considered objective,

as long as they imply quantitative measurements.

Leadership effectiveness refers to personal behaviors, which are completely within

the CEOs control. These behaviors include CEOs ability to carry out his or herresponsibilities, such as identifying a successor, meeting with key customers, improving

relationships with external stakeholders, energizing the organization, etc. Figure 3

identifies three key leadership effectiveness dimensions used to evaluate CEOs.

Research anddevelopment

Customer

satisfaction

Companyimage

Company

Morale

Organizationalflexibility and

agility


7/21

7

FIGURE 3. LEADERSHIP EFFECTIVENESSDIMENSIONS

Leads the development of appropriate strategies for

the enterprise; achieves support and commitment for

the strategies from management and the Board of

directors

Guarantees enterprise reputation, ethics, legal

compliance, customer relations.

Works collaboratively with Board members andcommittees, communicates information in a timely

manner to ensure full and informed consent about

matters of enterprise governance

Source: The Conference Board (2001).

These dimensions do not lend themselves to hard, quantitative measurement.

Nevertheless, there are robust methods for reliably and validly measuring these

variables as well. For instance, leadership behaviors can be measured through

behavioral rating methods that ask Board of directors members to indicate the

frequency with which the CEO engages in desired behaviors and to what perceived

effect.The three categories just identified (bottom-line impact, operational impact, and

leadership effectiveness) simply describe CEO performance in generic terms. The

specific dimensions and objectives used in a particular evaluation process will vary for

each company. Is appears clear that the scheme of measures and rewards chosen by a

company will always include, to some extent, both objective (such as bottom-line)

and subjective (e.g. leadership effectiveness) measures of performance. Financial

Board

relationship

Strategic

leadership

Enterprise

guardianship


8/21


9/21

9

only depends upon the principals evaluation. This is consistent with the generalrecommendation in the management literature against the use of self-evaluation to set

compensation, supported by Milkovich and Newman (1999), among the others.

If managers were evaluated on the basis of objective measures solely, the contract

could be precisely designed in order to include (almost) all the possible outcomes and

link them to the corresponding compensation2. Uncertainty increases when subjective

evaluations are relevant to the compensations definition, and the design of an ad hoc

contract becomes more difficult. The problem of the optimal contracts design becomes

even tougher when no objective measures of performance can be applied. In fact, most

managers cannot be evaluated on the basis of an objective criterion because the nature

of their job or the task they cover is not suitable for being exactly judged in terms of

numbers or measurable adjectives.The extent to which the principal is able to reward the agent as a function of a

subjective evaluation depends upon the degree to which the agent agrees with these

evaluations. Managers have their own opinions about their performance. In the

performance assessment process, individuals are asked to judge themselves on the basis

of non quantifiable criteria, and prior to the interview with their supervisor. When the

principals and the agents subjective evaluations concur, then one can implement the

optimal contract, just as if evaluations were objective and verifiable3. On the contrary,

when the principals and agents signals are uncorrelated, the optimal contract

compresses evaluation into two levels acceptable and unacceptable- with only the very

worst performances receiving the unacceptable ranking. The reason of this result lays on

the employees behavior when she feels that her evaluation is unfair. If the employeecannot justify the appraisal she is receiving on the basis of objective criteria, and does

not accept that the non-coded subjective opinion of her supervisor is worsening the

overall evaluations, she might react either by leaving the company, or by changing their

attitude in a negative way. In both cases, the relationship between the principal and the

agent is harmed and the benefit they get from their interaction diminishes. The

consequence is that employers tend to avoid assigning their subordinate extreme

(negative) evaluations. Take on example an appraisal in the rating format: employees

are evaluated on some standard measured on a scale like well above average, above

average, average, below average, well below average, but in case employer and

employees subjective evaluation do not coincide, the well below average rate would

never be used. Individuals in the upper two cases would be pooled together, and the

majority of the employees would receive the highest rating, a result that is consistent

with some of the evidence on performance rankings reported by Milkovich and

2 The problem of moral hazard cannot be eluded, so that even in presence of objective measures ofperformance only the optimal contract cannot be designed.

3 Here optimal contract is intended to be the best among all the possible contracts that can be signed in

presence of asymmetric information.


10/21

10

Newman (1999). The less significant differentiation among performances causes thesmoothing in compensations and the bias introduced in the contracts design.

On the other hands, companies that have shared values of what constitutes good

performance should show less pooling of evaluations, and more effective incentive pay.

In the limit, when the beliefs of the principal and agent are perfectly correlated, there

are no agency costs associated with the use of subjective evaluations: the employer

agrees on the evaluations received and aligns its behavior with the principals

expectations.

McLeods analysis justifies the tendency of employees to agree with the views of

their supervisors, and, on the other hand, the reluctance of supervisors to distinguish

between employees, particularly when it affects compensation, as remarked by

Prendergast (1999).

Baker, Gibbons and Murphy (1994) view subjective assessments for performance as

a means to mitigate the effect of distortionary objective measures. Compensation plans

based on objective measures, like sales or orders, for example, can induce individuals to

modify their behavior or to manipulate their numbers in order to get a higher bonus.

Take in example a company selling medical equipments, in which salespeople have

their variable part of the compensation linked to number of devices sold and installed at

the customers site. Technicians in charge of installation may be tempted or induced to

sign the technical report that guarantees that the equipment is up and running even when

not all the conditions are met. In this case, the firm will probably incur future costs: an

internal audit could detect the improper procedure and impose a sales reversal someperiod after.

The cause of this dysfunctional behavior is not the compensation system per se, but

rather the fact that the compensation system is based on an inappropriate performance

measure. The point is made clear by Baker, Jensen and Murphy (1998). Subjective

evaluations can help in easing the problem, if integrated with the objective ones. In our

examples, salespeople can be judged and in most companies they really are, not only

on the basis of the number of equipments sold but also for their contribution to

customers satisfaction, to marketing, to subordinates growth, etc. The set of skills that

can only be captured by subjective considerations includes many qualities, varying in

a wide range of nuances each. The final appraisal should differentiate among them and

define employees compensation accordingly, in order to induct the right incentives in

any different person.

The peculiar difference between Baker, Jensen and Murphys approach and

MacLeods is that, according to Baker, Jensen and Murphy, the principals subjective

evaluations end up in evaluations of agents subjective characteristics. Subjective

evaluations are viewed as appraisals of non-objective factors, like employees

dedication, passion, trustworthiness. On the other hands, just like in MacLeods model,


11/21

11

the problem of the correct definition of subjective evaluations in contractual termsarises. The employees contribution to firm value is difficult to compute, when objective

measures are considered inadequate or they are not available. The subjective assessment

is assigned to supervisors who are best placed to observe employees behavior and

opportunities. Even if such subjective assessments are imperfect, they may complement

or improve the available objective measures. Thus, an implicit contract based on

subjective performance assessments may augment or replace an explicit contract

based on objective performance measurements. The explicit contract is signed by

parties and can eventually be enforced by a court, while the implicit contract cannot,

so is vulnerable to reneging by the firm. Implicit contracts are sustained by trust among

parties, and by firms concern for its reputation in the labor market in case of unfair

treatment or discrimination against employees.Prendergast and Topel (1996) explicitly say that workers don't trust subjective

performance evaluation when they feel that supervisors indulge in favoritism. This may

happen when supervisor formulates her subjective performance assessment and does not

report it truthfully. The latter result is consistent with the conclusion in MacLeod that

the optimal contract in presence of subjective evaluations can be enforced as long as

employer and firm agree on these evaluations.

The economists interest in subjective evaluation has reduced the gap between

economic and managerial literature. Managerial literature has always emphasized the

importance of psychological factors, such as trust and fairness, in the definition of

incentives. The importance granted to subjective measures of performance is the

consequence of a natural evolution of the theory of compensation toward greaterconsideration for those psychological aspects.

3.2. The managerial approach

Although the traditional management accounting literature advocates the use of

financial performance measures, many writers attribute many problems to the use of

financial performance measures. Ridgway (1956) presents many examples where a

managers performance is maximised in the light of selected performance targets.

However, such high performance does not contribute to the companys overall

objectives. In addition, financial performance measures are frequently criticised on the

grounds that they can lead to many behavioural problems including behaviouraldisplacement, myopia (focusing on achieving results on the short term) and

dysfunctional behaviour in terms of budgetary slack and data manipulation. The bias

introduced in behaviour by performance measures is also described in economic terms

by Holmstrom (1979).

In addition, Eccles and Pyburn (1992) argue that financial performance measures are

lagging indicators since they determine the outcomes of managements actions after a


12/21

12

time period. Therefore, it is difficult to establish a relationship between managersactions and the reported financial results. Also, any corrective action has nothing to do

with the past and it will be difficult to identify which action leads to a particular result.

During the early 90s much of the academic literature claims that traditional

performance measurement is too financially biased, focusing only inside the

organization on cost and budget variance data. The attack to the traditional performance

measurements leads to supplement them with non-financial (and often intangible)

measures. These measures may include measures for productivity market effectiveness,

product leadership, personnel development, employees attitudes, public responsibility,

market share, product development, quality, inventory costs and employee turnover. In

addition, it has been argued that the intensity of competition leads to the use of

sophisticated control and performance evaluation systems (Otley, 1994, among theothers).

In this context, researchers end up developing more rounded and comprehensive

measurement frameworks, such as the balance scorecard (Kaplan and Norton, 1992).

Such a methodology widens the concept of performance measurement by making

executive look externally, at how customer and shareholders see the business, as well as

internally at process performance and the source of innovation and learning. The

balance scorecard initially includes four dimensions: the financial perspective (the

drivers of shareholder value), the customer perspective (the differentiating value

proposition), the internal perspective (how value is created and sustained) and the

learning and growth perspective (role for intangible assets: people, systems, climate and

culture).More recently, the concept of multiple stakeholders comes to the fore. Companies

can no longer be satisfied with only considering shareholders and customers. Employees

are also seen as important stakeholders, as are suppliers, regulators and the community

at large and these stakeholders need to be incorporated into the performance

measurement system. One of the changes is that companies move from purely internally

focused performance measures through multiple dimensional frameworks to having

measurement systems including in their focus the wants and needs of relevant

stakeholders. In addition to that, the emphasis in practices is on the transformations

rather than the individual stock measures. Examples of this second generation

measurement frameworks (Neely et al., 2003) include strategy maps, developed by

Kaplan and Norton (2000), success and risk maps (Neely, 2002), and the IC-Navigator

model (Roos et al., 1997). The detailed explanation of these measures goes beyond the

scope of the article. What is important to know is that all methodologies have a stronger

focus on long-term value creation potential, thus emphasizing subjectivity.

The weakness of the second generation measurement frameworks is their inability to

link the business-orientated methodology to real free cash flow, which is the current

cornerstone of market valuation. That is why there is a call, in management research, for


13/21

13

the venue of a new generation of performance measures, which would seek greaterclarity about the linkages between the non-financial and intangible dimensions of

organizational performance and the cash flow consequence of these.

Neely et al. (2003) list the new performance measures challenges (p. 135):a) Models must reflect the static and dynamic realities of organizations but at the same

time not lose appropriateness as a managerial tool;

b) We must move from data to information and must provide rigorous information

especially for the intangible value drivers in organizations;

c) The models must be practical and aligned with other organisational processes in

order to allow actions to be taken;

d) We must seek increasingly robust ways of demonstrating the cash flow implications

of the non-financial and intangible organisational value drivers.These measures may give a better indication of a companys underlying potential to

create value, rather than looking to, for example, the immediate stock price, which is

subject to market-wide volatility. The past obsession with pure financial performance is

now decreasing and there may be a recognition that there is a trade off between hitting

todays financial results and sustaining the capabilities and competences that allow

companies to compete effectively in the future.

Clearly identified measures for each objective greatly facilitate the sharing of

performance expectations between CEO and stakeholders. Nevertheless, clarity on

performance dimensions and objective helps, but does not guarantee, the

implementation of an effective CEO evaluation. In the early stages of designing a CEO

evaluation, a debate over the appropriate criteria for assessing performance is probable(and healthy), indicating that the relevant stakeholders are involved and are thinking

carefully about the process. Before the process can be enacted however, the CEO and

the Board in charge of CEOs evaluation must agree that the dimensions of performance

and objectives used are the right ones. The importance of agreement on performance

measures, so deeply stressed in the economic literature, is cited but not analytically

treated in the managerial contributions. Managerial literature should borrow the detail of

analysis and the results obtained from the economic literature, so that the managerial

studies may gain in validity.

4. Are objective and subjective measures of performance substitutes orcomplements?

In the last ten years the dominant trends in compensation have centered on finding

ways to enhance competencies in employees. A merit pay system links increases in base

pay (called merit increases) to how highly employees are rated on a subjective

performance evaluation. Merit increases may be linked to employees ability and

willingness to demonstrate key competencies. In the previous paragraph we mentioned


14/21

14

salespeople contribution to customer satisfaction and marketing. An example of a possible set of subjective criteria for measuring competencies in customer care is

illustrated in Table 1.

TABLE 1. Competency: Customer Care

1. Follows through on commitments to customers in a timely manner

2. Defines and communicates customer requirements

3. Resolves customer issues in a timely manner

4. Demonstrates empathy for customer feelings

5. Presents a positive image to the customer

6. Displays a professional image at all times

7. Communicates a positive image of the company and individuals to customers

Source: Milkovich and Newman (1999).

More recently, while economic literature has renewed its interest in subjective

measures of performance, managerial literature has started putting merit pay under

attack.

The main concern is in fact an old one: merit pay is unable to solve the problem of

optimal contract definition and does not guarantee the achievement of the desired goal

improving employees and firms performance. Already in 1992, Heneman had warned

about merit pay limitations and advocated the improvement in accuracy of

performance ratings, the allocation of merit money to truly reward performance and the

guarantee that the size of the merit increase differentiates across performance levels.

Even worse, incentive plans can lead to unexpected and undesired behaviors. We

already provided an example of this conflict in the previous section. What matters

remarking here is that employees and managers (or managers and firms) end up in

disagreement because the incentive system focuses only on one part of what the

company believes important to be successful. Employees, being rational, do more of

what the incentive system pays for.Bommer et al. (1995) give an important imprimatur to the debate. They think that

the objective/subjective distinction has been given too much attention, at the expense of

examining the construct validity of performance measures. The risk is to consider

objective and subjective criteria as substitutes before attesting their actual

interchangeability. In their meta-analytic analysis, the authors consider managerial

ratings as subjective measures on one hand, and objective employee performance

measures, on the other.


15/21

15

The economic studies we reviewed in the previous paragraph predict that objectiveand subjective measures can indeed be considered substitutes when trust or shared

values characterize the principal-agent relation. Bommer et al. suggest instead that the

consequent reference to a broad performance construct might be misleading. If the

population correlation between objective and subjective performance measures is of a

magnitude suggesting convergent validity, then the measures may be combined largely

without incident.

The correlation between the two types of measures can vary accordingly to specific

situations or in presence of peculiar factors (moderators). In example, the authors

expect objective and subjective measures to be more strongly related in samples using

salespeople than in samples using other functions employees. In sales samples

objective performance measures are generally easily assessed and readily available, providing sales managers with the information necessary to evaluate their personnel.

Salespeople are traditionally evaluated on sales output, so managers are likely to

consider it when evaluating performance.

Analogously, we might expect that objective and subjective indicators are more

strongly related in samples using a quantitative objective measure than in samples

utilizing qualitative measures. The objective measures content is related to job type, but

it goes beyond a distinction based strictly on sales or non-sales samples. Production

quantity information is likely to be more easily obtained than qualitative measures, and

the higher frequency of the quantity-related behaviors should make them easier to

observe.

Bommer et al. meta-analysis results in an overall mean correlation betweenobjective and subjective performance measures of 0.389, suggesting that the measures

are not interchangeable. The second important result is that one moderator affects the

strength of the relationship between objective and subjective measures: indeed, this

connection is stronger when the objective measure assesses performance quantity, rather

than performance quality. The third finding concerns the relationship between objective

and subjective measures when they refer to a unique performance construct: when

samples are restricted to meet this requirement, it appears that the measures are

reasonably substitutable.

The practical implication of Bommer et al. model is that, at the most basic level,

subjective measures should not be used as proxies for objective measures, if objective

performance is the behavior of interest. For example, if sales are the desired outcome,

organizations should not reward employees based on a supervisors overall performance

evaluation of that employee. Conversely, if a more broadly defined performance is

considered more important, it is equally inappropriate to reward employees solely on

gross sales. Although a universal substitutability is not advised on the basis of this

model, a restricted substitutability is acceptable. In case of production quantity, the

problems associated with using proxy measures may not be severe. This is also true for


16/21

16

other cases where it is the precise construct that is being measured through multiplemeans, i.e. through both objective and subjective measures.

A very recent study from Wall et al. (2004) goes deeper in the analysis of

performance indicators validity. Their primary concern is to address the threat to

validity that comes from potential error in subjective measures. Two types of errors are

particularly significant. First, if subjective performance measures contain random error,

for example because respondents remember figures incorrectly or because they refer to

the wrong period, then the effect will be to attenuate any real underlying relationship

with associated variable of interest (Type II error, or false negative). More severe is

the possibility of systematic bias creating relationships between practices and

performance that do not really exist (Type I error, or false positive). Authors lamentthat although recent research has begun to address the question of the validity of

subjective measures in the human resources management area, it has mainly focused on

the measurement of practices.

Validity requires not only objective and subjective measures of the same

performance construct to be associated (convergent validity), but also that their

association should be stronger than that between measure of related but different

constructs using the same method (discriminant validity). For example, the

relationship between objective and subjective measures of productivity, and between

objective and subjective measure of profit, should be stronger than the relationships

between subjective measures of productivity and profit, or between objective measures

of productivity and profit. In other words, different ways of measuring the sameconstruct should show stronger correspondence than the same way of measuring

different constructs as advocated by Bommer et al..

A second way of increasing the understanding of subjective performance measures

is to examine their construct validity. Even for a subjective performance measure with

good convergent validity, the amount of unique variance in that measure leaves plenty

of scope for relationships to be found with other variables. It is important to determine

not only whether objective and subjective measures of performance are associated with

one another, but also whether they are associated with other variables of interest in the

same way. Our initial question Are objective and subjective measures

interchangeable? can be reformulated as Does the use of a measure of subjective

performance lead to different findings than those based on objective measures of

performance? If there is no such difference, we can be more confident that the

subjective performance does not lead to erroneous conclusions. On the basis of two

samples comprising, on one hand, 80 single-site UK manufacturing companies and, on

the other, both single- and multi-site publicly quoted UK companies with more than 50

employees, Wall et al. demonstrate that convergent, discriminant and construct validity

are evident.


17/21

17

There are three main implications of Wall et al.s research. First, the degree ofequivalence between the findings for objective and subjective measures supports the use

of subjective performance measures in future studies where objective ones may not be

feasible or for certain levels of analysis that do not have separate objective measures.

The second implication is that there is a need to improve both types of measure.

Objective measures can be ameliorated by investigating the accounting decisions in any

given reporting period and eventually making convenient adjustments. Subjective

measure can be systematically improved by, in example, asking specific questions that

directly correspond to the objective performance index of interest, or investigating the

relative validity of subjective measure of different kinds, or also by providing

respondents with prior warning of the areas to be covered, or focusing on a particular

category (finance managers or CEO, e.g.). The third implication is that, when possible,investigators should use both objective and subjective measure of performance within

studies. Given each type of measure will contain its own error, more reliable estimate of

performance may be obtained by combining them. At the same time, equivalent results

from both types of measure would add credibility to any substantive findings, and

differential findings alert investigators of underlying problems.

5. Conclusions

The results from Wall el al. (2004) show that the indicators used to measure a

particular performance aspect suggest the types of error that would probably occur.

Ratings are subject to numerous well-documented sources of systematic bias andrandom error. Subjective evaluations are non-contractable, then any distortion

deliberately introduced by the evaluator may provoke a deviation from the optimal

contract. Table 2 illustrates some of the recurrent errors in the appraisal process.

TABLE 2. Common errors in the appraisal process

Halo error An appraiser giving favorable ratings to all job duties

based on impressive performance in just one job

function.

Horn error The opposite of a halo error. Downgrading an employee

across all performance dimensions exclusively because

of poor performance on one dimension.

First impression error Developing a negative or positive opinion of an

employee early in the review period and allowing that to

negatively or positively influence all later perceptions of

performance.


18/21

18

Recency error The opposite of first impression error. Allowing performance, wither good or bad, at the end of the

review period to play too large a role in determining an

employees rating for the entire period.

Leniency error Consistently rating someone higher than is deserved.

Severity error The opposite of leniency error. Rating someone

consistently lower than is deserved.

Central tendency error Avoiding extremes in ratings across employees.

Clone error Giving better ratings to individuals who are like the

rater in behavior and/or personality.

Spillover error Continuing to downgrade an employee for performanceerrors in prior rating periods.

Source: Milkovich and Newman (1999).

Objectives measures tend to be less prone to bias and random error, but they are not

a panacea. Performance constructs that can be measured by objective means tend to be

narrowly focused and are typically representative of low-order factor structures, as

stated by Gibbs, Merchant, and Vargus (2004), so that both theoreticians and

practitioners should not rely solely on objective measures for their supposedly superior

measurement properties.Many firms are implementing compensation plans that supplement financial metrics

with additional measures in order to assess performance dimensions that are not

captured in short-term financial results. These additional measures can take a variety of

forms, ranging from quantitative, nonfinancial metrics, such as employee and customer

survey results, to qualitative assessments of performance by the managers superior.

One critical implementation issue that arises when incorporating multiple

performance measures in reward systems is the determination of relative weights to

place on the various measures when determining compensation, as considered by Ittner

et al. (2003).

The other important question, which was more extensively treated here, is their

contingent effect of different kind of performance measures on the evaluation and thesuccessive rewards determination.

Both economic and managerial literature has provided an answer to this concern.

Economics, in particular, admits that subjective measures may substitute for the

objective ones when agents personal beliefs agree on the appraisal given by the

principal.


19/21

19

Managerial literature offers a more extensive answer to the dilemma. The authorswe reviewed here study the interrelation between objective and subjective factors and

give indications on the ideal conditions under which they can be considered

interchangeable. In agreement with the economic literature, there is a role for the

substitutability of objective and subjective measures of performance. Nevertheless, the

general recommendation is the analysis of each contingent situation so that the validity

of both the type of measures is not prevented or harmed. In line with the general

practice, objective and subjective evaluations should both enter in the compensation

definition. The close observation of results and the repetition of preventive experimental

studies may help improving the capacity to detect errors and biases compromising

objective and subjective measures validity.

Researchers should remember that it is better to imperfectly measure relevantdimensions than to perfectly measure irrelevant ones.


20/21

20

Riferimenti bibliograficiBaker, J., Gibbons, R., Murphy, K.J., 1994. Subjective performance measures in

optimal incentive contracts. Quarterly Journal of Economics, 109(439), 1125-56.

Baker, J., Jensen, M.C., Murphy, K.J., 1998. Compensation and incentives: practice

vs. theory. Foundations of Organizational Strategy, Michael C. Jensen, Harvard

University Press.

Bommer, W.H., Johnsons, J.L., Rich, G.A., et al., 1995. On the interchangeability of

objective and subjective measures of employee performance: A meta-analysis.

Personnel Psychology, 48, 587-605.

The Conference Board, 2001. Compensation committee of the board: best practices

for establishing executive compensation. Report No. R-1306-01-RR.

Eccles, R., Pyburn, P.J., 1992. Creating a comprehensive system to measure

performance. Management accounting (UK), 74(4), October, 41-44.

Gibbons, R., Murphy, K.J., 1990. Relative performance evaluation for chief

executive officers. Industrial and Labor Relations Review, 23(1), 116-155.

Gibbs, M., Merchant, K.A., Vargus, M.E., 2004. Determinants and effects of

subjectivity in incentives. The Accounting Review 79(2), 409-436.

Heneman, R., 1992. Merit pay: Linking pay increases to performance ratings.

Reading, MA: Addison-Wesley.

Holmstrom, B., 1979. Moral hazard and observability. Bell Journal of Economics

10(Spring), 74-91.

Ittner, C.D., Larcker, D.F., Meyer, M.W., 2003. Subjectivity and the weighting of

performance measures: evidence from a balanced scorecard. The Accounting Review

78(3), 725-754.

Kaplan, R.S., Norton, D.P., 1992. The balanced scorecard measures that drive

performance. Harvard Business Review, September/October, pp. 167-176.

Kaplan, R.S., Norton, D.P., 2000. The strategy focused organization: how balanced

scorecard companies thrive in the new business environment, Boston 2000.

MacLeod, W.B., 2003. Optimal contracting with subjective evaluation. American

Economic Review 93(1), 216-240.


21/21

Milkovich, G.T., Newman, J.M., 1999. Compensation. Chicago: Irwin.

Neely, A., Adams, C., Kennerley, M., 2002. The performance prism: the scorecard

for measuring and managing business success. London.

Neely, A., Marr, B., Goran, R., Pike, S., Gupta, O., 2003. Towards the third

generation of performance measurement. Controlling, Heft 3/4, Marz/April 2003.

Otley, D., 1994. Management control in contemporary organizations: towards a

wider framework. Management Accounting Review, 5, 289-299.

Prendergast, C. 1999. The provision of incentives in firms. Journal of Economic

Literature XXXVII, 7-63.

Prendergast, C., Topel, R.H. 1996. Favoritism in organizations. Journal of Political

Economy, 104(5), 958-978.

Ridgway, V.F., 1956. Dysfunctional consequences of performance measurements.

Administrative Science Quarterly, 1, 240-247.

Roos, J., Roos, G., Dragonetti, N.C., Edvinsson, L., 1997. Intellectual capital:

navigating the new business landscape. London.

Wall, T.D., Michie, J., Patterson, M., et al., 2004. On the validity of subjective

measures of company performance. Personnel Psychology 57(1), 95-118.

CIO Measurement

Documents

Transcript of CIO Measurement