1

Measuring & Managing Performance

EECS811 : SW Project ManagementPresenter : Amit Dandekar

Instructor : Dr. Hossein Saiedian

2

Contents

• Introduction• Measurement dysfunction defined• Measurement dysfunction model• Designing measurement system• Designing incentive system• Management by delegation• Popular measurement models & dysfunction• Conclusion

3

Introduction

• Measurement: The Good– always a constructive practice– no downside– requires little or no justification– popular belief

• Measurement: The Bad – most things extremely difficult to measure– enormous cost of measurement mechanisms– can inhibit productivity– unconstructive uses possible

4

Introduction

• Measurement: The ugly– dysfunction !!!– measurements unknowingly put to

unconstructive use– measurements do more harm than good– distorted incentive system

5

Measurement dysfunction defined• Measurement way it is suppose to be

– measure <parameter> in hopes of improving <goal>

– parameter gets better as goal comes near

• Measurement dysfunction– parameter gets better even if goal stays as

distant– this becomes a rule and not just an anomaly– measured standards actually interfere with

achievement of intended goals

6

Measurement dysfunction defined

Time

Level of performance

True performance

Measurement Indicators

7

Characteristics of measurement dysfunction• Key measure do not provide clear account of

performance• Substitute characteristics used as indicators

• measurement of timeliness of delivery as project success

• Assuming bi-directional nature of correlation between indicator and performance

• measurement standard based on physical appearance

• Measured standards interfere with achievement of intended effect

• Employee rewards not associated with performance of organization

8

Measurement dysfunction model• 3 roles

– principal– agent– customer

• Principal– typically a manager– motivated by profits

9

Measurement dysfunction model• Agent

– typically an employee– motivated by fondness for money– dislike for work– possibly by internal motivation

• Customer– purchaser of services generated– final judge of the value of efforts

10

Measurement dysfunction model• Principal cannot determine value of agents

production directly– must measure proxies of value

• Agent & principal have opposite interests– principle wants max value for min money– agent wants max money for min works

11

Measurement dysfunction model• Apply this model to a typical scenario

– DOD contracted a firm to develop software– DOD expects thorough unit testing to ensure

quality– roles defined per model

• principal: manager from the firm• agent: development engineer at the engineer• customer: DOD

– no of lines of code written is measurable – code coverage achieved via unit tests is not

easily measurable

12

Measurement dysfunction model• What DOD wants ?

– more effort on coding is better– more effort on testing is better– coding and testing , both activities are critical

13

Measurement dysfunction model• Ways engineer can allocate his effort

capacity• DOD’s most preferred allocation

14

Measurement dysfunction model• Effect of increasing engineers effort

capacity• DOD’s preference: best mix path

15

Measurement dysfunction model• Possible ways in which engineer can

achieve increased target level set by manager

16

Measurement dysfunction model• Can engineer meet higher

productivity targets by not achieving complete code coverage in unit tests?– how closely engineers activities are being

monitored ?– can manager measure code coverage achieved

by unit tests ?

• In most cases measuring engineer’s activities fully is not possible– too costly or impossible.

17

Measurement dysfunction model• Factors that affect measurement costs

– repetitiveness of the task• offer more opportunities for observing all phases of

agent’s work• permit use of statistical sampling techniques• inexpensive measurement enabling full supervision

– complexity of task• overtaxes principal’s cognitive capabilities• simple tasks easier to become fully supervised

18

Measurement dysfunction model• Factors that affect measurement costs

– newness of task• principal more likely to be familiar with older tasks• older tasks provide more measurement opportunity

over time• more likely to become fully supervised

– specialized knowledge requirement• difficult/expensive to find someone qualified to do

measurement & interpret results• typically include tasks that are pushing technology

beyond boundary• full supervision unlikely to be achieved

19

Measurement dysfunction model• Factors that affect measurement costs

– interdependence and in-separable effort• tasks of numerous workers tightly coupled• all have ability to hinder others tasks• impossible to distinguish how much effort each

member is contributing• full supervision unlikely

– environmental covariates & noise• outside factors affect agent’s performance• external random factors for which no measure

available affect performance• full supervision unlikely

20

Measurement dysfunction model• Measurement costs dictate supervision

model• Three ways of supervision based on

measurement costs– no supervision – full supervision– partial supervision

21

Measurement dysfunction model• No supervision

– nothing observed on effort allocation• impossible to measure / measurement cost too high

– agent does useful work due to internal motivation only

– payment not contingent upon performance– effort distribution along best path – typically short of customer expectation

• because agent dislikes work

22

Measurement dysfunction model• No supervision

23

Measurement dysfunction model• Full supervision

– principal can measure every critical dimension of performance

– principal sets target at ideal point on best mix path

• agent will meet target exactly– incentive distortion unlikely

• effort increase can be forced to be exactly on best mix path

– ideal but very uncommon in real life• finding and measuring all critical dimensions not easy• measure infeasible / too expensive

24

Measurement dysfunction model• Partial supervision

– most likely real world scenario– some but not all critical dimensions measured

• Dysfunction likely– incentive system likely to induce distortion in

effort mix– value to customer reaches a point where its

same /lower than unsupervised value– thus agent is given incentive to deliver less

values to customer – dysfunction !!!

25

Measurement dysfunction model• Partial supervision

26

Measurement dysfunction model• Applying model to our case study

– measuring code coverage achieved by unit tests is too expensive

– only partial supervision possible– incentive provided to engineer to write more code– engineer changes effort allocation changed to write more

code and less tests– engineer can easily slip under radar without writing

adequate unit tests – value to customer reaches a point where its same /lower

than value without additional incentive– engineer given incentive to deliver less values to

customer – dysfunction !!!

27

Designing measurement system• Comprehensiveness is desirable

– must measure all critical dimensions– essential to prevent dysfunction– difficult to verify performance of knowledge workers– quality is notoriously difficult to measure

• Motivational uses of measurement– agent can not be prevented from reacting to measure– results in competitive dynamics that distorts measure– Often causes warning signs to be concealed– extremely difficult to achieve informational only use

28

Designing measurement system• Use measurements for information only

– convince agents that measure is informational only

• promote people longer time• no sudden rising stars• reduces competition within employees

– aggregation of measurement• individual see measures at their or higher level • need procedural methods to enforce aggregation• gives manager a self assessment• managerial job shifts from evaluation to inspiration

and assistance to subordinates

29

Designing incentive system

• Identify mode of supervision possible– dictated by structure of measurement costs

Critical dimensions measured

Mode of supervision Target setting

All Full Maximize all measured dimensions

Some Partial Moderate Level,Agents discouraged from exceeding target performance level

None Unsupervised No target setting

30

Designing incentive system

• Common mistake– assuming full supervision and setting targets too

far out• Most experts agree that comprehensive

measurement impossible• Most of the times only partial supervision

possible• How to achieve better than moderate

performance ?– internal motivation & management by

delegation

31

Introducing internal motivation

Internal motivation External motivationBenefits include feeling of fulfillment,gratification from sharing organizational success

Incentives are external and of material nature

Considers effort on work as natural as play and rest

Workers dislike work

Self direction , self control to achieve objectives

Coerced, directed, threatened to achieve organizational objectives

Managers use communication, inspiration to motivate

Managers concerned with design & implementation of external motivation schemes

32

Management by delegation

• Management by delegation– management style that relies on internal

motivation– agents preferences changed as compared to

management by measurement– unsupervised effort allocation

• always moves along best mix path• dysfunction can not occur

– in conflict with measurement based management

• principal must choose between the two

33

Management by delegation

• Management by delegation involves costs– cost benefit analysis needs to be done

• Delegation costs– delegation involves changes in weighing of

agents utility components– needs taking time out of production activities

to promote group identity– extending lifetime employment

34

Management by delegation

• Factors that affect delegation costs– organizational size

• large organizations are less personal than small due to sheer number of employees and size

• difficult to inspire organizational identification and loyalty

– public or private organizations• employees of public bureaucracies less motivated• higher delegation costs in public organizations

– cultural homogeneity of organization• organizations with large cultural diversity have higher

delegation costs• difficult to align objectives of firm with employees

35

Management by delegation

• Factors that affect delegation costs– duration of relationship between principal and

agent• longer relationship will develop long term

relationships• principal and agents objectives can get closely linked

– frequency of interaction between employees• frequent interactions within employees help

organizational identification• delegation costs lower in geographically less

dispersed organizations– organizational prestige

• organizational identification easy to cultivate with firms that are widely known

36

Management by delegation

• Factors that affect delegation costs– extent to which employee needs are met

• non-material needs for companionship, quality of life• employee satisfaction with work environment lower

delegation costs

– Perceived level of mutual commitment• easier to cultivate loyalty if organizations takes

responsibility for employee well being• Japanese practice of assuring lifetime employment

greatly increases loyalty

37

Management by delegation

• Recommended management models

NoSupervision

Partial Supervision

Full Supervision

Best-case Full Supervision

Low delegation cost

Delegation Delegation Delegation or Measurement

Measurement

High delegation cost

Nothing works

Measurement with precaution against dysfunction

Measurement Measurement

38

Management by delegation

• What to do when nothing works ?– no supervision possible– delegation costs too high

• Convert the situation to manageable one– make delegation appropriate– make measurement appropriate

39

Management by delegation

• How to make jobs more measurable ?– standardization

• almost all processes repetitive at some level of abstraction

• identify phases and establish standard methods of execution

• decide appropriate product properties and agent behaviour at a certain stage

• e.g. 4 phases of RUP

40

Management by delegation

• How to make jobs more measurable ?– specification

• construct detailed model of the process• standardise every step in a process• variances from specification can be noted at every step

– subdivision• decompose jobs into subtasks and group similar tasks• grouping makes repetition & self similarity visible

making measurement easier• people doing similar activities can be assigned

overseers that have specialized knowledge as workers

41

Management by delegation

• How to make situation suitable to delegation?– promoting organizational intimacy

• make hierarchy flatter & work groups smaller• reconfigure work spaces to promote casual contatcs

between managers and workers in same team– promoting trust

• facilitate mutual trust between employer and employees

• introducing mutual vulnerabilities into relationship• investments that suggest permanence of employee

base such as investing heavily in training

42

Management by delegation

• How to make situation suitable to delegation?– better leadership

• skilful expression of vision• management styles that emphasize inspiring and

trusting employees rather than analyzing and coercing

43

Popular measurement models & dysfunction• Popular measurement systems

– Malcom Baldrige quality award– ISO 9000 certification– SEI SCE– Goal Question Metrics (GQM)

• Malcom Baldrige quality award– NIST designed award scoring system– 7 category, 1000 point, 3 level judging– 75% points allocated to process criteria– Only 25% to actual product quality– Site visits by experts and interviews by experts

44

Popular measurement models & dysfunction

• ISO 9000– five quality standards set by ISO– ISO establishes requirements for production

systems• not product standard

– documents all processes comprehensively– document what you do and do what you

document– ISO certification required to maintain market

access

45

Popular measurement models & dysfunction

• SEI SCE– Determine organization’s ability to deliver

particular product • on time• on budget• With high quality

– uses standard questionnaires & interviews– standard scoring algorithm used– Limited empirical data showing relation

between maturity levels and quality of product

46

Popular measurement models & dysfunction

• GQM (Goal Question Metric)– fashionable in software metrics circles– summary of six-step GQM process

• develop set of business and associated measurement goals

• generate questions to define those goals as completely as possible

• associate set of metrics with questions to answer questions in a measurable way

– advocates measuring high profile areas associated with organizational goals

– not concerned with making measures comprehensive

47

Popular measurement models & dysfunction

• Similarities among these popular models– audit from external examiner– measure process characteristics than product– standard and publicly known scoring algorithm– each firm has tremendous incentive to achieve

favourable result– each employ a measurement based control

system

48

Popular measurement models & dysfunction

• Problems with the popular models– cover only certain critical dimensions

• likely to lead to dysfunction– rely on point in time measurement

• firms learn how to exploit program measures• firms ramp up to corrupt one time measure

– examiners often turn consultants• conflict of interest

– costs do not necessarily outweigh benefits– each of these popular systems can lead to

measurement dysfunction !!!

49

Conclusion

• Measurement dysfunction always possible– comprehensive measurement extremely

difficult

• Never use measurement to monitor individual performance– individuals always react to it

• Internal motivation is a viable mean to achieve satisfactory effort levels– external motivation works only in limited

environments

50

References

• R. D. Austin, “Measuring and Managing Performance in Organizations”, Dorset House Publishing Company, 1996.