ITIL KPIs and the IT Balanced Scorecard

ITIL Key Performance Indicators and the IT Balanced

Scorecard

IT-Management 2

Supervisors: Prof. Dr. Andreas Breiter, Dipl.-Inf. Jörg Hofmann 28 August 2009

Tobias Hildebrandt Master Digitale Medien, Fachbereich 3

Universität Bremen Matrikelnummer: 2337614

[email protected]

Abstract This paper deals with the IT Service Management Framework ITIL and the business

performance measurement method Balanced Scorecard and its IT equivalent, the IT

Balanced Scorecard. The Balanced Scorecard aggregates low level measures, which

are called KPIs. Usually there are relations between these measures, which are

called cause-and-effect relationships. This paper tries to find out what features

current software implementations of the Balanced Scorecard and of the IT Balanced

scorecard offer, especially in regards to cause-and-effect relationships.

Keywords : ITIL, Performance measurement, Key Performance Indicator (KPI),

Balanced Scorecard (BSC), IT Balanced Scorecard (IT BSC), IT Management, IT

Service Management, IT Governance

Table of Contents 1 Introduction ........................................................................................................................ 1

2 Balanced Scorecard............................................................................................................ 3

2.1 Motivation ................................................................................................................... 3

2.2 Concepts ...................................................................................................................... 4

3 IT Balanced Scorecard ....................................................................................................... 8

3.1 Differences to the Business Balanced Scorecard ........................................................ 9

3.2 Considerations for implementation ........................................................................... 10

4 Linking ITIL Key Performance Indicators and the IT Balanced Scorecard .................... 11

4.1 ITIL Key Performance Indicators ............................................................................. 11

4.2 Combining ITIL Key Performance Indicators with the IT Balanced Scorecard....... 11

4.3 Translating the strategy ............................................................................................. 13

5 Cause-and-effect Relations .............................................................................................. 16

5.1 State of the art in research ......................................................................................... 17

5.2 Software implementation .......................................................................................... 18

5.3 How a software IT BSC could look like ................................................................... 27

6 Conclusion........................................................................................................................ 29

7 Sources ............................................................................................................................. 31

List of abbreviations BSC Balanced Scorecard CIO Chief Information Officer CobiT Control Objectives and Related Technology IT Information Technology IT BSC IT Balanced Scorecard ITIL IT Infrastructure Library KPI Key Performance Indicator OGC Office of Government Commerce

Introduction

1

1 Introduction The ITIL (Information Technology Infrastructure Library) framework is an approach to

IT Service management and according to [Th@] also the most widely accepted one.

It was developed by Great Britains’ Office of Government Commerce (OGC) and is a

documentation of best practices for IT Service Management. It consists mainly of a

series of advising books. The ITIL framework has been revised since its initial

development in the 1980s, the current version as this paper is being written is

Version 3 which was released in 2007. The OGC claims, that using ITIL might

provide benefits like reduced costs, improved IT services and improved customer

satisfaction. It provides best practices for all of the lifecycles of an IT service, which

according to [Th@] are Service Strategy, Service Design, Service Transition, Service

Operation and Continual Service Improvement. This paper will mostly deal with the

Continual Service Improvement phase.

Since its first introduction the ITIL framework has become more and more accepted,

being the most used framework for IT service management and a de facto standard

in this area nowadays. This success might be supported by the shift from IT

Management to IT Service Management.

While IT Management was more concentrated on infrastructure and the operational

level of IT, IT Service Management focuses on the services that are provided to its

customers. In the course of this shift, new methods were needed to cope with these

changes, and the ITIL framework was able to provide these methods. At the same

time, the importance of the alignment of business and IT increased, which made the

role of the IT Manager more important, leading to the nowadays common role of the

CIO (Chief Information Officer), who is often a member of the board of executives.

The next shift of roles and mindsets in IT that is frequently discussed is the IT

Governance, which is even more business oriented than IT Service Management.

While IT Service Management can still be considered to be in the IT domain, the IT

Governance responsibilities are to be found almost completely in the business

domain. According to [Va07] the ITIL framework is clearly a management framework,

and therefore it does not cover IT Governance – in contrary to other frameworks like

for example CobiT, which are explicitly frameworks for IT Governance.

Introduction

2

What ITIL is missing however is a business-centered way of controlling the IT and

evaluating its support in reaching the business goals. These aspects can be covered

by the IT Balanced Scorecard, which can be used to link the business goals and IT

Service Management. According to [Sa04@], the IT Balanced Scorecard is a method

that can be used to support IT Governance.

This paper tries to answer the question “What are the current features of (IT)

Balanced Scorecard software regarding cause-and-eff ect relationships

between measures in general and ITIL Key Performanc e Indicators in special

and how can they be improved? ”

In order to answer this question, firstly the concepts and methods of the Balanced

Scorecard (BSC) are explained in chapter 2, which is followed by remarks on the IT

Balanced Scorecard (IT BSC) in chapter 3. Afterwards methods to combine ITIL and

the IT BSC are discussed in chapter 4, which is then follow by extensive

considerations on cause-and-effect relationships in theory and in software products

(in chapter 5). This paper ends with the conclusion.

Balanced Scorecard

3

2 Balanced Scorecard The BSC is a method for strategic business performance measuring that was

suggested by Robert S. Kaplan and David P. Norton in 1992. Since then it received a

lot of attention by managers as well as by scientists of different sectors. These

sectors include big companies like banks, insurance companies or software

manufacturers, but also smaller or governmental enterprises, like for example

hospitals. This chapter first tries to explain the motivation that led to the development

of the BSC, and then shows the underlying concepts.

2.1 Motivation Historically, business performance measurement was based mostly on financial

techniques and measures like budget planning, Return on Investment (Ratio of

money earned or lost on an investment), Cash flow (movement of cash into or out of

a business) or earnings per share (Earned money divided by the number of shares)

[Ba01@].

The concentration on these aspects may lead to very short term thinking – usually

managers plan ahead at most until the next fiscal year, where a new budget plan is

being created. Long term goals and developments may be neglected by this

paradigm. This kind of thinking may also lead to actions and decisions which can

help in the short term, but impede long term improvements. One example might be

expenditures on employee trainings. When just focussing on “traditional” financial

performance measuring, a manager will, in many cases, come to the conclusion to

cut expenditures in this area – in doing so he will better be able to keep his budget

plan for the current year. And when a Return on Investment analysis on expenditures

on training is being calculated, there might not be any measurable or predictable

“return” at all, let alone in the current fiscal year. This is because it is very difficult to

calculate the detailed benefits of an improved employee training. But in cutting this

expenditure there might be a lot of additional costs in the future years coming up, for

example a lower productivity or a decrease in customer satisfaction due to a worse

service.

This is where the BSC comes into play. It tries to help to see a business from a more

“integrated” perspective, and also aims to put performance measurement on a more

long term strategic level.

Balanced Scorecard

4

Its’ basic concept is to select a hand full of goals from the following areas, which are

called “perspectives”: user orientation, business contribution, operational excellence

and future orientation (see figure 1).

Each of these perspectives has a mission. One example for such a mission might be

“be the supplier of choice for our customers” for the customer perspective. Besides

the overall mission, each perspective is made up of several subgoals that support the

mission. For each of these goals it has to be defined a way to measure the state of

their completion. In this example “reach a high customer satisfaction” might be a

good goal to reach the mission of the customer perspective. In this example “Score of

evaluation by customer” might be a way to measure this goal. [Ka92]

Figure 1: The four perspectives of a Balanced Scorecard

It is very important that these values on the lowest hierarchy level of the BSC are

actually measurable.

2.2 Concepts One of the reasons why the concept of the BSC is so successful may be that it claims

to help seeing a business from an integrated meta-perspective. It especially is

intended to help to see interconnections between areas of business (the

perspectives), which is illustrated by figure 2.

Balanced Scorecard

5

Figure 2: The interconnections of a Balanced Scorecard

One area that is often discussed about the BSC is the finding of cause-and-effect

relationships between the perspectives in general, and between the goals of them in

special.

These relations shall now be explained by one example. Figure 3 shows on the left

side a so-called “strategy map” (an overview of the perspectives of an organization

with its goals) of a BSC, which consists of four perspectives - each of which have two

subgoals.

Now one might come to the conclusion that there might be relationships between the

goals, for example between the customer support and the customer satisfaction. A

manager of a company could for example suspect that by improving his customer

support, a higher customer satisfaction might be the result. This possible relation was

added to the strategy map in figure 3. The right side of figure 3 offers a more detailed

view with metrics measuring the subgoals defined in the strategy map. The

superficial cause-and-effect relation defined between the goals “Customer Support”

and “Customer Satisfaction” may here be displayed more detailed by building

relations between the metrics themselves. For example assuming that the

imaginative company of this example is a data center, especially the metric “% SRs

Resolved on Initial Contact” might have a big effect on customer satisfaction. Maybe

this relation can even be specified more detailed: the relation may exist especially

with the measurement “Customer Satisfaction - Service Support”. But the finding of

cause-and-effect relationships usually does not end here – generally this is a process

with multiple steps.

The relations between goals that have been defined in figure 3 can now be used as a

base for finding more relations.

Balanced Scorecard

6

Figure 3: Strategy Map and IT BSC part 1

As pointed out in figure 3, there might also be a cause-and-effect relationship

between customer satisfaction and the revenue. Given that improving the customer

support leads to a higher customer satisfaction, it can further be concluded that this

may lead to increased revenue. Figure 4 links this relationship to the metric “IT

Revenue – actual”.

Balanced Scorecard

7

Figure 4: Strategy Map and IT BSC part 2

The examples that were pointed out here can be completed by many more relations

that might exist between these goals. Once found, these possible relations should

then be evaluated – possibly in group workshops with discussions, using historical

business data if available.

IT Balanced Scorecard

8

3 IT Balanced Scorecard After the concept of the BSC was introduced, it was soon picked up by different

sectors and areas. Thus, a BSC especially for IT Management / IT Governance was

created. The first suggestions to use the BSC in the IT context came up in 1992 by

Gold [Go92]. However, Willcocks found out in 1994 [Wi94], that Golds’ approach was

leading too much to see IT from an IT departments’ perspective. After that, the

concept of the IT BSC has been refined in 1997 [Va97] and 1998 [Va98] again. Since

then it has further been improved, for example by Wim Van Grembergen in 2000

[Va00], who pointed out how to link the IT BSC to the business BSC.

The main motivation to develop the IT BSC was to demonstrate the value of IT to the

business, address IT Governance issues and make IT more efficient and cut costs

[Cr07]. According to its popularity, [Cr07] created a hype cycle for the IT BSC. A hype

cycle is:

“[…] a graphic representation of the maturity, adoption and business

application of specific technologies […]”1.

This cycle (see figure 5) was published in 2007, according to which the IT BSC

already reached the “Plateau of productivity”. This is the last phase of a Hype Cycle

and means, that a product or method is no longer a hype theme which is heavily

discussed, but somehow accepted and also widely used.

1 [Un@]


9

Figure 5: Gardner Hype Cycle of the IT BSC [Cr07]

One of the points of motivation to use the IT BSC is that it delivers faster feedback of

the status of goals than traditional methods, what van der Zee expresses as follows:

“The time lag between business planning processes and IT planning

processes is often just too long to be acceptable.”2.

This chapter first points out the differences to the business BSC, and then states

considerations that should be taken into account when implementing an IT BSC.

3.1 Differences to the Business Balanced Scorecard One difference to the business BSC is that often the perspective “financial” is

replaced by “business contribution” [Va00]. While the naming of the perspective can

differ, the content can be considered the same (IT expenses and business value of IT

projects). The term “customer” from the customer perspective is often replaced by the

term “user” [Va00], which may sometimes fit better. The “learning and growth”

perspective is sometimes replaced by “future orientation”, which still contains

strategies like offering a good employee training, but also includes for example the

research of new technologies and applications in order to provide a better position for

the future.

2 [Va99], Page 141


10

3.2 Considerations for implementation There are some things to consider when implementing an IT BSC. For example

instead of just one IT BSC there can also be a cascade of hierarchical grouped IT

BSCs (which applies to the business BSC also) [Va99]. There could be for example

BSCs for each ITIL-Service or for different phases of service lifecycles, like a

scorecard for Service Design. These scorecards on the lower levels may then be

aggregated to a single high level IT BSC.

However, there are some critical success factors that need to be considered when

implementing an IT BSC. One of them is the communication between the IT

department and business managers, as J.T.M. Van der Zee states:

“Potential success is highly dependent on the communication skills of all who

are involved. In many organizations, business managers and IT planners have

shown that they are unable to express themselves in a common language, […]

so that successful links between business objectives, the IT strategy, and the

IT architecture were insufficiently build.”3.

This ability to communicate is especially crucial in the development phase of an IT

BSC. In a case study ([Va99]), the IT manager of a company was not involved in the

planning process of an IT BSC introduced in his company. In that example, the CIO

of said company was not a member of the board and not involved in any decision

making - neither in the initial development, nor in the process of finding cause-and-

effect relationships. This resulted in unrealistic time estimations of steps that needed

to be completed in order to meet goals from the BSC. For example some business

executives shared the opinion

“[…] that the replacement of a core information system could be done almost

overnight.” 4

, what resulted in a delay of more than one year. The conclusion of the study is that

communication barriers between disciplines have to be recognized and addressed

before developing a BSC, as

“[…] the Balanced Business Scorecard does not strictly enforce an integration

of business and IT management processes by itself.” 4.

When business- and IT managers are working together the probability that an IT BSC

is successful is higher.

3 [Va99], 2 4 [Va99], Page 153

Linking ITIL Key Performance Indicators and the IT Balanced Scorecard

11

4 Linking ITIL Key Performance Indicators and the I T Balanced Scorecard

This chapter tries to point out how to link ITIL Key Performance Indicators (KPIs) to

the concept of the IT BSC. After a short introduction about ITIL KPIs, the question is

tackled how to combine these with the IT BSC. This chapter ends with remarks on

how to translate the organizations strategy top-down up to low level measurements.

4.1 ITIL Key Performance Indicators The lowest level of performance measuring is defined by metrics, which are in the

context of BSCs most of the times called KPIs. Although Kaplan and Norton [Ka92]

did not speak of KPIs when they first introduced the BSC (instead, they used the

more generic term “measure”), the term was soon associated with the BSC.

Nowadays, in most cases the measures or metrics in a BSC are called KPIs. ITIL

also uses the term KPI to describe its metrics.

ITIL defines KPIs as atomic measures that provide low level measurements of how

well the ITIL services are running. For every KPI there should be a target value, to

which the current value is compared to – without a target, there would simply be no

way of telling if a goal is reached or not. There is a number of KPIs suggested in the

ITIL framework for every state of the service lifecycle. It is recommended by the ITIL

framework that these measurements are used to invoke a continual service

improvement – if the actual measures differ from the planned values, actions have to

be taken.

It has also been recommended frequently, that for every KPI a responsible person is

defined (the so called “metric owner”). This person is the responsible for gathering

the necessary data to keep up-to-date values of a metric and sometimes also for

reaching the goal that is measured by “his” KPI.

4.2 Combining ITIL Key Performance Indicators with the IT Balanced Scorecard

As the goals and strategies of the perspectives of the IT BSC have to be supported

by metrics, using ITIL KPIs for that may come to mind. And in fact many of the

metrics typically found in IT BSCs are recommend by the ITIL framework.


12

But often ITIL KPIs itself can not cover all aspects of an IT BSC; this may especially

be true for the financial perspectives. This is why ITIL KPIs are often coexisting

beside other KPIs or also purely financial measures (like displayed in figure 6).

Figure 6: Combining ITIL KPIS and the IT BSC

According to [Ke05] it is a deadly sin to organize an IT BSC in a way that it offers a

very IT-centric view of its performance - according to the authors an IT BSCs’

missions and goals should instead be business-centered. One might therefore come

to the conclusion that an IT BSC might run the risk of being too IT-centric by

focussing too heavily on ITIL KPIs (as they are often very operational and IT-

oriented). On the other hand [Ke05] says that another sin might be using measures

that lack standard metric definitions. This problem might occur less when using KPIs

that are recommended in the ITIL framework, as there is already existing knowledge

about them. And it should not to be forgotten, that even the OGC recommends in its

book on Continual Service Improvement [Ca07] to use KPIs generated from ITIL

services to measure higher level IT goals and later subsequently aggregate these in

IT BSCs.


13

A conclusion might be to use ITIL KPIs, but also other measures and KPIs where

adequate. If several KPIs are combined to measure one goal, the risk of having a too

short-time-oriented, too IT-focussed measurement can be reduced. When merging

several of these KPIs, the fusion of these might help to see the IT from a more

distanced, business-like perspective.

However, it has to be kept in mind to have a filtering process at work, which helps to

select just KPIs that are relevant for the respective business and IT goals.

4.3 Translating the strategy The translation of the business goals into IT goals may be a crucial step when

implementing an IT BSC. Figure 7 shows an example of such a transition process.

The financial perspective of the Business BSC has in this example, among others, a

business goal called “provide a good return on investment of IT-enabled business

investments”, which can be mapped to an IT goal “Improve IT’s cost-efficiency and its

contribution to business profitability”. This makes sense, as when the IT cost is going

down the investment in IT will also decrease, which may make it easier to get a

positive Return on Investment.

However, this figure covers only the “investment” part of the business goal to

increase the Return on Investment. However, the term Return on investment

indicates, that there is besides an investment also some kind of Return (or income).

However in this case the IT goal only covers the expenditures of IT, not its incomes

or benefits – therefore, the business goal is only partly translated into an IT goal. The

moneywise benefit of IT however may be difficult to measure, unless the affected

company is a data center, which has concrete revenues for its services. But even if

not, with ITIL successfully in place it should be possible to charge the business for

the IT services it consumed.

Returning to the example in figure 7, the IT goal “Improve IT’s cost-efficiency and its

contribution to business profitability” is being covered by two processes, one of them

is ”Identify and allocate costs”. This process makes sense, as if the cost-efficiency

has to be improved, the costs have to be known. This process is measured by a

bunch of KPIs (in figure 7 they are called “IT Performance Metrics”), which in this

case are not KPIs recommend by the ITIL framework, as the financial perspective is

not covered very deeply by ITIL.


14

Figure 7: Mapping of business goals to IT metrics (screenshot of the software Metricus)

Other IT goals and perspectives could probably be very well measured by ITIL KPIs –

for example the goals of the Internal perspective, as it focuses on internal business

processes. These internal business processes should be easily translatable into IT

processes, which may be managed using the ITIL framework anyway. In this case it

would make sense to use also KPIs that are recommended by the ITIL framework.

Another example of how to combine different types of KPIs could be the IT BSC for a

software developing company. The customer perspective for example could have as

a business goal “support the software development team in developing new

products”. This business goal could be translated into the more specific IT goal

“Support the development team by delivering the necessary tools at high availability”.

In order to measure the fulfilment of this goal, the processes could be for example

“find out which tools the development team needs”, “provide and maintain the

adequate tools” and “provide the availability of development tools”.

Concerning the process “provide and maintain the adequate tools”, one could think of

the following KPIs to measure the fulfilment of this process:

• Number of requests for change (regarding the tools)

• Average ratio of number of computers where provided tools are installed

compared to average number of computers existing in development

department


15

• Percentage of tools that are installed in the latest available product version

• Number of customer complaints (regarding the tools)

• Number of incidents (regarding the tools)

• Percentage of tools that are generally the most popular tools of their category

This list is just for the purpose of giving a general idea, one could certainly think of

more KPIs. What is noteworthy in this example is that metrics defined by the ITIL

framework (like number of requests, number of customer complaints or number of

incidents) coexist beside KPIs that have been developed for this specific process.

Cause-and-effect Relations

16

5 Cause-and-effect Relations The principle of cause-and-effect relationships in BSCs was explained in chapter 2.2.

There has some theoretical research being done in this field already, for example by

Torben Hügens [Hü08], whose results will be picked up later.

In comparison to this theoretical existing knowledge, there are a lot of software

manufacturers that claim to offer software products which support the creation and

maintenance of BSCs or even IT BSCs. This chapter will evaluate the product

“Metricus” regarding its functionality of the IT BSC and ITIL KPIs in general and then

regarding the finding and evaluation of cause-and-effect relations in special. This is

followed by a broader overview over the market of products, stating if the most

popular products realize the finding and evaluation of cause-and-effect relationships

in their software, and how. This chapter ends with suggestions on how the finding

and evaluating of cause-and-effect relations could be implemented in a software

product.

The selection criteria for choosing the tool “Metricus” for a more detailed evaluation

were primarily, that the software supports creating and maintaining a BSC. Besides

that another criterion was, as this paper specializes on ITIL KPIs and the IT BSC, that

ITIL KPIs and ITIL Service Dashboards are supported.

Three software products were evaluated for this: Hyscore BSC, Opalis Business

Process Centric IT Dashboard and Metricus.

Out of these, “Opalis Business Process Centric IT Dashboard” offered the support of

ITIL KPIs and ITIL Service dashboards, but no BSC. Hyscore BSC did this, but

without supporting ITIL KPIs and ITIL Service dashboards. Only Metricus managed to

support all three criteria (see figure 8).


17

Tool Balanced Scorecard ITIL KPIs,

ITIL Service Dashboards

Hyscore BSC Yes No

Opalis Business Process Centric IT

Dashboard No Yes

Metricus Yes Yes

Figure 8: Selection criteria for the software evaluation

5.1 State of the art in research One of the most comprehensive books in the area of cause-and-effect relationships

in BSCs is “Balanced Scorecard und Ursache-Wirkungs-beziehungen” from Torben

Hügens [Hü08]. He criticizes that the way of finding relations between measures

suggested by Kaplan and Norton - the strategy map - does actually not state

causalities, as it is not clear which metric is influencing the other. Further he criticizes

that this system is based on subjective expectations from managers, as the possible

relations have to be found manually, without a clear model to evaluate these

suggestions.

He further presents a model suggested by the consulting agency Horváth & Partners,

in which the possible relations are determined in the form of a workshop. This seems

better to Hügens, as this enables an extensive discussion about the causalities. It is

recommended however, to evaluate these possible relations afterwards. Regarding

the traditional ways of finding the relations, Hügens sees big problems in the fact that

they point out relations, but they do not quantify these. This means they do not show

how strong the dependency of a relation is. He concludes this by saying that the

status quo in research, as well as in current software products, provides only

methods to find qualitative, not quantitative relations. He further states that statistical

methods are available, which can help to find correlations between the measures.

These are according to the author already in use in order to find cause-and-effect

relationships, despite the fact that these methods can just find correlations, not

causalities.


18

He suggests to use a method called sequence analysis instead, which can help to

find out the time which passes between a cause and the effect it has on another

variable.

He concludes that there is no simple way to solve this challenge to find these

relations due to the complexity of the topic. He sees need for further research in this

area, among other things in how to simulate the effects. This might be one area

where software would be very helpful. He states that one way to solve this problem

might be qualitative reasoning, which is a technique that is usually used in physics,

and can help by means as simulation to find causalities even without much

quantitative data. Like Hügens, Akkermans et al. [Ak02] also suggest to build a model

of the expected relationships, and then test this model using simulation methods

which are fed by historical data available in the company. For doing so, they claim

that most relations between KPIs are not unidirectional, but that instead most metrics

will build cyclic loops of some kind. They tried to prove this by conducting a case

study with a company, for which they build a quantified simulation model. The

expected (loop) causalities where established in a workshop, the model was filled

with company data. The outcome of the study, among other things, established loop-

causalities, as well as the successful calculation of time lags between changes in

related KPIs. In summary, it can be said that the study proves that simulation can in

fact be very helpful to test out relationship models. Unfortunately, according to

Hügens, these simulation methods are hardly ever implemented in software products

(or at least were at the time his book was written) – if this is still the case will be

verified in the course of this chapter.

At least, according to Hügens, statistical methods like correlation analysis are already

in place and used in developing BSCs. If these methods are also embedded in

common BSC software will also be a topic of this chapter.

5.2 Software implementation Metricus

Metricus is a web-based Software-As-A-Service application for IT-Management by

the Dutch company ITpreneurs. It is a modular system that offers an IT BSC, service

Scorecards and dashboards for various ITIL processes such as for example Service

Desk and Change Management. These modules come with a set of over 600 KPIs

within a “best practice IT KPI library”.


19

Also available are modules to support IT Governance such as a module for project

management, as well as a module for green IT and one for Cost-control.

This evaluation is giving a general overview of the functionality first, afterwards the

features regarding relations are being discussed. The first steps when using Metricus

is the data entry. Here Metricus offers some user interfaces to enable the data entry

and processing from different sources, such as databases or CSV-files. When

building the scorecards, one can use a set of predefined KPIs or define his own

ones. When the system is configured and running, one can choose mainly between

several dashboards, ITIL-service scorecards and the IT BSC. The dashboards are

left aside for this evaluation, as they are not topics of this paper.

Figure 9 shows one of the ITIL-service scorecards, the change Management

Scorecard. It offers an easy to understand and comprehensive overview of the

perspectives, goals and measures.

Figure 9: Change Management Scorecard in Metricus

Figure 10 shows the user interface for IT BSCs, which has a similar structure as the

ITIL-service scorecards.


20

Figure 10: IT BSC in Metricus

Besides that view of the IT BSC, the online demo that was used for this evaluation

contained no features that seemed to support the finding or simulation of

relationships. However, the menu entry “Service Desk Benchmarking” contained a

window that was named “Cause and Effect”, which was unfortunately blank and did

not offer any options.

After asking ITPreneurs about that and about the overall possibilities of Metricus to

support the finding or evaluation of these relationships, the company responded that

Metricus offers the display of MS-Visio diagrams that show relationships between

measures. However, these diagrams have to be created by the customer, and also

the relationships itself have to be found, defined and tested by the customer.

Therefore it can be concluded that Metricus does not support the process of finding

relations.


21

Of course there is more BSC software that might support relations – according to

[Fi06@] in 2006 there were 23 vendors whose products were certified by the

Balanced Scorecard Collaborative, and around 80 more that were not. The most

popular ones are now described according to the features that are stated by their

publishers.

IBM Cognos 8

IBM Cognos 8 is a business intelligence tool that also contains a BSC. According to

IBM it offers cause-and-effect diagrams as well as impact analysis diagrams, which

should help to visualize how strong the impact of a change of one KPI on a related

KPI is. IBM states the following in its product brochures:

“Automatically generated HTML displays of the relationship between metrics

visually guide analysis to the root of performance problems.” 5.

IBM claims that its tools can answer the following questions:

“What are the factors driving the performance?

What other processes or metrics are affected?” 6.

It has, however, not further been researched for this paper, if the cause-and-effect

diagrams have to be created manually or if up to which extent Cognos can help to

find relations and causalities. In the whitepapers and videos IBM offers about Cognos

(at http://www-01.ibm.com/software/data/cognos/library.html) there is also no

mentioning of any tools than can simulate changes, so it can be assumed that this is

not supported.

CP-BSC

The tool CP-BSC by the company “Corporate planning” offers a software supported

BSC. It contains views for cause-and-effect relations and their impacts. However, it

can be assumed that as in Metricus these relations have to be modelled manually,

and that no simulation tool is supported:

„Es ist möglich, sowohl positive als auch negative Wirkungen mit

verschiedenen Ausprägungen (Stärken) zu definieren und zu

dokumentieren.“7.

5 [Co02@] 6 [Mo09@] 7 [Cp@]


22

Artemis Balanced Scorecard

The company Artemis offers a product called Artemis Balanced Scorecard. It offers,

among other, a view that shows cause-and-effect relations. As shown in figure 11,

these relations can also been shown quantitatively with a weighed index.

Figure 11: cause-and-effect relations in Artemis [Ar@]

However, again the product description might lead to the conclusion that the relations

have to be established manually:

„A simple yet powerful feature links Strategic Objectives, Measures (KPIs) and

Initiatives, allowing easy navigation and management of dynamic cause and

effect relationships.“ 8.

One might come to this conclusion because the words “navigation” and

“management” are used, instead of words like suggestion, finding, exploring or

simulation.

8 [Ar@]


23

SBS - STRAT&GO Business Scorecard

The product SBS-STRAT&GO Business Scorecard by the company PROCOS AG

offers, beside the common features for BSC software, possibilities to simulate the

relations between KPIs:

“[…] Simulation of scenarios using the defined hierarchical KPI relationships

[…]” 9.

Figure 12 shows that an extensive simulation of the impact of KPIs on other KPIs is

possible – regarding [St@] however, the relations itself have to be found and defined

manually, PROCOS offers workshops to support the customer in this task.

Figure 12: Cross-Impact-Analysis in STRAT&GO Balanced Scorecard [St@]

SAP SEM Business Planning and Simulation

SAP offers the tool SAP Strategic Enterprise Management (SAP SEM), which

includes a software enabled BSC. Regarding [Cr@] the tool can help in finding

relationships between KPIs:

9 [Sb@]


24

“It supports the development and maintenance process of a Balanced

Scorecard through the following functionality […] Definition of an influence

diagram (cause-and-effect linkage) to visualize dependencies among strategic

objectives on a Balanced Scorecard. “10.

However, it is not mentioned that this “definition of an influence diagram” is supported

by automatization features or that SAP SEM suggests possible relations. Therefore, it

can be assumed that the relations have to be defined manually. This assumption is

supported by the following statement:

“[…] Hypothesis definition through development of influence diagrams to

visualize dependencies among strategic objectives and KPIs in a Balanced

Scorecard. Influence diagrams are stored in the SEM database for shared

access (controlled by defined authorizations) [...]” 11.

However, Regarding [Sa04@2], the relationships between metrics can be used for

“what-if” simulations. And once defined, the relations can also be quantified:

“[…] Quantification of influence diagram cause-and-effect linkages by creating

a dynamic simulation model with SEM Business Planning and Simulation

functionality." 12.

Figure 13 shows the simulation functionality of SAP SEM.

10 [Cr@] 11 [No99@] 12 [No99@]


25

Figure 13: SAP SEM simulation functionality [Sa@]

In addition, according to [No99@] the created simulation models are even assessed

automatically on a regular basis, where actual metric values are compared to their

targets. If simulation models are no longer representing the reality, they are

automatically adapted.

SAS Balanced scorecard:

The company SAS offers a tool called Balanced Scorecard. SAS claims to offer

extensive automatization and simulation support for their tool:

“As noted before, users can create data-driven, dynamic links within strategy

diagrams to virtually any element within a SAS Strategic Performance

Management project (measures, objectives, initiatives etc…This means that

one can “logically” dig deeper into a specific piece of information using

appropriate software for developing insight and foresight.” 13.

13 [Cr07@]


26

In contrast to all other tools evaluated so far, SAS claims not only to be able to

simulate user defined relationships, but also to suggest and find these relations at

first hand:

„Correlation analysis can reveal previously unknown relationships between

metrics, making it clear which metrics are important and where to set

thresholds. This analysis shows a tight correlation between customer churn

and total revenue. Once cause-and-effect relationships are determined and

validated, their relationship to one another and the strengths of the

relationships can be displayed. Users can better predict potential outcomes

based on achieving certain results.“14.

The ability of finding relation is further described as follows:

“That is, the movement of one variable may have been caused by the

movement of another. Using advanced modelling techniques, these causal

relationships can then be isolated and highlighted. These capabilities can help

uncover cause-and-effect relationships. Regression analysis is a common

form of predictive modelling that can reveal previously unknown relationships

between KPIs in an easy and intuitive way. In the process, it reveals broader

and deeper insights into the way an organization really operates. To further

hone your strategy, a variety of analytical methods such as neural nets,

genetic algorithms, experimental design and optimization can be applied.” 15.

Simulation also seems to be supported:

“Example 2: Forecasting and “what-if analysis” help you model scenarios to

determine the best course of action. Change the values for one or more

variables to see the effect on the forecasted margin.“16.

And also the impact that a change in one KPI has on another KPI seems to be

simulatable in form of a sensitivity analysis:

“Applying rigorous analytics to business problems enables managers to more

quickly anticipate future challenges and opportunities with confidence assess

the impact of changing KPI values and respond more quickly with fact-based

decisions.” 17.

14 [Co@] 15 [Pr09@] 16 [Co@] 17 [Pr09@]


27

According to [In08@], every SAS program can be loaded into their data mining

workbench. One might expect that by means of data mining SAS can offer extensive

possibilities to find correlations between measures (for example by the association

method) or conduct simulations.

Conclusion

After evaluating Metricus practically by the means of an online demo, and the

features of six other tools according to the descriptions of their publishers, it can be

concluded that software implementations of the BSC leave in many cases a lot to be

desired. All of the evaluated tools deliver an implementation of the BSC itself via

some kind of dashboard or cockpit view with different degrees of comfort. But only

the tool from SAS really seems to help in finding and suggesting possible relations

between metrics. The situation regarding simulation features looks a bit brighter: here

at least the products from SBS, SAP and SAS seem to offer extensive tools to

simulate quantitative impact changes between KPIs. In the short evaluation that was

executed here, the tool from SAS seems to offer the best features. As the same

company also offers modules for ITIL KPIs and an IT BSC, it seems therefore much

more powerful than the practically evaluated Metricus, which lacks any features

regarding relation finding or simulation.

5.3 How a software IT BSC could look like It has to be noted up-front that it seems very unrealistic that a BSC system could find

out all important cause-and-effect relations with their linkage in terms of the time lag

between changes and in terms of their impact on each other without any human

intelligence. As the business reality is probably not a deterministic model that can

completely be calculated and predicted, it is and will probably stay a task of

managers to at least evaluate automatically suggested relations and also to suggest

ones that might not be found by correlation analysis. However a well working

methodology could be that a system suggests through correlation analysis or other

means an initial set of cause-and-effect relations. This list can then be expanded by

managers for example in the form of a workshop. For example could a software

solution, based on historical business data, create a list of all correlating measures

that exceed a predefined correlation threshold. This list could also contain the impact

that the metrics have on each other and the time lag that lies between cause and

effect.


28

Managers and employees can then use their knowledge to verify or falsify this list, set

the causalities of the suggested correlations and add own suspected relations. These

new relations can then again be subject of a correlation analysis. Once in place, this

system of causalities can then always be used for simulations, for example for

sensitivity analyses. And even if such a system fails to provide correct relations, at

least the process can help to sensibilize managers, as concluded by [Ho98].

The software from SAS seems to enable all these features - however, one could

imagine even more. One thing that could be useful is a feature to suggest possible

relations not only using historical data of one company, but maybe using data from

several different companies of the same sector or industry. Online based software-

as-a-service solutions like Metricus could collect the data from all its users in an

anonymized way to facilitate analysis and simulation by providing a bigger data pool.

This might especially be useful for companies that do not have much historical data

for KPIs yet, for example because they just introduced ITIL to their company. Such a

comparison within companies that operate in the same sector might be especially

easy if a company is using a lot of KPIs that are recommend by the ITIL framework or

that are very common. However, the suggested relations have to be evaluated

thoroughly, as even in companies of the same sector they might sometimes not be

directly transferable. What can be suggested for further research is to find out how

KPIs of companies of the same sector resemble each other.

Conclusion

29

6 Conclusion This paper began by pointing out the motivation for the change from IT Management

to IT Service Management and finally to IT Governance. It described the need for IT

Service Management frameworks like ITIL and performance measurement concepts

like the IT BSC. The motivation for and concepts of the BSC were explained, followed

by considerations on the IT BSC. It was further evaluated how to link ITIL KPIs to the

IT BSC and how to translate the business strategy into goals and measures of an IT

BSC. Then the state of the art in research regarding the finding and evaluation of

cause-and-effect relations in BSCs was illustrated, which made clear that the

research theories are somehow ahead of its implementations in software solutions.

The evaluation of the software IT BSC Metricus showed, that even as it offers good

support of ITIL and the evaluation of defined goals in an IT BSC, it lacked features

regarding cause-and-effect relation concepts, as most of the other evaluated

software solutions did too. Three of the seven evaluated products did offer

mentionable simulation support, however only one of the seven products (SAS

Balanced Scorecard) did show signs of features that help in finding possible

relations. This evaluation was followed by considerations on how software enabled

(IT) BSCs could be improved, which can be summarized as the implementation of

simulation and correlation analysis techniques (an approach that already has been

suggested by researchers). The research question of this paper

“What are the current features of (IT) Balanced Scorecard software regarding cause-

and-effect relationships between measures in general and ITIL Key Performance

Indicators in special and how can they be improved?”

has therefore been answered in the course of this software evaluation, as it pointed

out the available features by examining a few representative software products. The

part of the research question about possible improvements of software products has

been answered subsequently.

What remains to be seen is if and how fast the industry will adapt the new mindset of

IT Governance and replace or adjust its existing concepts of IT Service Management.

In this context, it will be interesting to see if ITIL can still be adequate in an

environment where IT Governance is in place – maybe the combination with more

business-centered concepts of performance measurement like the IT BSC can help

ensuring ITIL its popularity.

Conclusion

30

It is hard to predict future features of BSC software but it can be assumed, as there is

already indication for that in some products, that software solutions will be more and

more connected to business intelligence and data analysis tools. This linkage will

provide the desired functionalities by methods like data mining and simulation.

There is probably not much need for further theoretical basic research on finding and

evaluating cause-and-effect relations, as there has been done a lot of work already.

What can and should be done however is to research what mix of methods will

provide the best output for practically implemented software solutions. Then it can

only be hoped that this knowledge will be subsequently implemented by software

manufacturers.

Sources

31

7 Sources [Ak02] Henk Akkermans and Kim van Oorschot: Developing a Balanced Scorecard

with System Dynamics, Proceedings of the 20th International Conference of the System Dynamics Society, July 28 - August 1, at Palermo, Italy, 2002

[Ar@] Artemis Balanced Scorecard, http://sg.aisc.com/product/3, last checked on 8 August 2009

[Ba01@] D. R. Backinsell: Performance Measurement and Evaluation, http://www.gfkmysteryshopping.eu/imperia/md/content/gfk_nop/mysteryshopping/cs_performancemeasurement.pdf, 2001, last checked on 8 August 2009

[Ca07] Gary Case: Continual service improvement, The Stationary Office, Norwich, 2007

[Co@] Gary Cokins et. Al: Discover how analytics makes the difference between managing and improving performance, http://www.sas.com/solutions/pm/ pmsupp-analytics.pdf, last checked on 8 August 2009

[Co02@] Cognos Metrics Manager, https://www.bscol.com/pdf/Cognos_Facts.pdf, 2002, last checked on 8 August 2009

[Cp@] CP-BSC Balanced Scorecard, http://www.corporateplanning.com/d/pages/ soft/cpbsc.php, last checked on 8 August 2009

[Cr@] Creating Strategic Focus: The Balanced Scorecard, http://help.sap.com/ saphelp_sem40bcs/helpdata/de/5a/546637a04c2367e10000009b38f8cf/ content.htm, last checked on 8 August 2009

[Cr07] Alec Cram: The IT Balanced Scorecard Revisited, Information Systems Control Journal, Volume 5, Pages 33-36, Rolling Meadows, 2007

[Cr07@] Creating focus, alignment and agility with SAS® Strategic Performance Management, http://www.sas.com/resources/whitepaper/wp_3549.pdf, 2007, last checked on 8 August 2009

[Fi06@] Joseph M. Firestone: The Balanced Scorecard: Developments and Challenges, KMCI Online Press, 2006, http://www.adaptivemetricscenter.com/media/ BSCdevelopmentsandchallenges.pdf, last checked on 8 August 2009

[Go92] C. Gold: Total quality management in information services – IS measures: a balancing act, Research Note, Ernst & Young Center for Information Technology and Strategy, Boston, 1992

[Ho98] Peter Horvath and Lutz Kaufmann: Balanced Scorecard - ein Werkzeug zur

Sources

32

Umsetzung von Strategien, Harvard Business Manager, Heft 5, Pages 39-48, Hamburg, 1998

[Hü08] Torben Hügens: Balanced Scorecard und Ursache-Wirkungs-beziehungen, Auflage 1, Gabler Verlag, Wiesbaden, 2008

[In08@] Integrating Your Favorite SAS Program into a SAS Data Mining Workbench, http://www.sas.com/resources/whitepaper/wp_6874.pdf, 2008, last checked on 8 August 2009

[Ka92] Robert S. Kaplan and David P. Norton: The Balanced Scorecard-Measures that drive performance, Harvard Business Review, January-February, Pages 71-79, Boston, 1992

[Ke05] Jessica Keyes: Implementing the IT balanced scorecard: aligning IT with corporate strategy, CRC Press, Boca Raton, 2005

[Mo09@] Monitor.Manage.Perform. Scorecarding with IBM Cognos 8 Business Intelligence, http://www.ibm.com/software/data/cognos/pdfs/whitepapers/ wp_c8bi_scorecarding_monitor_manage_perform.pdf, 2009, last checked on 8 August 2009

[No99@] David Norton et al.: SAP Strategic Enterprise Management, http://www.hochschul-management.de/BSC-SAPSEM.pdf, 1999, last checked on 8 August 2009

[Pr09@] Predictive Performance Management, http://www.sas.com/apps/ whitepaper/index.jsp?cid=3496, 2009, last checked on 8 August 2009

[Sa04@] Mathias Sallé: IT Service Management and IT Governance: Review, Comparative Analysis and their Impact on Utility Computing, http://www.cioindex.com/nm/articlefiles/67864-ITServiceManagementITGovernance.pdf, 2004

[Sa04@2] SAP Strategic Enterprise Management (SAP SEM), http://www.sap.com/ germany/media/50071524.pdf, 2004, last checked on 8 August 2009

[Sa@] SAP SEM simulation functionality, http://help.sap.com/saphelp_sem40bcs/ helpdata/de/f8/4569377f2b8032e10000009b38f8cf/Image113.gif, last checked on 8 August 2009

[Sb@] SBS-Strat&GO Business Scorecard, http://www.procos.com/en/ pro_mod_sbs.asp, last checked on 8 August 2009

[St@] Strategieprozess innovativ unterstützt, http://www.procos.com/it/img/pdf/Strategieprozess_innovativ.pdf, last checked on 8 August 2009

[Th@] The Official ITIL® Website, http://www.itil-officialsite.com/home/home.asp, last checked on 8 August 2009

Sources

33

[Un@] Understanding Hype Cycles, http://www.gartner.com/pages/ story.php.id.8795.s.8.jsp, last checked on 8 August 2009

[Va00] W. Van Grembergen: The Balanced Scorecard and IT Governance, Information Systems Control Journal, Vol. 2, Pages 40-43, Rolling Meadows, 2000

[Va97]

W. Van Grembergen and R. Van Bruggen: Measuring and improving corporate information technology through the balanced scorecard technique, Proceedings of the Fourth European Conference on the Evaluation of Information Technology, October 1997, Pages 163-171, Delft, 1997

[Va98] W. Van Grembergen and D. Timmerman: Monitoring the IT process through the balanced scorecard, Proceedings of the 9th Information Resources Management (IRMA) International Conference, May 1998, Pages 105-116, Boston, 1998

[Va99] J.T.M. van der Zee et al.: Alignment is not enough: integrating business and information technology management with the balanced business scorecard, Journal of Management Information Systems archive, Volume 16, Issue 2, Pages 137–156, M. E. Sharpe Inc., Armonk, 1999

[Va07] Jan Van Bon: IT service management: an introduction, Van Haren Publishing, Zaltbommel, 2007

[Wi94] L. Willcocks: Information Management: The Evaluation of Information Systems Investments, Chapman & Hall, London, 1994

ITIL KPIs and the IT Balanced Scorecard

Documents

Transcript of ITIL KPIs and the IT Balanced Scorecard