Identifying Process Metrics Establishing accurate and reliable process measures.

Identifying Process Metrics

Establishing accurate and

reliable process measures

Aligning Strategy with Process Metrics

This is a process for ensuring alignment among the organization’s performance measures, strategic plans, improvement projects, and budgets. The items in blue were covered previously, and the items in grey will be covered in later class sessions.

BA 553: Business Process Management 2

1. Establish the organization’s key goals

2a. Develop and deploy the enterprise strategy to the process level

2b. Establish KPIs associated with the organization’s key goals, and measure performance in these

3. Establish process measures (if not already existing)

4. Recalibrate enterprise KPIs and align them with process-level metrics

5. Once the metrics are aligned at all levels, identify process improvement projects

6. Allocate budget aligned with the process improvements needed to achieve the strategic goals. This requires that the budgeting process be scheduled after the strategic planning process


What Are Process Measures? Measure 1

a. Dimensions, quantity, or capacity as ascertained by comparison with a standard.

b. The act of measuring

Process measures are measures that evaluate the efficiency (productivity) and effectiveness (quality) of the process

Some people refer to measures as metrics

Metric:a. Quantitative measure of the degree to which a system, component, or

process possesses a given attribute 2

b. Standard measurement of a known object or activity. For example, a company has a metric to calculate customer loyalty 3

Metrics are at the heart of a good, customer-focused process management system 4

1 http://www.answers.com/topic/measurement2 http://www.stsc.hill.af.mil/crosstalk/1995/03/Measure.asp3 Performance Dashboards (2006), Eckerson, W., Wiley & Sons, Hoboken, NJ.4 http://www.prosci.com/metrics.htm


Related Terms

Baseline metric (measure): The beginning point, based on an evaluation of the output over a period of time, used to determine the process parameters prior to any improvement effort; the basis against which change is measured1

Leading and lagging indicators: Most often used in finance, these are measures that precede or follow the process area being studied (see following slides)

1 http://dictionary.babylon.com/Baseline_measurement


Leading and Lagging Indicators

A lagging indicator is an indicator that summarizes past events rather than explicitly predicting futures ones. It’s a bit like looking in your rear-view mirror (you can see where you’ve been)

A leading indicator is an indicator used to predict the future. It’s like looking through the car’s windscreen (you can see that you’re headed in the right direction)

Process metrics are like looking within the car at how the car and driver are performing (keeping within the speed limit, sufficient gas, oil level okay, car not overheating)

http://en.wikipedia.org/wiki/Lagging_indicator

http://en.wikipedia.org/wiki/Leading_indicators


Leading and Lagging Indicators (Cont’d.)

They’re not necessarily the same as input and output measures for the process

Leading indicators show up in a process preceding the one being examined

Lagging indicators show up in a process following the one being examined

For a manufacturing process, examples of leading indicators include: Percent of personnel trained Number of new orders received Percent of inventory items with safety stock available

For a manufacturing process, examples of lagging indicators include: Profit or market share Customer satisfaction

These concepts will be re-examined in session 8


The Measurement Process

Checkaccuracy and

reliability

Understandhow metrics

impact behavior

Set upmeasurement

system

Collectbaseline data

Implementimprovement

Measureprogress

Decide whyto measure

Decide whatto measure


Why to Measure (General Purpose)

To help plan for improvement:

Clarifies the current status ("as is") – identifies the extent of the problem

Provides a basis for the "to be"

To show progress from improvement:

Justifies the cost of the improvement effort

Encourages the improvement team to keep going

Why Not to Measure To justify punishing someone. This is how measurement has

traditionally been viewed in industry.


Types of Measures for a Given Purpose

The goal for the improvement effort must be established prior to selecting metrics

If the intent is to improve process efficiency (productivity), the following types of measures might be collected:

Time-related metrics (cycle time, delays, on-time delivery, etc.)

Cost-related measures (total cost, procurement cost, etc.)

Utilization of people or equipment

If the intent is to improve process effectiveness (quality), the following types of measures might be collected:

Quality of incoming parts

Customer satisfaction

Measures of product defects (such as number of returns, cost of post-production customer service, etc.)

What to Measure

Explain these quotes:

“Not everything that can be counted counts, and not everything that counts can be counted.” 1


1. Cameron, William Bruce, Informal Sociology, a casual introduction to sociological thinking, Page 13, Random House, New York 1963.

2. “Measure What’s Important, NOT What’s Easy,” SFO Blog - Helping You Build Your Company into a Strategy Focused Organization, http://sfo-blog.typepad.com/sfo-blog/2011/08/measure-whats-important-not-whats-easy.html, accessed 2/15/15.

“Measure what’s important, not what’s easy.” 2


What to Measure

It is important to know what to measure

Once an organization learns how to measure what is important, it often finds most of the measures that have been historically collected are not helpful - people have been measuring the wrong things

It is common in industry to over-measure an area and let the data pile up without knowing how to use it

Often, the less the organization knows about how to improve, the more measures are collected

You need to focus measurement efforts on what will be useful to you - this is an area that requires a lot of judgment

What to Measure (Cont’d.)

Ensure you identify measures that are actually able to be measured:

A company’s measures related to its high-level vison and goals can be hard to measure if they are not SMART (specific, measurable, achievable, realistic, time-based). Example:

Vision: a university with a team-based environment among faculty – how can you measure this?

Goal: to encourage teamwork between professors

Measures: number or percentage of courses that are co-taught by 2 professors

Ensure your measures accurately reflect reality (slide 14)

Ensure your measures reinforce desired behaviors (slide 16)



The Politics of Process Measurement

Be aware that some people may worry about providing data regarding their performance, particularly in organizations that have used this type of information in the past for punishment

In these cases, it is best to try to collect the data in a way that is anonymous, so no one person’s name will be connected with particular data

If the culture of the organization isn’t taken into account, it’s possible to end up with faulty or inaccurate measures


Ensuring Metrics Reflect Reality

It is possible to establish metrics that are misleading. For example, assume you are collecting information on delays in the process from employees in a fear-based organizational culture:

In this situation, employees may not be willing to indicate that they’ve been waiting around for parts or approvals

The data collected will show few, if any, delays in the process, providing no information on which to base improvements

In addition, if the improvement team decided to directly observe employees working on the process, it would make employees uncomfortable and lead to a lack of trust

A possible solution is to involve some of the employees on the improvement team, so they understand and can share with fellow employees the purpose of the effort

Another option is to collect the information anonymously, on slips of paper put into a suggestion box indicating the total wait time per day per employee

Be Careful What You Measure:Measures Impact Employee Behavior



Ensuring Metrics Reinforce Desired Behaviors

It is possible to establish metrics that have the opposite of the desired effect on behavior. For example, an organization wishes to improve customer satisfaction with its IT help desk services:

Information is collected on the number of IT support desk calls answered by each employee each day

In this situation, help desk employees may feel pressured to work quickly, resulting in cutting the calls short

This will leave the customers unhappy, reducing customer satisfaction

Deming gave an example where a help desk employee said she was told her job was to take 25 calls per hour and to satisfy customers. Deming said “Is your job to make 25 calls per hour, or to give callers courteous satisfaction? It can’t be both!”

The problem is that the organization didn’t identify what was leading to poor satisfaction. If it was that many issues went unresolved, then placing a time constraint on the support personnel will not help


What to Measure: Types of Measures

Process input measures: these are measures of the inputs to the process

Process output measures: these are measures of the output of the entire process. Traditionally, if organizations measure anything at the process level, they measure only process outputs.

In-process measures: these are measures that exist part-way through the process to measure certain steps where problems might exist. These measures are used when just measuring at the end of the process (process outputs) doesn’t give us enough information.


What to Measure: Where in the Process

You will want to collect output measures to show improvement to the entire process, but you also want to collect in-process measures to show how the changes you made to parts of the process are improving the process outputs

Inputs OutputsSub-process A Sub-process B Sub-process C

M 1 M 2

In-process measures(collected from customers internalto the process)

Work Process

Input measures(collected

from the

customer forthe inputs -

you)

Output measures(collected fromcustomers in

next process, or externally)

M 3 M 4


What to Measure: Process Input Measures

These measures evaluate the effectiveness (quality) or other characteristics (for example, number or timeliness) of inputs to the process. Examples include:

Percentage of acceptable incoming parts

Status of equipment to be used in the process

Number of requests for maintenance

Percentage of maintenance requests that are correct/ complete

Number of various types of customer service requests (rush orders vs. regular, help with software vs. hardware, etc.)

Number of help desk support requests received

Percentage of employees requiring retraining in their job functions

Correctness of the procedures manual for doing the job


What to Measure: Process Output Measures

These measures evaluate the efficiency (productivity) and effectiveness (quality) of the entire process. Examples include:

Time needed to complete the process (cycle time)

Number of unpaid invoices over 20 days old

Percentage of errors in sales quotations

Waste (in dollars) caused by process errors

Labor cost to complete the process

Equipment lifecycle

Number of calls answered

Number of end user issues resolved


What to Measure: Output Measures (Cont’d.)

It’s important to take measures in such a way that not too many items will influence your measure: otherwise, your improvements may not show up

For example, if you improved the production process and then measured cost per unit built, you might not see the effect of your improvement, because the cost of raw materials, labor, or supplier parts could have risen

It would be better to measure the reduction in rework, or the number of parts coming off the line per day per worker


Questions on Process Measures

Example

Input measure: number of end user support requests received

Output measure: number of end user issues resolved

Questions

Do these input and output measures provide enough information to determine whether there is a problem with the process? Why?

Do these input and output measures provide enough information to understand what the cause of the problem might be? Why?


What to Measure: In-Process Measures

These measures evaluate the efficiency (productivity) and effectiveness (quality) of segments of the process. Examples include:

Percentage of phone calls not answered within 3 minutes

Number of letters retyped

Time between receiving product from production and completing the packaging

Number of times a certain task is repeated

Amount of travel required for a particular task (measured in distance or time)

Percentage of first pass yield

Number of times orders are expedited

Equipment utilization


What to Measure: Defining the Metric

Once a metric is selected for measurement, it is important to ensure it’s well-defined

For example, if “new product sales in dollars” is the selected metric, the following decisions could form part of the definition effort:

Result measured in dollars vs. units sold

How do define “new” product vs. product upgrade

Length of time product is considered “new”

Whether length of time is measured from product release, or from first sale

Whether sales to new customers constitute new product sales

Process Management Memory Jogger (2008), GOAL/QPC, Salem, NH.


How to Measure: Accurately and Reliably

ReliabilityReliability is a matter of whether a particular technique, applied repeatedly to the same object, would yield the same results.

AccuracyAccuracy refers to the extent to which the measure adequately reflects the real meaning of the concept being considered.

(From "The Practice of Social Research - Fifth Edition" by Earl Babbie, 1989).

Reliable but not accurateAccurate but not reliableAccurate and reliable


The Importance of Accuracy

A critical success factor in measurement is the accuracy of your measures

For example, assume you have a situation where there is a large backlog of IT service desk requests that aren’t being completed. If you make improvements to the service desk process, you would expect the backlog to be reduced

But what if employees begin to call in requests for support more frequently because they see that the problems are actually getting addressed?

This is an area where many items influence the data being collected, so you need to apply systems thinking to understand the items influencing your measures

What would be a better measure to determine the results of the improvement?


The Measurement Process

After defining the metric, the next step is to determine how to collect the measurement information

This is discussed in more detail in the course on process improvement

Once the metric information has been collected, it is important to link it to business performance

This is covered in more detail in the next presentation

Identifying Process Metrics Establishing accurate and reliable process measures.

Documents

Transcript of Identifying Process Metrics Establishing accurate and reliable process measures.