Real World Consulting Organizational Performance Measurement

© 2013 IBM Corporation

Real World ConsultingOrganizational Performance Measurement

Mark Tregar, MS


Agenda

1. Introduction and Bio2. Career Path3. What is Consulting Really?4. Example of Organizational Performance Management Project5. Questions?


Mark Tregar - Bio


Levels Across Firms: An example of my career path

CNA

Leaders

Analysts

Associate

Specialist

Assistant

IBM

Partners (B10+)

Associate Partners (B10)

Sr. Managing Consultant (B9)

Managing Consultant (B8)

Senior Consultant (B7)

Consultant (B6) – CBD Prog

Analyst/Assistant

Deloitte

Partner/Principal/Director

Sr. Manager

Manager

Senior Consultant

Consultant

Analyst

1

23

4

• My career trajectory has spanned 3 organizations in the past 8 years.• Possible UB graduates entry point: Between consultant and senior consultant!


Why Companies Hire Consultants

1)Staff augmentation

2)External change force aka “political cover”

3)Best practices across industries and functions

4)Analytical horsepower

5)Fresh perspective

6)Training and skillset growth of staff

Management consulting is the practice of helping organizations to improve their performance, primarily through the analysis of existing organizational problems and development of plans for improvement. Organizations may draw upon the services of management consultants for a number of reasons, including gaining external (and presumably objective) advice and access to the consultants' specialized expertise. http://en.wikipedia.org/wiki/Management_consulting

A Review of Performance Measures Work at USAID

November 21, 2013

USAID HR LOB CHRRPS Technical Integration Support Project


Table of Contents

1. Background 8

2. Purpose and Approach 9

1. Conceptual Design 9

2. Detailed Design 10

3. Data Gathering 13

4. Data Reporting and Analysis 14

3. Next Steps and Lessons Learned 16


The purpose of the Performance Measure effort was to aid the measurement of the overall health of HR operations.

The team identified a series of specific Key Performance Indicators (KPIs) and then selected the appropriate KPIs to provide an accurate measure of current HR operations for USAID.– The identified KPIs were reported and measured in a one-time scorecard using

Balanced Scorecard-based categories

– Our approach to selecting KPIs was adapted from the 2009 OPM Shared Service Center (SSC) Human Resource Line of Business (HR LOB) Benchmarking Process, which was piloted with 8 agencies and expanded to 12 in 2010

Conceptual Design

Detailed Design

Data Gathering

Data Reporting

and Analysis

Del

iver

abl

esSt

eps

Nov Dec-May May-Jul Jul-Sep

ID of 10 OHR Process and HR LOB System KPIs

Delivered: 5/24/13

Scorecard Template

Delivered: 12/12/12

Initial Performance Report

Delivered: 8/9/13

Figure 1. Our 4-Step Approach Adapted from OPM’s SSC HR LOB Benchmarking Steps


In Conceptual Design, IBM developed a conceptual model Scorecard Template and gathered feedback on inclusions.

Level 1 - Top OHR USAID Management Report/ Scorecard reports high-level results (i.e., Customer Service) with an overall status for a given time period.

Level 2 - HR Management and Staff Report/Scorecard reports at a lower level of results (i.e., results for a specific metric) for a given time period.

We envisioned a future state report/scorecard with two levels and four Balanced Scorecard Categories:


In Detailed Design, the team refined our original list of 135 KPIs, to a smaller set that were most relevant to USAID.

We cut our list to 49 preliminary KPIs* based on initial performance measure research and a series of meetings with key stakeholders.

– Our original list of 135 was an agency list with additional potential KPIs from IBM’s Benchmarking wizard

– In addition, we added KPIs based on our first focus group which included HR Staff members and meetings across HR.

To gather feedback we conducted workshops with the following groups: HR leaders, HR staff (Customer Service Group), 3 HR teams, and one contractor team.

135KPIs

54KPIs

End Result:

Final KPIs selected and then

measured

* The umber of KPIs ranged between 26 and 55 throughout the stakeholder engagement process.

45 KPIs

49* KPIs


In Detailed Design, we used a series of questions to help leadership stakeholders cut KPIs

Is the potential measure

Actionable?


Applicable?


Meaningful?

Is the potential measure Detailed?

Is the potential measure Easily Understood?

The data is under control of the agency (or other provider) currently available and practical to collect.

The measure is relevant to government processes and supports USAID HR goals.

The data will be meaningful to decision makers and the derived value is greater than cost of collecting.

The scope of the measure is defined and is quantifiable to the extent possible.

The data measured is clear and is easy to understand.

Discussion Items:

1.Which KPIs are most valuable for selection and use? If so, why?

2.Which KPIs should be removed from our potential list?

Example KPI: HR Servicing Ratio - This metric measures the number of employees receiving HR services from Agency HR employees. The metric provides insight into the size and support level of HR.

UB Discussion: What are some others that could be important to an agency?

Figure 1. Items to consider in reviewing KPIs:


Following Detailed Design discussions, the IBM team rated on Relevance and Data Availability

The feedback we received from the HR Director and targeted HR groups helped us understand the key priorities for OHR.

• Relevance Rating: (Likert scale of 1-5)– 1: Not Relevant to OHR Priorities– 2: Little Relevance to OHR Priorities– 3: Some Relevance to OHR Priorities– 4: Moderate Relevance to OHR Priorities – 5: Very Relevant to OHR Priorities

• Data Availability Rating: (Likert scale of 1-5)– 1: Data is not available in any format– 2: Data is available sporadically and is maintained less than monthly– 3: Data is available but in a static spreadsheet format– 4: Data is fully available in a database and available to the team indirectly – 5: Data is fully available in a database and available to the team directly

• Based on this understanding of USAID, we rated each KPI on Relevance and Data Availability

• The sum of these two ratings forms the Overall KPI Usage Score

• Each KPI was also cataloged into one of 12 categories (e.g., Time to Hire, Processing) to help understand KPI type

• The final KPI determination for use was based on Overall KPI Usage Score and representation in 12 categories.

• This allowed for targeting KPIs with available data, while avoiding oversampling in categories

Relevance Rating + Data Availability Rating = Overall KPI Usage Score


In Data Gathering, the team met with groups such as HR Training and Education (HR T&E) to locate and collect data

• In USAID, each KPI had a specific HR owner who had access to raw data and could provide data to the team.

• The team interviewed the data owners and tracked the status of data collection, the data source (identification of specific data base), and who could help us collect the data.

• The team also tracked the specific parameters of each data element (e.g., the date range, and specific queries used in WA, Peoplesoft, or other databases) in order to validate the metrics at a later date

List of Potential T&E KPIs Used in Discussion:

Automated ILTP Metrics, for example:-Percentage of staff currently entering ILTPs

Post-Course Training Evaluation Ratings

Percentage of students who feel the knowledge they gained from the training will be applied on their job

Percentage of students who feel training will improve their job performance

Percentage of staff taking USAID University Training

Sample KPI Discussion Questions:•In our current list of KPIs (right), which KPIs do you currently track?•If you track this information, where is it currently stored (e.g., Excel, a database)?•What metrics do you measure monthly? Annually?•Which KPIs are most important to your team? Least important? •In your ideal scorecard, which KPIs would be reported?


In Data Reporting and Analysis, the team successfully collected and analyzed all data

• Each KPI was reported across 3 periods of time• We compared each result to IBM’s Benchmark in terms of median and a

specific public sector median.• The data was shared with the HR director in the form of the Initial

Performance Report. The HR Director reported our findings in multiple sessions with her Agency Leadership Council (ALC) and the USAID Administrator, Dr. Rajiv Shah

• Similar data analysis can be found in annual HR LOB Reports– http://www.opm.gov/services-for-agencies/hr-line-of-business/benchmarking/payroll-be

nchmarking/2011report.pdf– http://www.opm.gov/services-for-agencies/hr-line-of-business/benchmarking/hr-benchm

arking/2010report.pdf


Sample of results in 2010 OPM HR LOB Report

– Agency HR Servicing Ratio: This metric measures the number of employees receiving HR services from Agency HR employees. The metric provides insight into the size and support level of HR.

– Source: MAESC 2010 HR LOB Benchmarking Report - http://www.opm.gov/services-for-agencies/hr-line-of-business/benchmarking/hr-benchmarking/2010report.pdf


HR Leaders are moving forward with a new BI Tool Deployment

Next Steps•HR leaders are moving forward with BI Tool Deployment based on our ideas

•We hope to work with USAID later this year

Lessons Learned

Do:– Get leadership’s input and approval before discussing the effort with other leaders and

managers.– Use multiple ways of gathering information. Even if you are promised data, it may not

be available as needed.– Create ratings based on your understanding of KPIs, and work to validate this

understanding throughout the process.– Encourage transparency with the initiative’s goals, expected outcomes, and level of

accountability.

Don’t:– Hold up your progress for one person.– Look for one person to have all the answers.– Commit to more KPIs or data collection than necessary for your primary objective.


Mark [email protected]

Thanks!


PBC 2012 General Information Utilization PA Results

Name:

Title:

Band/Year in Band:

SAM Name:

Are you a SAM?

Client Delivery Business Development Lead or Mentor People

Picture Here

SAM Area: Administrative Compliance:

PA Score Average: Recommended PBC Rating:

Practice Giveback Thought Leadership

PA Name ResultsRelation-ships

Prof Eff Leading Bus &

FMChargeable Target:

Chargeable:

Total (110% Target):

Comments:

Comments: Client Proposal (Type Work)

Individual Contribution

TCV Result Current Client

Role (months)

# of resources led/mentored

2nd Client

Role (months)

# of resources led/mentored

Giveback Role DateThought Leadership/Strategic

ContributionRole Date

Comments:

Comments:

Yes No

Comments: Comments:

Comments:

90%

59%

• Led USAID’s Performance Measures development task,

while providing competency management expertise in the launch of the USAID LMS.• Received positive client feedback for USAID performance such as: ”I've been impressed with the quality of his work products and his ability to dig right in…and figure it out.“• Provide project management expertise at USAID including Financial and labor planning, and creating Project Launch documentation.• Led WellPoint communications team in a complex and competitive environment.• Led CPAC team delivering all deliverables on time and on budget, and developed executable plans for future work• Analyzed and presented performance management statistics to NASA HR leaders and received high client praise.

Mark Tregar

Managing Consultant

8 / 9 MonthsStephen Hutton

USAID

WellPoint

1-2

Team Lead / DPM

Team Lead

1

Contributed to IBM with work on over 10 proposals including VA HR LOB, OPM TMA, DHS S&T SETA IDIQ, Dept. of Energy, VA VISN 5, and State of New York.

Overall, I was very successful in my first 9 months including solid client contributions, as well as B&P and MOS project success. At USAID, I received “O” ratings and high client satisfaction. At WellPoint, I stabilized a challenging client environment which supported our win. At CPAC, I completed all deliverables on time and on budget.

3rd Client – VA CPAC, Role: Team Lead; Resources led/mentored: 1Provided mentorship and leadership to all lower level staff, as well as clear and consistent feedback on all products and deliverables.

x 112%

costs) at DoE HQ.

Developed an improved Public Sector Competency Development Methodology

Project 517/Career Pro – Competency Team Lead

Lead

Team Lead

Sep-Jan 2013

Apr-Aug 2012

Social Media for Human Capital White Paper

Department of Energy (DoE) Vforce

Lead & Author

Contributor

Nov-2012 (In Review)

April 2012

OPM TMA (Recompete)

DHS S&T SETA IDIQ

Dept. of Energy (New)

Section Lead (Cost Estimate, Admin)

Section Lead

Contributor

Section Lead

Win

TBD

TBD

TBD

USAID

WellPoint

O

VG

O

VG

O

O

O

O

O

VG

Provided expert input across Federal B&P efforts: Led the VA HR LOB cost proposal input development by interviewing proposal leaders, and facilitated large proposal editing sessions. I authored a section of the OPM TMA proposal and co-authored the VA Social Media RFI response. I provided research and analysis for the Dept. of Energy solution and DHS S&T SETA IDIQ response.

VA HR LOB (New)


Overview of Competency Based Management

Knowledge - What you know about the work.

Skills - Individual know-how and techniques related to the work.

Ability - Your capability and capacity to accomplish the work.

Behaviors & Characteristics - Your attitude and state of mind.

Competencies embody an individual’s total experience, including both training and education.

A competency is an observable, measurable pattern of skills, knowledge, abilities, behaviors and other characteristics that an individual needs to perform work roles or occupational functions successfully.

19

© 2013 IBM Corporation20

Relationship between KSAs and Competencies

Competency: Internal Audit and ControlReview and validate

accounting and administrative controls to safeguard the integrity of programs and

achieve compliance in accordance with government

auditing standards

Knowledge

Understand accounting

and administrative

controls

Abilities

Able to audit standards

when dealing with financial information

SkillsOperate

internal audit software

programs to validate

compliance

A competency is more than just the sum of knowledge, skills, and abilities

A competency is a “bundle” of the KSAs associated with a given area of work. It is the description of the observable behaviors that makes it different from a single K, S, or A.


Competency Development Process

• Identify data collection methodology to meet project objectives

• Identify proper mix of interviews, focus groups, and surveys

• Identify & Select SMEs• Work with sponsor to

identify Resource Panel SMEs and Incumbent SMEs

• Identify required SME population details (i.e., numbers of staff, occupations, levels)

• Initiate effort by inviting attendees to Resource Panel

• Support effort with the use of an executive memo or message of support for effort

• Identify data collection methodology to meet project objectives

• Identify proper mix of interviews, focus groups, and surveys

• Identify & Select SMEs• Work with sponsor to

identify Resource Panel SMEs and Incumbent SMEs

• Identify required SME population details (i.e., numbers of staff, occupations, levels)

• Initiate effort by inviting attendees to Resource Panel

• Support effort with the use of an executive memo or message of support for effort

• Develop/refine competency model using Resource Panels

• Hold 2-3 facilitated sessions with SMEs

• Output: Straw man competency model

• Refine competency model using Incumbent SMEs via focus group or survey [initial validation]

• Refine model using SME input (i.e., ratings, behavioral examples) and factor analysis (100+ SMEs required)

• Output: Interim Competency Model

• Develop final competency model

• Analyze data and develop final model

• Present final model to sponsor

• Output: Final Competency Model

• Develop/refine competency model using Resource Panels

• Hold 2-3 facilitated sessions with SMEs

• Output: Straw man competency model

• Refine competency model using Incumbent SMEs via focus group or survey [initial validation]

• Refine model using SME input (i.e., ratings, behavioral examples) and factor analysis (100+ SMEs required)

• Output: Interim Competency Model

• Develop final competency model

• Analyze data and develop final model

• Present final model to sponsor

• Output: Final Competency Model

• Define Objectives• Build business case for

competency model• Link model to strategic

plan

• Collect Background Materials

• Request existing competency models, position descriptions, etc.

• Review previous related IBM competency models

• Research and Review• Review strategic plan, and

organizational background• Review related

occupational models

• Develop and Execute Communications Plan to Build Sponsor Support

• Deliver kick-off brief• Collect feedback and

create additional briefings

• Define Objectives• Build business case for

competency model• Link model to strategic

plan

• Collect Background Materials

• Request existing competency models, position descriptions, etc.

• Review previous related IBM competency models

• Research and Review• Review strategic plan, and

organizational background• Review related

occupational models

• Develop and Execute Communications Plan to Build Sponsor Support

• Deliver kick-off brief• Collect feedback and

create additional briefings

• Conduct final validation survey with Incumbent SMEs for preparation in deeper applications

• Develop and refine competency survey behavioral anchors

• Execute competency validation survey

• Note: This can double as a competency assessment (including proficiency items)

• Output: Validated Competency Model

• Apply model in workforce applications

• Includes workforce & succession planning, selection*, training needs assessment)

• Conduct final validation survey with Incumbent SMEs for preparation in deeper applications

• Develop and refine competency survey behavioral anchors

• Execute competency validation survey

• Note: This can double as a competency assessment (including proficiency items)

• Output: Validated Competency Model

• Apply model in workforce applications

• Includes workforce & succession planning, selection*, training needs assessment)

Plan InitiateDevelop and

RefineValidate and

Apply

Plan, envision, and communicate competency effort

Plan, envision, and communicate competency effort

Select and initiate specific competency modeling approach

Select and initiate specific competency modeling approach

Develop and refine competency modelsDevelop and refine competency models

Validate competency models and apply models in various applications

Validate competency models and apply models in various applications

11

22

33


• Behavioral Event Interviews (BEI) are a type of structured interview use for collecting competency data from SMEs

• BEIs can be used to collect additional qualitative information for use in competency model development.

• BEIs often use a Situation-Task-Action-Reasoning/Results format in response to a prompt (i.e., Tell me about a time when…)

• Behavioral Event Interviews (BEI) are a type of structured interview use for collecting competency data from SMEs

• BEIs can be used to collect additional qualitative information for use in competency model development.

• BEIs often use a Situation-Task-Action-Reasoning/Results format in response to a prompt (i.e., Tell me about a time when…)

Situation/Task

What was the situation or

context? What were you doing? What task were you working on?

Action Reasoning

What was the reasoning/

rationale that led to the action?

Results

What was the result/

outcome of the key

situation?

What did you do? What were the

steps you took to get to that effective

outcome?

An Example Of Data Collection for Competency Modeling


Class Discussion

1. Why do IBM consultants exist? What is their job – what is the end result for consultants?

•What types of competencies are useful?

•What are lacking in the workforce today?

Think of what this looks like on the job: Think of a recent situation where you helped a client, where you saw this happen (i.e., at work, in a movie). What types of behaviors helped? How would you classify those behaviors?


As you move up the career leader there is an expectation of deeper involvement, independence, and relationship building.

Project Planning

Work Product Development

Customer Management

What is this chart for?Many beginning practitioners do not have a solid idea as to what is expected of them

based on their particular level. This chart make the expectations more clear.

Communication


Applying Competency Models in Recruiting/Hiring

Behavioral Interviewing: Once a validated model is created with a full set of

behavioral anchors (i.e., a model with 1, 3, 5

benchmarks) you can create behaviorally-based

interview questions.

Behavioral Interviewing: Once a validated model is created with a full set of

behavioral anchors (i.e., a model with 1, 3, 5

benchmarks) you can create behaviorally-based

interview questions.

1-Not Capable1-Not

Capable

• Understand asset management principles to budgeting and programming projects

• Know about available tools (unit price books, other contract estimate tools) in order to conduct complex end-to-end cost estimates

• Understand asset management principles to budgeting and programming projects

• Know about available tools (unit price books, other contract estimate tools) in order to conduct complex end-to-end cost estimates

2-Minimally Capable

2-Minimally Capable

3-Capable3-Capable

4-Fully Capable4-Fully Capable

5-Exceeds Capability

5-Exceeds Capability

Competency: Financial ManagementInterview Question: “In regards financial management, can you tell me a time when you dealt with a large budget in a complex project that was delivered on time and on budget?

Please be specific as to size of budget, team and any constraints.”

Competency: Financial ManagementInterview Question: “In regards financial management, can you tell me a time when you dealt with a large budget in a complex project that was delivered on time and on budget?

Please be specific as to size of budget, team and any constraints.”

• Understand asset management principles for budgeting and programming projects and able to determine rules for use

• Apply available tools in basic situations (unit price books, other contract estimate tools) in order to conduct complex end-to-end cost estimates

• Understand asset management principles for budgeting and programming projects and able to determine rules for use

• Apply available tools in basic situations (unit price books, other contract estimate tools) in order to conduct complex end-to-end cost estimates

• Apply asset management principles to budgeting and programming projects

• Interpret and apply available tools (unit price books, other contract estimate tools) in order to conduct complex end-to-end cost estimates


• Interpret and apply available tools (unit price books, other contract estimate tools) in order to conduct complex end-to-end cost estimates


• Selects the best available tools (unit price books, other contract estimate tools) in order to conduct complex end-to-end cost estimates


• Selects the best available tools (unit price books, other contract estimate tools) in order to conduct complex end-to-end cost estimates

• Evaluate asset management principles and execution in budgeting and programming projects

• Evaluate the use of available tools (unit price books, other contract estimate tools) in order to conduct complex end-to-end cost estimates

• Evaluate asset management principles and execution in budgeting and programming projects

• Evaluate the use of available tools (unit price books, other contract estimate tools) in order to conduct complex end-to-end cost estimates


All Top Consulting Firms Use Competencies in Hiring

Situation/Task

What was the situation or

context? What were you doing? What task were you working on?

Action Reasoning

What was the reasoning/

rationale that led to the action?

Results

What was the result/

outcome of the key

situation?

What did you do? What were the

steps you took to get to that effective

outcome?

A pro-tip: Use the STAR response method. It will keep you from going beyond what people want.

• Companies are looking for examples of your past work. Remember, history repeats itself!

• What you did in the past, chances are you will continue – good and bad• Think about what they are hiring for (look at the job posting) then think

about:1. What competencies are do they need in their hires?2. What BEHAVIORS can I talk about?

Real World Consulting Organizational Performance Measurement

Documents

Transcript of Real World Consulting Organizational Performance Measurement