Results from the Performance Management Survey By Janice Rouiller and Laura Stouffer, SAIC and

1

Results from the Performance Management Survey

By Janice Rouiller and Laura Stouffer, SAICand

Amy Culbertson, OQM*

1 April 2004

*Note: All data analyses were conducted by SAIC.

2

Table of Contents• Purpose and Methodology………………………………………………………….3

• Respondent Information…………………………………………………………….6

• Perceptions of Cycle 2 PM Implementation……………………………….…….14

• Perceptions of Needs for Cycle 3…………………………………………………47

• Organizational Climate Perceptions………………………………………………63

• Customer Scorecard Ratings………………………………………………………83

• Summary of Comments..…………………………………………………………..96

• Summary………………………………………………..…………………………..102

• Appendix - Individual Comments from Survey…………………………………..107

3

Purpose

• Administer a customized version of the ORS Customer Scorecard to:• Gather feedback on past Performance

Management (PM) implementation efforts• Collect needs assessment information to tailor the

on-going implementation of PM• Gather data on the required Customer Scorecard

dimensions• Collect organizational culture change information

for the Evaluation Study

4

Methodology• Administer the survey at the completion of the

second cycle of Performance Management implementation on an organization-wide basis

• Which occurred in January 2004

• Designed and tested web survey during February 2004

• Survey went “live” on 2 March 2004• Email requests sent to all PM Team Leaders, Members, and

additional senior ORS/ORF managers• First reminder email sent 8 March• Second reminder email sent 15 March• Closed survey field period 22 March

• Data file sent to contractor for analysis

5

Survey Distribution

Number of email recipients 196

Number of respondents 70

Response Rate 36%

6

Respondent Information

7

Respondents by Role

3

43

19

7

00

5

10

15

20

25

30

35

40

45

50

ORSDirector/Associate

Director

DivisionDirector/Office

Director

PM Team Leader PM Team Member Other

N =69

Note: Multiple responses allowed. One respondent skipped this question.

Nu

mb

er

of

Resp

on

den

ts

8

Respondents by Organization

52

16

10

5

10

15

20

25

30

35

40

45

50

55

60

ORS ORF Other

N = 69

Note: One respondent skipped this question.

Nu

mb

er

of

Resp

on

den

ts

9

Respondents by Program Area

2

10

13

12

5

1515

0

3

6

9

12

15

Program andEmployeeResources

ScientificResources

SecurityServices

ManagementServices

Real Estateand Facilities

Not Part ofthe Services

Hierarchy

Don't Know

N = 68

Note: Multiple responses allowed. Two respondents skipped this question.

Nu

mb

er

of

Resp

on

den

ts

10

Experience with the PM Process

42%

75%

73%

85%

76%

0% 20% 40% 60% 80% 100%

Part of ASA Team inFY01

Part of ASA Team inFY02

Received FY03Measures Workshop

Attended FY03 BSCTraining

Attended FY03Orientation Training

Percent “Yes”

N = 66

N = 65

N = 63

N = 65

N = 65

11

Approximately what percent of time did you spend on PM-related activities during Jan 03 - Jan 04?

3

0

21

0

3

9

20

12

19

0

2

4

6

8

10

12

14

16

18

20

0-10% 11-21% 21-30% 31-40% 41-50% 51-60% 61-70% 71-80% 81-90% 91-100%

N = 69

Percent of time Spent

Nu

mb

er

of

Resp

on

den

ts

12

On average, how often did your PM team meet during the time period Jan 03 - Jan 04?

21

25

28

12

1

0

5

10

15

20

25

30

Several Timesa Week

Once a Week Biweekly Once a Month Once a Quarter Less thanOnce a Quarter

N = 69

How Often PM Team Met

Nu

mb

er

of

Resp

on

den

ts

13

0

4

5

6

13

6

8

12

4

9

2

00

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse

To what extent did your team members work between meetings to implement PM?

To No Extent

To a Great Extent

N = 69

Mean = 6.17

Median = 6

14

Perceptions of Cycle 2 PM Implementation(January 2003 - January 2004)

15

How helpful did you find the following in implementing PM?

4.17

5.03

5.26

5.39

5.46

5.54

5.82

5.93

5.97

6

6.28

6.34

6.88

6.91

6.94

7.07

8.13

1 2 3 4 5 6 7 8 9 10

Other PM Teams

PM Conference

Performance Management Website

Services Hierarchy

Design of Data Collection Forms

Process Behavior Chart Analysis

Measures Roadmap Template

PM Related Emails from OQM

Establishing Data Collection

ORS Customer Scorecards

Strategy Mapping

Graphing Data

Your Supervisor

PMPTemplate

PM Presentation Template

OQM Staff

PM Consultants

Not at all Helpful

Extremely Helpful

Mean Ratings

16

How helpful did you find the following tools in implementing PM?

6.94

5.82

6.28

6.91

5.39

1 2 3 4 5 6 7 8 9 10

PM PresentationTemplate forConference

Measures RoadmapTemplate

Strategy Mapping

PMP Template

Listings of theServices Hierarchy N =

68

Not at all Helpful

Extremely Helpful

Mean Ratings

N = 67

N = 69

N = 67

N = 68

17

8

4

5

3

6

5

9 9

4

6

7

2

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Listings of the Services Hierarchy

Not at all Helpful

Extremely

Helpful

N = 68

Mean = 5.39

Median = 6

18

1

2 2 2

7 7

8

12

11

13

2

00

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Performance Management Plan (PMP) Template

Not at all Helpful

Extremely

Helpful

N = 67

Mean = 6.91

Median = 8

19

4

3

4

5

8

7

5

13

10

8

2

00

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Strategy Mapping

Not at all Helpful

Extremely

Helpful

N = 69

Mean = 6.28

Median = 7

20

4

3

4 4

9

3

7

12

5

7

9

00

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Measures Roadmap Template

Not at all Helpful

Extremely

Helpful

N = 67

Mean = 5.82

Median = 7

21

2

1

3 3

4

6

7

14

8

13

7

00

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


PM Presentation Template for Conference

Not at all Helpful

Extremely

Helpful

N = 68

Mean = 6.94

Median = 8

22

How helpful did you find the following data analysis services in implementing PM?

6

5.54

6.34

5.97

5.46

1 2 3 4 5 6 7 8 9 10

ORS CustomerScorecards

Process BehaviorChart Analysis

Graphing Data

Establishing DataCollection

Methodologies

Design of DataCollection Forms

Not at all Helpful

Extremely Helpful

Mean Ratings

N = 67

N = 68

N = 68

N = 68

N = 67

23

5

7 7

0

4

5

9

12

5 5

3

5

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse

How helpful did you find the following data analysis services in implementing PM:

Design of Data Collection Forms

Not at all Helpful

Extremely

Helpful

N = 67

Mean = 5.46

Median = 6

24

3

8

4

0

5 5

11

15

5

6

3 3

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Establishing Data Collection Methodologies

Not at all Helpful

Extremely

Helpful

N = 68

Mean = 5.97

Median = 7

25

4 4

2

4 4

1

14

16

4

9

4

2

0

5

10

15

20

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Graphing Data

Not at all Helpful

Extremely

Helpful

N = 68

Mean = 6.34

Median = 7

26

5 5

3 3

5

4

9

8

3

6

10

7

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Process Behavior Chart Analysis

Not at all Helpful

Extremely

Helpful

N = 68

Mean = 5.54

Median = 6

27

5 5 5

1

2

7

5

10

9 9

5

4

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


ORS Customer Scorecards

Not at all Helpful

Extremely

Helpful

N = 67

Mean = 6.00

Median = 7

28

How helpful did you find the following communication vehicles in implementing PM?

5.03

5.93

5.26

1 2 3 4 5 6 7 8 9 10

PM Conference

PM Related Emailsfrom OQM

PM Website N = 68

Not at all Helpful

Extremely HelpfulMean Ratings

N = 68

N = 68

29

6

4

3

8

9

5

9

8

1

6

9

00

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse

How helpful did you find the following communication vehicles in implementing PM:

Performance Management Website

Not at all Helpful

Extremely

Helpful

N = 68

Mean = 5.26

Median = 5

30

3 3 3

6

9 9 9

13

5 5

3

00

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


PM Related Emails from OQM

Not at all Helpful

Extremely

Helpful

N = 68

Mean = 5.93

Median = 6

31

7

3

4

5

11

8

7 7

1

4

7

4

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


PM Conference

Not at all Helpful

Extremely

Helpful

N = 68

Mean = 5.03

Median = 5

32

How helpful did you find assistance from the following groups in implementing PM?

6.88

4.17

8.13

7.07

1 2 3 4 5 6 7 8 9 10

Your Supervisor

Other PM Teams

PM Consultants

OQM Staff N = 69

Not at all Helpful

Extremely Helpful

Mean Ratings

N = 69

N = 68

N = 68

33

0

1

3 3

8

7

8

15

13

9

1 1

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


OQM Staff

Not at all Helpful

Extremely

Helpful

N = 69

Mean = 7.07

Median = 8

34

01 1 1 1

5

1011 11

26

0

2

0

5

10

15

20

25

30

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


PM Consultants

Not at all Helpful

Extremely

Helpful

N = 69

Mean = 8.13

Median = 9

35

9

6

3

7

2

45 5

12

6

18

0

5

10

15

20

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Other PM Teams

Not at all Helpful

Extremely

Helpful

N = 68

Mean = 4.17

Median = 4

36

3

1

4

1

7

3

6

14

8

14

1

6

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Your Supervisor

Not at all Helpful

Extremely

Helpful

N = 68

Mean = 6.88

Median = 8

37

Associate Director Involvement in PM

7.48

5.52

1 2 3 4 5 6 7 8 9 10

What extent wasyour Associate

Director involved inyour PM processduring this PM

Cycle?

To no Extent

To a Great Extent

7.29

5.67

1 2 3 4 5 6 7 8 9 10

How helpful wasthe involvement of

your AssociateDirector in PMrelated matters

during this cycle?

FY02 FY03Not at all Helpful

Extremely Helpful

Mean Ratings

N = 69

N = 69

N = 62

N = 63

38

To what extent has your knowledge/skills/abilities (KSAs) improved in the following areas during this PM cycle?

4.04

5.27

5.32

6.24

6.02

6.22

6.16

6.37

1 2 3 4 5 6 7 8 9 10

Using Visio

Using PowerPoint

Using Excel

Working in a Team

Presentation, Preparation and Delivery

Methods to Analyze Data

Methods to Gather Data

BSC approach to PM N = 69

To No Extent

To a Great Extent

Mean Ratings

N = 69

N = 69

N = 69

N = 69

N = 69

N = 69

N = 68

39

3

1

4

2

8

9

11

12

8

5

4

2

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Balanced Scorecard Approach to PM

To No Extent

To a Great Extent

N = 69

Mean = 6.37

Median = 7

40

3 3

4

1

8

13

12 12

4

6

2

1

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Methods to Gather Data

To No Extent

To a Great Extent

N = 69

Mean = 6.16

Median = 7

41

2 2

5

4

6

12

13

12

6

5

2

1

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Methods to Analyze Data

To No Extent

To a Great Extent

N = 69

Mean = 6.22

Median = 7

42

5

2

4

2

5

11

13

15

4

3 3

2

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Presentation Preparation and Delivery

To No Extent

To a Great Extent

N = 69

Mean = 6.02

Median = 7

43

5

3

1 1

11

7 7

16

10

3 32

0

5

10

15

20

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Working in a Team

To No Extent

To a Great Extent

N = 69

Mean = 6.24

Median = 7

44

10

4

3

5

8

6

5

6

11

3

2

6

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Using Excel

To No Extent

To a Great Extent

N = 69

Mean = 5.32

Median = 5

45

13

1 1

5

7

9

7

5

9

3

2

7

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Using PowerPoint

To No Extent

To a Great Extent

N = 69

Mean = 5.27

Median = 6

46

14

45

32

3 3

9

21

5

17

0

5

10

15

20

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Using Visio

To No Extent

To a Great Extent

N = 68

Mean = 4.04

Median = 3

47

Perceptions of Needs for Cycle 3

48

In planning for the next PM cycle, how helpful would each of the following be to your PM Team?

5.4

5.61

5.63

6.02

6.05

6.08

6.18

6.27

6.75

7.35

7.4

8.31

1 2 3 4 5 6 7 8 9 10

Yearly Performance Mgmt Conf

Rotating Quarterly Presentation Meetings

Training on the BSC

Training on data collection systems and forms

Training on analyzing data through graphs

Training on ORS ABC Model

Training on process behavior charts

Training on customer assessment

Regular Meetings with Mgmt to discuss PM data

IT Support to establish Data Collection systems

OQM Staff

PM Consultants provided by OQM

Not at all Helpful

Extremely Helpful

Mean Ratings

49

In planning for the next PM cycle, how helpful would each of the following tools/resources be to your PM Team?

7.35

5.4

5.61

6.75

7.4

8.31

1 2 3 4 5 6 7 8 9 10

IT Support to Establish Data CollectionSystems

Yearly PM Conference


Regular Meetings with your Mgmt to discussPM Data

OQM Staff

PM Consultants provided by OQM N = 69

Not at all Helpful

Extremely Helpful

Mean Ratings

N = 69

N = 69

N = 68

N = 68

N = 67

50

0 0

21

4

23

9

18

26

2 2

0

5

10

15

20

25

30

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse

In planning for the next PM cycle, how helpful would each of the following tools/resources be to your PM Team:

PM Consultants Provided by OQM

Not at all Helpful

Extremely Helpful

N = 69

Mean = 8.31

Median = 9

51

0

2

1

2

5

8

7

14

13 13

3

1

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


OQM Staff

Not at all Helpful

Extremely Helpful

N = 69

Mean = 7.40

Median = 8

52

4

1

4

2

3

5

13 13

7

11

3 3

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Regular Meetings with your Management

Not at all Helpful

Extremely Helpful

N = 69

Mean = 6.75

Median = 7

53

8

1

8

3

1

6

3

9

6

8

13

2

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse



Not at all Helpful

Extremely Helpful

N = 68

Mean = 5.61

Median = 6

54

10

5

4

2

6

5

4

11

6

7 7

1

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Yearly Performance Management Conference

Not at all Helpful

Extremely Helpful

N = 68

Mean = 5.40

Median = 6

55

1 12

32

4

98

9

20

7

1

0

5

10

15

20

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


IT Support to Establish Data Collection Systems

Not at all Helpful

Extremely Helpful

N = 67

Mean = 7.35

Median = 8

56

In planning for the next PM cycle, how helpful would each of the following training sessions be to your PM Team?

6.08

6.27

6.18

6.05

6.02

5.63

1 2 3 4 5 6 7 8 9 10

ORS ABS Cost Accounting Model

Customer Assessment (interviews, focusgroups, surveys)

Analyzing Processes with Process BehaviorCharts

Analyzing Data through Graphs

Setting up Data Collection Systems andForms

Balanced Scorecard Methodology N = 67

Not at all Helpful

Extremely HelpfulMean Ratings

N = 68

N = 68

N = 68

N = 67

N = 67

57

7

2

5

4

8

6

5

11

5

8

3 3

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Balanced Scorecard Methodology

Not at all Helpful

Extremely Helpful

N = 67

Mean = 5.63

Median = 6

58

5

4

3 3

6

8 8

12

7 7

3

2

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Setting up Data Collection Systems and Forms

Not at all Helpful

Extremely Helpful

N = 68

Mean = 6.02

Median = 7

59

7

0

4 4

7

5

8

15

7

6

3

2

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Analyzing Data Through Graphs

Not at all Helpful

Extremely Helpful

N = 68

Mean = 6.05

Median = 7

60

6

1

4

3

5

7

8

15

7 7

3

2

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Analyzing Processes with Process Behavior Charts

Not at all Helpful

Extremely Helpful

N = 68

Mean = 6.18

Median = 7

61

5

2

3 3

5

6

9

10 10

9

3

2

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


Customer Assessment through Interviews, Focus Groups and Surveys

Not at all Helpful

Extremely Helpful

N = 67

Mean = 6.27

Median = 7

62

6

1

5

3

6

5

4

8

10 10

5

4

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse


ORS ABC Cost Accounting Model

Not at all Helpful

Extremely Helpful

N = 67

Mean = 6.08

Median = 7

63

Organizational Climate Perceptions

64

4.04

4.01

3.46

4.13

3.83

3.73

4.21

3.72

1 2 3 4 5

Actively Involved in Data Collection

PM Assist with A-76

PM Contributed to Improvements

Accountability is an Important Value

Results-Oriented Culture

I Understand Services Hierarchy

Committed to PM

See Value of PM

Organizational Climate PerceptionsSummary

Strongly Disagree

Strongly Agree

N = 68

N = 68

N = 68

N = 68

N = 68

N = 68

N = 67

N = 68

Neither Agree nor Disagree

65

4

89

19

27

10

0

5

10

15

20

25

30

1 2 3 4 5 DK NA

Fre

qu

ency

of

Res

po

nse

I see the value of performance management for improving business operations in my area.

Strongly Disagree

Strongly Agree

N = 68

Mean = 3.72

Median = 4


66

21

8

19

33

5

00

5

10

15

20

25

30

35

1 2 3 4 5 DK NA

Fre

qu

ency

of

Res

po

nse

My organization (ORS or ORF) is committed to the performance management effort.

Strongly Disagree

Strongly Agree

N = 68

Mean = 4.21

Median = 5


67

8

3

11

16

26

4

00

5

10

15

20

25

30

1 2 3 4 5 DK NA

Fre

qu

ency

of

Res

po

nse

I understand what the Services Hierarchy is and its purpose.

Strongly Disagree

Strongly Agree

N = 68

Mean = 3.73

Median = 4


68

1

6

14

2221

4

00

5

10

15

20

25

1 2 3 4 5 DK NA

Fre

qu

ency

of

Res

po

nse

The culture of my organization (ORS or ORF) is changing to be more results-oriented.

Strongly Disagree

Strongly Agree

N = 68

Mean = 3.83

Median = 4


69

12

12

20

26

7

00

5

10

15

20

25

30

1 2 3 4 5 DK NA

Fre

qu

ency

of

Res

po

nse

Managers in my organization (ORS or ORF) believe accountability is an important organizational value.

Strongly Disagree

Strongly Agree

N = 68

Mean = 4.13

Median = 4


70

6 6

15

23

17

10

0

5

10

15

20

25

1 2 3 4 5 DK NA

Fre

qu

ency

of

Res

po

nse

Performance management has contributed to improvements in my area.

Strongly Disagree

Strongly Agree

N = 68

Mean = 3.46

Median = 4


71

4 4

10

17

21

7

4

0

5

10

15

20

25

1 2 3 4 5 DK NA

Fre

qu

ency

of

Res

po

nse

Performance management will assist my organization if we have to go through an A-76 competition.

Strongly Disagree

Strongly Agree

N = 67

Mean = 4.01

Median = 4


72

0

5

10

19

33

10

0

5

10

15

20

25

30

35

1 2 3 4 5 DK NA

Fre

qu

ency

of

Res

po

nse

My PM Team (or the groups I am responsible for) are actively involved in data collection and analysis.

Strongly Disagree

Strongly Agree

N = 68

Mean = 4.04

Median = 4


73

Organizational Climate Perceptions by Program Area

74

Program and Employee Services Climate Perceptions

69%

81%

69%

69%

63%

69%

56%

69%

0% 20% 40% 60% 80% 100%


PM Assist with A-76





Committed to PM

See Value of PM

Percent Agree or Strongly Agree

N = 16

N = 16

N = 16

N = 16

N = 16

N = 16

N = 16

N = 16

75

Scientific Resources Climate PerceptionsPositive Responses

40%

67%

40%

53%

67%

53%

36%

73%

0% 20% 40% 60% 80% 100%


PM Assist with A-76





Committed to PM

See Value of PM


N = 15

N = 14

N = 15

N = 15

N = 15

N = 15

N = 15

N = 15

76

Security Services Climate Perceptions

40%

60%

40%

60%

80%

40%

20%

80%

0% 20% 40% 60% 80% 100%


PM Assist with A-76





Committed to PM

See Value of PM


N = 5

N = 5

N = 5

N = 5

N = 5

N = 5

N = 5

N = 5

77

Management Services Climate Perceptions

82%

73%

82%

64%

82%

55%

64%

91%

0% 20% 40% 60% 80% 100%


PM Assist with A-76





Committed to PM

See Value of PM


N = 11

N = 11

N = 11

N = 11

N = 11

N = 11

N = 11

N = 11

78

Real Estate and Facilities Climate Perceptions

92%

83%

67%

83%

67%

67%

67%

83%

0% 20% 40% 60% 80% 100%


PM Assist with A-76





Committed to PM

See Value of PM


N = 12

N = 12

N = 12

N = 12

N = 12

N = 12

N = 12

N = 12

79

Organizational Climate Perceptions by Role

80

Division Directors’/Office Directors’ Climate Perceptions

100%

100%

100%

100%

86%

100%

57%

86%

0% 20% 40% 60% 80% 100%


PM Assist with A-76





Committed to PM

See Value of PM


N = 7

N = 7

N = 7

N = 7

N = 7

N = 7

N = 7

N = 7

81

PM Team Leaders’ Climate Perceptions

83%

78%

83%

89%

67%

78%

55%

89%

0% 20% 40% 60% 80% 100%


PM Assist with A-76





Committed to PM

See Value of PM


N = 18

N = 18

N = 18

N = 18

N = 18

N = 18

N = 18

N = 18

82

PM Team Members’ Perceptions

56%

72%

47%

49%

63%

49%

53%

74%

0% 20% 40% 60% 80% 100%


PM Assist with A-76





Committed to PM

See Value of PM


N = 43

N = 42

N = 43

N = 43

N = 43

N = 43

N = 43

N = 43

83

Customer Scorecard Ratings

84

Customer Scorecard Ratings for PMOverview for Cycle 2

6.33

7.06

6.78

7.14

7.03

6.83

6.58

6.82

6.12

1 2 3 4 5 6 7 8 9 10

Handling of Problems

Competence

Convenience

Responsiveness

Availability

Reliability

Timeliness

Quality

Cost N = 68

Unsatisfactory Outstanding

Mean Ratings

N = 68

N = 67

N = 68

N = 68

N = 68

N = 68

N = 68

N = 65

85

0 1 2 2

7

13

64 4

36

2

0

5

10

15

20

25

30

35

40

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse

Cost Satisfaction Ratings


N = 68

Mean = 6.12

Median = 7

86

0

2 23

8

3

8

17

9

7

9

00

5

10

15

20

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse

Quality Satisfaction Ratings

Unsatisfactory

Outstanding

N = 68

Mean = 6.82

Median = 8

87

0

3

2

5

7

2

5

14

11

7

11

00

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse

Timeliness Satisfaction Ratings

Unsatisfactory

Outstanding

N = 67

Mean = 6.58

Median = 8

88

1 1 1

4

7

4

6

15

10

7

12

00

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse

Reliability Satisfaction Ratings

Unsatisfactory

Outstanding

N = 68

Mean = 6.83

Median = 8

89

0

2

3 3

6

3

11

15

9

12

4

00

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse

Availability Satisfaction Ratings


N = 68

Mean = 7.03

Median = 8

90

01

23

54

11

19

7

11

5

00

5

10

15

20

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse

Responsiveness Satisfaction Ratings

Unsatisfactory

Outstanding

N = 68

Mean = 7.14

Median = 8

91

0

2

0

8 8

3

7

16

7

10

7

00

5

10

15

20

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse

Convenience Satisfaction Ratings


N = 68

Mean = 6.78

Median = 8

92

1 1

5

2

7

2

6

17

8

15

4

00

5

10

15

20

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse

Competence Satisfaction Ratings

Unsatisfactory

Outstanding

N = 68

Mean = 7.06

Median = 8

93

2

3

1

3

7

0

9

13

9

8

9

1

0

3

6

9

12

15

1 2 3 4 5 6 7 8 9 10 DK NA

Fre

qu

ency

of

Res

po

nse

Handling of Problems Satisfaction Ratings


N = 65

Mean = 6.33

Median = 7

94

Customer Scorecard Ratings for PMTrend Analysis

7.03

7.06

7.36

6.57

6.83

6.58

6.82

6.12

1 2 3 4 5 6 7 8 9 10

Reliability

Timeliness

Quality

Cost

Cycle 1 Cycle 2

N = 68


Mean Ratings

N = 68

N = 67

N = 68

N = 37

N = 70

N = 67

N = 67

95

Customer Scorecard Ratings for PMTrend Analysis (cont.)

6.33

7.33

7.5

7.5

7.17

6.33

7.06

6.78

7.14

7.03

1 2 3 4 5 6 7 8 9 10

Handling of Problems

Competence

Convenience

Responsiveness

Availability

Cycle 1 Cycle 2

N = 68


Mean Ratings

N = 68

N = 68

N = 68

N = 65

N = 69

N = 68

N = 66

N = 69

N = 65

96

Summary of Comments

Note: 34 out of 70 respondents made comments (49%).

97

What did you value most about this cycle of performance management?

• Assistance provided by consultants and OQM staff (9)• Professional guidance and support from OQM staff.• Knowledgeable and helpful consultants.• Increased involvement of OQM staff in team meetings.

• Implementing measurement systems (6)• Developing systems to gather data.• Using data to improve processes.

• Knowledge and Experience Gained (4)• Experience has led to fuller appreciation of the Performance

Management Process.• Continued experience is leading to greater comfort with the

process.

Note: Comments were sorted into categories. Most frequently mentioned categories listed first.

98

What did you value most about this cycle of performance management? (cont.)


• Understanding our processes (4)• Greater understanding of the processes.• Actually seeing improvements in processes.

• Training Offered (2)• Measures Workshop and Balanced Scorecard Orientation.• Just in Time training.

99

What obstacles were most challenging to you during this cycle of performance management?


• Lack of time/resources (10)• Office is short staffed and workloads at all-time high• Difficulty scheduling OQM staff for timely meetings• Office identified late and struggled to “catch up” with process

• Data Collection and Measurement (9)• Manual data collection• Definition of measures difficult• Analysis of measures difficult

• Applying Balanced Scorecard Approach (6)• Lack of flexibility in application• Use of resulting data to make changes is absent

100

What obstacles were most challenging to you during this cycle of performance management? (cont.)


• Lack of management involvement (3)• PMP not reviewed• Close communication missing

• Conference (2)• Lack of management presence• Full presentation too time-consuming – focus should be on presentation

of a few key indicators

101

How can we help integrate PM into your daily business activities?

• Foster use of data. (7)• Show how data is being used by Service Groups and by

ORS/ORF Managers.• Follow up with Service Groups on implementation of objectives.

• Continue support currently provided. (6)• Maintain communication throughout the year.• PMP is very helpful in its present form.

• Increase management involvement in process. (2)

102

SummarySurvey Respondents

• Respondents were primarily from ORS (3/4th)• About half of the PM team leaders answered the survey • Smallest response from the security program area

• Respondents were experienced with PM (3/4th) and had received training (3/4th)

• About 74% of respondents spent 30% or less of their time during the past year on PM

• Team meetings typically occurred biweekly (40%) or once a month (36%)

103

Summary (cont.)

Feedback on Past Cycle

• The five most helpful tools/resources were the Consultants, OQM Staff, PM Presentation Template, PMP template, their supervisors

• Respondents reported just above mid-scale improvements to their KSAs for most topics listed

• Respondents valued assistance provided, implementing measurement systems, knowledge and experience gained, understanding of processes

• Obstacles included lack of time/resources, issues with data collection/measurement, applying the BSC approach

104

Summary (cont.)

Planning for Next PM Cycle

• Most helpful to teams for the future would be PM consultants, OQM staff, IT Support for data collection systems

• To foster the integration of PM into daily business operations, respondents said we need to help foster the use of data, continue support provided, increase management involvement in process

105

Summary (cont.)

Organizational Climate

• Majority of respondents either agree or strongly agree that:

• Their organization is committed to PM (76%)• They see the value of PM for improving business operations

(68%)• Managers in their organization believe accountability is

important (68%)• The culture of their organization is changing to be more

results-oriented (63%)

• Senior managers more strongly endorse these items compared to team leaders and team members

106

Summary (cont.)

Customer Scorecard Trend Analysis

• Ratings on all dimensions are lower in Cycle 2 compared to Cycle 1

• Current ratings:• Highest on responsiveness, competence, availability• Lowest on cost, handling of problems, timeliness

107

Appendix

Individual Comments From Survey

108

What did you value most about this cycle of performance management? Individual Comments

• Really customizing the strategic plan to our particular area . Implementing a system to gather data to understand our service delivery.

• Measuring customer service performance is valuable, however, I am not sure much was gained by this exercise over the past several years other than taking people away from assigned duties.

• Understanding the process.• Nothing.• News I can use!• Results from data collection - mostly trend reporting.• Nothing - I waste of time and tax payers money. • Amy is a wealth of knowledge and always eager to assist when

needed.• The increased involvement of the OQD staff in the team meetings.

109

What did you value most about this cycle of performance management? (cont.) Individual Comments

• Our OQM representative and PM consultant listened to our concerns about this method of measuring our productivity. They convinced us that this type of measurement and representation could represent our points of view and culture better than we imagined.

• Continuity from prior year. Beginning to provide a context, esp. in using strategy mapping.

• Our consultant was great. This is the first year I understood what we were doing.

• It spurred us to initiate and develop a new call center that we know use to keep track of our work and help keep more statistics on the work we do. The concept of coordinated, consistent performance mgt techniques and metrics is a good one.

• I support the formalized training for ABC, Balance Scorecard etc. However, it was extremely beneficial to have just in time training provided by Pat Patterson as we progressed through our PMP process.

110


• Learning about data collection and how to use that data to develop improvement plans.

• OQM supported a program review and a root cause analysis as part of this years PMP that was extremely useful. Really appreciate the expertise and responsiveness of both OQM and contract staff. Our contract staff is a real standout.

• The Measures Workshop and Balanced Scorecard Orientation.• Our consultant was extremely helpful - Pat Patterson. Carmen, Amy and

Joe were also extremely helpful even though they were not our consultant. They answered our questions before we got to Josh (who was not our OQM consultant) but he was helpful.

• I gained experience in and fuller appreciation for the importance of collecting and analyzing data about our Division's efficiency and effectiveness. The PMP has provided insights that are guiding us in improving both of these areas. The data we collected and analyzed had use beyond informing us about our own operation--when SDs external to our program recently requested detailed information about our operations, we were able to respond quickly and completely.

111


• It's become so familiar that I spend more time on the data versus the process.

• OQM and the consultant's support throughout the process.• Experience under my belt in all areas this cycle.• I valued the professional guidance provided by OQM. Ms. Amy

instilled the knowledge to our group on why we perform this task which gave more definition and purpose for doing it. She was outstanding and I would like to commend her for a job well done. From depositing her guidance and understanding into our group, I have learned the reasons for the PMP which will reveal to the manager how to effectively administer the business.

• Having a chance to make suggestions as to what can be improved in the process. Actually seeing some improvements in the different layers of the process - this is invaluable!

112

What obstacles were most challenging to you during this cycle of performance management?Individual Comments

• The OQM staff member for my team, was not committed, only attended less then half of the mtgs.

• Too many man hours are required for the PM Initiative during a time when FTEs are reduced and workloads are at an all-time high Question the cost effectiveness of the PM Initiative. Do we have any data the supports the cost (i.e., man-hours, OQM salary, consultants) of the PM Initiative?

• THE BSC is not being implemented right, nor are the results being used for anything. I think the current system is too prescriptive, it's not flexible enough for organizations to incorporate their own modifications or metrics. It has to be balanced scorecard, which is ok, but the not only way.

• The Associate Director never reviewed our template though given to him in the beginning months. Management in our office is in process of changing and I am apprehensive to new management.

113

What obstacles were most challenging to you during this cycle of performance management? (cont.) Individual Comments

• A culture that doesn't support accountability Resistance to change by people in the organization.

• Trying to identify our division and its measurable values.• I do not think we need the conference at the end. It was a farce to

have the ORSAC list who they wanted to present and then not one of them show up. It makes no sense and just takes up valuable time. I think that the Management Council should be consulted on what to do this year. Let them vote on what should be done, instead of us being told what to do, so there is buy in.

• The lack of time to fully define, measure, and implement results.• Defining and measuring unit cost.• Too much time spend on this project by team members.• Having to manually collect data.• Analyzing the data through graphs and making trend projections.

114


• Move from Discrete Service to Service Area review continues to prove problematic in applying Internal Business perspective to improve business performance.

• Trying to define meaningful measurements of productivity and progress within this model of activity.

• financial perspective. Hard to relate to effectiveness. Is a measure for the sake of measuring only. insistence on full up presentation preparation each year. Why not some key indicators only except for a select few chosen each year on rotating basis.

• Director of our organization did not find time to meet with our team even once, although we had a number of issues to discuss. Success of our service group is greatly dependent on close communication with upper management in ORF and with other NIH stakeholders. This is missing. We can only correct what we control; our suggestions for changing what we don't control are always ignored.

115


• I think the ASA/PMP is pushed down to too low a level. My five person office really shouldn't be doing it's own ASA/PMP. It's a time consuming dis-economy of scale. Our parent office should incorporate it into their ASA/PMP, we are just too small in numbers and dollars to get any significant value out of the time we spend on the ASA/PMP. We collected stats and performance metrics on our work even before the ASA/PMP, because that's what good mgt does, which was sufficient enough for us to track and improve our service. But it was less intensive than the current PMP.

• The lingo. If you do this on a daily basis it's easy to remember. The entire process of balanced scorecard, service hierarchy, etc. is hard to remember if you do not do it every day.

• Getting information to resolve issues.• OQM Staff member hardly ever attended any of our meetings.

Miscommunication regarding what was needed.

116


• Due to multiple layers of management involvement we did some re-tracking over materials that produced some levels of frustration. Some tracking forms proved less than useful.

• Our Team was not initially convened when others in our Division started; so we had to play catch-up. The service for which I was designated Team Leader passed to the MEO, so it was determined late in the cycle that we would not present.

• Time -our office was short staffed (and still is). Being able to get to have any quality focus was difficult -- This is where our consultant was extremely helpful. Because of the systems we used for counting (e.g. ADB) we had to do a lot of manual counting.

• Gathering data for entire discreet service.• After three years of ASA/PMP, I feel that our division has

addressed everything that was/could be of concern. I see PMP as now being in a maintenance mode for us since our process is fairly straightforward.

117


• Team member participation.• Last minute changes.• Reading the graphs.• Data collection - I waited too late before collecting it. Obtaining the

correct measures and objectives.• Trying to make the experience as interesting as possible since the

process could be boring at times.• Internal disagreements in OQM staff led to confusion on how

proceed with the process. Delay in proceeding forward due to schedule in-availability of OQM staff to meet.

118

How can we help integrate PM into your daily business activities? Individual Comments

• Managers realizing the importance of data for decision making.• Make the reasons of why we are doing this more concrete - more

specific examples of how it has helped others.• Understand what our division does and applying the PMP to the

function.• There must be something else that can be done instead of the

conference. We should be able to learn from each other on how to collect data.

• I have used the provided control charts to forecast and project overtime needs.

• Have upper management do this project.• Follow up on documentation of implementation of objectives, i.e.,

training records and frequencies of meetings/actions.• Insist on development of standard and specialized business tools and

services to support the service activity.

119

How can we help integrate PM into your daily business activities? (cont.) Individual Comments

• As far as I can tell the ASA/PMP is a waste of time, nothing is ever done with any of the information collected. The data is collected and analyzed and we look at it and say "oh that's nice" or "that's interesting", but no changes or new initiatives have ever started because of the data, not our level or at the ORS/ORF level. If there have been changes at any level I would be interested in knowing about them. Particularly at the ORS/ORF level.

• I already do by collecting data and putting it into spreadsheets almost monthly to track trends. I did not ever do this until the PMP program so it has helped me significantly. The data really does help in showing the value of my programs and where there are weaknesses and strengths!

• By being focused on true value added.• Maintain communication throughout the year.

120

How can we help integrate PM into your daily business activities? (cont.) Individual Comments

• The PMP is very helpful in its present form. Like virtually any good thing, however, it has the potential to grow to a point of diminishing returns, with gradual increases in requirements to collect, analyze and present data detracting from a Division's

• Unfortunately, the only way of assuring it gets tied in is to tie it to performance reviews.

• You already have.• Continue support you are currently providing.• Data review and collection.• You have helped.• We are integrated and know the cost of the activities of our service

group.

121

OtherIndividual Comments

• This is too long and I don't have time to fill it all out.

Results from the Performance Management Survey By Janice Rouiller and Laura Stouffer, SAIC and

Documents

Transcript of Results from the Performance Management Survey By Janice Rouiller and Laura Stouffer, SAIC and