1 Results from the Performance Management Survey Prepared by Janice Rouiller and Laura Stouffer, SAIC and Joe Wolski, OQM 9 May 2005 2 Table of Contents Survey Background....3 Respondent Information.7 Perceptions of Cycle 3 PM Implementation..15 Perceptions of Needs for Cycle 4...52 Organizational Climate Perceptions...68 Organizational Climate Perceptions by Program Area78 Organizational Climate Perceptions by Role84 Customer Scorecard Ratings..88 Summary of Comments.. Summary...... Appendix - Individual Comments from Survey...111 3 Survey Background 4 Purpose Administer a customized version of the ORS Customer Scorecard to: Gather feedback on past Performance Management (PM) implementation efforts Collect needs assessment information to tailor the on-going implementation of PM Gather data on the required Customer Scorecard dimensions Collect organizational culture change information for the Evaluation Study 5 Methodology Administer the survey at the completion of the third cycle of Performance Management implementation on an organization-wide basis Which occurred in February 2005 Designed and tested web survey during February 2005 Survey went live on 3 February, requests sent to all PM Team Leaders, Members, and additional senior ORS/ORF managers First remindersent 17 February 2005 Closed survey field period 24 February 2005 Data file sent to contractor for analysis 6 Survey Distribution Number ofrecipients186 Number of respondents 41 Response Rate 22% 7 Respondent Information 8 Respondents by Role N = 45 Note: Multiple responses allowed. Number of Respondents 9 Respondents by Organization N = 41 Number of Respondents 10 Respondents by Program Area N =41 Number of Respondents 11 Experience with the PM Process Percent Yes N = 41 N = 40 N = 36 N = 35 12 Approximately what percent of time did you spend on PM-related activities during FY04? N = 41 Percent of time Spent Number of Respondents 13 On average, how often did your PM team meet during FY04? N = 41 How Often PM Team Met Number of Respondents 14 To what extent did your team members work between meetings to implement PM? To No Extent To a Great Extent N = 41 Mean = 6.71 Median = 6.5 15 Perceptions of Cycle 3 PM Implementation (January January 2005) 16 How helpful did you find the following in implementing PM? Not at all Helpful Extremely Helpful Mean Ratings 17 How helpful did you find the following tools in implementing PM? N = 41 Not at all Helpful Extremely Helpful Mean Ratings N = 41 N = 38 N = 40 N = 41 18 How helpful did you find the following tools in implementing PM? Listings of the Services Hierarchy Not at all Helpful Extremely Helpful N = 41 Mean = 5.46 Median = 6 19 How helpful did you find the following tools in implementing PM? Performance Management Plan (PMP) Template Not at all Helpful Extremely Helpful N = 41 Mean = 7.15 Median = 8 20 How helpful did you find the following tools in implementing PM? Strategy Mapping Not at all Helpful Extremely Helpful N =40 Mean = 6.29 Median = 7 21 How helpful did you find the following tools in implementing PM? Measures Roadmap Template Not at all Helpful Extremely Helpful N = 41 Mean = 6.18 Median = 7 22 How helpful did you find the following tools in implementing PM? Initiative Roadmap Template Not at all Helpful Extremely Helpful N = 41 Mean = 6.25 Median = 7 23 How helpful did you find the following tools in implementing PM? Measures Reporting Form Not at all Helpful Extremely Helpful N = 41 Mean = 5.88 Median = 7 24 How helpful did you find the following tools in implementing PM? PM Presentation Template for Conference Not at all Helpful Extremely Helpful N = 38 Mean = 6.52 Median = 8 25 How helpful did you find the following data analysis services in implementing PM? Not at all Helpful Extremely Helpful Mean Ratings N = 41 N = 40 N = 41 26 How helpful did you find the following data analysis services in implementing PM: Design of Data Collection Forms Not at all Helpful Extremely Helpful N = 41 Mean = 5.38 Median = 5.5 27 How helpful did you find the following data analysis services in implementing PM: Establishing Data Collection Methodologies Not at all Helpful Extremely Helpful N = 41 Mean = 6.00 Median = 7 28 How helpful did you find the following data analysis services in implementing PM: Graphing Data Not at all Helpful Extremely Helpful N = 40 Mean = 6.06 Median = 7 29 How helpful did you find the following data analysis services in implementing PM: Process Behavior Chart Analysis Not at all Helpful Extremely Helpful N = 41 Mean = 5.86 Median = 7 30 How helpful did you find the following data analysis services in implementing PM: ORS Customer Scorecards Not at all Helpful Extremely Helpful N = 41 Mean = 6.45 Median = 7 31 How helpful did you find the following communication vehicles in implementing PM? N = 40 Not at all Helpful Extremely Helpful Mean Ratings N = 40 32 How helpful did you find the following communication vehicles in implementing PM: Performance Management Website Not at all Helpful Extremely Helpful N = 40 Mean = 4.78 Median = 4 33 How helpful did you find the following communication vehicles in implementing PM: PM Relateds from OQM Not at all Helpful Extremely Helpful N = 40 Mean = 5.74 Median = 6 34 How helpful did you find the following communication vehicles in implementing PM: PM Conference Not at all Helpful Extremely Helpful N = 40 Mean = 5.44 Median = 6 35 How helpful did you find assistance from the following groups in implementing PM? N = 39 Not at all Helpful Extremely Helpful Mean Ratings N = 40 N = 38 36 How helpful did you find assistance from the following groups in implementing PM? OQM Staff Not at all Helpful Extremely Helpful N = 39 Mean = 6.38 Median = 8 37 How helpful did you find assistance from the following groups in implementing PM? PM Consultants Not at all Helpful Extremely Helpful N = 40 Mean = 7.97 Median = 9 38 How helpful did you find assistance from the following groups in implementing PM? Other PM Teams Not at all Helpful Extremely Helpful N = 38 Mean = 4.96 Median = 6 39 How helpful did you find assistance from the following groups in implementing PM? Your Supervisor Not at all Helpful Extremely Helpful N = 38 Mean = 7.82 Median = 8 40 Immediate Supervisor Involvement in PM To no Extent To a Great Extent Not at all Helpful Extremely Helpful Mean Ratings N = 38 N = 39 41 What extent was your immediate supervisor involved in your PM process during this PM cycle? To No Extent To a Great Extent N = 38 Mean = 7.53 Median = 8 42 How helpful was the involvement of your immediate supervisor in PM related matters during this cycle? Not at all Helpful Extremely Helpful N = 39 Mean = 7.60 Median = 8 43 To what extent have your knowledge/skills/abilities (KSAs) improved in the following areas during this PM cycle? N = 38 To No Extent To a Great Extent Mean Ratings N = 40 N = 39 N = 38 N = 39 N = 38 44 To what extent have your knowledge/skills/abilities (KSAs) improved in the following areas during this PM cycle? Balanced Scorecard Approach to PM To No Extent To a Great Extent N = 38 Mean = 6.03 Median = 7 45 To what extent have your knowledge/skills/abilities (KSAs) improved in the following areas during this PM cycle? Methods to Gather Data To No Extent To a Great Extent N = 40 Mean = 6.18 Median = 7 46 To what extent have your knowledge/skills/abilities (KSAs) improved in the following areas during this PM cycle? Methods to Analyze Data To No Extent To a Great Extent N = 40 Mean = 5.68 Median = 7 47 To what extent have your knowledge/skills/abilities (KSAs) improved in the following areas during this PM cycle? Presentation Preparation and Delivery To No Extent To a Great Extent N = 40 Mean = 5.54 Median = 7 48 To what extent have your knowledge/skills/abilities (KSAs) improved in the following areas during this PM cycle? Working in a Team To No Extent To a Great Extent N = 39 Mean = 5.54 Median = 6 49 To what extent have your knowledge/skills/abilities (KSAs) improved in the following areas during this PM cycle? Using Excel To No Extent To a Great Extent N = 38 Mean = 5.03 Median = 6 50 To what extent have your knowledge/skills/abilities (KSAs) improved in the following areas during this PM cycle? Using PowerPoint To No Extent To a Great Extent N = 39 Mean = 4.79 Median = 5 51 To what extent have your knowledge/skills/abilities (KSAs) improved in the following areas during this PM cycle? Using Visio To No Extent To a Great Extent N = 38 Mean = 4.19 Median = 5 52 Perceptions of Needs for Cycle 4 53 In planning for the next PM cycle, how helpful would each of the following be to your PM Team? Not at all Helpful Extremely Helpful Mean Ratings 54 In planning for the next PM cycle, how helpful would each of the following tools/resources be to your PM Team? N = 40 Not at all Helpful Extremely Helpful Mean Ratings N = 40 N = 37 N = 39 N = 38 N = 37 55 In planning for the next PM cycle, how helpful would each of the following tools/resources be to your PM Team: PM Consultants Provided by OQM Not at all Helpful Extremely Helpful N = 40 Mean = 7.84 Median = 9 56 In planning for the next PM cycle, how helpful would each of the following tools/resources be to your PM Team: OQM Staff Not at all Helpful Extremely Helpful N = 40 Mean = 6.57 Median = 7 57 In planning for the next PM cycle, how helpful would each of the following tools/resources be to your PM Team: Regular Meetings with your Management Not at all Helpful Extremely Helpful N = 37 Mean = 7.03 Median = 8 58 In planning for the next PM cycle, how helpful would each of the following tools/resources be to your PM Team: Rotating Quarterly Presentation Meetings Not at all Helpful Extremely Helpful N = 39 Mean = 5.94 Median = 7 59 In planning for the next PM cycle, how helpful would each of the following tools/resources be to your PM Team: Yearly Performance Management Conference Not at all Helpful Extremely Helpful N = 38 Mean = 5.59 Median = 7 60 In planning for the next PM cycle, how helpful would each of the following tools/resources be to your PM Team: IT Support to Establish Data Collection Systems Not at all Helpful Extremely Helpful N = 37 Mean = 7.47 Median = 8 61 In planning for the next PM cycle, how helpful would each of the following training sessions be to your PM Team? N = 38 Not at all Helpful Extremely Helpful Mean Ratings N = 39 N = 40 N = 39 N = 40 62 In planning for the next PM cycle, how helpful would each of the following training sessions be to your PM Team? Balanced Scorecard Methodology Not at all Helpful Extremely Helpful N = 38 Mean = 5.97 Median = 6 63 In planning for the next PM cycle, how helpful would each of the following training sessions be to your PM Team? Setting up Data Collection Systems and Forms Not at all Helpful Extremely Helpful N = 39 Mean = 6.45 Median = 7 64 In planning for the next PM cycle, how helpful would each of the following training sessions be to your PM Team? Analyzing Data Through Graphs Not at all Helpful Extremely Helpful N = 40 Mean = 6.58 Median = 8 65 In planning for the next PM cycle, how helpful would each of the following training sessions be to your PM Team? Analyzing Processes with Process Behavior Charts Not at all Helpful Extremely Helpful N = 39 Mean = 6.54 Median = 8 66 In planning for the next PM cycle, how helpful would each of the following training sessions be to your PM Team? Customer Assessment through Interviews, Focus Groups and Surveys Not at all Helpful Extremely Helpful N = 40 Mean = 6.86 Median = 8 67 In planning for the next PM cycle, how helpful would each of the following training sessions be to your PM Team? ORS ABC Cost Accounting Model Not at all Helpful Extremely Helpful N = 40 Mean = 6.56 Median = 8 68 Organizational Climate Perceptions 69 Organizational Climate Perceptions Summary Strongly Disagree Strongly Agree N = 40 N = 39 N = 40 N = 39 N = 40 Neither Agree nor Disagree 70 I see the value of performance management for improving business operations in my area. Strongly Disagree Strongly Agree N = 40 Mean = 3.59 Median = 4 Neither Agree nor Disagree 71 My organization (ORS or ORF) is committed to the performance management effort. Strongly Disagree Strongly Agree N = 39 Mean = 4.03 Median = 4 Neither Agree nor Disagree 72 I understand what the Services Hierarchy is and its purpose. Strongly Disagree Strongly Agree N = 40 Mean = 3.50 Median = 4 Neither Agree nor Disagree 73 The culture of my organization (ORS or ORF) is changing to be more results-oriented. Strongly Disagree Strongly Agree N = 40 Mean = 3.70 Median = 4 Neither Agree nor Disagree 74 Managers in my organization (ORS or ORF) believe accountability is an important organizational value. Strongly Disagree Strongly Agree N = 39 Mean = 3.84 Median = 4 Neither Agree nor Disagree 75 Performance management has contributed to improvements in my area. Strongly Disagree Strongly Agree N = 40 Mean = 3.21 Median = 4 Neither Agree nor Disagree 76 Performance management will assist my organization if we have to go through an A-76 competition. Strongly Disagree Strongly Agree N = 40 Mean = 3.56 Median = 4 Neither Agree nor Disagree 77 My PM Team (or the groups I am responsible for) are actively involved in data collection and analysis. Strongly Disagree Strongly Agree N = 40 Mean = 4.15 Median = 5 Neither Agree nor Disagree 78 Organizational Climate Perceptions by Program Area 79 Program and Employee Services Climate Perceptions Positive Responses Percent Agree or Strongly Agree N = 7 80 Scientific Resources Climate Perceptions Positive Responses Percent Agree or Strongly Agree N = 6 81 Security and Emergency Response Services Climate Perceptions Positive Responses Percent Agree or Strongly Agree N = 8 82 Management Services Climate Perceptions Positive Responses Percent Agree or Strongly Agree N = 3 83 Real Estate and Facilities Climate Perceptions Positive Responses Percent Agree or Strongly Agree N = 4 84 Organizational Climate Perceptions by Role 85 Division Directors/Office Directors Climate Perceptions Percent Agree or Strongly Agree N = 5 86 PM Team Leaders Climate Perceptions Percent Agree or Strongly Agree N = 7 87 PM Team Members Perceptions Percent Agree or Strongly Agree N = 27 N = 26 N = 27 88 Customer Scorecard Ratings 89 Customer Scorecard Ratings for PM Overview for Cycle 3 N = 40 UnsatisfactoryOutstanding Mean Ratings N = 40 N = 39 N = 38 N = 40 N = 39 N = 40 90 Cost Satisfaction Ratings UnsatisfactoryOutstanding N = 40 Mean = 5.92 Median = 7 91 Quality Satisfaction Ratings UnsatisfactoryOutstanding N = 40 Mean = 6.77 Median = 8 92 Timeliness Satisfaction Ratings UnsatisfactoryOutstanding N = 39 Mean = 6.67 Median = 7 93 Reliability Satisfaction Ratings UnsatisfactoryOutstanding N = 38 Mean = 6.82 Median = 8 94 Availability Satisfaction Ratings Unsatisfactory Outstanding N = 40 Mean = 7.16 Median = 8 95 Responsiveness Satisfaction Ratings UnsatisfactoryOutstanding N = 40 Mean = 7.16 Median = 8 96 Convenience Satisfaction Ratings UnsatisfactoryOutstanding N = 39 Mean = 7.23 Median = 8 97 Competence Satisfaction Ratings UnsatisfactoryOutstanding N = 40 Mean = 7.71 Median = 8 98 Handling of Problems Satisfaction Ratings UnsatisfactoryOutstanding N = 40 Mean = 7.18 Median = 8 99 Customer Scorecard Ratings for PM Trend Analysis Unsatisfactory Outstanding Mean Ratings N = 37 N = 70 N = 67 N = 40 N = 39 N = 38 N = 68 N = 67 N = 68 100 Customer Scorecard Ratings for PM Trend Analysis (cont.) Unsatisfactory Outstanding Mean Ratings N = 68 N = 66 N = 69 N = 65 N = 40 N = 39 N = 40 N = 68 N = 69 N = 68 N = 65 101 Summary of Comments 102 What did you value most about this cycle of performance management? Knowledge and Experience Gained (4) Results show that we have improved since last year. Surveys have provided an opportunity to find out how customers perceive quality of service. Data gathering has helped quantify quality of service. Assistance provided by consultants and OQM staff (3) Interaction with OQM staff & consultants in analyzing performance. Knowledgeable and helpful consultants. Measurement Tools (3) The use of data reporting template and the initiatives template. The Measures Reporting Form allowed us to look at the objectives and measures in a more relevant perspective. Note: Comments were sorted into categories. Most frequently mentioned categories listed first. 103 What obstacles were most challenging to you during this cycle of performance management? Note: Comments were sorted into categories. Most frequently mentioned categories listed first. Lack of Time and/or Resources (6) Lack of time to devote to PMP and keep up with workload Short notice OQM website is out of date Miscellaneous (4) Learning to use Visio and Excel Preparing the report Lack of Management Involvement (3) Not having discussions about PM in staff meetings Managers need to use data in decision making Data Collection and Measurement (2) Obtaining data Lack of commitment from team members (2) Getting all team members to contribute 104 How can we help integrate PM into your daily business activities? Increase support currently provided (6) Need help in learning how to distribute data collection throughout the division. Need help collecting and interpreting data. Miscellaneous (3) Work to establish timelines so that we see the year at a glance. Continue emphasis on the importance of performance management. 105 Summary 106 Summary Survey Respondents Response rate of 22% Respondents were primarily from ORS (73%) The majority of respondents were team members (68%) About 75% of respondents were experienced with PM (i.e., participated in FY03) About 60% of respondents attended FY04 training About 63% of respondents spent 20% or less of their time during the past year on PM Team meetings typically occurred once a month (41%), biweekly (22%), and quarterly (20%) 107 Summary (cont.) Feedback on Past Cycle Most Helpful Tools/Resources PM consultants Service group supervisors PM template PM presentation template ORS Customer Scorecards OQM Staff. These are the same tools/resources receiving the highest ratings in FY03. KSA Improvements Just above mid-scale improvements noted on all topics with the following exceptions: PowerPoint and Visio. Perhaps respondent familiarity with these tools has leveled off. 108 Summary (cont.) Planning for Next PM Cycle Future needs Most helpful to teams for the future would be PM consultants IT support for data collection systems Regular meetings with management to discuss PM data Training on customer assessment Training on analyzing data These requests are similar to previous years 109 Summary (cont.) Organizational Climate Majority of respondents either agree or strongly agree that: PM Teams are actively involved in data collection (85%) Their organization is committed to PM (77%) Managers in their organization believe accountability is important (64%) They see the value of PM for improving business operations (60%) The culture of their organization is changing to be more results-oriented (58%) 110 Summary (cont.) Customer Scorecard Trend Analysis Ratings on dimensions show minor variance from cycle to cycle but none of the differences are statistically significant. Current ratings: Highest on competence, convenience, and handling of problems, Lowest on cost 111 Appendix Individual Comments From Survey 112 What did you value most about this cycle of performance management? Individual Comments The opportunity to find out how my customers perceive the quality of the job I'm doing. Data gathering enabled us to quantify that construction inspection was decreasing our submittal review response time. Our consultant Janice Rouiller. The survey provided verification of the quality of service DMCS is providing the NIH community. The numbers show that last year's improvements have not only been sustained, but improved upon. Honestly, I must state NOTHING! This ploy that the PMP process is management tool is hog-wash, the true intent of the PMP is to tie it to how much time staff spend in each function and to cut staff. Interaction with OQM staff & consultants in analyzing performance. 113 What did you value most about this cycle of performance management? (cont.) Individual Comments Patrick Patterson, is the consultant with whom our team works on PMP. His skills & knowledge of all aspect of the program are excellent. He ensures that we see the process & purpose rather than just do it. The willingness of the ORS Scientific resources to be first in the rotation of PM conferences. The use of data reporting template and the initiatives template. Meeting as a group to plan ahead. the introduction of the use of the Business Framework and the concept of the Balanced Scorecard. It was helpful to focus just on the high impact objectives. The Measures Reporting Form allowed us to look at the objectives and measures in a more relevant perspective. 114 What obstacles were most challenging to you during this cycle of performance management? Individual Comments Preparing the report is painful. Being taken away from my primary duties created undue stress and hardship when OQM wanted their deadlines met, yet they couldn't seem to get their parts done in a timely fashion. We did not have the time to do the customer survey focus groups we had planned on doing. OQM website is out of date and needs updating. The challenge of continuing performance gains without losing any quality. learning to use Excel and Visio. Lack of time to devote to PMP AND perform ones own work. PMP is almost a full time job. This is an OQM function and should be performed by that staff. Supervisors and managers should use data more often in their decision-making. 115 What obstacles were most challenging to you during this cycle of performance management? (cont.) Individual Comments Commitment from all the team members. obtaining data. Getting all team members to contribute to the tasks of identifying relevant data and analyzing it. The way the PMP was created in Excel with an entire page in one row of cells made it extremely time consuming & tedious to update. Switching the rules in the middle of the game! We went from PMP, to score card, then to the dash board. These aberrations made staff wonder what value all of this has to our the ultimate duty to provide the service to the Institutes. Not having discussions about the performance of service groups/discrete services in staff meetings. Lack of a program area manager who accepts PM plans and manage their implementation. 116 What obstacles were most challenging to you during this cycle of performance management? (cont.) Individual Comments Concentration on the Measures Reporting Form throughout the year caused the PMP to be neglected. At the end of the cycle, it took a lot of time to update the PMP to match the Measures Reporting Form. The short notice and scheduling a report due for the week of January 17 was inane! 117 How can we help integrate PM into your daily business activities ? Individual Comments We have always been collecting and analyzing data to improve our program--no value added from this formal process. In my opinion performance management might be applicable for some businesses in private industry but not for the US Gov,t. It already is, and was before I was tasked with being a part of this initiative. Work to establish timelines so that we see the year at a glance. Several times my consultant worked with me to make deadlines at the last minute. Implement the ORF design review and inspection processes Already done. Have the ORS Office of Quality Management (OQM) staff do the work. In many agencies and in private sector companies this function is not put on the backs of the line staff but is performed by either consultants or an office such as OQM. 118 How can we help integrate PM into your daily business activities ? (cont.) Individual Comments Need help in learning how to distribute data collection throughout the division. Follow up with Quarterly team meetings. Continue emphasis on the importance of performance management to efficient 7 effective business operations. Need help collecting observation type data. Need help interpreting data.