The Development of a Pay-for-Per-formance Appraisal System for Pub-lic Park and Recreation Agencies: A Case Study

Volume 26, Number 4pp. 126-156

Journal of Park and Recreation AdministrationWinter 2008

Michael A. MulvaneyWilliam R. McKinneyRichard Grodsky

EXECUTIVE SUMMARY: Well-designed employee performance appraisals provide agencies with information that can guide administrative and devel-opmental decision-making about their most important asset—their human resources. Administratively, performance appraisals serve as the formal evaluation tool used by managers when making decisions about distribution of pay increases. Developmentally, performance appraisals assist agencies in identifying issues such as employee training needs, selection, and promo-tional opportunities (Milkovich & Newman, 2005). Despite its importance, an agency’s performance appraisal system is often viewed by employees and management as a frustrating and unfair process. These frustrations are largely attributed to performance appraisal instruments that are not job related, have confusing or unclear rating levels, and are viewed as subjective and biased by staff (Mathis & Jackson, 2006). A study was undertaken to respond to this issue by identifying the steps involved in creating a pay-for-performance system for a public park and recreation agency. This case study (1) identified a systematic procedure for creating performance appraisal instruments, (2) described the appropriate training necessary for those conducting a perfor-mance appraisal interview, (3) implemented the performance reviews using the developed instruments and the results of the appraisal training, and (4) determined how much of a merit increase is to be given for different levels of performance. Emergent themes from focus group interviews with employees identified differences between employee attitudes toward the original pay-for-performance system and the newly developed system, highlighting the value of employee participation in the development of the appraisal system. Results of the case study are analyzed and discussed.

Authors: Michael. A. Mulvaney is with the Department of Recreation Administration, Eastern Illinois University, 600 Lincoln Avenue, Charleston, IL 61920.Email: mamulvaney@eiu.edu. Phone: (217) 581-6589. William

127

McKinney is with the Department of Recreation, Sport, and Tourism at the University of Illinois. Richard Grodsky is the Executive Director for the Elmhurst Park District.

Keywords: Compensation, Pay-for-Performance, Human Resource Development, Recreation Management

“Performance appraisal” has become a general description for a variety of activities through which agencies provide feedback to their employees, develop competencies, enhance performance, and distribute rewards (Grote, 2000). An agency’s performance appraisal system impacts individual and organizational operations by guiding decisions about compensation and merit pay increases, training and development opportunities, performance improvement, termination, promotion, organizational climate, and financial management.

Social psychologists using experimental methods have found that performance-based pay enhances effort and improves work quality (Landau & Leventhal, 1976; Mitra, Gupta, & Jenkins, 1997). Economic theorists have argued that performance-contingent rewards motivate effort and attract talent when effort and talent are not easily increased or advanced (Holstrom, 1979; Zenger, 1992). Authors studying organizational management have also cited the benefits of distributing rewards based on performance (Milkovich & Newman, 2005). The results of these findings have worked their way into professional practice: employer surveys indicate that almost 90% of U.S. agencies use a pay-for-performance system to award pay increases (Cascio, 2006).

Despite the many uses of a performance appraisal system, the process is often seen by park and recreation administration and staff as a painful annual exercise in which the manager rates the performance of subordinates over the past 12 months and uses that information for administrative and developmental decisions. Furthermore, poorly designed and administered pay-for-performance systems are likely to be demotivating and generate anxiety among employees (Bassett, 1994).

Dolan (1996) identified five characteristics of deficient pay-for-performance systems: (1) the performance measure(s) are unclear or not job specific, (2) supervisors may not be adequately trained to evaluate employee performance, (3) the incentive value of the reward offered may be too low, (4) the link between performance levels and rewards may be weak or unnoticeable, and (5) the agency failed to address the issue of redlining (i.e., employees moving out of their job’s pay range).

Recognizing that one of the major difficulties with performance appraisal stems from the variety of objectives the appraisal is supposed to address (development, promotion, termination, staff training, etc.) and that salary decisions account for nearly 80% of appraisal uses (Mathis & Jackson, 2006; Thomas & Bretz, 1994; Smith, Hornsby, & Shirmeyer, 1996), this study provides an overview of the collaborative steps involved in creating a performance appraisal system used for merit pay increase decisions within a park and recreation agency. The purpose of this study was to describe the development process and procedures undertaken for a public park and recreation agency’s performance appraisal instrument, illustrating it with a representative case study. The value of employee participation throughout the appraisal system’s development is also highlighted.

128

Review of Related Research

This study is based on a strategic human resource management (SHRM) concept that places great importance on employees and managers in the success of the agency’s operations (Tompkins, 2002; Wright & McMahan, 1992). SHRM recognizes the value of an agency’s material resources (i.e., financial and physical) but asserts that it is equally, if not more, important to give attention to the agency’s human resources. This rationale is appropriate within a service field such as public parks and recreation, in which the human resources put the material resources into use and convert them into leisure programs and services and for which human resources typically account for more than 60% of an agency’s yearly operational budget (Chelladurai, 1999; Edginton, Hudson, & Lankford, 2001; McKinney & Yen, 1989).

Pay-for-performance systems have been described as one of the most effective methods of motivating and increasing employee performance (Levy & Williams, 2004; Moss & Martinko, 1998; Nadler, 1979; Podsakoff & Farh, 1989). Pay-for-performance plans relate adjustments in an employee’s base pay directly to his or he job performance. These plans theoretically forge a link between pay expenditures and individual productivity and communicate a useful message to employees (Campbell, Campbell, & Chia, 1998). A well-developed pay-for-performance appraisal instrument addresses the norm of distributive justice, or the commonly held belief that individuals should be rewarded in proportion to their contributions. Thus, an adequately developed appraisal instrument potentially diffuses employee concerns about equity and fairness while motivating employees to increase their performance.

A well-designed appraisal instrument helps an agency distinguish outstanding performers from those who are below average. Agencies that recognize and discriminate among employee contribution levels create an atmosphere conducive to individual and agency growth (Chen & Min, 2004). In contrast, an agency using a generic, organization-wide appraisal form that does not rate employees according to job-specific criteria, and is more subjective in content and implementation, faces rater error, deficient criteria, and erroneous merit pay decisions (Campbell, Campbell, & Chia, 1998). Human resources represents one of the largest expenses in a public park and recreation agency; therefore, the need to develop and use an accurate pay-for-performance appraisal instrument reflects the increasing demand for wise fiscal management practices in the public sector (Bartlett & McKinney, 2004; Edginton, Hudson, & Lankford, 2001; Edginton & Williams, 1978).

A pay-for-performance plan moves from an entitlement orientation where all employees receive the same annual raise for simply showing up for work to a pay system that varies with individual performance. However, creating an effective, valid, and legally defensible pay-for-performance plan requires three things: (1) clearly defined, job-specific performance criteria that enable creation of an appraisal instrument that describes different performance levels from low to high, (2) a well-defined performance appraisal interview process, and (3) equitable decisions about how much of a merit increase will be given for different levels of performance (Milkovich & Newman, 2005).

129

The Importance of Staff Involvement

Even the most well developed pay-for-performance system is predisposed to problems if it is viewed negatively by staff (Cardy & Dobbins, 1994; Keeping & Levy, 2000; Murphy & Cleveland, 1995). In particular, employee reactions to performance appraisal have been closely linked to acceptance and use of an appraisal system as well as the validity of an appraisal (Bernadin & Beatty, 1984; Cardy & Dobbins, 1994; Murphy & Cleveland, 1995). Cardy and Dobbins (1994) asserted that “with dissatisfaction and feelings of unfairness in process and inequity in evaluations, any appraisal system will be doomed to failure” (p. 54). Furthermore, Murphy and Cleveland (1995) contended that “reaction criteria are almost always relevant, and an unfavorable reaction may doom the most carefully constructed appraisal system” (p. 314). Thus, it seems clear that among performance appraisal researchers, the assessment of reactions to the performance appraisal instrument and interview is important (Keeping & Levy, 2000).

Research on performance appraisal reactions has identified two general areas: satisfaction with the appraisal instruments and fairness of the appraisal (Giles & Mossholder, 1990; Greller, 1978; Keeping & Levy, 2000; Smither, 1998). Satisfaction, the most widely studied reaction, has been primarily conceptualized into two subcategories: satisfaction with the appraisal interview or session and satisfaction with the overall appraisal system (Keeping & Levy, 2000). Satisfaction with the appraisal interview refers to the employee attitudes about the structure and implementation of the performance review. Satisfaction with the overall appraisal system is the level of satisfaction an employee has with the agency’s entire appraisal system (Keeping & Levy, 2000). Satisfaction with the overall appraisal system represents a more global measure of the appraisal system, including the interview and subsequent actions following the interview (Keeping & Levy, 2000).

Fairness of the appraisal has also been conceptualized into two subcategories: procedural justice and distributive justice (Keeping & Levy, 2000). In this case study, procedural justice is defined as the perceived fairness of the processes and procedures used in the agency’s performance appraisal system. Distributive justice is defined as perceived fairness in the distribution of outcomes (i.e., amounts of merit salary increases) based on the agency’s performance appraisal system.

Focus of the Study

A strategic human resource management (SHRM) framework was applied to the development of a performance appraisal system for the Elmhurst (Illinois) Park District, the public park and recreation agency serving as the representative case study. SHRM places great importance on employees and managers in the success of the agency’s operations (Tompkins, 2002; Wright & McMahan, 1992). It recognizes the value of an agency’s material resources (i.e., financial and physical) but asserts that it is equally, if not more, important to give attention to the agency’s human resources.

The Elmhurst Park District sought to accomplish three goals with the development of a new performance appraisal system. First, the park district wanted to establish a high-performance culture by improving employee motivation and job performance while controlling costs. Second, Elmhurst Park District sought to increase employee

130

buy-in of their performance appraisal system. Specifically, managers wanted to improve employee perceptions of the fairness, accuracy, and overall satisfaction of the agency’s performance appraisal system. Third, managers expressed concern about the need for employees to become and remain aware of the highly demanding nature of today’s park and recreation users. From increasing utilization of new technologies to agency restructuring to meet residents’ leisure needs, Elmhurst Park District employees have been required to learn and adapt at a rapid pace. A recognized method to address these issues is establishment of a valid pay-for-performance plan that rewards high-performing employees while not rewarding employees whose performance falls below an established standard.

This case study explored the process of developing pay-for-performance instruments, conducting performance reviews, and implementing a model for the distribution of merit pay increases. Once the pay-for-performance system was implemented, the study assessed employee attitudes about the newly developed pay-for-performance system compared to the agency’s previous system. Employee satisfaction with the performance appraisal system and their perceptions of the procedural and distributive justice of the system were assessed. Specifically, this study used focus group interview data to address the following questions:

1.) What specific steps are involved in developing a pay-for-performance instrument for a public park and recreation agency?

2.) What procedures are effective for conducting the performance appraisal review?

3.) What steps are involved in developing a model for distributing merit pay increases for a public park and recreation agency?

4.) How do employee attitudes about the new pay-for-performance appraisal instrument differ compared to the previous pay-for-performance appraisal system?

5.) How do employee attitudes about the new performance review session differ compared to the previous performance review session?

6.) How do employee perceptions of the procedural justice of the new pay-for-performance system differ compared to the previous performance review system?

7.) How do employee perceptions of the distributive justice of the new pay-for-performance system differ compared to the previous appraisal system?

Although a case study with a limited number of respondents cannot be widely generalized, the results should be of interest to management researchers and directors of park and recreation agencies who think critically about how to increase employee performance and improve management of pay-for-performance dollars available within the agency’s operating budget. The need to develop a pay-for-performance appraisal system that motivates staff, is cost effective, and assists the agency in meeting its goals is arguably a problem, or opportunity, that has applications across the field of parks and recreation.

131

MethodsCase Study Site

The Elmhurst Park District was selected as the representative public park and recreation agency for this case study. The park district was established in 1920 in portions of Cook and DuPage counties in the Illinois; it oversees approximately 330 park acres, 25 buildings, and 27 park sites and serves approximately 44,500 people. The district employs approximately 58 full-time, 235 part-time, and 325 seasonal employees; is governed by seven publicly elected commissioners; and has an equalized assessed valuation of approximately 1.8 billion dollars.

Prior to the study, the Elmhurst Park District’s performance appraisals were performed annually. A generic (agency-wide) performance appraisal instrument was used for all employees. The instrument was divided into two sections. The first section asked supervisors to evaluate the skills/capabilities that affected an employee’s job performance in the following areas: job knowledge, productivity, attendance, planning, communication, attitude, dependability, leadership/subordinate development, creativity, quality of work, and public contact. These skills/capabilities were applied across the agency to every full-time employee. This universal application was problematic because managers were forced to evaluate employees on several skills/capabilities that did not apply to their specific jobs (e.g., overnight facility maintenance staff evaluated on public contact, lower-level employees evaluated on creativity, senior-level employees on productivity). These skills/capabilities were evaluated on a six-point scale (1= needs improvement; 6 = exceptional).

The second section asked the supervisor to identify the level (on a five-point scale) that best represented the employee’s overall performance for the year. This single scale provided the basis to determine merit pay increases for every full-time employee. The merit pay-increase model was categorized into five performance levels. The employee’s rating on the single scale was matched to the model to determine his or her merit pay increase percentage. For example, if an employee whose pay was below the midpoint of his or her pay range received a rating of 3, that employee would receive a 3% pay increase. An employee above the midpoint of the pay range who received a rating of 3 would receive a 2% increase. Exceptions to these procedures occurred when an employee reached the top of his or her pay range and was unable to receive the specified raise without moving outside the pay range for the job title. In such cases, the employee’s pay is frozen until cost-of-living increases to the entire pay range are applied (usually annually) and the employee’s pay moves back into the range.

Data Collection ProceduresQualitative data was collected to compare employee attitudes between the

original performance appraisal system and the newly developed system. Specifically, focus group interviews with full-time employees of the Elmhurst Park District were conducted. Such interviews are useful in encouraging members of the focus group to state feelings, perceptions, attitudes, and beliefs that they might not express if interviewed individually, for fear of employer retaliation.

The focus groups for this study were chosen based on their position in the agency. Specifically, every employee who supervised and evaluated full-time employees was included in the focus groups. Based upon the supervisory criteria, 17 employees were

132

included in the focus groups. To reduce employee anxiety associated with having the supervisor and subordinate in the same focus group sharing sensitive information, and in an effort to increase the accuracy of the interview data, employees were divided into two focus groups.

The first focus group contained eight director-level positions (recreation director, park director, finance director, information services director, organizational planning director, marketing and communication director, project development director, and a senior accountant). Employees in this focus group had from two to 17 years of experience within their respective job titles. Supervisory responsibility in these positions varied from supervising one employee to a position overseeing seven employees.

The second focus group consisted of nine mid-level management positions (office manager, facility manager, recreation manager, parks manager, administrative services manager, fitness/racquet sports manager, operations manager, human resources manager, and a golf course manager). Job experience within this group ranged from less than a year to seven years of experience. Supervisory responsibility also varied, with one manager responsible for just one full-time employee while three managers supervised more than five employees.

Data from the interviews was collected by note-taking procedures. Note taking was selected over tape recording for the data-collection method because the content discussed in the interviews was sensitive. To help ensure validity of the note-taking procedures, a triangulated approach was adopted: three investigators recorded the interview responses, and comparisons among the three investigators were made to check for accuracy. Consistent with Lincoln and Guba’s (1985) approach to ensuring the quality or trustworthiness of the qualitative data, member checks were used to solicit feedback on the interview findings from the respondents. Specifically, the focus group members were asked to review their recorded responses at the conclusion of each interview session. In addition, the interviewees reviewed the final research report for accuracy and completeness.

Data AnalysisThe focus group interview data was examined using an inductive thematic analysis

approach. The raw interview data was reviewed to identify patterns that described and organized employee reactions to the original system and newly developed performance appraisal system. To identify themes or patterns generated from focus group interviews, coding was used (Boyatzis, 1998) to break down the data into manageable segments, leading to the identification of themes (Schwandt, 2001; Strauss & Corbin, 1998).

Two steps were employed to enhance the accuracy and reliability of the data analysis procedures. First, a code book was developed by the authors that included a list of themes and their translations. This code book provided the authors an ongoing reference and assisted in consistently identifying emerging patterns in the interview data. Second, adopting a procedure recommended by Schwandt (2001) to improve the reliability of qualitative data analysis, the authors separately coded the interview data and later reviewed it to identify any discrepancies.

133

Outcomes

Defining Job Performance and Creating an Appraisal InstrumentAn employee’s job description is often used to identify job performance standards

(Bommer, Johnson, Rich, Podsakoff, & MacKenzie, 1995; Borman & Motowidlo, 1993; Heneman, 1986; Lance, Teachout, & Donnelly, 1992; Mathis & Jackson, 2006; Rotundo & Sackett, 2002; Schmidt & Kaplan, 1971; Scullen, Goff, & Mount, 2000; Van Scotter, Motowidlo, & Cross, 2000; Vance, MacCallum, Coovert, & Hedge, 1988). In particular, the job description must clearly identify the major job domains and tasks of a job. Accurate, systematically developed, and job-specific descriptions provide the basis for defensible, job-related human resource decisions (i.e., pay-for-performance decisions) (Mathis & Jackson, 2006). Performance standards can flow directly from a job description and indicate how performance is measured in key areas of the job description (Wojcik, 2000).

Researchers have suggested conducting a thorough job analysis to define the appropriate content domains and tasks for the job description (Viswesvaran & Ones, 2000). Thus, conducting job analyses for every full-time position in the Elmhurst Park District appeared to have merit as a starting point for creating job-specific pay-for-performance appraisal instruments. In describing this initial step (and the subsequent steps) in the development of a pay-for-performance system, this study provides an overview of the research that guided the step(s) and a description of what occurred during the implementation of each of the six steps.

Job analysisJob analyses are a systematic way to gather and analyze information about the

content of jobs (Mathis & Jackson, 2006). The job analysis process should identify the jobs under review, the participants involved in conducting the job analysis, a review of existing documentation including current job descriptions, the identification of the major job content domains contained within the job, and a list of tasks to be fulfilled under each domain. (Mathis & Jackson, 2006). Job descriptions often summarize the elements identified in a job analysis.

In completing job analyses, research has indicated that involvement of employees at all levels facilitates acceptance of the system and increases cooperation (Levy & Williams, 1998; Longenecker & Fink, 1999, 2003). When employees are allowed to participate in the development of the appraisal system, their sense of ownership increases. Employee involvement in the performance appraisal and development process is critical. Longenecker and Fink (1999) caution against attempts to save time by bypassing employee and manager input because it can lower the system’s credibility. If managers, acting alone, produce a system that does not meet staff needs, it damages the perceived connection between pay and performance and loses the performance-enhancing effects of employee commitment to agency goals. Lack of employee involvement creates a domino effect; researchers have found if either the managers or employees are dissatisfied with the system, the effectiveness of the overall appraisal and feedback process is diminished (Gosselin & Murphy, 1994; Petit & Haines, 1994; Tziner & Kopelman, 2002; Wiersma & Latham, 1986).

134

Subscribing to the approach to defining job performance suggested by Viswesvaran and Ones (2000), job analyses were completed for every full-time job title in the Elmhurst Park District. To complete each job analysis, a meeting between the employee(s) and the supervisor was conducted for every full-time position. In conducting the job analyses, the employee(s) and supervisor collectively reviewed the current job description. Job descriptions of similar positions from other agencies were also reviewed to guide employees in brainstorming a list of job domains and tasks performed. During this meeting, the employee(s) and supervisor collectively identified between six and ten job domains that represented the major content areas of responsibility for the job (programming, human resource management, finance, etc.). Once the general content domains were identified, the employee(s) and supervisor discussed and identified a list of specific tasks within each job domain. On average, 10 to 15 tasks were identified for each job domain. Consistent with previous research, the task descriptions (1) began with an action verb, (2) included only one specific task, and (3) stated what the employee did (Brademas, Lowrey, Gress, & Bostrom, 1981; Drauden & Peterson, 1974; Grant, 1988; Mathis & Jackson, 2006).

Rating of task statementsOnce an agreed-upon list of job domains and task statements was developed,

weights were assigned to each task to further describe the significance of each task to the job. This procedure is advocated by a number of researchers as a means of ensuring the validity of job descriptions (Brademas, Lowrey, Gress, & Bostrom, 1981; Drauden & Peterson, 1974; Grant, 1988; Mathis & Jackson, 2006).

Each employee and supervisor independently reviewed the list of task statements and rated each statement on two, 7-point scales. The first scale, “importance” (1 = low, 7 = high), rated their perception of the importance of each task statement to overall job performance. The second scale, “time/frequency” (1 = low, 7 = high), assessed the time/frequency that each task required in comparison to all other tasks. The values from each scale were multiplied and a total weight for each job task was created with a possible low of 1 (1 x 1 = 1) to a possible high of 49 (7 x 7 = 49). Then the employee and supervisor each independently completed the task statement rating form.

Next, a meeting was scheduled between the employee(s) and supervisor to review the weights. The intent of this discussion following the independent rating by the supervisor and employee(s) was to agree on the overall importance and the amount of time that should be spent on each task. If the supervisor and employee(s) had any disagreements about the overall weight of a task, the ultimate decision was that of the supervisor. However, the discussion between the supervisor and employee(s) on this disagreement prompted an in-depth discussion about the significance of each task. This discussion was guided by research suggesting that if employees have a clear perception of their tasks and the importance that managers place on these tasks, it’s likely the employees will successfully accomplish the tasks (Mathis & Jackson, 2006). After discussing any discrepancies in the weight assignments, a final list of all task statements with an agreed upon weight for each task statement was completed.

Creation of appraisal instrument The information collected during the job analysis provides the content for the

appraisal instrument (Milkovich & Newman, 2005). Performance appraisal instruments

135

are often divided into two general formats: ranking and rating. Ranking formats require the rater to compare employees against each other on a performance measure. Rating formats have two elements: (1) they require raters to evaluate employees on some absolute standard rather than relative to other employees, and (2) each performance standard is measured on a scale on which appraisers can check the point that best represents the employee’s performance level (Milkovich & Newman, 2005).

In deciding which appraisal format is most appropriate for an agency Keeley (1978) and Tziner and Kopelman (2002) suggest that an understanding of the type of tasks being performed is needed. Because the task statements developed during the job analysis phase included written statements of what the employee does, the Elmhurst Park District incorporated an anchored rating format. An anchored rating format describes performance variation along a continuum from good to bad (Milkovich & Newman, 2005).

The type and number of descriptors used in anchoring the continuum provide the major differences in rating scales. Organizational research has shown that the reliability of a performance appraisal instrument is strongest when using between three and seven descriptive anchors (Arvey & Murphy, 1998). The Elmhurst Park District selected a three-anchor approach. As a result, each task was evaluated against three anchors on the performance continuum: (1) below standard, (2) meets standard, and (3) exceeds standard. A “not applicable” rating was created to address tasks that were not observed or performed during the evaluation period.

Performance Appraisal Interview

Identifying ratersOnce the performance appraisal instrument has been developed, the agency must

identify who will rate the performance (i.e., supervisor, subordinate, co-workers) (Milkovich & Newman, 2005). According to research, a variety of rater methods have been implemented in organizations, including 360-degree feedback, a system that uses supervisors, peers, customers, subordinates, and oneself as raters (see Harris & Schaubroeck, 1988; Jackson, Schuller, & Rivero, 1989; Landy & Farr, 1980; Ones, Schmidt, & Viswesvaran, 1996). In deciding which approach to take, Milkovich and Newman (2005) recommend that organizations identify individuals who possess the most complete information on the performance of the ratee. Research has found the immediate supervisor to be the most frequently used rater (Ones, Schmidt, & Viswesvaran, 1996). In addition, a comparison of the reliability of raters suggests that supervisor ratings tend to be more reliable than those from other sources (Ones, Schmidt, & Viswesvaran, 1996).

The Elmhurst Park District identified the immediate supervisor as the individual who possessed the most complete information on the performance of the ratee and were therefore selected as the evaluator. This approach was chosen for two reasons. First, the ratee and supervisor were collectively involved in the job analysis and the development phases of the performance appraisal instrument. This collaborative approach gave the supervisor and each employee more knowledge about the job content and tasks and the importance of each task to be rated. Second, a majority of the supervisors at the Elmhurst Park District had considerable prior experience in

136

rating employees, thus giving them some firm ideas about what level of performance is required for each task to be completed by the employee (see Jackson, Schuller, & Rivero, 1989; Milkovich & Newman, 2005).

Rater trainingThe next step in the construction of the performance appraisal system is to

understand how and where the raters make mistakes (Milkovich & Newman, 2005) and how to reduce those errors. One approach to limiting errors in the performance appraisal process is through appraiser training (Bernadin & Pence, 1980; Wilson & Western, 2001; Zedeck & Cascio, 1982). Surprisingly, managers frequently report that they receive very little training beyond a description of the rating form (Longenecker & Fink, 1999). An effective formal performance appraisal system cannot exist without ongoing education for all key appraisers in the appraisal process (Longenecker & Fink, 1999). Developing the skills necessary to conduct effective performance appraisals, including issues of psychometric errors, can be accomplished through appraiser training (Schweiger & Sumners, 1994).

Training sessions subscribing to the previously mentioned principles were conducted for employees of the Elmhurst Park District. Employees were provided with a 3-hour session that focused on three categories: (1) rater-error training, (2) performance dimension training, and (3) performance standard training. During the rater-error training session, employees were introduced to several psychometric errors (e.g., leniency, halo effect, recency) and offered suggestions to address those problematic areas. The performance dimension training session involved a collective discussion on performance dimensions between raters and ratees. Finally, the performance standard training provided raters with a standard of comparison or frame of reference for making appraisal decisions. Because the Elmhurst Park District adopted a three-anchored approach, a discussion on how “below standard,” “meets standard,” and “exceeds standard” are defined within the agency was conducted.

Performance appraisal interviewA large body of research over the last four decades indicates that employee

participation in the interview is associated with a variety of desirable appraisal related outcomes, including appraisal system fairness, appraisal satisfaction, supervisory support, satisfaction with supervisors, appraisal system acceptance, and greater acceptance of feedback (Roberts, 2003). In particular, self-evaluation provides employees with the opportunity to systematically assess their performance. A common method to facilitate self-evaluation is to require employees to complete their own appraisal and present the draft for discussion with the supervisor. The supervisor can review the draft with the employee and compare the employee’s self-appraisal ratings to the supervisor’s appraisal ratings of the employee.

Adopting this participative process of self-appraisal, a trial run was conducted for the new performance appraisal instrument. The trial run allowed the Elmhurst Park District to test the appraisal process and instrument with no consequences assigned to the results. The trial run was conducted instead of an actual live implementation because of the developmental timeline of the new appraisal system. In particular, the Elmhurst Park District uses to a 12-month review period to evaluate their employees for merit pay

137

increase decisions. The development of the new appraisal system occurred during the last six months of the 12-month review period. Thus, formally implementing the new appraisal system as the evaluation system for the entire 12-month period would have been unfair to the employees because the performance standards created in the new system were not identical to the performance standards from the previous system.

The trial run began with the employee and supervisor independently completing the appraisal instrument. To assist in the final calculations, the appraisal instruments were created in a Microsoft Excel format. Paper copies of the spreadsheet-formatted instrument were printed and provided to the employee and supervisor.

Once the employee and supervisor had independently completed the appraisal instrument, they met and discussed the ratings. During that time, the employee and supervisor agreed on a rating for each task statement. The appraisal was then completed and signed by both the employee and supervisor. The Elmhurst Park District Human Resource Specialist applied the previously established weights for each task statement to determine a final percentile score for each employee once the final ratings had been determined (Figure 1).

Statistically, this procedure involved (1) multiplying the score of the task (i.e., 1 = below standard, 2 = meets standard, 3 = exceeds standard) by the weight of each task (tasks receiving a “not applicable” rating were voided from the computations), (2) determining the total points possible (e.g., the sum of each task’s weight multiplied by 3 = exceeds standard), and (3) dividing the total possible points by the total points earned by the employee to obtain a final percentile score. For example, in Figure 1, the Division Manager’s final percentile score was 68.97% .

A Model for the Distribution of Merit Pay IncreasesA review of the literature identified three elements to include in the development

of a model for the distribution of merit pay increases: pay differentials, forced distribution, and pay adjustment matrices (Martocchio, 2001). According to Milkovich and Newman (2005), those elements help guide management decisions in delivering performance-based salaries while safeguarding against “redlining” (i.e., an employee moving above the pay range for the particular job). Accordingly, those three elements guided the development of a merit pay increase model for the Elmhurst Park District.

Pay differentialsPay differentials represent the varying levels of merit increases that are based

on an employee’s job performance (Milkovich & Newman, 2005). An agency should consider the pay differentials it uses when rewarding various levels of job performance. An agency must ask, “How much do we pay the higher performers compared to the lower performers?” (Martocchio, 2001). For example, how much more does an employee receive if he or she “exceeds standard” compared to “meets standard”? This amount needs to appropriately distinguish the various levels of performance in order for the pay-for-performance system to correctly serve as a motivator for staff. In looking at this issue, researchers have suggested that an approximately 3% pay increase is needed between the levels of performance as perceived by the employees (see Arvey & Murphy, 1998).

138

Figu

re 1

. Fi

naliz

ed D

ivis

ion

Man

ager

of R

ecre

atio

n Pe

rform

ance

App

rais

al In

stru

men

t

Per

sonn

el M

anag

emen

t

Rec

ruit,

hir

e an

d tr

ain

staf

fM

anag

e an

d ev

alua

te s

taff

Prov

ide

ongo

ing

dire

ctio

n, f

ores

ight

and

m

otiv

atio

n to

sta

ffPr

epar

e fo

r an

d co

nduc

t sta

ff m

eetin

gs a

nd

trai

ning

s as

nee

ded

Adh

ere

to a

nd e

nfor

ce s

taff

com

plia

nce

with

or

gani

zatio

nal p

olic

ies

and

proc

edur

es, a

nd

the

stra

tegi

c pl

anE

nsur

e th

at e

mpl

oyee

s ar

e pr

ovid

ed n

eces

sary

tr

aini

ng in

usa

ge o

f to

ols,

equ

ipm

ent,

ma-

chin

ery

and

pers

onal

pro

tect

ive

equi

pmen

tA

sses

s fu

ture

div

isio

n st

affin

g re

quir

emen

tsF

inan

cial

Man

agem

ent

Prep

are

and

mon

itor

divi

sion

bud

get

Coo

rdin

ate

purc

hase

of

divi

sion

sup

plie

s,

mat

eria

ls a

nd e

quip

men

tD

evel

op a

nd im

plem

ent p

rogr

am p

rici

ng

polic

ies

Man

age

inte

rnal

and

ext

erna

l ass

ista

nce

pay-

men

ts a

nd d

elin

quen

t acc

ount

sM

anag

e bi

d sp

ecs

for

maj

or p

urch

ases

Org

aniz

atio

nal P

lann

ing

Est

ablis

h an

d m

onito

r di

visi

on g

oals

and

ob

ject

ives

Serv

e as

mem

ber

of th

e Pa

rks

and

Rec

reat

ion

Serv

ices

Man

agem

ent T

eam

and

oth

er a

p -po

inte

d co

mm

ittee

ass

ignm

ents

Not

A

pplic

able

Not

A

pplic

able

Not

A

pplic

able

Bel

ow

Stan

dard

x x x B

elow

St

anda

rdx

Bel

ow

Stan

dard

x x

Mee

ts

Stan

dard

x x xM

eets

St

anda

rd x

Mee

ts

Stan

dard

Exc

eeds

St

anda

rd x

Exc

eeds

St

anda

rd x x x

Exc

eeds

St

anda

rd

Scor

e

2 1 2 1 3 1 2Sc

ore

1 3 3 2 3Sc

ore

1 1

Wei

ght

21 49 49 16 49 30 10W

eigh

t

36 25 6 12 5W

eigh

t

36 49

Wei

ghte

d Sc

ore

42 49 98 16 147

30 20W

eigh

ted

Scor

e36 75 18 24 15

Wei

ghte

d Sc

ore

36 49

Tot

al

Pos

sibl

e63 14

714

7

48 147

90 30T

otal

P

ossi

ble

108

75 18 36 15T

otal

P

ossi

ble

108

147

139

Figu

re 1

. Fi

naliz

ed D

ivis

ion

Man

ager

of R

ecre

atio

n Pe

rform

ance

App

rais

al In

stru

men

t (Co

nt.)

Atte

nd s

taff

mee

tings

Prep

are

annu

al t-

shir

t bid

Dev

elop

and

impl

emen

t cus

tom

er s

ervi

ce

stan

dard

sC

omm

unic

atio

ns/C

usto

mer

Ser

vice

Dev

elop

and

mai

ntai

n co

oper

ativ

e re

latio

n-sh

ips

and

effe

ctiv

e or

al a

nd w

ritte

n co

mm

unic

atio

ns w

ith in

tern

al a

nd e

xter

nal

cust

omer

sPr

omot

e D

istr

ict p

rogr

ams

to p

atro

ns, g

uest

s an

d st

aff

Prep

are

wri

tten

and

verb

al b

ulle

t poi

nts,

up

date

s an

d re

port

s as

req

uire

d.A

ct a

s a

liais

on to

affi

liate

s, c

omm

unity

gr

oups

and

gov

ernm

enta

l uni

tsD

esig

n an

d di

stri

bute

info

rmat

ion

for

publ

ic

dist

ribu

tion

Coo

rdin

ate

seas

onal

bro

chur

e pr

oduc

tions

Safe

ty

Enc

oura

ge a

nd d

emon

stra

te s

afe

wor

k ha

bits

th

roug

h us

e of

est

ablis

hed

safe

ty p

rogr

am

guid

elin

esSe

rve

as m

embe

r of

the

Dis

tric

t’s C

risi

s M

anag

emen

t Tea

mM

aint

ain

CPR

and

AE

D c

ertifi

catio

n, a

nd

ensu

re th

at a

ll st

aff

with

in s

uper

visi

on d

o th

e sa

me

Not

A

pplic

able

Not

A

pplic

able

x

Bel

ow

Stan

dard

B

elow

St

anda

rd

x M

eets

St

anda

rdx x x

Mee

ts

Stan

dard

x x x

xE

xcee

ds

Stan

dard

x x xE

xcee

ds

Stan

dard

2 1 3

Scor

e

2 2 2 3 3 3Sc

ore

2 2 2

30 15 25

Wei

ght

49 36 16 16 9 20W

eigh

t

49 21 49

60 15 75

Wei

ghte

d Sc

ore

98 72 32 48 27 60W

eigh

ted

Scor

e38 42 98

90 45 75

Tot

al

Pos

sibl

e14

7

108

48 48 27 60T

otal

P

ossi

ble

147

63 147

140

Figu

re 1

. Fi

naliz

ed D

ivis

ion

Man

ager

of R

ecre

atio

n Pe

rform

ance

App

rais

al In

stru

men

t (Co

nt.)

Pro

gram

min

g

Dev

elop

pro

gram

s to

cre

ate

lifet

ime

user

sD

evel

op a

nd im

plem

ent a

con

sist

ent c

usto

mer

fe

edba

ck p

rogr

amE

nsur

e sa

fe a

nd a

ppro

pria

te p

rogr

am f

acili

ties

Man

ager

pro

gram

incl

usio

n pr

oces

sR

egis

trat

ion

Man

age

prog

ram

reg

istr

atio

n pr

oces

sM

anag

e pr

ogra

m c

ance

llatio

n/ex

pans

ion/

ad-

ditio

n pr

oces

sM

anag

e pe

rmit

and

rent

al r

egis

trat

ions

Not

A

pplic

able

Not

App

licab

le

Bel

ow

Stan

dard

B

elow

St

anda

rd

Mee

ts

Stan

dard

Mee

ts

Stan

dard

x

Exc

eeds

St

anda

rdx x x x

Exc

eeds

St

anda

rd x x

Scor

e

3 3 3 3Sc

ore

2 3 3

Wei

ght

36 15 24 36W

eigh

t

36 25 16T

OT

AL

ST

OT

AL

P

OSS

IBL

EP

ER

-C

EN

TA

GE

(%

)

Wei

ghte

d Sc

ore

108

45 72 108

Wei

ghte

d Sc

ore

72 75 48 1895

2748

68.9

7

Tot

al P

ossi

ble

108

45 72 108

Tot

al

Pos

sibl

e10

875 48 2748

Res

pons

e Sc

ale:

0 =

Not

App

licab

le

The

task

is n

ot p

erfo

rmed

/not

obs

erve

d1

= B

elow

Sta

ndar

d

T

he p

erfo

rman

ce is

bel

ow s

tand

ards

2 =

Mee

ts S

tand

ard

The

per

form

ance

mee

ts s

tand

ards

3 =

Exc

eeds

Sta

ndar

d

The

per

form

ance

exc

eeds

sta

ndar

ds

141

In addressing pay differentials, researchers have also introduced the “just noticeable differences” (JND) concept (Heneman & Ellis, 1982; Katkowski, Medsker, & Pritchard, 2002; Krefting & Mahoney, 1977; Mitra, Gupta, & Jenkins, 1997). JND refers to the minimum pay increase that employees will perceive as making a meaningful change in compensation and thereby enhance their motivation to improve job performance (Heneman & Ellis, 1982). JND as a concept also has been represented as “smallest meaningful pay increase” (Krefting & Mahoney, 1977). Rooted in psychophysical research that centered on identifying JND in sounds, weights, and other environmental stimuli, the JND concept has been used in agency compensation programs to investigate appropriate levels of merit pay increase amounts based on performance (Katkowski, Medsker, & Pritchard, 2002).

In an attempt to further understand JND, Katkowski, Medsker, and Pritchard (2002) empirically reviewed 11 JND studies. When looking at the pay increase percentage corresponding to a JND, Katkowski et al. (2002) drew from nine of the 11 studies (only nine studies provided enough information to be included in the analysis) and generated an overall JND range of 6% to 8%. Thus, based on those findings, approximately 6% to 8% is recommended for employees who exceed performance standards, to adequately address employee perceptions of the effort/input to outcome ratio.

Guided by the research on pay differentials and JND, a preliminary pay increase model was developed for the Elmhurst Park District. The model divided performance levels into three categories: below standard, meets standard, and exceeds standard. Using Katkowski et al.’s (2002) meta-analysis findings, an initial list of merit pay percentage increases was proposed to the park district. The proposal recommended that employees within the “exceeds standard” category receive an 8% pay increase, employees within the “meets standard” category receive a 4% pay increase, and employees within the “below standard” category receive no increase.

Discussions with the Elmhurst Park District staff prompted an adjustment to the preliminary model. Specifically, the district did not feel that employees within the “below standard” category should receive no merit pay increase. Recognizing the almost constant increase in the cost of living index and that all employees of the district were recognized as the best available when hired, employees felt that a 1% increase should be awarded to the lowest performers. This view led to the revision of the preliminary model. Thus, the “exceeds standard” category was reduced to a 7% merit raise and the “below standard” category was increased to 1%. These percentages served as the foundation for the remaining steps in the development of the district’s merit pay increase distribution model.

Forced distributionAn agency must consider and identify an approach to determining placement

of employees into various levels of performance. A widely used method to complete this task is forced distribution, which enables evaluators to assign a predetermined percentage of employees to each of the various performance categories (Martocchio, 2001). The distribution of employee performance is represented in a bell-shaped curve (Mathis & Jackson, 2006) that allows pay increase amounts to be statistically assigned to various points on the curve.

142

The rationale for forced distribution is that it eliminates clustering among employees at the top of the distribution (rater leniency), at the bottom of the distribution (rater severity), or in the middle (central tendency) (Cascio, 2006). Because the distribution of ratings/scores is forced into a bell-shaped curve, a relatively small portion of employees will be identified as truly outstanding, a relatively small portion as unsatisfactory, and the remaining portion will be somewhere in the middle. Guralnik, Rozmarin, and So (2004) suggest that this process is particularly helpful when pay increase budgets are limited for an agency. Forced distribution also makes managers identify high, average, and low performers. Thus, high performers can be rewarded and developed while low performers can be encouraged to improve or asked to leave.

A few criteria made the forced distribution technique appealing to the pay-for-performance system for the Elmhurst Park District. First, forced distribution is most effective in an agency with more than 50 employees (Martochhio, 2001). The Elmhurst Park District has almost 60 full-time employees. Next, companies that benefit from forced distribution typically have growth and performance-driven cultures that belong in a highly demanding industry. This criterion is particularly fitting for most public park and recreation agencies (including the Elmhurst Park District) who are regularly confronted with increasing and ever-changing needs of the publics they serve. From population shifts (i.e., demographics such as an increase in the community’s aging population) to budgetary constraints, public park and recreation agencies must be concerned with the entire publics’ needs while being increasingly faced with demands to identify and establish creative revenue sources.

In an effort to remain consistent with the performance levels from the appraisal instruments, three levels of performance (below standard, meets standard, and exceeds standard) were identified in the distribution. Scores below –1.0 (standardized Z scores) were categorized as below standard, scores falling between +/– 1.0 were categorized as meeting standard, and scores above +1.0 were categorized as exceeding standard.

Using the performance appraisal scores from the Elmhurst Park District’s trial run, employees were placed on the distribution curve. Six employees fell into the “below standard” category, 1 employees into the “meets standard” category, and seven into the “exceeds standard” category. Upon reviewing these results, the park district moved the cutoff point for exceeds standard to +0.50 Z score. The move was based on the park district’s assertion that it hired “above average” employees. It was believed that some of the employees at the Elmhurst Park District who were forced into the “meets standard” category would likely exceeded the standard at another agency and were, in fact, performing above standards. Therefore, the cutoff point was lowered, which resulted in sixteen employees falling into the “exceeds standard” category and 35 employees in the “meets standard” category. Results of this distribution are provided in Figure 2.

Once the employees were plotted on the distribution curve, a series of distribution analyses was conducted to determine the average merit pay increase for the agency. The results of these analyses determined the average merit pay increase to be over 4.5%. Unfortunately, the Elmhurst Park District’s merit pay increase pool was restricted to 3.5% to 4%. Like most public park and recreation agencies, the Elmhurst Park District’s merit pay increase resource pool was determined by the district’s board of commissioners and was based on change in the local/national consumer price index and the overall financial health of the agency. As a result of this problem, the Elmhurst Park District established a pay adjustment matrix for their agency.

143

Pay adjustment matricesIn addition to addressing employee attitudes (e.g., about JND) and perceptions

of fairness of the pay-for-performance system, a merit pay system must be cost efficient. Of particular concern is the problem of redlining employees who have moved out of their job title’s acceptable pay range. Several agencies have adopted pay adjustment matrices to safeguard employee movement through the assigned pay ranges (Martocchio, 2001; Mathis & Jackson, 2006; Milkovich & Newman, 2005).

A pay adjustment matrix consists of two dimensions: the employee’s current position in the pay range and the employee’s performance rating (Mathis & Jackson, 2006). These two variables influence the amount of the performance-based pay increase. In developing an agency’s pay adjustment matrix, compa-ratios are often used. Compa-ratios help organizations assess how they pay employees in relation to the midpoint of the pay range. A compa-ratio of 1.0 would correspond to an employee who is in the middle of his or her pay range. An employee with a compa-ratio of 1.0 is receiving, on average, the market rate for that job (Bergmann & Scarpello, 2001).

Using compa-ratios, an agency creates a matrix that includes various pay increase amounts based on the employee’s current position in the pay range (e.g., a compa-ratio between 1.0 and 1.1) and his or her performance rating (e.g., a “meets standard” rating) (Bergmann & Scarpello, 2001). The matrix helps prevent redlining of employees while controlling costs for the agency. For example, as an employee moves to a higher position in the pay range (e.g., from a compa-ratio of 1.1 to a compa-ratio of 1.2), he or she would be awarded a smaller percentage pay increase for the same performance rating.

In addressing employee perceptions of the fairness of this approach, Bergmann and Scarpello (2001) suggested that the matrix must be constructed to show that a given level of performance would be granted a pay increase of a similar dollar amount because of that employee’s current position in the pay range. For example, an employee making $30,000 annually and receiving a 7% pay increase would receive a $2,100 raise, while an employee who is higher in the pay range (making $55,000 annually)

Figure 2. Elmhurst Park District’s Forced Distribution of Peformance Appraisal Scores

Number of Employees n = 6 n = 33 n = 16

Standardized Z Scores -1.0 0.50 1.0Performance Appraisal Scores 66.60 72.21 74.05

144

and receiving a 3.5% increase for the same performance would receive a $1,925 raise. When comparing the two raises over the course of a year, the difference comes to less than 50 cents per day. Thus, although the percentage is significantly less, the absolute amount is approximately the same. The compa-ratio suggests that higher performance is to be expected of more senior employees and that a smaller percentage increase is available to those reaching the top of their pay range for their job title (Mathis & Jackson, 2006).

Guided by the compensation literature and in response to the Elmhurst Park District’s need to monitor and control merit pay costs, a pay adjustment matrix was developed. The pay adjustment matrix had three performance levels and four pay quartiles. The JND and pay differential research served as the foundation to the matrix and were used to establish the percentages in the first quartile. In particular, an employee with a compa-ratio of .80 was eligible to receive a 1% (below standard), 4% (meets standard), or a 7% (exceeds standard) merit pay increase. Percentages were inserted into the remaining performance levels and quartiles based on the Elmhurst Park District’s pay ranges and merit pay resource pool while adequately maintaining just noticeable differences in pay increases based on performance. The finalized merit pay increase model is provided in Figure 3.

1.20

1.10

1.00

.90

.80

Figure 3. Finalized Merit Pay Increase Model

Q4

Q3

Q2

Q1

Com

pa-R

atio

(P

erce

nt t

o M

idpo

int)

0.00

0.00

0.00

1.00

Does Not Meet Standard

1.00

2.00

3.00

4.00

Meets Standard

2.00

3.50

5.50

7.00

Exceeds Standard

Performance Appraisal Rating

Figure 4 shows the results of the revised pay-for-performance plan of the Elmhurst Park District. Column 1 shows the percentile score, arranged from highest to lowest, of each employee on the performance appraisal. The percentile score was determined by multiplying the actual performance score (1 = below standard, 2 = meets standard, 3 = exceeds standard) by the weight given to each performance criteria or task statement contained in the job-specific performance appraisal instrument. The maximum score was three times the weight given for each task statement. The score earned by the employee was divided by the maximum score possible for the appraisal instrument to obtain the finalized percentile score. Column 2 is the percentile score converted to standardized Z scores and divided according to the forced distribution model adopted by the Elmhurst Park District. Thus, six scores fell below the –1.0 Z score, 33 scores were within the range of –1 to .50 Z score, and 15 scores had a Z score greater than .50.

145

Column 3 shows the compa-ratio for the performance appraisal rating, which signifies each staff person’s merit pay increase. As previously identified, the actual pay increase received by each staff person was based not only on performance rating but also on that employee’s position in the pay range for his or her job title; that is, an employee who exceeded performance standards and was at the top of the pay range received a smaller percentage pay increase than an employee who merely met performance standards but was at the midpoint of his or her pay range. The percentage increase may be less for an employee at the top of his or her pay range, but as discussed in the section about compa-ratios, the actual dollar amount might be greater. The new model, using the forced distribution and compa-ratios, resulted in an overall pay increase of 3.67%—an acceptable range for the Elmhurst Park District.

Performance Appraisal Percentile Score*

79.4177.9677.8477.7876.1575.2175.1074.0373.9673.7673.3872.9172.7072.6272.4972.1671.7171.2871.2070.8370.7270.6370.6170.1670.0269.9969.9569.8469.79

Figure 4. Elmhurst Park District Performance Appraisal Results

Converted Z Score

2.452.062.032.011.571.321.290.980.970.950.820.690.640.610.580.490.370.250.230.130.100.070.07-0.05-0.09-0.10-0.11-0.14-0.15

Percentage Pay Raise*

3.50 %5.50%2.00%7.00%7.00%7.00%5.50%7.00%7.00%5.50%7.00%7.00%2.00%3.50%7.00%1.00%4.00%4.00%1.00%4.00%4.00%3.00%3.00%3.00%4.00%1.00%4.00%2.00%2.00%

146

69.7669.6169.5069.3569.1569.0569.0268.9768.9268.5768.3467.9467.6067.5567.4467.3067.1866.9666.9465.3765.1764.3464.1664.1261.56

Figure 4. Elmhurst Park District Performance Appraisal Results (Cont.)

-0.16-0.20-0.23-0.27-0.33-0.35-0.36-0.38-0.39-0.48-0.55-0.65-0.75-0.76-0.77-0.83-0.86-0.92-0.93-1.35-1.41-1.63-1.68-1.69-2.38

4.00%3.00%3.00%4.00%4.00%3.00%3.00%4.00%3.00%4.00%3.00%4.00%4.00%4.00%2.00%4.00%4.00%4.00%4.00%1.00%0.00%1.00%1.00%1.00%0.00%

Focus Group Results

This study also used focus group interviews to examine employee reactions to the appraisal system.

Performance Appraisal Instrument and Review SessionResults from the interviews suggested a change in employee satisfaction with

the newly developed performance review session compared to the original system. In particular, a recurring theme emerging from the data was employee dissatisfaction with the original pay-for-performance interview sessions. Predominant coded patterns identified from the focus group interview data on the original system included themes describing the appraisal review session as confusing, poorly developed, and viewed with negatively by staff. Consider the comments by employees:

The entire system and interview are confusing to understand—how are raises

determined from this? Because we don’t have a clear understanding of the

[performance appraisal] results, we don’t know what to tell our staff. Leaving

147

staff “in the dark” creates skepticism and distrust in the results. (Female,

mid-level facility manager)

I dread it. I don’t know what to expect going in [to the appraisal interview].

Of course I want a good evaluation for obvious reasons, but it seems like

regardless of what I feel my performance was for the year, the interview

session throws me an unexpected curve ball. (Male, mid-level operations

manager)

In contrast, when asked about the newly developed instrument, a majority of employees felt that the interview session and system were greatly improved. A common theme emerging from both focus groups was the participative process of the new appraisal instruments. Consider the following statement:

Since the employee was involved in creating the instrument, it has become

much easier and less confrontational to comment on specific [performance]

ratings since they [the employees] were part of the process to identify what

was included on the performance appraisal. (Female, mid-level recreation

manager)

Procedural Justice of Performance Appraisal SystemInterview results identified changes in employee perceptions of the processes

involved in creating and administering the newly developed pay-for-performance system. Several employees commented on the role of the job analyses and their influence on their staff’s views of justice. Consider the comments made by some employees:

Staff initially were wondering why we needed to do this [job analyses]….

However, once we sat down and discussed the employee’s job, she and I were

provided with better insight on what she was doing and what I [rater] thought

she was doing.… The staff saw value in the weighting of tasks by time and

importance as we often agreed that a majority of their tasks were important,

but not conducted as often. (Male, operations manager)

The job description revisions were positive even though they took some time

to complete. I supervise employees and have a supervisor, and the process was

beneficial on both levels. I was able to obtain input from staff about what they

felt they needed to be doing while communicating my expectations. On the

other end, I was able to share [with my supervisor], where I felt I was spending

most of my time and energy. She was then able to let me know if I “was on

track,” which was helpful. I am hopeful this process will reduce some of those

gray areas of the evaluation process. (Male, recreation director)

148

Despite the majority of employees expressing positive comments about the procedures used in the new pay-for-performance instrument, some staff expressed concern. Consider the comment made by an upper-level office manager:

The proof is in the pudding, and it is still too early to tell. It does not seem

realistic to ding them on one particular task, but we are. We have delineated

multiple tasks into multiple statements. Our previous instrument was

[broader]. (Female, upper-level accountant) When looking at the way supervisors conducted the performance review and

rated their employees, an interesting finding emerged. Some supervisors felt that their subordinates rated themselves more critically than they did. For example, one director stated,

I was surprised at how some of my staff self-rated themselves compared

to the scores I gave them—mine were actually higher. This made for an

interesting [performance appraisal interview] discussion. I was pleasantly

surprised—I thought the discussion was valuable in coming to an agreement.

(Male, facilities director)

Distributive Justice and Overall SystemFocus group interview responses about the original system identified dissatisfaction,

confusion, and overall frustration. For example, several employees questioned how the original performance appraisals were linked to the merit pay increase model. One employee asked,

If I am outstanding, shouldn’t I receive a 5% raise? I don’t recall ever

receiving a 5% raise even when I received an outstanding appraisal. Where is

the motivation in that? (Female, administrative manager)

Consistent with the administrative manager, another employee stated,

It [pay distribution model] must just be a guide because I can’t remember

ever getting the appropriate raise based on my evaluation…. The toughest

part is evaluating my employees and not being able to tell them what their

performance has earned them. (Female, mid-level recreation manager)

When asked about this issue, the Elmhurst Park District administration identified the problem as the agency’s inability to adjust pay ranges at an annual rate of +5%. Elmhurst Park District’s pay ranges were adjusted each year based on a local cost-of-living index, and these adjustments had not been 5% (or higher) in recent years. As a result, administrators were forced to adjust the pay raises for employees at the top of their pay ranges who received an outstanding rating.

149

Upon completing the trial run of the new performance appraisal instrument, interviewees were shown how their ratings would be applied to the merit pay increase and compa-ratio distribution model. Next, interview data on employee reactions and distributive justice perceptions of the newly developed system was examined. Several interesting patterns in the new system’s distributive justice emerged. A majority of the coded themes centered on the three elements of the new distribution model. For example, a director stated,

I agree with the pay adjustment matrix idea. I believe it is a necessity that

must be present for the system to work correctly. Everyone cannot make the

same [percentage raise], and certain people may get mad, but they are going

to have to deal with it. (Male, upper-level director)

In contrast, an upper-level administrator was cautious about the implementation of compa-ratios and pay adjustment matrices:

Implementing the compa-ratio will be difficult. Staff don’t like it [in the

original model] now, and I don’t think they will like it in the new one. I

understand its importance, but I think staff will have a hard time understanding

it. A lot will depend on how fair this [the new system] is and how it gets

communicated to staff. (Male, upper-level parks manager)

When asked why employees did not understanding the agency’s current and/or new models, the upper-level administrator suggested that the problem was a result of the current model’s inability to adequately control merit pay increases. In addition to Elmhurst’s pay ranges failing to keep pace with the top raise amounts for employees above the midpoint, the previous system failed to adopt a forced distribution approach. Consequently, the park district had difficulty linking the appropriate merit pay raise increase to the performance level. Several employees therefore disregarded the existing distribution model or attributed their confusion about the link between the model and their merit raise to not completely understanding the system.

The upper-level administrator’s conclusions were supported by several employees who liked the forced distribution concept. The employees interviewed felt that forced distribution would help discriminate among performance levels. Consider the following statement:

Forced distribution is an excellent tool to adjust pay in an objective and

quantifiable manner. We clearly meet the criteria by having over 50 employees

… in our appraisal system. I also think the new performance instruments

make our performance evaluations much more objective than before.…

Forced distribution is what we need to make fair pay decisions, and I think

it helps make our evaluation process easier to understand and use. (Female,

accountant staff)

150

Discussion

Guided by a strategic human resource management framework asserting that agencies should align their pay system to support their strategic objectives, this study sought to provide an overview of the steps involved in creating job-specific, content-valid, pay-for-performance appraisal instruments, an appraisal review process, and a distribution model for merit pay increases. As suggested by Milkovich and Newman (2005), a valid pay-for-performance plan requires (1) clearly defined, job-specific criteria that enable creation of an appraisal instrument that describes different performance levels from low to high, (2) a well-defined performance appraisal interview process, and (3) equitable decisions about how much of a merit increase will be given for different levels of performance.

Based on research findings that employee participation leads to elevated levels of employee acceptance and on research suggesting that fairly distributed merit pay increases improve staff and agency performance, this study adopted a participatory approach in creating the Elmhurst Park District’s pay-for-performance appraisal system. In addition, focus group interviews and observations were used to analyze employee satisfaction with the appraisal interviews/sessions and perceptions about fairness of the appraisal system.

Results from the focus group interviews suggested that, overall, employees were satisfied with the new system, distribution model, and its fairness as an evaluation tool. For example, several employees indicated that the previous system lacked one or more of the three elements associated with the new merit pay distribution model (i.e., pay differentials, forced distribution, or pay adjustment matrices). In contrast, focus group results suggested that employees had a more positive and slightly cautious attitude toward the new system. The new system and model allowed for merit pay increases that were recognizable by staff and clearly tied to the performance ratings. Employees believed that the new system greatly improved on the subjective nature of the previous instrument but failed to completely remove all subjectivity in the process. However, the use of forced distribution as a method of distributing employees across performance levels was well received as an approach to further removing subjectivity from merit pay distribution decisions.

In addition to the positive role of forced distribution, the reduction from five performance levels to three was well received by the employees. Employee performance appraisal results indicated a tight cluster of scores near the mean, and research has advocated for the use of fewer performance-based pay distinctions in these situations (Martocchio, 2001; Zenger, 1992). In particular, research has found an increase in perceived inequity comparisons by employees when more than three performance distinctions are made in an organization with minimal score variance (Zenger, 1992). These inequity perceptions can be troublesome for the agency: studies have identified several costly outcomes to the employer, including low effort, turnover, job dissatisfaction, poor interpersonal relations with colleagues, and sabotage (Arvey & Murphy, 1998; Levy & Williams, 2004).

In summary, the overall results of the focus group interviews suggested that employees were pleased with the new system but were concerned about some of the system’s implications. Specifically, concerns were expressed about the pay adjustment

151

matrix and pay differentials. However, despite these concerns, interviewees believed that the system was a positive addition to the Elmhurst Park District and its operations. Results suggested that employees of the park district were optimistic about the motivational aspects of the new system, particularly when recognizing the obvious need to maintain pay equity within the agency while adhering to the agency’s merit increase pool restrictions.

Management Implications, Limitations, and Future ResearchIn describing the cognitive and affective value of employee participation in

development of an appraisal system, Roberts (2003) identifies three benefits:

1) Employee participation in creating an appraisal system is a key intrinsic motivational strategy for facilitating worker growth and development. Specifically, employee participation is an effective tool for enhancing job-related autonomy, a necessary precondition for employee growth.

2) Participation provides employees with a voice into the appraisal process. With employee participation, staff are empowered to rebut ratings, documentation, or verbal feedback. Thus, employees are more likely to accept performance ratings, even adverse ones, if they believe that the decision-making process is fair. Based on the assumption that employees possess valid, unique, and relevant performance information that is unavailable to or unobservable by the rater, the quality and quantity of performance appraisal information increases with employee participation. Subsequently, information from employees leads to a more accurate and valid rating. The employee attains ownership over the process and has a personal stake in the success of the system, which enhances employee acceptance.

3) Employee participation generates an atmosphere of cooperation and employee support.

These findings are further supported by a comment from one of the supervisors:

It was somewhat difficult to tell Employee A that he is a 1 [below standard rating], but it was worth it. I was surprised—Employee A showed little disagreement with the rating [during the performance review session], and I saw immediate and continued improvement in Employee A’s attitude and performance.

The supervisor’s feedback identified specific job-deficient areas where the employee could improve. The employee’s participation in the appraisal process and the clearly stated areas for improvement led to more positive perceptions about fairness and to acceptance of the supervisor’s ratings. These high levels of employee acceptance and understanding will likely enhance the employee’s motivation and job performance.

This case study of the Elmhurst Park District sought to provide an empirically grounded overview of the initial steps involved in developing a pay-for-performance system for public parks and recreation agencies. Future research with different and larger samples is needed to further understand pay-for-performance appraisal practices

152

in the leisure service field. Although park districts represent the norm in the state of Illinois, public park and recreation departments housed within municipal or county government represent a predominant type of leisure service organization in the United States. As a case study, this research was limited to performance appraisal practices in a park district within the state of Illinois. Additional research examining the development of a pay-for-performance system within a municipal or county park and recreation department is needed.

In addition, research that examines the social context of performance appraisal development in public park and recreation agencies could provide additional insight into the role of employee participation. As research in the management field has suggested (see Levy & Williams, 2004), research examining the effects of the social context of the agency, such as feedback culture, group dynamics, politics, impression management, and other environmental variables, is needed and can help provide a richer understanding of this important management issue.

The trial-run approach of the performance appraisal process was limitation of the study. Although the performance appraisal process was “live,” it was without consequences (i.e., results were not used to make merit pay increase decisions). Employees were aware of the lack of consequences in the trial run. Future studies will attempt to address the success and continual acceptance of the performance appraisal system adopted by the Elmhurst Park District.

This study adopted a two-pronged approach to further understanding performance appraisal systems in public park and recreation agencies, using the Elmhurst Park District as a case study. First, the study sought to identify the steps needed to develop a pay-for-performance appraisal system for a public park and recreation agency. Next, the study examined employee reactions to a system that adopted these steps. The study identified positive employee reactions to a performance appraisal system that incorporated the steps, and strong evidence was found for increased satisfaction and improved perceptions of procedural justice in the new system.

References

Arvey, R.D., & Murphy, K.R. (1998). Performance evaluation in work settings. Annual Review of Psychology, 49, 141-168.

Bartlett, K. R., & McKinney, W. R. (2004). A study of the role of professional development, job attitudes, and turnover among public park and recreation employees. Journal of Park and Recreation Administration, 24(4), 63-81.

Bassett, G. (1994). Merit pay increases are a mistake. Compensation and Benefits Review, 26(2), 20-22.

Bergmann, T. J., & Scarpello, V. G. (2001). Compensation decision making (4th edition). Orlando, FL: Harcourt College Publishers.

Bernadin, H. J., & Pence, E. C. (1980). Effects of rater training: Creating new response sets and decreasing accuracy. Journal of Applied Psychology, 6, 60-66.

Bommer, W. H., Johnson, J. L., Rich, G. A., Podsakoff, P. M., & MacKenzie, S. B. (1995). On the interchangeability of objective and subjective measures of employee performance: A meta-analysis. Personnel Psychology, 48, 587-605.

153

Borman, W. C., & Motowidlo, S. J. (1993). Expanding the criterion domain to include elements of contextual performance. In: Schmidt, N., & Borman, W. C. (Eds.), Personnel selection in organizations, 71-98.

Boyatzis, R.E. (1998). Thematic analysis and code development: Transforming qualitative information. Thousand Oaks, CA: Sage Publications.

Brademas, D. J., Lowrey, G. A., Gress, K., & Bostrom, D. (1981). A model job analysis procedure for the park and recreation profession. Urbana-Champaign, IL: Office of Recreation and Park Resources.

Campbell, D.J., Campbell, K.M., & Chia, H.B. (1998). Merit pay, performance appraisal, and individual motivation: An analysis and alternative. Human Resource Management, 37, 2, 131-146.

Cardy, R. L., & Dobbins, G. H. (1994). Performance appraisal: alternative perspectives. Cincinnati, OH: South-Western Publishing.

Cascio, W. F. (2006). Managing human resources: Productivity, quality of work life, profits. Boston, MA: McGraw-Hill Irwin.

Chelladurai, P. (1999). Human resource management in sport and recreation. Champaign, IL. Human Kinetics.

Chen, H.M. & Min, K.J (2004). The role of human capital cost in accounting. Journal of Intellectual Capital, 5, 1, 116-131.

Dolan, K. A. (1996). When money isn’t enough. Forbes, 11/18, 164-170.Drauden, G., & Peterson, H. (1974). A domain sampling approach to job analysis. St.

Paul, MN: Test Validation Center.Dyer, L., & Reeves, T. (1995). Human resource strategies and firm performance:

What do we know and where do we need to go? International Journal of Human Resource Management, 6(3), 656-670

Edginton, C. R., Hudson, S. D., & Lankford, S. V. (2001). Managing recreation, parks, and leisure services: An introduction. Champaign, IL: Sagamore Publishing.

Edginton, C. R., & Williams, J. G. (1978). Productive management of leisure service organizations: A behavioral approach. New York, NY: John Wiley & Sons. Giles, W. F., & Mossholder, K. W. (1990). Employee reactions to contextual and

session components of performance appraisal. Journal of Applied Psychology, 75, 371-377.

Gosselin, A., & Murphy, K.R. (1994). L’Echec de l’evaluation de la performance. Gestion, 17-26.

Grant, P. C. (1988). What use is a job description? Personnel Journal, 67(2), 44-54.Greller, M. M. (1978). The nature of subordinate participation in the appraisal interview.

Academy of Management Journal, 21, 646-658.Grote, D. (2000). Public sector organizations. Public Personnel Management, 29, 1,

1-20.Guralnik, O., Rozmarin, E., & So, A. (2004). Forced distribution: Is it right for you?

Human Resource Development Quarterly, 15(3), 339-345.Harris, M. M., & Schaubroeck, J. (1988). A meta analysis of self-supervisor, self-peer

and peer-supervisor ratings. Personnel Psychology, 4, 43-62.Heneman, R. L. (1986). The relationship between supervisory and results-oriented

measures of performance: A meta-analysis. Personnel Psychology, 39, 811-826.

154

Heneman, H. G., & Ellis, R. A. (1982). Correlates of just noticeable differences in pay increases. Labor Law Journal, 33, 533-538.

Holstrom, B. (1979). Moral hazard and observability. Bell Journal of Economics, 10(1), 74-91.

Jackson, S. E., Schuller, R. S., & Rivero, J. C. (1989). Organizational characteristics as predictors of personnel practices. Personnel Psychology, 42, 727-786.

Katkowski, D. A., Medsker, G. J., & Pritchard, K. H. (2002). Literature review of “acceptable” or “just noticeably different” pay increases. Human Resources.

Keeley, M. (1978). A contingency framework for performance evaluation. Academy of Management Review, 3, 428-438.

Keeping, L.M., & Levy, P.E. (2000). Performance appraisal reactions: Measurements, modeling, and method bias. Journal of Applied Psychology, 85, 5, 708-724.

Krefting, L. A., & Mahoney, T. A. (1977). Determining the size of a meaningful pay increase. Industrial Relations, 16(1), 83-93.Lance, C. E., Teachout, M. S., & Donnelly, T. M. (1992). Specification of the criterion

construct Space: An application of hierarchial confirmatory factor analysis. Journal of Applied Psychology, 77, 437-452.

Landau, S. B., & Leventhal, G. S. (1976). A simulation study of administrators’ behavior toward employees who receive job offers. Journal of Applied Social Psychology, 6(4), 291-306.

Landy, F. S., & Farr, J. L. (1980). Performance rating. Psychological Bulletin, 87, 72-107.

Levy, P.E., & Williams, J.R. (1998). The role of perceived system knowledge in predicting appraisal reactions, job satisfaction, and organizational commitment. Journal of Organizational Behavior, 19, 53-65.

Levy, P. E., & Williams, J. R. (2004). The social context of performance appraisal: A review and framework for the future. Journal of Management, 20(6), 881-905.

Lincoln, Y.S., & Guba, E.G. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage Publications.

Longenecker, C.O., & Fink, L.S. (1999). Creating effective performance appraisals. Industrial Management, 41, 5, 18-24.Longenecker, C.O., & Fink, L.S. (2003). Benchmarks for effective performance rating

instruments. Journal of Compensation and Benefits, 19, 2, 24-31.Martocchio, J.J. (2001). Strategic compensation: A human resource management

approach. Upper Saddle River, NJ: Pearson Prentice Hall.Mathis, R.L., & Jackson, J.H. (2006). Human resource management (11th ed.). Thomas

Learning, Mason, OH.McKinney, W. R., & Yen, T. H. (1989). Personnel management in large U. S. park and

recreation organizations. Journal of Park and Recreation Administration, 7(2), 1-25.

Milkovich, G. T., & Newman, J. (2005). Compensation (8th Ed.). Boston, MA: Irwin McGraw-Hill Companies, Inc.

Mitra, A., Gupta, N., & Jenkins, G. D. (1997). A drop in the bucket: When is a pay raise a payraise? Journal of Organizational Behavior, 18(2), 117-137.

155

Moss, S.E., & Martinko, M.J. (1998). The effects of performance attributions and outcome dependence on leader feedback behavior following poor subordinate performance. Journal of Organizational Behavior, 19, 3, 259-274.

Murphy, K. R., & Cleveland, J. N. (1995). Understanding performance appraisal: social, organizational, and goal-based perspectives. Thousand Oaks, CA: Sage Publications.

Nadler, D.A. (1979). The effects of feedback on task group behavior: A review of the experimental research. Organizational Behavior and Human Performance, 25, 123-128.

Ones, D. S., Schmidt, F. L., & Viswesvaran, C. (1996). Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology, 81(5), 557-574.

Petit, A., & Haines, V. (1994). Trois instruments d’evaluation du rendement. Gestion, 59-68.

Podsakoff, P.M., & Farh, J.L. (1989). Effects of feedback sign and credibility on goal setting and task performance. Organizational Behavior and Human Performance, 23, 309-338.

Roberts, G. E. (2003). Employee performance appraisal system participation: A technique that works. Public Personnel Management, 32(1), 89-99.

Rotundo, M., & Sackett, P. R. (2002). The relative importance of task, citizenship, and counterproductive performance to global ratings of job performance: A policy-capturing approach. Journal of Applied Psychology, 87, 66-80.

Schmidt, F. L., & Kaplan, L. B. (1971). Managerial talent, pay, and age. Journal of Vocational Behavior, 1, 129-135.

Schwandt, T.A (2001). Dictionary of qualitative inquiry (2nd Ed.). Thousand Oaks, CA: Sage Publications.

Schweiger, I., & Sumners, G.E. (1994). Optimizing the value of performance appraisals. Managerial Auditing Journal, 9, 8, 3-7

Scullen, S. E., Goff, M., & Mount, M. K. (2000). Understanding the latent structure of job performance ratings. Journal of Applied Psychology, 24, 419-434.

Smith, B.N., Hornsby, J.S., & Shirmeyer, R. (1996). Current trends in performance appraisal: An examination of managerial practice. SAM Advanced Management

Journal, 1, 20.Smither, J. W. (1998). Lessons learned: Research implications for performance appraisal

and management practice. IN: J. W. Smither (Ed.), Performance appraisal: State of the art in practice. San Francisco, CA: Jossey-Bass.

Strauss, A., & Corbin, J. (1998). Basics of qualitative research: Techniques and procedures for developing grounded theory (2nd Ed.). Thousand Oaks, CA: Sage Publications.

Thomas, S.L., & Bretz, R.D. (1994). Research and practice in performance appraisal: Evaluating employee performance in America’s largest companies. SAM Advanced Management Journal, 59, 2, 28-34.

Tompkins, J. (2002). Strategic human resources management in government: Unresolved issues. Public Personnel Management, 31, 1, 95-111.

156

Tziner, A., & Kopelman, R.E. (2002). Is there a preferred performance rating format? A non-psychometric perspective. Applied Psychology: An International Review, 51, 3, 479-503.

Van Scotter, J. R., Motowidlo, S. J., & Cross, T. C. (2000). Effects of task performance and contextual performance on systematic rewards. Journal of Applied Psychology, 85, 526-535.

Vance, R. J., MacCallum, R. C., Coovert, M. D., & Hedge, J. W. (1988). Construct validity of multiple job performance measures using confirmatory factor analysis. Journal of Applied Psychology, 73, 74-80.

Viswesvaran, C., & Ones, D. S. (2000). Perspectives on models of job performance. International Journal of Selection and Assessment, 8(4), 216-226.

Wiersma, V.C., & Latham, G.P. (1986). The practicality of behavioral observation scales, expectation scales and trait scales. Personnel Psychology, 39, 619-628.

Wilson, J.P., & Western. S. (2001). Performance appraisal: An obstacle to training and development? Career Development International, 6, 2/3, 93-102.

Wojcik, J. (2000). Focus on performance. Business Insurance, July, 20.Wright, P. M., & McMahan, G. C. (1992). Theoretical perspectives for strategic human

resource management. Journal of Management, 18(2), 295-320Zedeck, S., & Cascio, W. (1982). Performance appraisal decision as a function of rater

training and purpose of the appraisal. Journal of Applied Psychology, 67(6), 752-758.

Zenger, T. R. (1992). Why do employees only reward extreme performance? Examining the relationships among performance, pay and turnover. Administrative Science Quarterly, 37(2), 198-219.