Journal of Management Education - SAVIE · 10.1177/1052562905280099JOURNAL OF MANAGEMENT...

http://jme.sagepub.com

Journal of Management Education

DOI: 10.1177/1052562905280099 2006; 30; 455 Journal of Management Education

Dalia Rachman-Moore and Ron S. Kenett The Use of Simulation to Improve the Effectiveness of Training in Performance Management

http://jme.sagepub.com/cgi/content/abstract/30/3/455 The online version of this article can be found at:

Published by:

http://www.sagepublications.com

On behalf of:

OBTS Teaching Society for Management Educators

can be found at:Journal of Management Education Additional services and information for

http://jme.sagepub.com/cgi/alerts Email Alerts:

http://jme.sagepub.com/subscriptions Subscriptions:

http://www.sagepub.com/journalsReprints.navReprints:

http://www.sagepub.com/journalsPermissions.navPermissions:

http://jme.sagepub.com/cgi/content/refs/30/3/455 Citations

at TELE-UNIVERSITE on July 15, 2009 http://jme.sagepub.comDownloaded from

http://www.obts.org/

http://jme.sagepub.com/cgi/alerts

http://jme.sagepub.com/subscriptions

http://www.sagepub.com/journalsReprints.nav

http://www.sagepub.com/journalsPermissions.nav

http://jme.sagepub.com/cgi/content/refs/30/3/455


10.1177/1052562905280099JOURNAL OF MANAGEMENT EDUCATION / June 2006Rachman-Moore, Kenett / SIMULATION AND TRAINING IN MANAGEMENT

SIMULATION

THE USE OF SIMULATION TO IMPROVETHE EFFECTIVENESS OF TRAINING INPERFORMANCE MANAGEMENT

Dalia Rachman-MooreCollege of Management

Ron S. KenettKPA Ltd., and University of Torino

Performance management is an important managerial tool that directsemployees’goals and behavior toward the organization’s strategic goals. Thisarticle focuses on simulation-based training in performance management sys-tems. The simulation developed at the School of Business Administration of theCollege of Management in Israel is based on a typical life insurance firm. It isdesigned to promote understanding of the process of performance manage-ment. The simulation involves assessing performance of supervisors usingmultirater assessment and an organizational Balanced Scorecard. Studentfeedback showed that the simulation significantly enhanced the effectivenessof the performance-management teaching program.

Keywords: performance management; balanced scorecard; multi-rater as-sessment; training; simulation; business education; teachingmethods

455

Authors’Note: The development of this simulation was funded by the Research Unit of Schoolof Business Administration, the College of Management, Academic Studies, Rishon LeZion,Israel. We thank Prof. Tamir Agmon for encouraging us to write up this simulation and Prof.Micha Etgar and Prof. Arie Reichel for providing feedback on earlier versions. We would alsolike to thank the anonymous reviewers for their helpful comments. Correspondence concerningthis article should be addressed to Dr. Dalia Rachman-Moore, School of Business Administra-tion, College of Management, Academic Studies, 7 Yitzhak Rabin Boulevard, P.O. Box 9017,Rishon Lezion, 75190, Israel; e-mail: [email protected]. URL of the Performance Manage-ment simulation: http://www.kpa.co.il/english/organizational/index.php.

JOURNAL OF MANAGEMENT EDUCATION, Vol. 30 No. 3, June 2006 455-476DOI: 10.1177/1052562905280099© 2006 Organizational Behavior Teaching Society



Teaching MBA students presents a special challenge. Most students wantto study a variety of management areas through a practical approach gearedtoward applying the acquired knowledge in a realistic work environment. Acombination of an experiential approach and lectures enables students toexamine the process of business decision making in a practical manner andenhances their ability to apply concepts and tools presented in class (e.g.,Alderfer, 2003; Benek-Rivera & Mathews, 2004; Hedges & Pedigo, 2002;Kern, 2000; Ledman, 2001; Licari & Ovedovitz, 2002; K. S. McDonald &Mansour-Cole, 2000; McKone & Bozewicz, 2003; Muir, 2001; Trefry,2002).

This article presents a simulation through which MBA students can expe-rience the process of performance management as a managerial tool to helporganizations reach strategic objectives. The simulation was conducted suc-cessfully as part of an MBA program at a school of business administration.A first report on this simulation was presented in Kenett and Rachman-Moore (2003).

This article describes performance management, explains the simulationand its underlying principles, and finally offers conclusions and recommen-dations based on classroom observations and student feedback.

Background of Performance Management

Fisher, Schoenfeldt, and Shaw (2003) defined performance management as

The integration of performance appraisal systems with broader human resourcesystems as a means of aligning employees’work behaviors with organizationalgoals. Performance management should be an ongoing, interactive processthat is deigned to enhance employee capability and facilitate productivity.(p. 482)

Performance management is a process that expands on performance appraisal.It is a managerial tool that links employee tasks and goals with organizationalgoals and utilizes ongoing performance feedback to direct employee behav-ior toward realization of organizational goals. This new generation of assess-ment systems focuses on organizational development and application oforganizational strategy to human resources (Kenett & Maisel, 1998; Kenett,Waldman, & Graves, 1994; Waldman & Kenett, 1990).

Bevan and Thompson (1991) identified performance management ascomprising

• a shared vision of organizational objectives communicated to all employees

456 JOURNAL OF MANAGEMENT EDUCATION / June 2006



• the setting of individual performance measurement targets that relate to theoperating unit and wider organizational objectives

• a regular formal review of progress to measure performance, identify trainingand development needs, and determine reward outcomes

• an evaluation of the process’s effectiveness and contribution, to facilitateimprovement of the process.

Many studies have examined the effectiveness of performance manage-ment. Several of these have produced positive findings:

• Performance management facilitates the implementation of business strategyby indicating what to measure, determining appropriate means of measure-ment, setting targets, and linking the measures with organizational performance(Scheiner, Shaw, & Beatty, 1991).

• Performance management improves the organization’s performance(D. McDonald & Smith, 1995), improves processes within the organization(Rummler & Brache, 1990), improves employee performance (Egan, 1995;Lane, 1994; Longenecker & Fink, 1999) and team performance (Kenett et al.,1994; Lawler, 1994), and eases implementation of changes in the organiza-tional culture (Wellins & Schulz Murphy, 1995).

Numerous initiatives have been put forward to manage performance. Theperformance management simulation that we developed incorporates theBalanced Scorecard and the multirater assessment methods. The BalancedScorecard concept was introduced by Kaplan and Norton (1992) to enhancetraditional management-accounting reporting systems. It links organiza-tional strategy and performance management by translating organizationalstrategy into measures aligned with organizational goals. The measures cap-ture the organizational vision and the strategies of individual units and of theoverall organization. It facilitates the implementation of an organizational (orbusiness) strategy and enhances employee assimilation of the strategy.

Implementing a Balanced Scorecard requires an organization to set per-formance measures from four perspectives: financial (e.g., revenue), cus-tomer relations (e.g., customer loyalty), internal business processes (e.g.,number of processing errors), and learning and growth (e.g., amount of train-ing undertaken). Performance targets are then set for each area. Some mea-sures reflect results from the past and are called “lag indicators” (e.g., finan-cial measures), while others predict future results and are called “leadindicators” (e.g., customer satisfaction).

The relationships among these four perspectives play an important role inthe organization’s strategic map. Feedback derived from such measuresenables an organization to refine its standards, objectives, and targets; totrack gaps and deviations; and to compare performance between employees

Rachman-Moore, Kenett / SIMULATION AND TRAINING IN MANAGEMENT 457



and units. The Balanced Scorecard also provides a basis for discussion at alllevels of the organization’s hierarchy on topics such as formulating objec-tives, targets, measures, and organizational learning (see Norton & Kaplan,2000).

The multirater method of performance management utilizes performancefeedback from several sources. The 360-degree assessment method is onemultirater technique. This approach seeks the feedback of people (sources)who interact with the employee being assessed. Sources may be internal(such as an employee’s supervisor[s], peers, and subordinates) or external(such as an employee’s suppliers and clients). Ghorpade (2000) noted thatalmost all Fortune 500 companies use multirater assessment.

Hoffman (1995) described several advantages of multirater assessment:obtaining a broader picture of employee performance, identifying perfor-mance thresholds, defining corporate competencies, increasing the focus oncustomer service, supporting team initiatives, creating a highly involvedworkforce, decreasing hierarchical structure and promoting a streamlinedstructure, detecting barriers to success, assessing developmental needs, andavoiding discrimination and bias.

Most of the studies described in the book edited by Tornow and London(1998) and in the literature survey by Ghorpade (2000) show that feedbackbuilt on multirater assessment is important in identifying training and devel-opment needs; however, it lends itself less to administrative and organiza-tional decisions, such as layoffs.

One factor behind the increasing use of multirater assessment is the beliefthat feedback from a variety of sources improves performance. Several stud-ies show that organizations can improve performance by adopting multiraterassessment (Atwater, Roush, & Fischthal, 1995; Milliman, Zawacki,Schults, Wiggins, & Norman, 1995; Reilly, Smither, & Vasilpopoulos,1996). However, Kluger and DeNisi’s (1996) survey of 600 companies thatuse multirater assessment reveals that only one third show increased perfor-mance. One reason for this poor result is inadequate training of managers.

THE NEED TO TRAIN MANAGERSIN PERFORMANCE MANAGEMENT

Lack of training and guidance for managers often leads to undesirableresults when using performance management. Although several studiesshow that performance management helps implement business and organiza-tional strategy and also to improve individual performance, it is not alwayssuccessful. Flynn (1995) and Glendinning (2002) indicated in their studiesthe importance of training managers to successfully implement performance




management. Glendinning (2002), in an extensive literature survey,described the advantages of the performance management method and notedthat incorrectly implementing the method can damage the organization.

Management training is also essential in multirater assessment. Klugerand DeNisi (1996) argued that lack of such training was one factor in the rela-tively low percentage of organizations showing increased performance withthe use of multirater assessment. Ghorpade (2000) presented the dilemmasand paradoxes associated with using multiple sources of assessment andemphasized the importance of training managers.

Implementing an effective performance appraisal is a complex task, espe-cially when performance management incorporates multirater assessmentand the Balanced Scorecard. As such, it is not always successfully achieved.Therefore, effective training of managers and MBA students is important forimplementing performance management. The academic training of MBAstudents requires exposure to theories, tools, and hands-on experience.

Universities and colleges play an important role in preparing skilled man-agers for industry, service, and public organizations. These institutions needto examine ways to enhance learning to improve the “product” they supply.This “product” includes the knowledge, tools, and experience studentsacquire while studying for an MBA. Simulations have been effective in manycountries. The experiential approach demonstrates the integration of man-agement concepts within a simulated business setting (Miller, 1991). The useof simulation facilitates student learning (Wenzler & Charlier, 1999) andincreases understanding (Klein, 1984). The frame of reference provided in asimulated business setting gives students some of the experience they need intheir quest for business education (Licari & Ovedovitz, 2002). The type ofsimulation presented here can be performed in any organization interested inperformance management with multirater assessment and the BalancedScorecard. The simulation is based on a fictitious company that could belocated anywhere.

Purposes of the Simulation

The performance management simulation gives students the opportunityto gain hands-on experience in performance management. The students maphuman resources of a typical insurance company; analyze, classify, and inter-pret multiple assessment data; and discuss operational decisions to promoteachievement of strategic goals. The simulation is focused mainly on the indi-vidual level but also extends to the unit and organizational level.




PRIMARY OBJECTIVES OF THE SIMULATION

The simulation enables students to

• understand and interpret varied data and measures (objective and subjective)from different assessment sources and classify them according to BalancedScorecard perspectives.

• assess employee strengths, weaknesses, and ability to meet targets and objec-tives; classify the employee according to performance and potential.

• understand the implications of individual assessments on employee and organi-zational development processes (such as identifying strengths, developmentaland training needs, and coaching or monitoring requirements).

• understand the implications of mapping human resources in terms of organiza-tional and administrative decisions (such as compensation, incentives, promo-tion, recruitment and selection needs).

SECONDARY OBJECTIVES OF THE SIMULATION

In addition to the above objectives, the simulation also provides studentsexperience in assessing an organization at various levels:

• unit level• organizational level

The Simulation

The simulation applies the performance management method to a lifeinsurance supervisor in a fictitious insurance company called “Insurance forAll,” which has several branches. The simulation analyzes rating informationregarding life insurance supervisors in four geographical areas labeled A, B,C, and D. Each of these areas has an area manager responsible for five lifeinsurance supervisors. Thus, there are four area managers and 20 supervi-sors. Every supervisor is responsible for at least 10 insurance agents. The lifeinsurance supervisors are employees of the company; however, the insuranceagents are not. All 20 supervisors are assessed, using specially designedquestionnaires, by their area manager, their insurance agents, and a corporateservice coordinator, who processes insurance claims received from theinsurance supervisors.

The assessment of a supervisor enhances learning with respect to (a) goalsetting, (b) the use of individual employee assessments for developmentalpurposes, and (c) the use of assessments for organizational and administra-tive decisions. It also deepens the students’ understanding of organizationalstrategy and their ability to assess progress toward organizational goals.




There are four sources of assessment for each supervisor:

• an objective quantitative computer report issued by the financial department. Itincludes performance measures, performance results, and targets.

• multirater appraisals of subjective behavior. Appraisals are obtained from threesources (the area manager, the sales agents, and the service coordinator) andbased on a questionnaire about job behaviors (“behavioral items”).

The simulation consists of 20 individualized information packs. Eachpack contains assessment data for one insurance supervisor. The contents ofthe information packs are distributed gradually to the students throughout thesimulation and include

• a description of the company (including its values, strategic goals, and targets)• a job analysis for a life insurance supervisor• performance measures and targets for a life insurance supervisor• three blank multirater assessment forms (the subjective behavioral question-

naires) and the objective quantitative performance measures (the blank com-puter report)

• classification of the performance measures from the computer report and thequestionnaires in terms of Balanced Scorecard perspectives

• the assessment data from the four sources. The data differed for each of thesupervisors.

GENERATION OF THE ASSESSMENT DATA

Six types of supervisors were defined according to the model proposed byKenett and Maisel (1998). The model classifies employees according to cur-rent performance (above expectation, as expected, below expectation) andpotential performance (yes—the employee has potential, no—the employhas no potential). Potential is determined using competency models designedto support the organization strategy:

• Bright Stars: Employees ranked highest in terms of performance and potential.They should receive first priority in developmental assignments and be first inline for promotion.

• Potential Stars: Those whose performance is average but whose potential maybe high. They are strong developmental candidates. Their performance is alsoexpected to be exceptional if their potential is assessed correctly.

• Futures: Employees, probably newly appointed, who have not yet performedbut whose potential is highly rated. It is expected that their performance willimprove with time.

• Performance Stars: Employees whose performance is highly rated but whosepotential is low. They are also developmental candidates whose real potentialmust be tested.




• Blockers: Employees whose performance is acceptable and who do not appearto have the potential to make a greater contribution. These employees mayblock positions and opportunities for others and may frustrate required changesto keep their jobs and their workforce.

• Readjusts: Employees whose performance and potential are low and need read-justment. A short term of corrective action or an alternative career direction(within or outside of the organization) is recommended.

This classification helps the students create a human resources strategythat includes developmental planning, such as promoting “stars” from withinthe organization or through external recruiting. It also includes creating asuitable succession plan, providing guidance or developing an alternativecareer plan for Blockers.

The 20 supervisors are randomly classified into the six employee typesusing a uniform distribution (i.e., each supervisor type has a 1/6 probability ofoccurrence). Every supervisor’s performance evaluation package containsassessments from the four sources and spans the past 6 months. The assess-ment results are designed to reflect the supervisor’s type using type-dependentprobability distributions.

To produce the simulation data, expected and potential performance lev-els are defined for each of the six types. For example, Blockers have accept-able performance but no potential, and Futures perform below expectationsbut score high on potential. Based on these definitions, a probability distribu-tion is defined for each of the performance measures and the behavioral items(from the objective quantitative report and the subjective behavior appraisalquestionnaires). Actual scores are generated using these distributions. There-fore, per supervisor type, there are differences in the performance manage-ment assessment data reflecting random statistical differences. The data inthe performance evaluation packs of the 20 supervisors were computed usingMINITAB v13.2.

STUDENT BACKGROUND

The MBA course on performance management is an advanced course.Students taking this course are interested in human resource management. Ofthe students, 80% are female, and 30% are immigrants to Israel. The courseprerequisites are courses in quantitative and qualitative research methods,finance, accounting, marketing, information systems, strategy, industrial psy-chology, organizational theory, organizational behavior, and human resourcemanagement. Before the simulation begins, three weekly meetings of 3 aca-demic hours each are held, discussing human resource strategy, performancemanagement, and the Balanced Scorecard. The MBA program is an evening




program, and, in addition to their studies, most students work full-time. Gen-erally, these students are in the early stages of their careers and hold midlevelmanagement positions in human resource management (such as trainingmanager, recruitment and selection coordinator). The average age is about 26years. The simulation is designed for graduate students in the MBA programwith work experience so that they can understand the basic elements of orga-nizational strategy, human resource strategy, and performance management.

CLASS SIZE

The optimal class size is 20 students, so that each student receives the per-formance evaluation pack of one insurance supervisor. The simulation can bealso used for a class of up to 40 students, where 2 students receive the sameperformance evaluation pack and work together on the required tasks. Eigh-teen students participated in the simulation exercise reported here.

DURATION OF THE SIMULATION

The simulation is conducted over five weekly meetings of 3 academichours each.

Procedure

In the first 2 weeks of the simulation, the students are required to workduring class in small groups of two to four students. Starting the 3rd week,students analyze the assessment data of one insurance supervisor. During the3rd, 4th, and 5th week of the simulation, students complete five structuredtasks in class and submit them for evaluation 1 week after completion of thesimulation.

To facilitate class discussion, each of the 20 supervisor files has a codeassigning it to one of the six types. Only the instructor knows the codes. Foreach structured task, six to eight students are chosen (based on the code) topresent their results. This covers all the types.

FIRST WEEK: ESTABLISHING THE VISION, GOALS,AND STRATEGY OF THE LIFE INSURANCE COMPANY

One week before the first simulation session, each student receives a briefdescription of the company, its vision, and background information on theinsurance market, including market changes and challenges.

In the classroom, the students are asked to recommend organizational val-ues and goals for the company to help achieve its vision. They are further




asked to suggest appropriate strategic initiatives and objective quantitativeperformance measures and targets. After the discussion, the students receivea copy of our answers to these questions. We state that our answers should beused to provide a uniform basis for the rest of the simulation.

The students are then asked to classify the objective performance mea-sures according to the Balanced Scorecard’s perspectives. Following the dis-cussion, we distribute our classification of the performance measures.

SECOND WEEK: JOB ANALYSIS OF THE LIFEINSURANCE SUPERVISOR, THE COMPENSATION SYSTEM,AND THE FOUR APPRAISALS FOR THE SUPERVISOR

Job analysis. Based on the data distributed in the first meeting, partici-pants are asked to analyze the life insurance supervisor’s job. The analysisincludes role definition, descriptions of tasks and the knowledge, skills, val-ues, behaviors, attitudes, and personality required for the job.

After the discussion, we distribute an example of job analysis with adetailed description of the behaviors and processes required from the super-visor to achieve organizational goals.

We initiate a discussion of the company’s compensation scheme, includ-ing a comprehensive description of our recommended scheme. It is recom-mended that the salary scheme in this company have two components: onefixed and the other variable. The variable component depends on meeting tar-gets and satisfactory behavior with regard to job performance. For the sake ofuniformity, a specific compensation scheme is adopted for the rest of thesimulation.

Objective quantitative performance measures for the supervisor. The stu-dents are asked to derive quantitative performance measures and targets forthe supervisors from the organizational goals and targets. Following class-room discussion, we distribute our suggestions for quantitative measures andtargets that can be issued by the computer for the next 6 months. Using thesuggested quantitative report components as a basis, the students classify theirmeasures according to the four Balanced Scorecard perspectives (finance, cus-tomer relation, business process, learning and growth). A copy of our classifi-cation is distributed. In the simulation, the quantitative performance measuresand targets (the same for all 20 supervisors) emerge from the company’sgoals and targets and the supervisor’s job analysis. Sample measures include

• financial performance: 10% growth in premiums over 6 months• customer relations: 20% growth in customer numbers over 6 months




• internal business processes: meeting payment deadlines for more than 90% ofpremiums

• learning and growth: at least one course per employee over 6 months

Subjective behavior appraisal for the supervisor from the area manager,agents, and service coordinator. The students receive questionnaires fromthe three sources. The questionnaires are based on Behavioral ObservationScales (Latham & Wexley, 1982). A 5-point frequency-of-occurrence scaleis attached to each behavioral item (1 = almost never, 2 = rarely, 3 = some-times, 4 = often, 5 = almost always).

Subjective behavior appraisal received from the area manager. The areamanager completes a questionnaire for each of his or her supervisors. Thequestionnaire contains 78 behavioral items classified according to behav-ioral performance dimensions such as responsibility, communication, ser-vice to agents, and so on.

Subjective behavior appraisal received from the agents. Every supervisorhas 10 agents, each of whom completes one questionnaire. The agent ques-tionnaire contains 52 behavioral items derived from the area manager’s ques-tionnaire, and the items are classified according to the corresponding behav-ioral performance dimensions.

Subjective behavior appraisal from the service coordinator. The question-naire completed by the service coordinator contains 28 behavioral items andis derived from the area manager’s questionnaire. It is classified according tothe corresponding behavioral performance dimensions. Only the relevantdimensions appear in the service coordinator’s questionnaire.

Classification of the items from the three subjective appraisal question-naires according to the four Balanced Scorecard perspectives. The studentsare asked to classify the items from the three questionnaires according to thefour Balanced Scorecard perspectives. Following classroom discussion, wedistribute a copy of our suggestions to the students.

Thus, the students have two classifications of the items from the three sub-jective behavior appraisals:

• one classification according to behavioral performance dimensions• one classification according to the four Balanced Scorecard perspectives.




THIRD WEEK: ANALYZING THE BEHAVIORAPPRAISAL DATA FROM THREE SOURCES

Each supervisor file includes the assessment data from the area managerand the service coordinator, and the means and standard deviations of theappraisal data submitted by the agents. Furthermore, the file includes statisti-cal reports for the behavioral performance dimensions and a report for eachof the four Balanced Scorecard perspectives (see Table 1). For each BalancedScorecard perspective and behavioral performance dimension, these reportscontain the number of behavioral items, the average of the scores, and thepercentage of items occurring with the highest (5) and lowest (1) frequencies.Based on these data, the students are asked to complete a structured task inclass and submit it at the end of the simulation.

Task 1: Analyzing the subjective behavior appraisal data. The studentsare asked to

• categorize data from each source into behavior appraisal dimensions and Bal-anced Scorecard perspectives.

• compare the results from the three sources according to behavioral dimensionsand Balanced Scorecard perspectives, and offer possible explanations fordifferences.

• describe strengths and weaknesses of the supervisor in terms of behavioraldimensions and Balanced Scorecard perspectives.

• Students present their results for discussion.


TABLE 1Example of a Statistical Report Categorized

According to a Balanced Scorecard Perspective

Supervisor 1: 12 Years in the Company

Area Service Sales Agents’ SD ofBalanced Manager’s Coordinator’s Appraisals Agents’Scorecard perspective Appraisal Appraisal (M) Appraisals

Learning and growth measureM 3.35 3.79 3.10 .31N 34 14 20 20% indicating very high 8.8% 0% 30% 0%% indicating not at all 0% 0% 30% 100%



FOURTH WEEK: ANALYZING THE QUANTITATIVEREPORT ON THE SUPERVISOR AND COMPARINGIT WITH THE BEHAVIOR APPRAISAL DATA

Each supervisor file also includes a quantitative computer-generatedreport that presents the following, for each of the objective performancemeasures:

• the semiannual performance target range for the supervisor• the semiannual performance results for the supervisor• the percentage of change in the supervisor’s performance compared to that of

the previous 6-month period• the mean performance results for all the supervisors over the 6-month period

(referred to as “mean supervisor performance”)• the ratio of the supervisor’s results for each performance measure over the cor-

responding “mean supervisor performance” (expressed as a percentage andreferred to as the supervisor’s “relative performance location”).

Using these data, the students are asked to complete the second structuredtask.

Task 2: Analyzing a quantitative report. The students are asked to assessthe supervisor in two ways:

• in terms of how the supervisor’s performance compared with the BalancedScorecard targetsFor each performance measure, the students are required to assign:

• A meeting-the-target score ranging from 1 to 3 (1 = below the targetrange, 2 = in the range, 3 = above the target range)

• A meeting-the-target index for each of the four perspectives of theBalanced Scorecard (computed as the mean of the meeting-the-tar-get scores for the performance measures associated with each Bal-anced Scorecard).

• by classifying the relative performance location in terms of the Balanced Score-card perspectives.

The students are required to calculate four indices, one for each of the Bal-anced Scorecard perspectives. The indices are computed as the mean of thesubject supervisor’s relative performance location in each performance mea-sure. A score of more than 100% describes performance above the norm, anda score of less than 100% describes performance below the norm.

Students present their results for discussion.




Task 3: Integrating the quantitative report with the behavior appraisaldata. Students are asked to describe the consistencies and inconsistenciesbetween the objective quantitative report and the three multirater assess-ments, classified in terms of the Balanced Scorecard perspectives.

For example, the objective quantitative report finds that the supervisor didnot meet the customer-related targets, while the behavioral assessment datashows that the supervisor performed above average. Possible reasons for thisinconsistency are considered. Are the targets too high? Did the marketchange, resulting in unreasonable targets? Is there a problem with customerrelations in the organization? The addition of the three behavioral assess-ments helps the students understand whether customer relations in the orga-nization are adequate. The discussion lends itself to a variety of interpreta-tions and possible explanations for the inconsistencies (including biasedevaluations, rating errors, etc.)

Task 4: Mapping the supervisor according to performance level andpotential. Students are asked to determine the supervisor’s type according tothe classifications of performance and potential (e.g., Blockers, PerformanceStars) and to suggest an appropriate developmental process. Students presenttheir findings for discussion.

FIFTH WEEK: ANALYZING THE ASSESSMENTDATA AT THE AREA LEVEL

We distribute three tables of data on all the supervisors. The first twotables are based on the objective quantitative report: the first contains themeeting-the-targets indices, the second table contains an index of the super-visor’s relative performance location. The third table presents the subjectivebehavior appraisal indices for all the supervisors. The data in all the tables isclassified according to the Balanced Scorecard perspectives. The studentsare asked to use the data to complete Task 5.

Task 5: Analyzing the assessment data for each area. For each sales area,the students are asked to use the objective quantitative report and the subjec-tive behavior appraisal data to determine performance in the area, in terms of

• meeting the targets• its performance compared to other areas• its strengths and weaknesses

The students’ findings and interpretations are discussed.




TASK EVALUATION

The simulation is a course requirement for all students. One week after theend of the simulation, the students submit the five structured tasks outlinedabove. The overall weight of the tasks is 10% of the final course grade. Thesimulation is considered a group exercise to which everyone contributes.

Observed Contribution of the Simulation to Student Learning

The students’ progress was monitored throughout the simulation, duringperformance of the tasks and the associated classroom discussions. It wasobserved that the simulation enhances several aspects of the learning process.

The students learn through experience to define quantitative goals and tar-gets for an individual employee. In particular, they learn to

• practice job analysis (in this case, for the supervisor position).• focus on the important aspects of performance management in this position.• derive quantitative targets for a specific position from the organization’s goals

and classify them according to the Balanced Scorecard perspectives.

The students gain a deeper understanding of assessment tools. In particu-lar, they learn to

• analyze multirater data by examining subjective behavior appraisalquestionnaires.

• classify the subjective behavior appraisal data measures according to the Bal-anced Scorecard perspectives.

The students learn to assess employee performance (in this case, the per-formance of each supervisor) by analyzing assessment information from thearea manager and the objective quantitative report. This includes learning to

• ascertain the employee’s strengths and weaknesses.• determine whether the employee is meeting targets in terms of the Balanced

Scorecard perspectives.• compare the assessment of whether the targets are being met (in terms of the

Balanced Scorecard perspectives) with the area manager’s behavioral assess-ment and propose reasons for the differences.

• analyze the relative performance location of an employee compared to othersholding the same position, in terms of the Balanced Scorecard perspectives.

• compare the assessment of whether the targets are being met (in terms of theBalanced Scorecard perspectives) and the relative performance location with thearea manager’s behavioral assessment and propose reasons for the differences.

• Draw conclusions regarding the employee’s potential and performance.




The students gain experience using multirater feedback as a tool for mak-ing operational decisions. Using the supervisors as an example, they learn to

• compare subjective behavior appraisal data from multirater sources (in thiscase, from the service coordinator, the area manager, and the agents), andpropose possible explanations for differing evaluations of the employee’sperformance.

• classify an employee according to potential and performance on the basis ofobjective quantitative data and multirater feedback.

• recommend career development pathways, including promotion and succes-sion plans.

The simulation gives students experience in analyzing an organizationthrough performance management of the supervisors. The students also gaininsights into performance management at the unit level from the exercises inwhich they classify the performance assessments according to BalancedScorecard criteria and aggregate them over each sales area. In particular, theylearn to

• assess whether targets at the operating-unit level (i.e., the four sales areas) arebeing met.

• compare the relative performance location with that of other operating units.• understand the implications of a subordinate’s performance on a manager’s per-

formance rating, through examining the relationship between the supervisors’performance and that of the area manager.

Student Feedback

The effectiveness of a teaching method can be assessed through studentfeedback. After one simulation, a feedback form was distributed to the stu-dents. Of the 18 participants, 15 completed the questionnaire (an 83%response rate). The questionnaire had two parts.

FIRST PART - OPEN QUESTION

Students were asked if the simulation enhanced learning and to describehow it did so. All the students responded, using similar wording, that the sim-ulation contributed significantly to their understanding of the theoreticalbackground (i.e., it increased their understanding of Balanced Scorecard andmultirater feedback theory). The students also agreed that the simulationincreased their understanding of the implementation of performance man-agement, and their ability to relate to the field.




The students also noted several other ways in which the simulation con-tributed to learning:

“[I learned how to] understand the data from the assessment questionnaires.”“[The simulation enabled me to] understand the connections between three differ-

ent areas: performance management, recruiting and selection.”“The simulation provided tools to implement the multirater assessment method

and to analyze different types of data (objective and subjective) in performancemanagement.”

“[I learned how] employee assessment is derived from the strategic goals.”“[I experienced] mapping the employee according to potential and performance.”“[I gained an] understanding of the disparities between the viewpoints of raters

holding different positions. In other words, performance appraisal is a compli-cated process.”

“The simulation helped me to understand the Balanced Scorecard.”

In summary, most students reported that their understanding of perfor-mance management and their ability to implement it increased, particu-larly with respect to identifying employees’ strengths and weaknesses,defining organizational objectives, and mapping employees’ potential andperformance.

SECOND PART - CLOSED QUESTIONS

The feedback questionnaire contained nine questions measuring the sim-ulation’s contribution to students’ learning, using a 5-point Likert-type scale(1 = not at all, 2 = low, 3 = moderate, 4 = high, 5 = very high). The questionswere derived from the simulation’s goals; each question reflected attainmentof a goal. The results of the questionnaire are shown in Table 2.

Table 2 shows that the simulation’s most important contribution was pro-viding students with tools and knowledge regarding the assessment of indi-vidual employee performance. A considerable majority (87% to 93%) of stu-dents noted that the simulation made a high or very high contribution to theirlearning in the following three respects:

• identifying employees’ strengths and weaknesses• mapping employees according to potential and performance• assessing whether targets and organizational objectives had been met.

The simulation also enhanced learning at the unit level. A large majorityof students noted that the simulation made a high or very high contribution to

• assessing whether the area manager met his or her goals (93%)• identifying the area manager’s strengths and weaknesses (73%).




472

TA

BL

E 2

Mea

n, M

edia

n, a

nd S

tand

ard

Dev

iati

on o

f th

e St

uden

t F

eedb

ack

% o

f stu

dent

sin

dica

ting

Sim

ulat

ion

"hig

h" o

rG

oals

Item

Goa

lM

Mdn

SD“

very

hig

h”

Pri

mar

yA

sses

smen

t and

map

ping

the

empl

oyee

(su

perv

isor

)go

als

1D

id y

ou le

arn

to a

sses

s w

heth

er th

e em

ploy

ee (

supe

rvis

or)

met

the

stra

tegi

c go

als?

4.13

4.5

293

.42

Did

you

lear

n to

iden

tify

the

stre

ngth

s an

d w

eakn

esse

s of

the

empl

oyee

(su

perv

isor

)?4.

404

.63

93.4

3D

id y

ou le

arn

to m

ap th

e em

ploy

ee (

supe

rvis

or)

acco

rdin

g to

per

form

ance

and

pot

entia

l?4.

334

.72

86.7

The

impl

icat

ions

of m

appi

ng h

uman

reso

urce

s on

org

aniz

atio

nal d

ecis

ions

4D

id y

ou le

arn

to r

ecog

nize

the

impl

icat

ions

that

map

ping

the

hum

an r

esou

rce

has

on3.

674

.72

66.7

orga

niza

tiona

l dec

isio

ns (

such

as

layo

ffs,

pro

mot

ions

, rec

ruiti

ng, t

rain

ing)

?

Seco

ndar

yA

sses

smen

t of t

he a

rea

man

ager

goal

s5

Did

you

lear

n to

ass

ess

whe

ther

the

area

man

ager

met

the

stra

tegi

c go

als?

4.00

4.3

893

.46

Did

you

lear

n to

iden

tify

the

stre

ngth

s an

d w

eakn

esse

s of

the

area

man

ager

?3.

804

.94

73.3

Ass

essm

ent o

f the

com

pany

7D

id y

ou le

arn

how

to a

sses

s w

heth

er th

e pr

oble

ms

in th

e co

mpa

ny w

ere

a re

sult

of th

e3.

644

.84

57.2

area

man

ager

, the

em

ploy

ee (

supe

rvis

or)

or o

utsi

de in

flue

nces

?8

Did

you

lear

n to

ass

ess

whe

ther

the

com

pany

met

its

stra

tegi

c go

als?

3.53

41.

1253

.39

Did

you

lear

n ho

w to

iden

tify

the

stre

ngth

s an

d w

eakn

esse

s of

the

com

pany

?2.

933

.96

26.7



The simulation also contributed to students’ understanding of how map-ping the employee according to potential and performance can affect organi-zational decisions. About two thirds (67%) of the students noted that the sim-ulation made a high or very high contribution to understanding this topic. Thesimulation played a lesser role in learning to assess the overall performanceof the company. Even so, more than one fourth of the students (27%) noted ahigh or very high contribution in this area. Furthermore, it promoted learningwith respect to detecting problems in the company, with 57% of the studentsnoting that the simulation made a high or very high contribution to theirlearning. It also contributed to the ability to assess whether strategic organi-zational objectives were being met, with 53% of students considering thesimulation effective in this respect.

Conclusions

From the open and closed questions in the feedback questionnaire, it isclear that the students’ reaction to the simulation was positive. The simula-tion focused mainly on methods for assessing employees at the individuallevel, understanding the process and implications of performance manage-ment, and the relationship between performance management and meetingthe organization’s goals. The feedback results indicate that these purposeswere achieved. Performance management is also valuable in terms of therelationship between performance appraisal and meeting targets derivedfrom strategic goals. The simulation enhanced learning in this area, as it wasfound that most students considered the simulation to make a high or veryhigh contribution to understanding this relationship.

With respect to the unit and organizational levels, student feedback wasless positive, because the simulation did not include sufficient data to enableparticipants to deeply analyze performance at these levels. However, despitethe lack of the data, the students noted that the simulation contributed towardassessing the performance of the manager and, to a lesser degree, towardassessing the performance of the company. It may be that the positive feed-back in these areas resulted from the classroom discussions. In the future, tosimulate the use of performance management at the unit and organizationallevels, we will need to expand the simulation and provide more data.

The student feedback presented here comes from the first performance ofthe simulation. In the two subsequent rounds (involving about 20 studentseach), only the same open question was asked, and the responses were almostidentical to those of the first simulation.




Academic studies have shown the importance of performance manage-ment as a tool to ensure that employee goals and behavior are directed towardachieving organizational goals. Practical experience in performance man-agement is an essential factor in developing future managers and may make asignificant contribution to the industry in the future. This is particularly truegiven that the vast majority of failures in the implementation of performancemanagement have resulted from inadequate training of mangers and employ-ees. The student feedback makes it clear that simulation plays a very impor-tant role in developing an understanding of performance management toolsand their implementation. Therefore, it is recommended that experientialapproaches, such as the one outlined here, be added to the theoretical andresearch components of MBA programs.

References

Alderfer, C. P. (2003). Using experiential methods to teach about measurement validity. Journalof Management Education, 2(5), 520-532.

Atwater, L. E., Roush, P., & Fischthal, A. (1995). The influence of upward feedback on self andfollower ratings of leadership. Personnel Psychology, 48, 35-39.

Benek-Rivera, J., & Mathews, V. E. (2004). Active learning with Jeopardy: Students ask thequestions. Journal of Management Education, 28(1), 104-118.

Bevan, S., & Thompson, M. (1991). Performance management at the crossroads. PersonnelManagement, 23, 36-39.

Egan, G. (1995). A clear path to peak performance. People Management, 18, 34-37.Fisher, C. D., Schoenfeldt, L. F., & Shaw, J. B. (2003). Human resource management (5th ed.).

Boston: Houghton Mifflin.Flynn, G. (1995). Training is the key to successful performance management. Personnel Journal,

75(2), 26.Ghorpade, J. (2000). Managing five paradoxes of 360-degree feedback. Academy of Manage-

ment Executive, 14(1), 140-150.Glendinning, P. M. (2002). Performance management: Pariah or messiah. Public Personnel

Management, 31(2), 161-178.Hedges, P., & Pedigo, K. (2002). What’s happening in your neighborhood? An experiential exer-

cise for teams. Journal of Management Education, 26(3), 322-333.Hoffman, R. (1995). Ten reasons you should be using 360-degree feedback. HRMagazine, 40(4),

82-85.Kaplan, R. S., & Norton, D. P. (1992). The balanced scorecard: Measures that drive performance.

Harvard Business Review, 70(1), 71-79.Kenett, R. S., & Maisel, M. (1998). Process performance, appraisal and employee development

planning. In C. Madu (Ed.), The handbook of total quality management (pp. 344-364). Lon-don: Kluwer Academic.

Kenett, R. S., & Rachman-Moore, D. (2003). On the use of simulations to improve the effective-ness of training in performance appraisal systems. Mashabei Enosh, 185, 30-43.




Kenett, R. S., Waldman, D. K., & Graves, S. B. (1994). Process performance appraisal systems:A working substitute to individual performance appraisal (in Hebrew). Total Quality Man-agement, 5, 267-280.

Kern, J. A. (2000). Manufacturing power relations: An organizational simulation. Journal ofManagement Education, 24(2), 254-275.

Klein, R. D. (1984). Adding international business to core program via simulation game. Journalof International Business Studies, 15(1), 151-159.

Kluger, A. N., & DeNisi, A. (1996). The effect of feedback interventions on performance: A his-torical review, a meta-analysis, and a preliminary feedback intervention theory. Psychologi-cal Bulletin, 119, 254-284.

Lane, L. M. (1994). Old failures and new opportunities: Public sector performance management.Review of Public Personnel Administration, 14(3), 26-44.

Latham, G. P., & Wexley, K. N. (1982). Increasing productivity through performance appraisal.Reading, MA: Addison-Wesley.

Lawler, E. (1994). Performance management: The next generation. Compensation and BenefitsReview, 26(3), 16-20.

Ledman, R. E. (2001). The family impact expatriate assignment: An experimental exercise.Journal of Management Education, 25(3), 341-351.

Licari, A., & Ovedovitz, A. C. (2002, June). A comparison between European and American col-lege students in their concept acquisition of management principles using a business gamesimulation technique. In N. Delener & C. Chao (Eds.), Global business and technology asso-ciation proceedings: Beyond boundaries (pp. 746-752). Rome: Italy.

Longenecker, C., & Fink, L. (1999). Creating effective performance appraisals. Industrial Man-agement, 41(5), 18-25.

McDonald, D., & Smith, A. (1995). A proven connection: Performance management and busi-ness results. Compensation and Benefits Review, 27(1), 59-62.

McDonald, K. S., & Mansour-Cole, D. (2000). Change requires intensive care: An experientialexercise for learners in university and corporate settings. Journal of Management Education,249(1), 127-148.

McKone, K., & Bozewicz, J. (2003). The ISM simulation: Teaching integrated management con-cepts. Journal of Management Education, 27(4), 497-516.

Miller, J. A. (1991). Experiencing management: A comprehensive “hands-on” model for intro-ductory undergraduate management courses. Journal of Management Education, 15(2), 151-169.

Milliman, J. F., Zawacki, R. A., Schults, B., Wiggins, S., & Norman, C. A. (1995). Customer ser-vice drives 360-degree goal setting. Personnel Journal, 74(6), 136-142.

Muir, C. (2001). Responding to public issues: An exercise in managerial communication. Jour-nal of Management Education, 25(5), 572 - 578.

Norton, D. P., & Kaplan, R. S. (2000). Strategy-focused organization: How balanced scorecardcompanies thrive in the new business environment. Boston: Harvard Business SchoolPublishing.

Reilly, R. R., Smither, J. W., & Vasilpopoulos, N. L. (1996). Longitudinal study of upward feed-back. Personnel Psychology, 49, 599-612.

Rummler, G. A., & Brache, A. P. (1990). Improving performance. San Francisco: Jossey-Bass.Scheiner, C. E., Shaw, D. O., & Beatty, R. W. (1991). Performance measurement and manage-

ment: A tool for strategy execution. Human Resource Management, 30(3), 279-301.Trefry, M. G. (2002). The carrier truck: An exercise exploring organizations as open systems.

Journal of Management Education, 26(4), 462 - 481.




Tornow, W. W., & London, M. (Eds.). (1998). Maximizing the value of 360 degree feedback. SanFrancisco: Jossey-Bass.

Waldman, D. K., & Kenett, R. S. (1990). Improve performance by appraisal. HRMagazine,35(7), 66-69.

Wellins, R. S., & Schultz Murphy, J. (1995). Reengineering: Plug into the human factor. Trainingand Development, 49(1), 33-37.

Wenzler, I., & Charlier, D. (1999). Why do we bother with games and simulation: An organiza-tional learning perspective, Simulation & Gaming, 30(3), 375-384.




Journal of Management Education - SAVIE · 10.1177/1052562905280099JOURNAL OF MANAGEMENT...

Documents

Transcript of Journal of Management Education - SAVIE · 10.1177/1052562905280099JOURNAL OF MANAGEMENT...