1 The Power of Peer Evaluations: Why Don’t They Get More Respect? Allen I. Kraut Professor...
-
Upload
steven-mckenzie -
Category
Documents
-
view
215 -
download
3
Transcript of 1 The Power of Peer Evaluations: Why Don’t They Get More Respect? Allen I. Kraut Professor...
1
The Power of Peer Evaluations: Why Don’t They Get More Respect?
Allen I. Kraut Professor Emeritus of Management Baruch College, CUNY
Presented at METRONew York, NYMay 8, 2013
2
3
4
Example of Peer Assessment In Industry
Reginald Jones, GE CEO in 1978, asked key executives:“Suppose you and I were killed in a plane crash this week. Who should be the next chairman of GE?”
Answers: “Jack Welch”… “Jack Welch “… “Jack Welch”
Recounted in J. Sonnenfeld’s The Hero’s Farewell (1991) from report in R. Vancil’s Passing the Baton (1987) .
5
Non-use of Peer Appraisal SystemsIs “ A Curious Paradox”*
“Long been known … Peer assessments… are among the most accurate assessments….predicting future performance….
Yet only a handful of organizations are using them.”
* From Latham &Associates (1993)
6
Three Recent Talent Management Books,With Not One Index Entry on Peer Evaluation
7
Research* Shows Peer Evaluations Useful in Predicting Success
Have High Reliabilities (in .80s) Even after short acquaintanceship Consistent across groups
Good Validities (.30s & 40s) For Training, Promotion, & Job Performance Unaffected by Friendship Patterns Same whether Research or Administrative Use Better than Staff/Observer ratings
*See Kane & Lawler, (1978), Downey, R. G., & Duffy, P. J. (1978).
8
“Modest” Correlations Have Powerful Practical EffectsExample for r=0.30 *
Predictor Outcome
Low High
High
Low
• From R. Rosenthal & R. L. Rosnow, “Essentials of Behavioral Research” (1984), on Binomial Effect – Size Display, pp.209-211
65%65%35%
35%65%
9
Peer Evaluations Have Higher Predictive Validity Than Most Other Measures
For Overall Job Performance
Top Measures (Partial list)* Validity (r)
General Mental Ability .51 Peer Ratings .49 Interview (Unstructured) .38 Assessment Center .37
* From Meta-Analytic study by F. L. Schmidt & J. E. Hunter (1998), Psych. Bulletin, on 19 predictors
10
Peer Evaluations: Three Types
Peer Nominations: Name Top People
(and Sometimes the Bottom Ones)
Peer Ratings: All Judged on a Common Scale
Peer Rankings: Listed from Highest to Lowest
11
Study #1
A Powerful and Simple Way to Predict Executive Success: Results From a 25-Year Study of Peer Evaluations
12
Evaluation in Month-Long Training Program, At Integrated Fortune 100 Corporation
Middle Managers: (“Minors, Triple AAA”)
N = 184, 2nd & 3rd Level Managers Class size = 16
Executives: (“Major League”)
N = 99, higher level (titles like Director, V-P)
Class size = 12 From All Divisions and Functions Attended Program 1967 through 1968 Peer Ratings Done in Third Week
13
Form of Evaluation
Rating (Forced)
Mean= 2.93HI LO SD=
0.77
1 2 3 4 5
14
Participants’ Reactions to Peer Evaluation
Two Major Complaints:
1. Great deal of work to rate all course-mates on many attributes (11, plus 2 “Overall”)
2. Resent giving low ratings to fellow attendees
15
Two Overall Predictors
General Impression“Considering all of the (specific) factors, how would you rate this person?”
Capacity for Advancement“Indicate your judgment of the potential of this person for the position of general manager of a new (company) division or its corporate staff equivalent?”
Implied administrative set, but actually locked up for research
16
Development of Nomination Rating
What if we asked only for top nominations? Smaller number of ratings No “low” evaluations
To develop a “nomination” rating:- Count as “high” any rating of “1” or a “2” - For each person, divide number of
nominations by maximum possible
17
Two Forms of Evaluation
Rating
Mean= 2.93HI LO SD= 0.77
1 2 3 4 5
Nomination
Mean= 0.30SD=0.26
HI 1 0 LO 1 2 3 4 5
18
Peer Ratings and Peer Nominations Are Highly Correlated
Rating Vs.
Nomination
For Total Sample, N = 283
General Impression 0.92
Capacity for Advancement 0.93
19
Study # 1
Used Two Criteria of Success
After 25+ Years Highest Career Level Achieved? Became a Corporate Officer? (Less than One per 5,000 employees)
20
Peer Evaluations PredictHighest Career Levels Achieved
Correlation with Highest Level
Middle Managers Executives
Attribute Rating Nomination Rating Nomination
General Impression .43* .38* .29* .26*
Capacity for Advancement .42* .38* .26* .21
p < .05; Note: evaluations were not related to year of leaving company.
21
A Criterion of Really Long-Term Executive Success
Only 5.7% Became Corporate Officers (16 of 283)
These are the “Major Leaguers” Who Made it to The “Hall of Fame”
22
“High” Nominees More LikelyTo Become Corporate Officers
Correlation With
Attribute Rating Nomination
General Impression .22* .23*
Capacity for Advancement .25* .24*
* p < .05, N = 283
Note: Maximum possible r estimated at .49
23
Top Nominations on “Capacity for Advancement”Much More Likely to Become Corporate Officers
0 2 4 6 8 10 12 14 16
Bottom 30%
Middle 40%
Top 30%
Became Corporate OfficerPeer Nomination
PERCENT
Total (n=283) = 5.7%
14.1%
2.7%
1.1%
24
Another Look at Future Corporate Officers
6%
19%
75%
Their Peer Nominations Were
In Top 30%
In Middle 40%
In Bottom 30%
Of Those Who Became Officers (n=16)
25
An Army Study of Long-Term Success, Shows Peer Nominations Valid PredictorsR. J. Gerard, 2002
370 USMA Grads, Tracked 1976-2000 Peer Nominations of Leadership (top ¼; bottom ¼)
Better than Tactical Officer Ratings, Cadet Rank, or GPA
In Predicting Career Success Highest Rank Achieved (r = .33**) Top Service School Attended (r = .30**) Largest unit Commanded (r = .33**)** = <.01
26
Study # 2, Same Industry Population, Shorter Time, Differing Criteria
After Two Years Promotion (in Levels) Performance Appraisal
After Eight years Promotion (in Levels) Performance Appraisal Pay Increase (%)
27
Peer Ratingsa Predict Short-Term Success,But Vary By Criteria, and Population
a almost identical to peer nominations
Criterion Correlation with Criterion Attribute Middle Managers Executives
Levels Moved: M = 1.75, SD = 1.55 M = 1.00, SD = 1.75b
Promotions (2 Yrs.) General Impression .38* .10Capacity for Advancement .41* .12
Performance Appraisal (2 Yrs.)General Impression .12* .43*Capacity for Advancement .14* .37*
Pay % Increase (After 8 Yrs.)General Impression .41* .23*Capacity for Advancement .41* .24*
*p < .05; Note: evaluations were not related to year of leaving company. B Wide due to demotions
28
Peer Ratingsa Prediction of Managers’ Mid-Term Promotions Illusory?
a almost identical to peer nominations
Correlation with Promotions
After 2 yrs. After 8
Yrs. Attribute General Impression .38* .41*
Capacity for Advancement .41* .42*
After 8 Yrs.ControllingFor First 2 yrs.
.12
.17
Conclusion?•Peer Prediction most Valid in Fixed Short-Term
•Career Movement like a Tournament in Short-Term, But a Marathon over Long-Term
*p < .05
29
Overall Conclusions
Again, Peer Evaluations Predict Career Success Peer Nominations as Effective as Peer Ratings
Are Simpler Avoid Devaluing Colleagues
Proof of Effectiveness Varies with Situation Basis of Selection into Group Homogeneity of the Group Criteria Used, Restriction of Range
Peer Evaluations Yield Major Benefits in Near-Term Help Put Light on Promising Long-Term Candidates
A Simple, Easy Method Can Be Quite Powerful!
30
Some Limitations of This Research
Setting in a Large Stable Hi-Tech Firm Long Careers in Same Company All Were Male Some Data NA, Pay Grades Shift to “Bands”
Note: Data are “Old” is Not a Limitation
31
Today’s Organizational Conditions Favor Peer Evaluations For Selection
Greater Emphasis on Teams Group’s Opinion More Important Than in Past Peer Acceptance is Critical
Less Time to Develop Track Records Peers Can Make Relevant Judgments
Real Issue may be “Political,” not Scientific
32
Recommendation: Use Peer Nominations More Often
Include in Training Programs Ask Only For Top Nominations Do Not Confuse with “Feedback For Development”
Use in Formal Assessment Centers Use in 360 Evaluations (not Just Development) In Current Organizational Hierarchies (Gingerly) Include Data in Executive Succession Planning Word Evaluations for Specific Criteria Desired
(“to Head Division X, Start New Product Line, etc.) Respect Evaluation’s Short “Shelf Life” (2 to 4
Years)
33
An Example: Peer Ratings Are Used For Promotion Of State Troopers in Minnesota
Ratings on Seven Scales (on BARS) Also, "I Would Feel Fine Reporting to Him/Her?” Used together with Supervisor Ratings, and
Tests: Analytic Skills, Work Styles; Experience
Reactions? Peer Input seen As Valuable Candidates Comfortable with This
Source: Ron Page
34
35
If Peer Evaluations Are So Good, Why Aren’t They Used More??
Maybe They Are Being Used, and I Don’t Know?
Some Other Possibilities...?
36
Don’t Show Up in Top Journals, Because They Lack “Theory”
“In general, journals publish validity studies only when a case can be made for a contribution to scientific understanding.”
Paul Sackett P. 773, In J. C. Scott and D. H. Reynolds, 2010
37
Squeezed Out by Other TopicsIn Today’s Zeitgeist?
Some Examples:
Work-family balance Emotional Labor Workplace Incivility Employee Engagement Organizational Citizenship Behavior
38
Profitability to Con$ultant$ ??
Proven, but Simple, Powerful Method Vs. More Complex, Larger Efforts
Competency Modeling 360 Feedback Programs Talent Management Programs Et Cetera
39
‘Political’ Concerns Management Giving Up Their Power?
Friendship Biases, “Popularity” Contests?
Get Divisiveness, Peer Competitiveness?
Reaction to Changing Rules of Game?
Individualistic American Culture Hostile to Peer Inputs?
40
Your Thoughts??
…and Thank You!