Post on 11-Mar-2018
Do first impressions last? The impact of initial assessments and subsequent performance on promotion decisions
Dirk Black Tuck School of Business
Dartmouth College dirk.e.black@tuck.dartmouth.edu
Marshall Vance Leventhal School of Accounting
Marshall School of Business University of Southern California
mvance@marshall.usc.edu
March 2016
*We acknowledge helpful comments on an earlier version of this paper from Eric Allen, Shane Dikolli, Ken Merchant, Bill Tayler, and participants at the BYU Accounting Research Symposium. Dirk Black acknowledges the support of the Tuck School of Business at Dartmouth College. Marshall Vance acknowledges the support of the Leventhal School of Accounting at the Marshall School of Business at USC. All errors are our own.
Do first impressions last? The impact of initial assessments and subsequent performance on promotion decisions
ABSTRACT This paper examines supervisors’ use of performance measures to update initial beliefs about employee ability over time. Using archival performance and job assignment data, we examine the weighting for promotion decisions of assessments made before employees begin working, and objective performance data collected over time. We find strong evidence that employers’ initial assessments about employee ability predict workers’ future career attainment. While we find evidence consistent with employers using objective performance measures to update expectations about employee ability and sort employees into appropriate job roles, we also find that, controlling for current and prior objective performance measures, managers continue to rely on initial assessments for promotion decisions up to six years later. Managers’ weighting of objective performance measures in promotion decisions appears justified in our setting based on a strong association between current (and past) performance and future performance. However, after the first year, initial assessments of ability have no predictive value for future performance. Thus, it appears managers in our setting over rely on initial impressions, long after they are useful for sorting employees on ability. We also document that supervisors differentially weight objective performance measures based on whether realized performance is consistent with or contrary to initial assessments of ability. Prior research has documented career “fast tracks,” whereby firms identify high-quality employees early and promote these workers quickly. Our findings suggest that workers may be placed on fast tracks even before they start working and are promoted more quickly, at least in part, due to biased performance evaluations and not on merit alone. JEL Classifications: G30; M41; M51; M54 Keywords: Promotions; performance evaluation; employer learning; dynamic; ability
1
1. Introduction
Promotion and job assignment policies are fundamental control mechanisms to ensure
employees are able to perform the necessary functions to help organizations succeed (Merchant
and Van der Stede 2012). While promotion schemes can be used to provide effort incentives
(Prendergast 1999), recent literature suggests that the primary importance of promotions is to
sort employees to the job/level that will result in the highest return on an employee’s ability and
skills (Gibbons and Waldman 1999; Grabner and Moers 2013). The process of matching
employees through promotions to jobs for which they are best suited occurs over time as firms
learn about employees’ ability and human capital acquisition (Baker et al. 1988). Employers
form initial expectations about workers’ ability based on the limited information available when
workers enter the labor market. Initial expectations about ability may be formed based on
education, interviews, test scores, etc. Over time, employers update these expectations as
workers gain experience in the labor market through observing successive on-the-job
performance, and promote employees accordingly (e.g., Murphy 1986; Gibbons and Waldman
1999). In this paper we examine two primary questions: First, how do managers weight initial
assessments of employee ability versus subsequent objective performance when making
promotion decisions? Second, how do these relative weights change over time as firms
accumulate information about employee ability?
While the theoretical literature has long recognized that the performance evaluation
underpinning employer learning and employee promotion is fundamentally a dynamic process
(Holmstrom 1999), there is little empirical research to show how managers use performance
measures over time to update expectations about employee ability. Prior research on employer
learning generally assumes beliefs about worker ability are revised through a Bayesian updating
2
process, which implies that managers rationally incorporate new information and weight it
against prior signals based on the relative informativeness of each (Farber and Gibbons 1996;
Altonji and Pierret 2001; Lange 2007; Grabner and Moers 2013; Kahn and Lange 2014). In
contrast, research in judgement and decision making documents cognitive limitations and biases
that influence the way individuals update beliefs to reflect new information (Bonner 2008). To
our knowledge, prior research does not examine the impact of these behavioral factors on firms’
ability to learn about employee quality or their effect on promotion decisions.
Several prior studies examine supervisors’ use of discretion for rewarding employees,
and document evidence consistent with cognitive limitations or bias influencing subjective
evaluations (Bailey et al. 2011; Bol 2011; Bol and Smith 2011; Woods 2012; Anderson et al.
2014). Our study differs from prior research in this area in important ways. Prior research
examines discretionary performance evaluation in the context of bonuses, while we study
promotion decisions. For bonuses, the purpose of evaluation is to infer effort, whereas for
promotions the purpose is to infer ability (Moers 2005; Gibbs 2008). This suggests a short-term
focus for bonus decisions, but a long-term focus, and perhaps a dynamic element for promotion
decisions. Thus, whereas previous studies focus on the static effect of a performance measure on
subjective evaluation on an unrelated task, we investigate how objective performance measures
are used to update beliefs about worker ability as a whole, and how their use changes as firms
accumulate experience with workers.1
1Anderson et al. (2014) also examine how supervisors’ use of performance measures changes over time, but their focus is on supervisors’ experience with the rating task over a relatively short window (five quarters), while we focus on supervisors’ experience with the worker over a relatively long time period (up to six years). Anderson et al. (2014) explicitly assume limited if any changes in supervisors’ impressions of worker ability over their study period, whereas examining how firms update beliefs about worker ability is a primary focus of our paper.
3
We examine our research questions using extensive performance and promotion data
from Minor League Baseball (MiLB). In doing so, we follow recent research in business and
economics that makes use of sports settings to answer important economic questions (Chapman
and Southwick 1991; Spurr and Barber 1994; Edmans et al. 2007; Berger and Pope 2011; Pope
and Schweitzer 2011; Cadman and Cassar 2013; Allen et al. forthcoming). Our research setting
offers several unique advantages. First, promotions in our setting are relatively frequent, and
follow a well-defined hierarchical path, which increases the power of our tests due to reduced
measurement error in our dependent variable (promotion events). Second, we are able to observe
detailed objective measures of individual employee output over time. Third, for each employee
in our sample, we obtain assessments of worker ability prior to the employee’s entering the labor
market. This is possible because in our setting workers enter the labor market through a formal
draft; workers deemed to have greater ability are selected earlier, and we can observe each
worker’s draft position. This unique setting therefore allows us to map our empirical work
closely to established theoretical models in which employers form initial expectations of
employee ability and update their priors over time based on observed performance (e.g., Gibbons
and Waldman 1999). Fourth, we are able to hold constant the basic nature of the work task for all
employees across each hierarchical level.
We obtain promotion and performance data for over twelve thousand unique employees
(over forty thousand employee-year observations) in approximately thirty different organizations
over the period 1987-2014. We find strong evidence that employers’ initial assessments about
employee ability predict promotion decisions, even after several years on the job. Across every
hierarchical level, unconditional promotion rates for employees with initial assessed ability in the
top quartile are significantly higher than for employees in the bottom quartile, with differences
4
ranging from 25-67%. These differences in promotion rates lead to dramatic differences in career
attainment: Employees initially assessed to be in the top quartile of their cohort are nearly 8.5
times more likely to reach the highest job assignment level within five years following entering
the labor market.
We find that both initial subjective assessments and subsequent objective performance
measures are associated with promotion decisions, but the relative weight on each changes over
time as employers have successive opportunities to observe on-the-job performance. We find
that the weight placed on initial ability assessments is larger than for on-the-job performance
following a worker’s first year on the job by about 12%. In subsequent years, the weight on
objective performance increases, while the weight on the initial assessment decreases, so that
objective performance receives almost twice as much weight in promotion decisions as initial
assessments following a second year of employment, and over 3.7 times as much weight after
five years of performance data have been observed. While the relative weight placed on
objective performance increases over time, initial assessments of ability remain significant
predictors of promotion decisions up to six years later, and persist even after controlling for
current as well as past objective performance. We also find that initial assessments influence
promotion decisions by affecting the sensitivity of promotions to current objective performance
measures. We observe greater weight on performance when it is consistent with expectations
implied by initial assessments; for workers who were initially assessed in the top quartile of their
cohort, above-average performance receives relatively greater weight, and below-average
performance receives relatively lesser weight. For workers initially assessed in the bottom
quartile, this pattern is flipped.
5
That employers still rely on their initial assessments for promotion decisions after several
years is perhaps surprising given that in our setting performance is readily observable by
managers, and worker production is easily captured by the performance measurement system.
We therefore examine whether the relative weights placed on initial assessments and objective
performance are justified by their respective ability to predict future performance. We find that
current performance is significantly associated with future performance in each of a worker’s
first six years on the job. Moreover, controlling for current performance, prior performance (up
to two years prior) is also associated with future performance. Thus, objective performance
measures appear to be useful inputs for promotion decisions in our setting. We also find that
initially assessed ability predicts future performance prior to the first year on the job. However,
once a worker has at least a year of on-the-job experience, we find no association between initial
assessments and future performance, even though initial assessments are related to promotion
decisions in each year.
This paper makes several contributions to the literature. We contribute to the literature on
the use of performance measures for promotion decisions (e.g., Gibbs 1995; Campbell 2008;
Cichello et al. 2009; Grabner and Moers 2013; Chan 2015). We extend these prior studies by
examining the use of performance measures over time, recognizing that employer learning is a
dynamic process. We find that the effect of objective performance on promotions follows an
inverted U-shaped pattern; the weight on objective performance increases over the first few years
on the job and then declines thereafter. This pattern is consistent with employers using observed
performance to learn about worker ability; but after a point, employers’ “make up their minds”
about worker ability, and beliefs are less susceptible to revision. How readily employers revise
their initial beliefs about worker ability after observing performance is critical for the provision
6
of incentives within firms; if employers quickly incorporate performance measures into
expectations about worker ability, which in turn impact job assignment and salary decisions, the
payoffs for effort are relatively high. Conversely, if employers are slow to update their beliefs
and continue to hold onto initial impressions about workers, incentives to work hard are muted.2
We also add to the literature on supervisors’ discretionary use of performance measures
for rewarding employees (Ittner et al. 2003; Gibbs et al. 2004), and particularly the growing
number of studies documenting the impact of managers’ cognitive biases on the performance
evaluation process. All but a few studies in this area examine discretion in determining bonus
payments as opposed to making promotion decisions. However, promotion decisions are a
particularly interesting outcome for studying the weighting of performance measures because
promotion decisions are especially likely to involve discretion (Bol 2008; Campbell 2008), and
the long-term compensation (and therefore, incentive) implications of promotions are large
relative to bonuses (e.g., Baker et al. 1994).3 Thus, empirical examination of how managers
translate performance measures and other information into promotion decisions is particularly
important for understanding explicit and implicit contracting arrangements within firms
(Pendergrast 1999).
Moreover, prior studies on cognitive biases in performance evaluation (and the large
literature on confirmation bias more generally) generally use lab experiments (Bonner 2008). In
contrast, we document evidence consistent with confirmation bias using archival data on
performance and job assignments. Prior research investigates the moderating effect of incentives
2 The speed of employer learning also has important implications for a number of other issues in labor economics, such as the signaling role of education (e.g., Spence 1973) and the returns to investing in human capital (Lange 2007; Kahn and Lange 2014). 3 Bonus calculations commonly rely at least in part on explicit formulas to determine the relation between performance measures and bonuses. To our knowledge, explicit formulas are rarely used for promotion decisions.
7
for rating accuracy as well as the persistence of biases over time. In lab settings, it is difficult to
manipulate these dimensions sufficiently to approximate conditions found in the field.4 In our
setting, promoting the best players through the minor league system and up to the major leagues
is critical to the success of the organizations we study, and extensive attention and resources are
devoted to performance evaluation and job assignment decisions. Moreover, we observe workers
and promotion decisions over multi-year periods. Our use of archival data from a setting
characterized by strong incentives to accurately assess worker ability and multiple years
observing employees makes our findings especially notable relative to the prior literature.
We extend the literature in labor economics on employer learning and careers within
organizations by providing evidence on how readily supervisors “change their minds” about
worker ability after forming an initial opinion. Our findings have implications for the
interpretation of serial correlation in promotion rates (i.e., “fast tracks”) documented in prior
research (Baker et al. 1994). In our setting, fast tracks appear to be established before employees
start working, and some employees may stay on the fast track despite, not because of, their
performance.
2. Background and Hypothesis Development
2.1 Employer learning about worker ability
A fundamental role of job assignments within organizations is to sort employees into
positions for which they are best suited to achieve organizational objectives. That is, skill levels
vary across employees, and jobs vary in the skills required for success. A common perspective
with which to view hierarchies within firms is that jobs at higher levels place greater demands on
4 More generally, there are important tradeoffs and relative advantages and disadvantages for both lab experiments and field studies (Falk and Fehr 2003; Bonner 2008; Charness and Kuhn 2011). We view these alternative research methods as complimentary.
8
employees’ skills, and thus promotions are a means of matching higher-skilled employees to jobs
where their skills yield a greater return to the firm (e.g., Gibbons and Waldman 1999). While
early theoretical work on job assignments focuses on static models (e.g., Rosen 1982; Waldman
1984), more recent literature in labor economics examines firms’ learning about employee ability
as a dynamic process. When an employee first enters the labor market, firms are uncertain about
the worker’s ability, but form initial expectations about ability based on observed signals such as
education, test scores, or interviews. Over time, firms update expectations about worker ability
through observing measures of employee output. In a seminal empirical study, Baker et al.
(1994, p. 903) examine the internal labor market within a large firm, and conclude that “career
dynamics are driven by the firm’s learning about and selecting on ability.”
Several studies provide evidence of employers learning about workers’ ability over time,
although typically in the context of wage distribution. Farber and Gibbons (1996) and Altonji
and Pierret (2001) use responses from the National Longitudinal Survey of Youth to examine the
determinants of wages. They find evidence consistent with hypotheses that firms have limited
information about employee quality at initial stages of careers, and therefore distinguish among
workers on the basis of observable variables that are correlated with productivity (such as
education). Over time, however, wages become increasingly correlated with scores for cognitive
aptitude (which employers are assumed not to observe), suggesting that firms learn about
employee quality as workers gain experience in the labor market. Performance measurement
systems can facilitate the process of employer learning. For example, prior research in
accounting finds that promotion likelihood is positively associated with observable measures of
performance (Campbell 2008; Cichello et al. 2009; Grabner and Moers 2013). In general,
however, observed performance measures reflect both worker ability and noise, so firms are
9
expected to update beliefs about employee ability imperfectly. Moreover, worker ability is
expected to change over time, due for example to human capital acquisition (Becker 1965). In a
recent study on employer learning and its implications for wage dynamics, Kahn and Lange
(2014, p. 1577) conclude that employers learn about worker ability over time, but they “make
substantial errors in wage setting” as they try to hit a moving target of worker ability.
The prevailing assumption in much of the prior literature is that managers incorporate
available information in a manner consistent with Bayes Rule. Under a Bayesian updating
framework, employers form a prior distribution (or, informally, a “prior”) for worker ability
based on whatever signals are initially available when the worker enters the labor market. Each
subsequently observed performance measure provides a basis for employers to update beliefs
about the distribution of worker quality. Both the initial information and subsequent performance
measures are noisy signals for worker ability; how quickly employers update their beliefs
depends on the relative informativeness of each.That is, after n performance periods, a firm’s revised
beliefs about worker ability are formed based on a weighted average of the prior distribution and n
performance assessments, with weights determined based on informativeness.
Consistent with Bayesian updating, we expect promotion likelihood to be positively
associated with current objective performance measures to the extent these measures are
informative about future ability. Similarly, we also expect managers to weight past objective
measures in promotion decisions. When the primary purpose of promotions is to sort employees
by their level of ability, previous performance may be useful because expanding the performance
window is likely to better reflect employee ability than using only a single year (Gibbs 2008).
However, to the extent workers’ effective ability evolves over time, prior measures become
outdated. We therefore expect current performance to be more informative about effective
ability, and thus more heavily weighted, compared to each prior measure individually. However,
10
the collective information contained in prior measures may outweigh current performance and
receive weight in promotion decisions accordingly.
We also expect firms’ initial assessments about employees’ ability to be positively associated
with promotion decisions, at least in the early years of workers’ careers. However, on-the-job
performance is likely to be more informative about worker ability than initial assessments for at
least two reasons. First, as workers’ effective ability develops (or in some cases, declines) over
time, initial assessments become outdated and therefore less informative about future ability.
Second, initial assessments are based on performance environments further removed from the
job assignment for which the worker is being considered.5 Under a Bayesian framework, in
settings where performance measures are informative about worker ability, employers can be
expected to quickly revise noisy and outdated first impressions, so that after relatively few
periods, promotion decisions are based on realized on-the-job performance and not initial
assessments. While it is not clear ex ante how long initial assessments will remain informative
about worker ability, we expect each successive measure of on-the-job performance will be
strictly more informative than an initial assessment, and therefore will receive greater weight in
promotion decisions.
The strength of the signals upon which initial assessments are made is likely to differ
depending on the channel through which workers enter the labor market. Assessments for
workers who attended college are likely to be more informative because less projection is
required for older players, and the performance environment and skill required for college play is
more similar to the professional level than is high school. As well, having already attended high
5 In our setting, initial assessments are primarily based on observing performance on a similar task but against much weaker competition and less frequently. It is difficult to assess a player’s ability when they are playing against competition that does not have professional-level potential. Also, amateur playing seasons are much shorter (fewer games), and it is difficult to project how a given player will respond to the physical demands of a longer season.
11
school, college players are likely to have been observed by potential employers for longer.
Therefore, we expect initial assessments for former college players to receive greater weight in
promotion decisions than for their high-school counterparts.
In contrast to the Bayesian updating perspective, psychological research suggests people
may weight signals based on the order in which they are received, and not just on the properties
of the signals per se. A number of studies provide evidence for “confirmation bias,” a cognitive
bias which occurs when individuals anchor on the information they receive first, so that
subsequent information gathering and processing are biased to support initially formed beliefs.6
In the context of Bayesian updating, such anchoring suggests that initial signals may lead to
irrationally strong prior beliefs (i.e., overestimating the informativeness of initial signals),
causing managers to revise expectations of employee ability too slowly after observing on-the-
job performance. In contrast to rational Bayesian updating, if managers are susceptible to a
“primacy effect”, the importance of initial assessments for promotion decisions will decline
slowly if at all. Moreover, managers may place increased weight on past measures of
performance relative to current measures.
Confirmation bias also manifests when individuals differentially weight signals that are
supportive of a previously held belief compared to signals that go against those beliefs (e.g.,
Bamber et al. 1997). This suggests that managers may place different weight on observed
performance measures depending on how these measures compare to managers’ initial
assessments about players’ ability.
In summary, economic theory predicts that if the purpose of promotions is to sort
employees by expected ability, the importance (or weighting) of different signals for promotion
6 See Nickerson (1998) and Rabin and Schrag (1999) for reviews of laboratory evidence of confirmation bias.
12
decisions should reflect the relative informativeness of the signals about worker ability. While
the relative informativeness of initial assessments versus on-the-job performance measures may
change over time as employee ability develops, after controlling for informativeness, the relative
order in which the signals are received should not influence their use in promotion decisions (or
their use in updating beliefs about worker ability more generally).
In contrast, psychology theory predicts that temporal order may matter for how signals
are used for belief updating. If supervisors anchor on initial assessments of worker quality, over
time promotion decisions may continue to reflect greater weight on initial assessments than is
warranted based on informativeness, and the weighting of on-the-job performance measures may
depend on their perceived consistency with the initial assessment. In our empirical tests
described below, we examine the extent to which the implied weighting for promotion decisions
of initial assessments and on-the-job performance reflect their respective informativeness as
predicted by economic theory, or is consistent with order effects as suggested by the psychology
literature.
3. Research setting
To examine our research questions, we obtain performance and job assignment data for
professional baseball players employed in Minor League Baseball (MiLB), which is a
hierarchical system of developmental leagues below Major League Baseball (MLB). We
acknowledge that professional baseball is a departure from “typical” business environments of
more general interest to researchers. We also recognize that professional athletes may exhibit
personality and behavioral tendencies that differ from a more general population of workers.
However, we emphasize that we examine promotion decisions concerning professional athletes,
13
not the athletes’ performance or behavior per se. Thus, while the specific job task, and the
workers who perform it, may be unique to our setting, we expect the task of evaluating
performance and making job assignments accordingly to be much more general.
MiLB teams are typically independently owned, but each is affiliated with an MLB
franchise through a formal agreement called a Player Development Contract. This agreement
specifies that personnel decisions are made by the MLB franchise (i.e., “parent”). Thus, while
MiLB teams operate as businesses in their own right, from a personnel perspective the primary
purpose of these teams and leagues is to develop players and prepare them to play at the Major
League level.
Within MiLB there are seven levels, ranging from “Rookie” to “Triple-A,” with the
primary distinction across levels being the ability of the players assigned to teams in each level.
When players enter professional baseball, they are generally assigned to a team at the lower
levels of MiLB, but can work towards the major league level through a succession of
promotions. While the possibility of promotion provides very strong effort incentives for MiLB
players, the primary purpose of promotions from the parent organization’s perspective is a means
of developing players’ ability by matching them to job assignments for which they are best
suited.
Workers enter the labor market through a formal draft process, which occurs in June of
each year.7 The draft is organized into rounds, and within each round each team may select one
player. The draft continues for 40 rounds. Each MLB organization employs a scouting
department to evaluate potential draftees for ability and potential to succeed in professional
baseball. Thus, the draft order provides a ranking of expected ability (or potential future ability) 7 The draft process applies only to U.S. residents. International players may negotiate and sign a contract with any MLB organization.
14
based on assessments made prior to a player’s entering the labor market. Once a team selects a
player in the draft, the team obtains exclusive rights to sign the player. Before a contract is
signed, the player and team may negotiate a signing bonus.8 The employment contract itself is
collectively bargained, and stipulates that the parent organization retains exclusive rights to the
player for seven years (i.e., workers are not free to move from one organization to another) at
salaries determined by the level of job assignment. 9
We focus our analysis on MiLB pitchers, whose job entails preventing the opposing
hitters from scoring points, or “runs.” Performance evaluation in MiLB is ubiquitous. In addition
to direct observation by coaches and supervisors, nearly every conceivable outcome is recorded
(e.g., runs, hits, walks, strikeouts, etc.). While objective performance measures are readily
available, these measures capture effective ability with noise in that recorded outcomes do not
perfectly reflect ability. For example, a well-pitched ball may still be struck hard or away from a
fielder resulting in scoring, or a poorly pitched ball may be swung at and missed.
4. Sample Selection and Variable Measurement
We obtain draft position, performance, and job assignment data for all affiliated minor
league pitchers who began their careers between 1987 and 2013.10,11 A primary purpose of our
study is to investigate the role of initial assessments and on-the-job performance, and therefore
we focus our analyses on players in their early careers; we limit our sample to players who have
8 If the player chooses not to sign a contract with the team that selected him, he must wait until the following year’s draft to be selected by a different team before entering professional baseball. 9 At the lowest MiLB level, players are paid $1,100/month (for 5 months of the year); at the highest level, the salary is $2150/month. By comparison, the minimum annual salary for MLB players is approximately $500,000/year (source: http://www.providencejournal.com/article/20150221/Sports/150229787 accessed 3/15/2016). 10 We obtained our data through a licensing agreement with the Society for American Baseball Research (SABR). 11 We begin our sample in 1987 because this is the first year of the draft in its current basic format. Previously drafts were held twice each year, and each draft consisted of multiple phases, making it more difficult to compare draft position, or proxy for initial assessments of ability, across years. The last sample year is 2013 because we require future promotion and performance data from 2014, the last year of available data from SABR.
15
six years of experience in professional baseball or fewer.12 We analyze promotions for pitchers
only in our tests because the job roles and promotion opportunities for pitchers are relatively
homogeneous compared to players in other positions, allowing for more standardized evaluation
criteria.13 We remove players for whom we cannot observe draft information. This exclusion,
which removes observations for undrafted players (predominantly internationally-signed
players), results in a sample of 41,043 observations for 12,444 players.
Our measure of promotion is based on a comparison of an employee’s primary job level
in a given year with his primary job level in the subsequent year. Specifically, Promotion is an
indicator variable equal to one if a player’s primary job level is higher in year t+1 than for year t.
If a player works at multiple job levels in a given year, we designate the level at which the player
played most, based on innings pitched, as the primary job level.
We measure Performance as the average number of runs attributable to the pitcher per
nine innings played (i.e., the standard length of a baseball game), for the primary job level each
year. This measure, commonly referred to as “Earned Run Average” (ERA) is a widely-used
measure of pitching performance, and summarizes how effectively a pitcher performed his
primary job task, which is to prevent the opposing team from scoring runs. To facilitate
interpretation of our results, we multiply ERA by -1 so that higher levels of our measure reflect
better performance, and we standardize the resulting measure. PriorPerformance is the
standardized negative of a player’s cumulative ERA for all years prior to the current year.
12 In our data, on average players reach their highest job assignment level after about 2 years of experience, and the 75th percentile reach their highest job level after 3 years of experience. 13 For example, while the job task of pitchers can be easily summarized as preventing runs, other positions are evaluated with differing emphasis on offense, defense, and running ability. Moreover, because each team needs only a single player at each non-pitcher position (along with a couple of multi-position backup players), non-pitchers’ promotion opportunities are more likely to be impacted by the performance of other players at their same position at higher job levels.
16
While we base our measure of performance on ERA because of its wide usage and
straightforward interpretation relative to a team’s objective for winning games, in general pitcher
performance can be measured in a variety of ways, and it is likely teams consider many different
performance measures in making promotion decisions. Our dataset enables us to capture many of
these common measures (e.g., wins, strikeouts, walks, etc.). We focus on a single performance
measure because our objective is to study the relative use of initial assessments and objective
performance compared to a benchmark of their relative informativeness; we also study how the
use of these inputs changes over time. Thus, a single measure of performance is sufficient for our
purpose.14
Our proxy for teams’ InitialAssessment of player ability is the log of draft position. As
discussed above, draft position reflects the order in which players are selected by teams, and thus
lower numbers reflect players taken earlier, and having higher assessed ability. To aid
interpretation and comparability with Performance, we multiply by -1 and standardize this
measure.
We use several control variables we expect are related to job assignment decisions. We
control for Age (measured as the standardized log of a player’s age in years as of the midpoint in
the season) because younger players may be less developed and thus less likely to be prepared
for higher-level job assignments. Similarly, we use an indicator for whether a player was drafted
following playing baseball in college (College), as former college players are expected to be
more developed players, all else equal. We include LeaguePerformance, which is the
standardized negative of the overall ERA for the league in which a player competes. To the
extent teams use relative performance information when making promotion decisions (Chan 14 Because we do not observe all possible performance measures available to teams, we do not attempt to address how the weight on initial assessments compares to the combined weight on all objective performance measures.
17
2015), we expect LeaguePerformance to be negatively associated with promotion likelihood. We
measure Experience as the number of prior years of professional experience (i.e., Experience is
equal to 0 in a player’s first year). While players with low experience are likely to be less
developed, holding job level constant, players with more experience are more likely to have been
passed over previously, and thus have a lower likelihood of being promoted in the future. In tests
for which we partition our sample by years of Experience, we alternatively control for Repeat,
which is an indicator for whether a player’s primary job level in the current year is the same as
for the previous year.
5. Research Design and Results 5.1 Descriptive analysis
Table 1 presents descriptive statistics for the full sample and for each level within the
MiLB hierarchy for the regression variables used in the study.15 Level 7 (AAA) has the fewest
observations, while Level 4 (Low A) has the most. From the descriptive statistics we learn that
promotions are a regular occurrence: the overall probability of promotion across all levels is
51%. The highest probability of promotion exists in Level 3 (Short-season A) at 66%, while the
lowest probability of promotion exists in Level 7 (AAA) at 19%. These findings are roughly
consistent with achieving promotions becoming relatively more difficult as employees advance
within the organization. The average player in our sample is approximately 23 years old
(untabulated), and 76% of the players were drafted out of college.
Table 2 presents correlation statistics (Pearson below, Spearman above) for the regression
variables used in the study. We find a positive pairwise relation between 𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 and
𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖+1. In addition, we find that earlier drafted players (𝐼𝑃𝑃𝑃𝑃𝑃𝐼𝐼𝐼𝐼𝑃𝐼𝐼𝑃𝑃𝑃𝑃𝑖) are more 15 There are seven levels within MiLB’s hierarchy: Rookie, Advanced Rookie, Short-season A, Low A, High A, AA, and AAA.
18
likely to be promoted, but older (𝐼𝐴𝑃𝑖𝑖) and more experienced (𝐸𝐸𝐸𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖) players are less
likely to be promoted. Players drafted out of college are less likely to be promoted (𝐶𝑃𝐼𝐼𝑃𝐴𝑃𝑖).
We examine relations between performance, initially-assessed worker quality, promotion
likelihood, and future performance in greater depth using multivariate regression techniques in
Tables 4-7.
Table 3, Panel A presents employee outcomes by initial assessment quartile. We examine
four potential outcomes following each worker-year observation: workers may be promoted, stay
in their current job level, be demoted, or exit the labor market. We note that our sample is
predominantly composed of relatively highly assessed workers, due to the fact that lower
assessed workers are less likely to sign a contract and enter professional baseball and also have a
higher exit probability. Workers with the highest initial assessments relative to their cohort (i.e.,
players in the highest (earliest) initial assessment quartile—Q4), have the highest promotion
likelihood in the full sample, and in each level within the organizational hierarchy. For the full
sample, the promotion likelihood for workers assessed in Q4 compared to workers assessed in
Q1 is greater by approximately 25% ((.557/.444)-1). This pattern exists for each hierarchical
level. For the full sample, the likelihood of either repeating a level or being demoted is
increasing in initial assessment, while the likelihood of exiting the labor market is decreasing in
initial assessment, perhaps due to lower-assessed players, viewing their potential for future
promotion to be lower, opting to exit the labor market rather than to repeat a level or to be
demoted.
Table 3, Panel B presents the average percentage of workers eventually reaching each
organizational level. For workers who skip a level, we consider them to have reached the level
they passed over. Within each column, the percentage of workers to eventually reach a given
19
level is strictly increasing by quartile of initially assessed ability. Moreover, the gap in career
attainment from Q1 to Q4 is monotonically increasing at higher levels in the hierarchy. While all
workers who enter the labor market reach at least level 1, the gap from Q1 to Q4 in reaching at
least level 2 is nearly 11% ((97.08/87.47)-1). That gap increases monotonically with each level;
top-quartile-assessed players are 5.6 times (37.20/6.64) more likely to reach level 7, the highest
level of MiLB, and 8.43 times (15.59/1.85) more likely to reach MLB. Together, Panels A and B
of Table 3 provide striking descriptive evidence for initial assessments, made prior to workers
entering the labor market, having a dramatic impact on career attainment.
5.2 Promotion weight on initial assessments vs. on-the-job performance
The preceding analyses provide strong descriptive evidence for the importance of initial
assessments in determining worker career attainment in our setting. This association may result
because initial assessments are informative about ability throughout players’ MiLB careers.
However, even in this scenario supervisors need not directly rely on initial assessments for
promotion decisions; if initial assessments predict performance, and supervisors base promotions
on performance, we will similarly observe a relation between initial assessments and promotions.
Alternatively, supervisors may anchor on initial assessments, such that these prior beliefs about
ability influence promotion decisions even after initial assessments are no longer informative
about ability. To examine these alternatives, we estimate linear probability models (LPM) of
promotion with panel ordinary least squares regressions. Using an LPM allows us to include year
and organizational hierarchy level (Level) fixed effects and avoid the proliferation of parameters
problem for probit and logistic models with fixed effects identified in prior work (Greene 2004),
20
and facilitates interpretation of interaction terms included in some of our models.16 Our primary
model is the following:
𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖+1 = 𝛽0 + 𝛽1𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 + 𝛽2𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 + 𝛽3𝐼𝑃𝑃𝑃𝑃𝑃𝐼𝐼𝐼𝐼𝑃𝐼𝐼𝑃𝑃𝑃𝑃𝑖
+ 𝛽4𝐿𝑃𝑃𝐴𝐿𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 + 𝛽5𝐸𝐸𝐸𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 + 𝛽6𝐼𝐴𝑃𝑖𝑖 + 𝛽7𝐶𝑃𝐼𝐼𝑃𝐴𝑃𝑖 + � 𝛽𝑗𝑗
𝑌𝑃𝑃𝑃𝑖
+ � 𝛽𝑘𝑘
𝐿𝑃𝐿𝑃𝐼𝑖 + 𝜀𝑖𝑖+1 (1)
where the variables are defined as above and in the Appendix.
Table 4 Panel A presents results for the weighting of objective performance and initial
assessments of worker quality for promotion decisions using our overall sample. Columns 1 and
2 examine the weight for Performance and InitialAssessment separately to provide a baseline for
their respective importance for promotion decisions. We find that a one-standard-deviation
increase in Performance is associated with an increase in promotion probability of roughly 14.1
percent, while a one-standard-deviation increase in Initial Assessment is associated with an
increase of roughly 8.0 percent. Both estimates are significant at the 1% level. Column 3
includes both Performance and InitialAssessment together, along with PriorPerformance and
control variables. We find that in this combined model, the effects of Performance and
InitialAssessment remain mostly unchanged. Also, our results suggest supervisors weight past as
well as current performance in promotion decisions, as predicted. However, the weight on prior
performance is much smaller than for current performance (.0205 vs. .1496, both statistically
significant), suggesting that supervisors perceive current performance to be much more
16 A common reason cited for using probit or logistic models rather than LPM is that the latter can result in predicted probabilities outside the interval [0,1]. Our primary LPM model results in predictions outside this interval for approximately 670 of our 29,716 regression observations (2.25%). More to the point, however, since our focus is on estimating the partial effect of initial assessments and subsequent performance on promotion probability, and NOT generating predicted probabilities, that some predicted values are outside of the [0,1] interval is of minimal importance (Wooldridge 2002). In unreported robustness tests, we repeat our main analyses in Table 4 using probit and logistic models and our inferences are similar except for an insignificant positive coefficient on 𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 ∗ 𝐸𝐸𝐸𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 in the probit and logistic specifications for Table 4a, Column 5.
21
informative than prior performance. For control variables, our results provide evidence that
managers use relative performance evaluation, as holding Performance constant, higher
LeaguePerformance is associated with a lower promotion probability. Experience is negatively
associated with promotion likelihood, though not significantly so, indicating that holding Level
fixed, promotion probability declines over time. We find that older players are less likely to be
promoted, all else equal, while College is associated with increased promotion likelihood.
In Column 4 of Panel A, we examine whether the importance of initial assessments is
greater for workers with college experience. Consistent with supervisors perceiving that for
college players the initial assessment is relatively more informative about worker quality, we
find a significant positive interaction between College and InitialAssessment.
In Column 5, we investigate how the weights for Performance, PriorPerformance, and
InitialAssessment vary over time by introducing interactions between these variables and
Experience. We find that the weight on Performance and InitialAssessment declines over time,
while the weight on PriorPerformance increases. This pattern is consistent with on-the-job
performance measures being useful for supervisors to learn about worker quality, and thus
supervisors shift weight from initial assessments to observed performance over time. That the
weight on Performance decreases and PriorPerformance increases is expected from a Bayesian
updating perspective if current performance is roughly equally informative about worker quality
each year, but supervisors have an increasing amount of informative prior performance signals to
include in promotion decisions as they gain experience with workers.
We next examine whether the weights observed in the preceding discussion are consistent
with economic theories that suggest that when making promotion decisions, supervisors weight
signals based on informativeness about worker ability. To proxy for informativeness, we
22
estimate similar models as in Table 4 Panel A, but we estimate future (next year’s) Performance
rather than promotion probabilities in Table 4 Panel B. This approach is similar to that used by
Ittner et al. (2003) in assessing whether supervisors’ weights of objective vs. subjective measures
are driven by informativeness for bonus decisions. For this analysis, we make a (perhaps strong)
assumption that supervisors should promote employees based on expected near-term future
performance. For these regressions, we additionally include fixed effects for future Level because
we expect future performance to be systematically related to the level at which an employee
performs. Overall, the idea behind the regressions in Table 4 Panel B is to provide a benchmark
for the general reasonableness of the relative promotion weights on Performance and
InitialAssessment that we observe in Table 4 Panel A, with the caveat that the models employed
in across these two panels employ different dependent variables.
In Panel B, we see that across each model the coefficient on Performance is roughly
similar to that in the respective models in Panel A, which is consistent with supervisors
weighting current objective performance measures according to their informativeness about
ability. For PriorPerformance, the coefficients are much larger than what we observe in the
promotion models, suggesting that supervisors under-weight prior performance in promotion
decisions relative to its ability to signal worker quality. Strikingly, for InitialAssessment, we
observe a negative association with future performance, in contrast to the relatively large positive
weight placed on this measure by supervisors in promotion decisions. A potential explanation for
this result is that supervisors over-rely on initial assessments so that all else equal, higher
assessed workers are promoted more than their current skillsets warrant; thus, these workers
underperform in the future. This large estimated difference in coefficients for the promotion
23
model vs. the future performance model is supportive of supervisors anchoring on initial
assessments, leading to biased promotion decisions.
In Table 5 Panel A, we test for the possibility that supervisors anchor on initial
assessments, which results in them giving differing importance to observed performance
depending on whether performance is consistent with prior beliefs. Specifically, we replace
Performance and InitialAssessment with separate measures of Performance for workers in each
quartile (by cohort) of initial assessment. We also include an indicator PerformanceGood, for
whether performance is greater than the average for peer workers in the same league/year and
interact PerformanceGood with each measure of Performance by quartile of InitialAssessment.
In Column 1, we find a large main effect of PerformanceGood, even after including continuous
measures of performance (PerformanceQ1-PerformanceQ4), suggesting that supervisors use
above-average performance as a heuristic for worker quality above and beyond performance per
se. The coefficients on PerformanceQ1 – PerformanceQ4 represent the weight on observed
below-average performance for each quartile. We find that the weight placed on “bad
performance” is decreasing for workers who were initially assessed as higher quality, suggesting
that supervisors discount bad performance if it runs counter to their expectations for the worker.
Moreover, the interaction terms, which represent the incremental weight placed on good
performance across quartiles, are negative and significant for Q1 and Q2 (below median
InitialAssessment), negative and insignificant for Q3, and significantly positive for Q4. This is
consistent with supervisors discounting good performance for workers for whom they have low
expectations and emphasizing good performance for workers who are expected to be the best. At
the same time, supervisors emphasize bad performance for workers for whom expectations are
low, and appear to discount bad performance for workers expected to be of higher quality. This
24
pattern, depicted in Figure 1, is consistent with supervisors’ anchoring on initial assessments
affecting their ability to use objective performance measures to update beliefs about worker
ability, and provides at least some explanation for the dramatic impact of initial assessments on
worker career outcomes documented in Table 3.
In Column 2, we include InitialAssessment to observe the main effect on promotions
separate from the effect due to differential weighting based on performance consistent with
expectations. While the pattern across quartiles is no longer monotonic (and in general the
weights on “bad performance” across quartiles are similar), a comparison of the difference in
weight on bad and good performance for Q1 vs. Q4 workers reveals the same pattern as in
Column 1. Specifically, for low-expected-quality employees, bad performance is weighted 47%
greater than good performance, while for high-expected-quality employees bad performance
receives roughly 32% less weight than good performance. Moreover, the difference in total
weight on good performance for Q4 vs. Q1 workers is economically large (82% greater) and
significant (p = .017).
In Table 5 Panel B, we perform similar analyses to Table 5 Panel A using future
performance as the dependent variable. As in Table 4 Panel B, Performance is positively related
to future performance, while InitialAssessment is negatively relative to future performance. We
find an inverted U-shaped pattern in the incremental weights on good performance, but find little
evidence of significantly different weighting of good versus bad performance in predicting future
performance. Comparing these results with the results in Table 5 Panel A again suggests the
possibility of managers’ exhibiting bias in promotion decisions relative to variables that predict
employees’ future performance.
5.3 Promotion weights by worker experience
25
In Table 6 Panel A, we repeat our model of promotion decisions after partitioning our
sample based on Experience to examine more directly how weights change over time using
seemingly unrelated regression. Notably, for employees in their first year of experience
(Experience=0), InitialAssessment receives greater weight in promotion decisions than
Performance (the difference is significant at the 5% level). However, the importance of
InitialAssessment declines sharply with Experience, while for Performance the coefficients
follow an inverted-U pattern over time. Thus, it appears supervisors at least partially recognize
the declining usefulness of initial assessments, and adjust their use of it accordingly. However,
across each column, we continue to find a significant association between InitialAssessment and
Promotion.
In Table 6 Panel B, we estimate models of future performance across partitions of
Experience. We find that InitialAssessment is significantly associated with Performance in a
worker’s first year on the job, validating the ability of teams’ scouting departments to identify
superior players and draft accordingly. However, following a worker’s first year on the job, and
continuing through four years of experience, InitialAssessment has no predictive value for future
performance, notwithstanding the positive weights on InitialAssessment in the promotion
models. Following an employee’s fifth year, initial assessments are negatively associated with
future performance. This negative association is again consistent with managers’ using measures
(InitialAssessment) for promotion decisions that seem inconsistent with future performance, and
may indicate bias.
5.4 Promotion weights by organizational level
In the preceding analyses, a possibility is that supervisors give workers with higher initial
assessments more difficult job assignments to stretch them, which may be optimal from a
26
developmental standpoint. While we control for level fixed effects in the above analysis, it may
be the case that the effect of InitialAssessment on promotions and future performance differs
across levels. To examine this possibility, we repeat our regressions after partitioning on Level.
The results, presented in Table 7 Panels A and B, are consistent with those of partitions on
Experience. For promotion decisions (Table 7 Panel A), we find larger coefficients on
InitialAssessment at the lower levels, although the effect persists even at the highest level of
MiLB. The importance of Performance for promotion decisions is generally increasing across
levels, although the coefficient declines sharply at the highest level. For future performance
(Table 7 Panel B), we again find that Performance is a significant predictor of future
performance, while InitialAssessment is generally not. We observe a significant negative
coefficient on InitialAssessment for the highest level of MiLB, consistent with Table 6 Panel B
and with managers’ making promotion decisions using inputs that are at best not predictive, and
at worst negatively predictive, of future employee success.
6. Conclusion
In this paper we bring together streams of research from labor economics on careers in
organizations and employer learning and from accounting on the use of performance measures.
The economics literature documents that firms learn about worker ability, which shapes workers’
career progression, but does not examine how firms learn. The accounting literature documents
that supervisors’ cognitive limitations and biases influence performance evaluation, but does not
examine how these limitations impact workers’ careers. By shedding light on how supervisors
use objective performance measures for promotion decisions, and their over-reliance on initial
impressions long after they are informative about worker ability, we make important
contributions to both literatures.
27
Prior research on employer learning has generally relied on either broad survey data to
answer high-level questions about career progression or has used detailed personnel records from
a single firm to descriptively examine promotion patterns within firms. A central feature of the
theoretical models in this field is that performance measurement is a dynamic process. One
implication from this research is that an important role of performance measurement systems is
to facilitate employer learning about employees’ ability over time. However, to this point the
empirical research in this area has not examined how supervisors weight performance signals
over time, or how these signals are used to update initial assessments of ability. By utilizing a
unique dataset that combines detailed promotion data across hierarchical levels with
corresponding performance measures over time, we are able to provide evidence on this
important topic.
Our study contributes to the judgement and decision making literature by using archival
data to document supervisors’ over-reliance on initial assessments even in the face of
informative objective performance measures. Moreover, we find this effect occurs in part due to
differential weighting of objective performance measures depending on whether or not realized
performance is consistent with initial expectations. The majority of evidence for supervisors’
biased performance evaluations is from experimental research generated in lab settings. In two
recent studies, Woods (2012) and Anderson et al. (2014) use field data to document that
supervisors allow ancillary performance measures to influence ratings on an unrelated
dimension, consistent with confirmation bias. Our study compliments and extends this prior
research. First, whereas Woods (2012) and Anderson et al. (2014) both examine performance
appraisal for bonus computations, our study examines promotion decisions. Also, both Woods
(2012) and Anderson et al. (2014) examine settings in which the ratings task was recently
28
introduced as part of an overhaul to an existing incentive system, and both studies posit that the
unfamiliarity with the task contributes to supervisors tendency to anchor on ancillary measures.
In our setting, the task of evaluating worker ability to determine job assignments is essentially
the same over a 27-year sample period, yet we find evidence of confirmation bias that persists
even after up to six years of on-the-job performance has been observed. Our results suggest that
supervisors are slow to revise beliefs about worker ability even when the evaluation task is
familiar and incentives to promote the best worker are strong. We leave it to future research to
explore alternative means of mitigating supervisors’ confirmation bias in promotion settings.
29
References Allen, E., Dechow, P., Pope, D., Wu, G., 2014. Reference-dependent preferences: Evidence from marathon runners. Management Science, Forthcoming. Altonji, J., Pierret, C., 2001. Employer learning and statistical discrimination. The Quarterly Journal of Economics 116, 313-350. Anderson, S., Dekker, H., Sedatole, K., Wiersma, E., 2014. Field evidence of bias in subjective ratings. Working paper. Bailey, W., Hecht, G., Towry, K., 2011. Dividing the pie: The influence of managerial discretion extent on bonus pool allocation. Contemporary Accounting Research 28, 1562-1584. Baker, G., Gibbs, M., Holmstrom, B., 1994. The internal economics of the firm: evidence from personnel data. The Quarterly Journal of Economics 109, 881-919. Baker, G., Jensen, M., Murphy, K., 1988. Compensation and incentives: practice vs. theory. Journal of Finance 43, 593-616. Bamber, E.M., Ramsay, R., Tubbs, R., 1997. An examination of the descriptive validity of the belief-adjustment model and alterative attitudes to evidence in auditing. Accounting, Organizations and Society 22, 249-268. Becker, G., 1965. A theory of the allocation of time. The Economic Journal 75, 493-517. Berger, J., and Pope, D., 2011. Can losing lead to winning. Management Science 57(5), 817-827. Bol, J., 2008. Subjectivity in compensation contracting. Journal of Accounting Literature 27, 1-24. Bol, J., 2011. The determinants and performance effects of managers performance evaluation biases. The Accounting Review 86, 1549-1575. Bol, J., Smith, S., 2011. Spillover effects in subjective performance evaluation: Bias and the asymmetric influence of controllability. The Accounting Review 86, 1213-1230. Bonner, S.E., 2008. Judgement and decision making in accounting. Upper Saddle River, NJ: Pearson Education, Inc. Cadman, B., Cassar, G., 2013. Explicit and implicit incentives: longitudinal evidence from NCAA football head coaches employment contracts. AAA 2014 Management Accounting Section (MAS) Meeting Paper. Available at SSRN: http://ssrn.com/abstract=2310164 or http://dx.doi.org/10.2139/ssrn.2310164.
30
Campbell, D., 2008. Nonfinancial performance measures and promotion-based incentives. Journal of Accounting Research 46, 297-332. Chan, E., 2015. Promotion, relative performance information, and the Peter Principle. Working paper. Chapman, K.S., and Southwick, L., 1991. Testing the matching hypothesis: The case of Major-League Baseball. The American Economic Review 81, 1352-1360. Charness, G., Kuhn, P., 2011. Chapter 3 – Lab labor: What can labor economists learn from the lab? Handbook of Labor Economics 4a, 229-330. Cichello, M., Fee, C. E., Hadlock, C., Sonti, R., 2009. Promotions, turnover, and performance evaluation: evidence from the careers of division managers. The Accounting Review 84, 1119-1143. Edmans, A., García, D., and Norli, Ø., 2007. Sports sentiment and stock returns. The Journal of Finance 62, 1967-1998. Falk, A., Fehr, E., 2003. Why labour market experiments? Labour Economics 10, 399-406. Farber, H., Gibbons, R., 1996. Learning and wage dynamics. The Quarterly Journal of Economics 111, 1007-1047. Gibbons, R., Waldman, M., 1999. A theory of wage and promotion dynamics inside firms. The Quarterly Journal of Economics 114, 1321-1358. Gibbs, M., 1995. Incentive compensation in a corporate hierarchy. Journal of Accounting and Economics 19, 247-277. Gibbs, M., 2008. Discussion of nonfinancial performance measures and promotion-based incentives. Journal of Accounting Research 46, 333-340. Gibbs, M., Merchant, K., Van der Stede, W., Vargus, M., 2004. Determinants and effects of subjectivity in incentives. The Accounting Review 79, 409-436. Grabner, I., Moers, F., 2013. Managers’ choices of performance measures in promotion decisions: An analysis of alternative job assignments. Journal of Accounting Research 51, 1187-1220. Greene, W., 2004. The behaviour of the maximum likelihood estimator of limited dependent variable models in the presence of fixed effects. Econometrics Journal 7, 98-119. Holmstrom, B., 1999. Managerial incentive problems: a dynamic perspective. Review of Economic Studies 66, 169-182.
31
Ittner, C., Larcker, D., Meyer, M., 2003. Subjectivity and the weighting of performance measures: evidence from a balanced scorecard. The Accounting Review 78, 725-758. Kahn, L., Lange, F., 2014. Employer learning, productivity, and the earnings distribution: evidence from performance measures. Review of Economic Studies 81, 1575-1613. Lange, F., 2007. The speed of employer learning. Journal of Labor Economics 25, 1-35. Merchant, K., Van der Stede, W., 2012. Management control systems: performance measurement, evaluation, and incentives. Prentice Hall, Third Edition, London. Moers, F., 2005. Discretion and bias in performance evaluation: The impact of diversity and subjectivity. Accounting, Organizations and Society 30, 67-80. Murphy, K.J., 1986. Incentives, learning, and compensation: A theoretical and empirical investigation of managerial labor contracts. The Rand Journal of Economics, 59-76. Nickerson, R., 1998. Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology 2, 175-220. Pope, D. and Schweitzer, M., 2011. Is Tiger Woods loss averse? Persistent bias in the face of experience, competition, and high stakes. American Economic Review 101(1), 129-157. Prendergast, C., 1999. The provision of incentives in firms. Journal of Economic Literature 37, 7-63. Rabin, M., Schrag, J., 1999. First impressions matter: A model of confirmatory bias. The Quarterly Journal of Economics 114, 37-82. Rosen, S., 1982. Authority, control, and the distribution of earnings. The Bell Journal of Economics 13, 311-323. Spence, M., 1973. Job market signaling. The Quarterly Journal of Economics 87, 355-374. Spurr, S., Barber, W., 1994. The effect of performance on a worker’s career: Evidence from minor league baseball. Industrial and Labor Relations Review 47, 692-708. Waldman, M., 1984. Job assignments, signaling, and efficiency. The RAND Journal of Economics 15, 255-267. Woods, A., 2012. Subjective adjustments to objective performance measures: The influence of prior performance. Accounting, Organizations and Society 37, 403-425. Wooldridge, J., 2002. Econometric analysis of cross section and panel data. Cambridge, MA: MIT Press.
32
Appendix – Variable Definitions Variable Name Definition Data Source 𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖+1 Indicator variable equal to one if player i’s
primary job level (i.e., the level at which player i accumulates the most playing time in a given year) in year t + 1 is higher than the primary job level in year t.
SABR
𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 Earned-run average (ERA) of player i during year t. ERA is a commonly-used summary performance measure for pitchers. This measure is calculated as the average number of runs attributed to the pitcher per nine innings pitched (i.e., the typical length of a baseball game). We multiply this measure by -1 so that higher values of Performanceit indicate better performance.
SABR
𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 Cumulative earned-run average (ERA) of player i prior to year t. This measure is calculated as the total cumulative earned runs attributed to the pitcher per nine innings pitched (i.e., the typical length of a baseball game). We multiply this measure by -1 so that higher values of Performanceit indicate better performance.
SABR
𝐼𝑃𝑃𝑃𝑃𝑃𝐼𝐼𝐼𝐼𝑃𝐼𝐼𝑃𝑃𝑃𝑃𝑖 The natural logarithm of the overall pick number with which player i is selected in a given year. We multiply this measure by -1 so that higher values of Initial Assessment indicate more favorable initial assessments of player ability.
SABR
𝐿𝑃𝑃𝐴𝐿𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 Average earned-run average (ERA) of the league in which player i plays during year t.
SABR
𝐸𝐸𝐸𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 Prior labor market experience, in years. SABR 𝐼𝐴𝑃𝑖𝑖 The natural logarithm of the age in years of
player i during year t. SABR
𝐶𝑃𝐼𝐼𝑃𝐴𝑃𝑖 Indicator variable equal to one if a player was drafted out of college; zero otherwise.
SABR
33
Figure 1: Differential Weighting of Performance Based on Consistency with Prior Expectations
Note: This figure depicts regression estimates for the weighting of objective performance for promotion decisions, based on Table 5, Panel A Columns 1 (above) and 2 (below). The blue line represents the regression coefficient on Performance that is below the league average, estimated separately by quartiles of initial assessment, with Q1-Q4 increasing in initial assessment. The red line represents the regression coefficients for performance that is above the league average.
-0.05
0
0.05
0.1
0.15
0.2
Q1 Q2 Q3 Q4
Wei
ght o
n Pe
rfor
man
ce
Initial Assessment Quartile
Promotion Weight on Good vs. Bad Performance
Bad
Good
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
Q1 Q2 Q3 Q4
Wei
ght o
n Pe
rfor
man
ce
Initial Assessment Quartile
Promotion Weight on Good vs. Bad Performance
Bad
Good
34
Table 1 – Descriptive Statistics
Notes: This table presents descriptive statistics for the variables used in the empirical tests. Statistics are presented for the full sample and for each hierarchical level within an organization. All variables are defined in the Appendix.
Full SampleVARIABLE N Mean Stdev P25 Median P75Promotion it+1 41,043 0.510 0.500 0.000 1.000 1.000Performance it 41,043 0.000 1.000 - 0.393 0.140 0.598PriorPerformance it 29,716 0.000 1.000 - 0.396 0.071 0.525InitialAssessment i 41,043 - 0.000 1.000 - 0.744 - 0.228 0.506LeaguePerformance it 41,043 0.000 1.000 - 0.626 0.141 0.713Experience t 41,043 1.631 1.530 0.000 1.000 3.000Age it 41,043 0.000 1.000 - 0.608 0.035 0.638College i 41,043 0.761 0.426 1.000 1.000 1.000
Job Level 1 (Rookie)VARIABLE N Mean Stdev P25 Median P75Promotion it+1 4,656 0.614 0.487 0.000 1.000 1.000Performance it 4,656 0.012 1.342 - 0.462 0.279 0.825PriorPerformance it 1,111 - 0.751 1.781 - 1.427 - 0.434 0.287InitialAssessment i 4,656 - 0.237 0.889 - 0.904 - 0.451 0.223LeaguePerformance it 4,656 0.462 1.119 - 0.036 0.807 1.338Experience t 4,656 0.348 0.773 0.000 0.000 0.000Age it 4,656 - 1.191 0.915 - 1.947 - 1.372 - 0.509College i 4,656 0.496 0.500 0.000 0.000 1.000
Job Level 2 (Advanced Rookie)VARIABLE N Mean Stdev P25 Median P75Promotion it+1 4,433 0.621 0.485 0.000 1.000 1.000Performance it 4,433 - 0.232 1.171 - 0.712 - 0.059 0.524PriorPerformance it 1,488 - 0.595 1.675 - 1.364 - 0.387 0.518InitialAssessment i 4,433 - 0.284 0.812 - 0.890 - 0.485 0.138LeaguePerformance it 4,433 - 1.056 1.067 - 1.877 - 1.078 - 0.240Experience t 4,433 0.388 0.652 0.000 0.000 1.000Age it 4,433 - 0.700 0.747 - 1.237 - 0.590 - 0.147College i 4,433 0.720 0.449 0.000 1.000 1.000
Job Level 3 (Short Season A)VARIABLE N Mean Stdev P25 Median P75Promotion it+1 6,268 0.658 0.474 0.000 1.000 1.000Performance it 6,268 0.092 1.020 - 0.342 0.256 0.742PriorPerformance it 2,314 - 0.282 1.390 - 0.904 - 0.145 0.560InitialAssessment i 6,268 - 0.206 0.823 - 0.811 - 0.384 0.220LeaguePerformance it 6,268 0.474 0.704 0.124 0.594 0.892Experience t 6,268 0.464 0.787 0.000 0.000 1.000Age it 6,268 - 0.374 0.581 - 0.718 - 0.335 0.013College i 6,268 0.852 0.355 1.000 1.000 1.000
35
Table 1 – Descriptive Statistics (continued)
Job Level 4 (Low A)VARIABLE N Mean Stdev P25 Median P75Promotion it+1 8,213 0.571 0.495 0.000 1.000 1.000Performance it 8,213 0.098 0.873 - 0.254 0.209 0.626PriorPerformance it 7,675 0.006 1.092 - 0.564 0.046 0.673InitialAssessment i 8,213 - 0.041 0.939 - 0.745 - 0.238 0.456LeaguePerformance it 8,213 0.378 0.579 0.005 0.401 0.764Experience t 8,213 1.358 0.915 1.000 1.000 2.000Age it 8,213 - 0.129 0.680 - 0.537 - 0.039 0.357College i 8,213 0.746 0.435 0.000 1.000 1.000
Job Level 5 (High A)VARIABLE N Mean Stdev P25 Median P75Promotion it+1 8,108 0.446 0.497 0.000 0.000 1.000Performance it 8,108 0.001 0.916 - 0.379 0.131 0.543PriorPerformance it 7,855 0.117 0.741 - 0.302 0.102 0.536InitialAssessment i 8,108 0.080 1.011 - 0.672 - 0.132 0.641LeaguePerformance it 8,108 0.001 0.983 - 0.732 0.139 0.752Experience t 8,108 2.188 1.172 1.000 2.000 3.000Age it 8,108 0.356 0.685 - 0.083 0.395 0.830College i 8,108 0.799 0.401 1.000 1.000 1.000
Job Level 6 (AA)VARIABLE N Mean Stdev P25 Median P75Promotion it+1 6,119 0.367 0.482 0.000 0.000 1.000Performance it 6,119 0.003 0.845 - 0.346 0.121 0.492PriorPerformance it 6,062 0.155 0.544 - 0.191 0.132 0.485InitialAssessment i 6,119 0.262 1.094 - 0.566 0.041 0.875LeaguePerformance it 6,119 - 0.084 0.651 - 0.494 - 0.064 0.303Experience t 6,119 3.160 1.215 2.000 3.000 4.000Age it 6,119 0.840 0.712 0.363 0.868 1.346College i 6,119 0.816 0.388 1.000 1.000 1.000
Job Level 7 (AAA)VARIABLE N Mean Stdev P25 Median P75Promotion it+1 3,246 0.194 0.396 0.000 0.000 0.000Performance it 3,246 - 0.135 0.823 - 0.536 - 0.055 0.362PriorPerformance it 3,211 0.148 0.451 - 0.155 0.129 0.433InitialAssessment i 3,246 0.533 1.249 - 0.415 0.258 1.286LeaguePerformance it 3,246 - 0.936 0.874 - 1.700 - 0.878 - 0.132Experience t 3,246 3.845 1.130 3.000 4.000 5.000Age it 3,246 1.241 0.727 0.771 1.283 1.706College i 3,246 0.863 0.344 1.000 1.000 1.000
36
Table 2 – Correlations Notes: This table presents pairwise correlation statistics for the variables used in the empirical tests. Pearson correlations are presented below the diagonal. Spearman correlations are presented above the diagonal. * indicates statistical significance at the 0.05 level. All variables are defined in the Appendix.
VARIABLE (1) (2) (3) (4) (5) (6) (7) (8)
(1) Promotion it+1 0.3231* 0.0589* 0.0907* 0.0970* -0.2560* -0.2473* -0.0270*
(2) Performance it 0.2863* 0.1673* -0.0244* 0.2405* -0.1019* -0.0262* 0.0615*
(3) PriorPerformance it 0.0464* 0.1638* -0.0913* 0.0526* -0.1243* 0.0513* 0.1397*
(4) InitialAssessment i 0.0878* -0.001 -0.0336* -0.0359* 0.1471* -0.0452* -0.1795*
(5) LeaguePerformance it 0.0883* 0.2019* 0.0517* -0.0299* -0.1721* -0.1937* -0.0632*
(6) Experience t -0.2563* -0.0606* -0.0382* 0.1345* -0.1562* 0.6355* -0.0696*(7) Age it -0.2346* -0.002 0.0955* -0.0487* -0.1680* 0.6393* 0.4653*(8) College i -0.0270* 0.0572* 0.1271* -0.1870* -0.0557* -0.0713* 0.4940*
37
Table 3 – Employee Descriptive Statistics by Initial Assessment Quartile and Job Level
Panel A: Employee Outcomes by Initial Assessment Quartile and Job Level Notes: This table presents the percentage of each employee outcomes by initial assessment quartile and job level. Employee outcomes include promotion, repeating the level, demotion, and exit from professional baseball.
Full SampleInitial Assessment Quartile N Promoted Repeat Demote ExitQ4 16,454 0.557 0.249 0.066 0.128Q3 12,203 0.496 0.225 0.055 0.223Q2 8,402 0.469 0.202 0.044 0.286Q1 3,984 0.444 0.186 0.042 0.327
Job Level 1 (Rookie)Initial Assessment Quartile N Promoted Repeat Demote ExitQ4 1,437 0.711 0.153 0.013 0.123Q3 1,324 0.574 0.182 0.014 0.230Q2 1,139 0.576 0.144 0.008 0.272Q1 756 0.557 0.144 0.003 0.296
Job Level 2 (Advanced Rookie)Initial Assessment Quartile N Promoted Repeat Demote ExitQ4 1,273 0.738 0.123 0.029 0.110Q3 1,345 0.625 0.146 0.025 0.204Q2 1,164 0.560 0.133 0.018 0.289Q1 651 0.490 0.141 0.026 0.343
Job Level 3 (Short Season A)Initial Assessment Quartile N Promoted Repeat Demote ExitQ4 1,982 0.789 0.082 0.024 0.106Q3 2,053 0.668 0.090 0.033 0.209Q2 1,543 0.555 0.111 0.021 0.312Q1 690 0.486 0.117 0.039 0.358
Job Level 4 (Low A)Initial Assessment Quartile N Promoted Repeat Demote ExitQ4 3,210 0.637 0.176 0.054 0.134Q3 2,522 0.557 0.168 0.037 0.238Q2 1,704 0.511 0.157 0.029 0.303Q1 777 0.479 0.140 0.026 0.355
Job Level 5 (High A)Initial Assessment Quartile N Promoted Repeat Demote ExitQ4 3,578 0.516 0.248 0.063 0.173Q3 2,397 0.416 0.250 0.056 0.278Q2 1,482 0.387 0.237 0.057 0.319Q1 651 0.309 0.240 0.069 0.382
Job Level 6 (AA)Initial Assessment Quartile N Promoted Repeat Demote ExitQ4 3,081 0.426 0.350 0.094 0.130Q3 1,731 0.323 0.371 0.101 0.205Q2 972 0.282 0.377 0.105 0.237Q1 335 0.299 0.379 0.107 0.215
Job Level 7 (AAA)Initial Assessment Quartile N Promoted Repeat Demote ExitQ4 1,893 0.228 0.544 0.156 0.072Q3 831 0.149 0.557 0.183 0.111Q2 398 0.138 0.555 0.173 0.133Q1 124 0.169 0.548 0.177 0.105
38
Table 3 – Employee Descriptive Statistics by Initial Assessment Quartile and Job Level (continued)
Panel B: Career Attainment by Initial Assessment Quartile
Notes: This table presents the percentage of employees in each initial assessment quartile who reach at least the job level indicated in each column.
Job Level 1 2 3 4 5 6 7 8Initial Assessment QuartileQ4 100.00 97.08 93.48 88.30 77.30 58.82 37.20 15.59Q3 100.00 93.55 85.96 73.55 55.91 35.16 17.35 4.78Q2 100.00 90.50 79.55 62.55 44.74 25.65 11.38 2.64Q1 100.00 87.47 72.33 55.00 36.16 17.33 6.64 1.85
Average Percentage of Players Reaching Each Level
39
Table 4 – Promotion weight on initial assessments vs. on-the-job performance
Panel A: Promotion Decisions Notes: This table presents the results of estimating variants of equation 1 using ordinary least squares regression. All continuous variables are standardized to facilitate comparison of weights (coefficients) in the promotion decision. All variables are defined in the Appendix. A constant term, as well as year and job level (e.g., A, AA, AAA) fixed effects, are included in each estimation, but omitted for the sake of brevity. Robust t-statistics based on standard errors clustered at the organization level are presented below regression coefficients. ***, **, and * indicate statistical significance at the 0.01, 0.05, and 0.10 levels. 𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖+1 = 𝛽0 + 𝛽1𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 + 𝛽2𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 + 𝛽3𝐼𝑃𝑃𝑃𝑃𝑃𝐼𝐼𝐼𝐼𝑃𝐼𝐼𝑃𝑃𝑃𝑃𝑖 +𝛽4𝐿𝑃𝑃𝐴𝐿𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 + 𝛽5𝐸𝐸𝐸𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 + 𝛽6𝐼𝐴𝑃𝑖𝑖 + 𝛽7𝐶𝑃𝐼𝐼𝑃𝐴𝑃𝑖 + ∑ 𝛽𝑗𝑗 𝑌𝑃𝑃𝑃𝑖 + ∑ 𝛽𝑘𝑘 𝐿𝑃𝐿𝑃𝐼𝑖 + 𝜀𝑖𝑖+1 (1)
Dependent Variable = VARIABLE (1) (2) (3) (4) (5)Performance it 0.1411*** 0.1496*** 0.1496*** 0.1719***
(50.8105) (37.4916) (37.1781) (26.7247)PriorPerformance it 0.0205*** 0.0210*** 0.0067
(6.9594) (7.2122) (1.4121)InitialAssessment i 0.0802*** 0.0631*** 0.0484*** 0.0912***
(23.9493) (20.4121) (10.0349) (17.1234)College i 0.0213**** InitialAssessment i (4.4838)Performance it -0.0101**** Experience it (-3.7040)PriorPerformance it 0.0100**** Experience it (3.8510)InitialAssessment i -0.0114**** Experience it (-5.8670)LeaguePerformance it -0.0108** -0.0109** -0.0112**
(-2.3957) (-2.4038) (-2.5197)Experience t -0.0027 -0.0020 -0.0008
(-0.9204) (-0.6757) (-0.2811)Age it -0.0643*** -0.0653*** -0.0614***
(-11.6313) (-11.8745) (-11.2265)College i 0.0228** 0.0187* 0.0210**
(2.4086) (1.9626) (2.2189)YEAR FE YES YES YES YES YESJOB LEVEL FE YES YES YES YES YESN 41,043 41,043 29,716 29,716 29,716R2 0.1531 0.0990 0.1670 0.1674 0.1691
Promotion it+1
40
Table 4 – Promotion weight on initial assessments vs. on-the-job performance
Panel B: Future Performance
Notes: This table presents the results of estimating variants of equation 1 with future performance as the dependent variable using ordinary least squares regression. All continuous variables are standardized to facilitate comparison of weights (coefficients) in the promotion decision. All variables are defined in the Appendix. A constant term, as well as year, job level (e.g., A, AA, AAA), and next job level fixed effects, are included in each estimation, but omitted for the sake of brevity. Robust t-statistics based on standard errors clustered at the organization level are presented below regression coefficients. ***, **, and * indicate statistical significance at the 0.01, 0.05, and 0.10 levels.
𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖+1 = 𝛽0 + 𝛽1𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 + 𝛽2𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 + 𝛽3𝐼𝑃𝑃𝑃𝑃𝑃𝐼𝐼𝐼𝐼𝑃𝐼𝐼𝑃𝑃𝑃𝑃𝑖 +𝛽4𝐿𝑃𝑃𝐴𝐿𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 + 𝛽5𝐸𝐸𝐸𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 + 𝛽6𝐼𝐴𝑃𝑖𝑖 + 𝛽7𝐶𝑃𝐼𝐼𝑃𝐴𝑃𝑖 + ∑ 𝛽𝑗𝑗 𝑌𝑃𝑃𝑃𝑖 + ∑ 𝛽𝑘𝑘 𝐿𝑃𝐿𝑃𝐼𝑖 + 𝜀𝑖𝑖 (1)
Dependent Variable = VARIABLE (1) (2) (3) (4) (5)Performance it 0.1631*** 0.1559*** 0.1559*** 0.1805***
(13.3958) (10.0577) (10.0402) (7.5263)PriorPerformance it 0.0622*** 0.0622*** 0.0382**
(5.9955) (5.9147) (2.6262)InitialAssessment i -0.0222*** -0.0202** -0.0210 -0.0147
(-3.1872) (-2.6707) (-1.4740) (-1.0967)College i 0.0012* InitialAssessment i (0.0632)Performance it -0.0115* Experience it (-1.4018)PriorPerformance it 0.0191*** Experience it (2.4789)InitialAssessment i -0.0011* Experience it (-0.3210)LeaguePerformance it 0.0131 0.0131 0.0121
(1.6119) (1.6067) (1.4924)Experience t -0.0150** -0.0149** -0.0125*
(-2.1990) (-2.1283) (-1.7856)Age it -0.0287* -0.0288* -0.0277*
(-1.9355) (-1.9907) (-1.8553)College i 0.0232 0.0229 0.0225
(1.2450) (1.1375) (1.1954)YEAR FE YES YES YES YES YESJOB LEVEL FE YES YES YES YES YESNEXT JOB LEVEL FE YES YES YES YES YESN 32,507 32,507 22,952 22,952 22,952
R2 0.0438 0.0270 0.0466 0.0466 0.0470
Performance it+1
41
Table 5 – Promotion weight on initial assessments vs. on-the-job performance
Panel A: Promotion Decisions and Initial Assessment Quartile
Notes: This table presents the results of estimating variants of equation 1 replacing Performance with separate performance variables by initial assessment quartile (PerformanceQ1 – PerformanceQ4) and interactions of these performance variables with an indicator for above-average performance, PerformanceGood, using ordinary least squares regression. All continuous variables are standardized to facilitate comparison of weights (coefficients) in the promotion decision. All variables are defined in the Appendix. A constant term, as well as year and job level (e.g., A, AA, AAA) fixed effects, are included in each estimation, but omitted for the sake of brevity. Robust t-statistics based on standard errors clustered at the organization level are presented below regression coefficients. ***, **, and * indicate statistical significance at the 0.01, 0.05, and 0.10 levels.
Dependent Variable = VARIABLE (1) (2)PriorPerformance it 0.0158*** 0.0186***
(5.1435) (6.3434)InitialAssessment i 0.0582***
(16.7377)PerformanceQ1 it 0.1312*** 0.0983***
(18.4683) (14.4097)PerformanceQ2 it 0.1114*** 0.0898***
(11.0689) (10.0939)PerformanceQ3 it 0.1027*** 0.0927***
(12.2578) (12.2720)PerformanceQ4 it 0.0575*** 0.0822***
(8.8810) (12.0047)PerformanceGood it 0.1556*** 0.1571***
(20.3195) (20.3485)PerformanceQ1 it -0.1346*** -0.0315* PerformanceGood it (-7.5612) (-1.5482)PerformanceQ2 it -0.0655*** 0.0050* PerformanceGood it (-3.2472) (0.2547)PerformanceQ3 it -0.0043 0.0257* PerformanceGood it (-0.2604) (1.5966)PerformanceQ4 it 0.1257*** 0.0393*** PerformanceGood it (8.2309) (2.4085)LeaguePerformance it 0.0006 -0.0001
(0.1327) (-0.0258)Experience it -0.0048 -0.0017
(-1.6927) (-0.5885)Age it -0.0794*** -0.0693***
(-14.8291) (-12.4687)College i 0.0088 0.0221**
(0.9066) (2.2893)YEAR FE YES YESJOB LEVEL FE YES YESN 29,716 29,716R2 0.1747 0.1831
Promotion it+1
42
Table 5 – Promotion weight on initial assessments vs. on-the-job performance
Panel B: Future Performance and Initial Assessment Quartile Notes: This table presents the results of estimating variants of equation 1 with future performance as the dependent variable and replacing Performance with performance by initial assessment quartile (PerformanceQ1 – PerformanceQ4) and interactions of PerformanceGood with performance sorted by initial assessment quartile using ordinary least squares regression. All continuous variables are standardized to facilitate comparison of weights (coefficients) in the promotion decision. All variables are defined in the Appendix. A constant term, as well as year, job level (e.g., A, AA, AAA), and next job level fixed effects, are included in each estimation, but omitted for the sake of brevity. Robust t-statistics based on standard errors clustered at the organization level are presented below regression coefficients. ***, **, and * indicate statistical significance at the 0.01, 0.05, and 0.10 levels.
Dependent Variable = VARIABLE (1) (2)PriorPerformance it 0.0629*** 0.0619***
(6.1157) (5.9391)InitialAssessment i -0.0153*
(-1.7608)PerformanceQ1 it 0.1368*** 0.1500***
(3.2933) (3.7132)PerformanceQ2 it 0.1028*** 0.1137***
(2.8454) (3.0069)PerformanceQ3 it 0.1233*** 0.1287***
(3.4080) (3.4220)PerformanceQ4 it 0.1760*** 0.1683***
(5.7733) (5.4600)PerformanceGood it -0.0040 -0.0057
(-0.1882) (-0.2594)PerformanceQ1 it 0.0755 0.0424* PerformanceGood it (1.1756) (0.6545)PerformanceQ2 it 0.0932** 0.0682* PerformanceGood it (2.1677) (1.5906)PerformanceQ3 it 0.0474 0.0355* PerformanceGood it (1.2383) (0.8630)PerformanceQ4 it -0.0245 -0.0016* PerformanceGood it (-0.7102) (-0.0422)LeaguePerformance it 0.0129 0.0129
(1.6174) (1.6151)Experience it -0.0132* -0.0146**
(-1.9200) (-2.1481)Age it -0.0266* -0.0292*
(-1.8719) (-1.9903)College i 0.0272 0.0236
(1.4336) (1.2938)YEAR FE YES YESJOB LEVEL FE YES YESNEXT JOB LEVEL FE YES YESN 22,952 22,952R2 0.0467 0.0468
Performance it+1
43
Table 6 – Promotion weights on initial assessments vs. on-the-job performance, partitioned by Experience
Panel A: Promotion Decisions
Notes: This table presents the results of estimating variants of equation 1 by years of player experience relative to the initial assessment player i using seemingly unrelated regression to enable statistical tests across years relative to the draft. All continuous variables are standardized to facilitate comparison of weights (coefficients) in the promotion decision. All variables are defined in the Appendix. A constant term, as well as year and job level (e.g., A, AA, AAA) fixed effects, are included in each estimation, but omitted for the sake of brevity. Robust z-statistics based on standard errors clustered at the organization level are presented below regression coefficients. ***, **, and * indicate statistical significance at the 0.01, 0.05, and 0.10 levels.
𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖+1 = 𝛽0 + 𝛽1𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 + 𝛽2𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 + 𝛽3𝐼𝑃𝑃𝑃𝑃𝑃𝐼𝐼𝐼𝐼𝑃𝐼𝐼𝑃𝑃𝑃𝑃𝑖 +𝛽4𝐿𝑃𝑃𝐴𝐿𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 + 𝛽5𝐸𝐸𝐸𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 + 𝛽6𝐼𝐴𝑃𝑖𝑖 + 𝛽7𝐶𝑃𝐼𝐼𝑃𝐴𝑃𝑖 + ∑ 𝛽𝑗𝑗 𝑌𝑃𝑃𝑃𝑖 + ∑ 𝛽𝑘𝑘 𝐿𝑃𝐿𝑃𝐼𝑖 + 𝜀𝑖𝑖+1 (1)
Dependent Variable = Experience = 0 1 2 3 4 5VARIABLE (1) (2) (3) (4) (5) (6)Performance it 0.1123*** 0.1561*** 0.1637*** 0.1590*** 0.1239*** 0.1042***
(27.9271) (24.5805) (17.0275) (18.0047) (11.4535) (8.2153)Performance it-1 0.0207*** 0.0095 0.0036 -0.0046 0.0016
(5.3582) (1.2604) (0.4409) (-0.5361) (0.2021)Performance it-2 0.0157*** 0.0227*** 0.0063 0.0171
(4.3488) (3.9999) (0.6272) (1.5647)Performance it-3 0.0153*** -0.0021 0.0035
(3.2561) (-0.1630) (0.3212)Performance it-4 0.0022 0.0179*
(0.3665) (1.8560)Performance it-5 -0.0040
(-0.3956)InitialAssessment i 0.1256*** 0.0891*** 0.0704*** 0.0530*** 0.0330*** 0.0348***
(34.2637) (19.3269) (7.9797) (7.6051) (4.2161) (4.4863)LeaguePerformance it -0.0173* -0.0149* -0.0059 -0.0143* -0.0007 -0.0194
(-1.8333) (-1.7137) (-0.8684) (-1.9419) (-0.0669) (-1.4816)Repeat it -0.0535*** 0.0436*** -0.0005 -0.0047 -0.0166
(-3.6007) (2.9456) (-0.0323) (-0.2741) (-0.9714)Age it 0.0012 -0.0586*** -0.0597*** -0.0676*** -0.0627*** -0.0966***
(0.1104) (-6.3312) (-4.0837) (-5.3786) (-3.8253) (-6.3987)College i 0.1278*** 0.0429*** 0.0001 0.0150 0.0065 0.0299
(6.8391) (2.8683) (0.0055) (0.7088) (0.2769) (1.3006)X 2 -stat (Performance it = Innate i ) 4.42** 66.10*** 37.91*** 85.16*** 70.10*** 17.84***Z-stat (Performance it - Innate i ) = (Performance it-1 - Innate i )YEAR FE YES YES YES YES YES YESJOB LEVEL FE YES YES YES YES YES YESN 41,043 41,043 41,043 41,043 41,043 41,043R2 0.1596 0.1469 0.1380 0.1465 0.1213 0.1343
Promotion it+1
N/A 7.50*** 1.44 0.55 -1.22-0.89
44
Table 6 – Promotion weights on initial assessments vs. on-the-job performance, partitioned by Experience
Panel B: Future Performance
Notes: This table presents the results of estimating variants of equation 1 with future performance as the dependent variable by years of player experience relative to the initial assessment player i using ordinary least squares seemingly unrelated regression to enable statistical tests across years relative to the draft. All continuous variables are standardized to facilitate comparison of weights (coefficients) in the promotion decision. All variables are defined in the Appendix. A constant term, as well as year, job level (e.g., A, AA, AAA), and next job level fixed effects, are included in each estimation except for Column 1, but omitted for the sake of brevity. Robust z-statistics based on standard errors clustered at the organization level are presented below regression coefficients. ***, **, and * indicate statistical significance at the 0.01, 0.05, and 0.10 levels.
𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖+1 = 𝛽0 + 𝛽1𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 + 𝛽2𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 + 𝛽3𝐼𝑃𝑃𝑃𝑃𝑃𝐼𝐼𝐼𝐼𝑃𝐼𝐼𝑃𝑃𝑃𝑃𝑖 +𝛽4𝐿𝑃𝑃𝐴𝐿𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 + 𝛽5𝐸𝐸𝐸𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 + 𝛽6𝐼𝐴𝑃𝑖𝑖 + 𝛽7𝐶𝑃𝐼𝐼𝑃𝐴𝑃𝑖 + ∑ 𝛽𝑗𝑗 𝑌𝑃𝑃𝑃𝑖 + ∑ 𝛽𝑘𝑘 𝐿𝑃𝐿𝑃𝐼𝑖 + 𝜀𝑖𝑖 (1)
Dependent Variable = Performance it
Experience = 0 0 1 2 3 4 5VARIABLE (1) (2) (3) (4) (5) (6) (7)Performance it 0.1474*** 0.1722*** 0.1776*** 0.1171*** 0.1144*** 0.1453***
(10.5249) (7.2212) (5.7845) (4.3848) (4.0893) (4.7286)Performance it-1 0.0498*** 0.0589*** 0.0472*** 0.0522** 0.0312*
(2.9983) (3.6406) (2.7186) (2.1290) (1.7122)Performance it-2 0.0618*** 0.0230 -0.0112 -0.0226
(4.7951) (0.9389) (-0.3851) (-0.8933)Performance it-3 0.0107 0.0117 0.0440*
(0.6734) (0.4849) (1.6677)Performance it-4 -0.0067 0.0191
(-0.3363) (0.7408)Performance it-5 -0.0148
(-0.6997)InitialAssessment i 0.0950*** -0.0014 -0.0148 -0.0126 -0.0156 -0.0223 -0.0412***
(8.4209) (-0.1048) (-0.9981) (-0.8554) (-1.1662) (-1.4356) (-2.7586)LeaguePerformance it 0.2044*** 0.0474*** 0.0129 -0.0177 0.0716*** 0.0151 0.0179
(17.7672) (3.3569) (0.9198) (-1.0639) (3.3622) (0.7011) (0.5691)Repeat it 0.0188 0.0307 -0.0147 0.0456 0.0090
(0.4996) (0.8704) (-0.4418) (1.6108) (0.1883)Age it 0.1128*** -0.0258 -0.0307 -0.0329 -0.0062 0.0453 -0.0382
(6.2310) (-1.2729) (-1.0408) (-1.4079) (-0.2244) (0.9902) (-0.6719)College i 0.1827*** 0.0363 0.0240 0.0499 0.0109 -0.0302 -0.0007
(5.4975) (1.0430) (0.6185) (1.1245) (0.2203) (-0.5932) (-0.0090)X 2 -stat (Performance it = Innate i ) N/A 68.03*** 45.66*** 21.10*** 23.41*** 16.22*** 26.38***Z-stat (Performance it - Innate i ) = (Performance it-1 - Innate i )YEAR FE YES YES YES YES YES YES YESJOB LEVEL FE YES YES YES YES YES YES YESNEXT JOB LEVEL FE NO YES YES YES YES YES YESN 12,437 32,507 32,507 32,507 32,507 32,507 32,507R2 0.0587 0.0581 0.0584 0.0747 0.0530 0.0417 0.0541
Performance it+1
0.10 1.12N/A N/A 1.16 0.07 -1.27
45
Table 7 – Promotion weights on initial assessments vs. on-the-job performance, partitioned by Job Level
Panel A: Promotion Decisions
Notes: This table presents the results of estimating variants of equation 1 using ordinary least squares regression. All continuous variables are standardized to facilitate comparison of weights (coefficients) in the promotion decision. All variables are defined in the Appendix. A constant term, as well as year fixed effects, are included in each estimation, but omitted for the sake of brevity. Robust t-statistics based on standard errors clustered at the organization level are presented below regression coefficients. ***, **, and * indicate statistical significance at the 0.01, 0.05, and 0.10 levels. 𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖+1 = 𝛽0 + 𝛽1𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 + 𝛽2𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 + 𝛽3𝐼𝑃𝑃𝑃𝑃𝑃𝐼𝐼𝐼𝐼𝑃𝐼𝐼𝑃𝑃𝑃𝑃𝑖 +𝛽4𝐿𝑃𝑃𝐴𝐿𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 + 𝛽5𝐸𝐸𝐸𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 + 𝛽6𝐼𝐴𝑃𝑖𝑖 + 𝛽7𝐶𝑃𝐼𝐼𝑃𝐴𝑃𝑖 + ∑ 𝛽𝑗𝑗 𝑌𝑃𝑃𝑃𝑖 + ∑ 𝛽𝑘𝑘 𝐿𝑃𝐿𝑃𝐼𝑖 + 𝜀𝑖𝑖+1 (1)
Dependent Variable = Job Level = 1 2 3 4 5 6 7VARIABLE (1) (2) (3) (4) (5) (6) (7)Performance it 0.0619*** 0.1040*** 0.1274*** 0.1956*** 0.1723*** 0.1820*** 0.0824***
(5.8788) (6.1080) (12.9419) (32.3579) (21.8520) (12.7806) (9.1636)PriorPerformance it 0.0160** 0.0207** 0.0158* 0.0194*** 0.0344*** 0.0184* 0.0274
(2.0395) (2.5957) (1.9340) (4.2570) (3.8649) (1.6964) (1.4607)InitialAssessment i 0.1071*** 0.0957*** 0.1067*** 0.0705*** 0.0738*** 0.0609*** 0.0337***
(4.8189) (5.6526) (7.8998) (10.7631) (12.4969) (12.7346) (4.9861)LeaguePerformance it -0.0129 -0.0045 0.0263 -0.0070 -0.0082 -0.0159 0.0001
(-0.8682) (-0.2188) (1.6495) (-0.4656) (-1.4214) (-1.1339) (0.0144)Experience it -0.0509** -0.0317 -0.0360*** -0.0027 0.0164*** -0.0103 0.0024
(-2.2921) (-1.3173) (-2.9421) (-0.3462) (2.7449) (-1.2924) (0.3794)Age it -0.0958*** -0.1130*** -0.1282*** -0.0295*** -0.0684*** -0.0361** -0.0725***
(-3.2940) (-5.8761) (-6.3753) (-2.7846) (-7.5633) (-2.4899) (-6.8328)College i 0.0111 0.0096 0.0237 0.0596*** 0.0395** -0.0120 -0.0082
(0.2687) (0.2740) (0.6443) (3.2416) (2.4525) (-0.4809) (-0.4267)F-stat (Performance it = Innate i ) 3.76* 0.10 1.30 214.81*** 105.92*** 65.70*** 23.51***YEAR FE YES YES YES YES YES YES YESN 1,111 1,488 2,314 7,675 7,855 6,062 3,211
R2 0.1322 0.1683 0.1727 0.1431 0.1399 0.1370 0.0766
Promotion it+1
46
Table 7 – Promotion weights on initial assessments vs. on-the-job performance, partitioned by Job Level
Panel B: Future Performance
Notes: This table presents the results of estimating variants of equation 1 with future performance as the dependent variable using ordinary least squares regression. All continuous variables are standardized to facilitate comparison of weights (coefficients) in the promotion decision. All variables are defined in the Appendix. A constant term, as well as year and next job level (e.g., A, AA, AAA) fixed effects, are included in each estimation, but omitted for the sake of brevity. Robust t-statistics based on standard errors clustered at the organization level are presented below regression coefficients. ***, **, and * indicate statistical significance at the 0.01, 0.05, and 0.10 levels. 𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖+1 = 𝛽0 + 𝛽1𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 + 𝛽2𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 + 𝛽3𝐼𝑃𝑃𝑃𝑃𝑃𝐼𝐼𝐼𝐼𝑃𝐼𝐼𝑃𝑃𝑃𝑃𝑖 +𝛽4𝐿𝑃𝑃𝐴𝐿𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 + 𝛽5𝐸𝐸𝐸𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 + 𝛽6𝐼𝐴𝑃𝑖𝑖 + 𝛽7𝐶𝑃𝐼𝐼𝑃𝐴𝑃𝑖 + ∑ 𝛽𝑗𝑗 𝑌𝑃𝑃𝑃𝑖 + ∑ 𝛽𝑘𝑘 𝐿𝑃𝐿𝑃𝐼𝑖 + 𝜀𝑖𝑖 (1)
Dependent Variable = Job Level = 1 2 3 4 5 6 7VARIABLE (1) (2) (3) (4) (5) (6) (7)Performance it 0.1317*** 0.1973*** 0.1580*** 0.1871*** 0.1614*** 0.1014*** 0.1005***
(7.5318) (6.7936) (6.5250) (6.7740) (5.9375) (3.9366) (3.3938)PriorPerformance it 0.0692*** 0.0598*** 0.0719** 0.1275**
(4.5436) (3.3858) (2.6538) (2.1509)InitialAssessment i -0.0254 -0.0219 0.0122 -0.0179 0.0053 -0.0198 -0.0225*
(-0.8776) (-0.9856) (0.5823) (-0.9591) (0.5326) (-1.5357) (-1.7401)LeaguePerformance it 0.0563* -0.0104 0.1027*** -0.0203 0.0320 -0.0388 0.0281
(2.0045) (-0.4643) (3.7654) (-0.5782) (1.6938) (-1.3112) (1.1752)Experience it -0.1501*** -0.1181*** -0.0044 -0.0157 -0.0139 -0.0239 -0.0176
(-3.1669) (-3.3033) (-0.1979) (-0.8619) (-0.8218) (-1.6134) (-0.6461)Age it -0.0794 -0.0406 0.0196 -0.0326 -0.0316 0.0058 0.0213
(-1.4502) (-1.0441) (0.6026) (-1.5274) (-0.9517) (0.1863) (0.5662)College i 0.0237 -0.0032 0.0741 0.0192 0.0604 0.0085 -0.0261
(0.3945) (-0.0494) (1.4803) (0.5491) (1.1564) (0.1736) (-0.4621)F-stat (Performance it = Innate i ) 23.95*** 30.32*** 20.51*** 55.29*** 27.20*** 15.83*** 12.01***YEAR FE YES YES YES YES YES YES YESNEXT JOB LEVEL FE YES YES YES YES YES YES YESN 3,640 3,459 4,900 5,927 5,890 5,030 2,941
R2 0.0562 0.0910 0.0401 0.0494 0.0511 0.0365 0.0457
Performance it+1