A Study of the Construct Differential Validity of a ...
71
University of Central Florida University of Central Florida STARS STARS Retrospective Theses and Dissertations Spring 1982 A Study of the Construct Differential Validity of a Performance A Study of the Construct Differential Validity of a Performance Appraisal System Appraisal System Hughette I. Crumpler University of Central Florida Part of the Industrial and Organizational Psychology Commons Find similar works at: https://stars.library.ucf.edu/rtd University of Central Florida Libraries http://library.ucf.edu This Masters Thesis (Open Access) is brought to you for free and open access by STARS. It has been accepted for inclusion in Retrospective Theses and Dissertations by an authorized administrator of STARS. For more information, please contact [email protected]. STARS Citation STARS Citation Crumpler, Hughette I., "A Study of the Construct Differential Validity of a Performance Appraisal System" (1982). Retrospective Theses and Dissertations. 617. https://stars.library.ucf.edu/rtd/617
Transcript of A Study of the Construct Differential Validity of a ...
University of Central Florida University of Central Florida
STARS STARS
Retrospective Theses and Dissertations
Spring 1982
A Study of the Construct Differential Validity of a Performance A Study of the Construct Differential Validity of a Performance
Appraisal System Appraisal System
Hughette I. Crumpler University of Central Florida
Part of the Industrial and Organizational Psychology Commons
Find similar works at: https://stars.library.ucf.edu/rtd
University of Central Florida Libraries http://library.ucf.edu
This Masters Thesis (Open Access) is brought to you for free and open access by STARS. It has been accepted for
inclusion in Retrospective Theses and Dissertations by an authorized administrator of STARS. For more information,
please contact [email protected].
STARS Citation STARS Citation Crumpler, Hughette I., "A Study of the Construct Differential Validity of a Performance Appraisal System" (1982). Retrospective Theses and Dissertations. 617. https://stars.library.ucf.edu/rtd/617
A STUDY OF THE CONSTRUCT DIFFERENTIAL VALIDITY OF A PERFORMANCE APPRAISAL SYSTEM
BY
HUGHETTE r. CRUMPLER B.S. Florida Southern College~ 1974
HES IS
Submitted in partial fulfillment of the requirements for the degree of Master of Science in Industrial Psychology
in the Graduate Studies Program of the College of Social Sciences University of Central Florida
Orlando, Florida
Spring Semester 1982
ACKNOWLEDGEMENT
To my fami 1 y for a 11 they have done for me throughout the yea rs,
with special appreciation for their encouragement.
Thanks to Dr. Edwin C. Shirkey for his guidance.
Thanks to Richard B. Dillard and Robert A. Cohen for thei~
support.
iii
TABLE OF CONTENTS
INTRODUCTION
Differential Validity .. Performance Appraisals
METHOD
Subjects ateria s .
Procedure
RESULTS
Oblique Factors .. Orthogona1 Factors
DISCUSSION .
CO CLUSION .
ABLES .
FIGURES
REFERE CES •.
i"
. . .
. . . . . . ..
.
Page
1 8
16
16 17 19
20
21 23
25
30
31
60
62
INTRODUCTION
In recent years, there are two subjects in the personnel
psycho1ogy field which have received considerable attention in the
literature, specifically the concept of differential validity and
secondly a multipurposed instrument known as the performance
appra·sal. It is important to understand the history, present status,
and lega ·mplications of these two topic.s as they relate to the
thesis proposed in this paper, i.e., the need to view these two issues
simultaneously as they relate to each other.
D1fferential Validity
First the notion of differential validity wi11 be reviewed as it
has received heated debate in the journals. The debate centers over
hether di ferential validity, as it is presently defined, exists in
actual data or is merely a theoretical construct. As aptly summarized
by Linn (1978), the hypothesis of differential validity was widely
accepted in the 60's. However, as empirical evidence began to be
tested, it became evident that it w,as not a common pl ace phenomenon.
Since then, Boehm (1 972) found very little evidence of differential
validity, while others concluded that it is at best an isolated
phenomenon. Linn also notes that Schmidt, Berner, and Hunter (1973)
referred to it as a "ps,eudo-prob 1 em 0 ( p. 5) . Bray and Moses ( 11 972.)
concluded 'the closer the study design comes to the ideal, the less
likelihood there is of finding differential validity (p. 554)."
2
There have been several definiti:ons of differential validity.
However, in all cases, the concept is defined in the context of a
predictive, validity paradigm. The classical validation model
determines a simple index of the relationship between the predictor
and criterion without regard for intervening vari ab 1 es. Dunn1ette.' s
{1963) validation model allows for predictors to be differentially
valid for different groups of individuals. It recognizes the
existence of moderator variables (Sanders, 1955), also known as
1 population control variables~ (Gayl ord & Garrell, 1948), "subgrouping
variables" (Frederiksen & Melville, 1954), 0 referrant variables"
(Toops 1959), 11 predictabi1 ity variables 111 (Ghise11 i, 1956), 11modifier
variables" (Grooms & Endler, 1960), and 11 nomologizer variables"
(Johnson, 960) .
he operation of race or sex as moderators has caused great
concern duringi recent years in the personnel! community and has
produced a considerable search for potential discrimination in the
use of employment and training selection devices. 801ehm (1972)
defines two kinds of situations -- differential validity and single
group validity which are of concern. Differential validity exists
where there is a signi,ficant differe,nce between the correlation
coefficient of a selection device and a criteria obtained for one
ethnic group and the correlation of the same device with the same
criterion obtained for the other group, and the validity coefficients
are significantly different from zero for one or both groups. She
also describes single group validity as where a oiven predictor exhibits
validity significantly different from zero for one 9roup only, and
3
there is no significant difference between the two validity coefficients.
Bray and Moses (1972) predicted "the controversy centering on the
possible differential validity of selection tests for majority and
minority groups will abate." They took the risk of being resoundingly
wrong and were.
Schmidt, Berner, and Hunter (1973) questioned the existence of
differences in va idity as a substantive phenomena and recommended
psychologists direct their efforts to the study and determination
o test fairness. Whi e Fox and Lefkowitz (1974) 0 confirmed the
ex·stence of differential validity ... and demonstrated the
efficiency of the moderator variable approach by means of the
independent correlation and regression analysis 11, they went so far
as to say ''there ex1sts sufficient empirical evidence to expect
differential validity as the rule in employment testing rather than
the exception." Citing Campbell (1969), Flaugher & Norris (1969),
Kirpatrick, Ewen, Barrett & Katzel1 (1968), Lefkowitz (1972), Lopez
(1966) O'Leary, Farr & Bartlett (1970), Ruda & Albright (1968),
and Wollowich (1969), they called for "the routine validation of
selection tests separately for different ethnic groups. 111
Kirchner {1975) reflects these comments and claims "most
industrial psychologists would tend to disagree.'' He raises questions
of additional moderator variables, restriction of ranges and
appropriateness of combining job categories.
Lefkowitz and Fox (1975) rebuttle Kirchner's criticisms and
emphasize the issue has been put simply and c1early by D.unnette (1974)
who, in the process of advocating maximally individualized personnel
4
decision systems, emphasized differential treatment of minority and
non-minor"ty group members when it is believed the accuracy of
decisions may be enhanced by so doing.
Fincher (1975) takes a different approach by stating ''if the
substantive issue of test bias is to be resolved, a comprehensive
effort must begin with the construction and development of psychological
tests and carry through to their uses and applications in personnel
decisions. The traditional paradigm for predictive validity must be
broadened to reincorporate basic principles of test construction and
take a more sophisticated view of personnel decision-making. Regression
techniques alone, regardless of how sophisticated, are not sufficient.
He cogently says to ensure test fairness, item content and standardiza
tion procedures should be accompanied by systematic efforts: a) to
assure competency in interpretation and use, and b) to get on with
the business of job analyses and criterion development.
Still focusing on the previously discussed items of differential
validity and single group validity, Katzell and Dyer (1977) analyzed
thirty-one investigations of test validity in samples of black and
white workers. The results werie considered somewhat i nconcl usi ve
as the studies did not permit a rigorous test of hypotheses, holding
that there are no ethnic differences in employment test validity. They
called not for more research but better research where the problem
would focus on the entire prediction system, not just validity but also
5
regression lines, standard errors, intercepts, cutoffs, and utilities.
Boehm (1977) supports this change in direction. 11 Differential
prediction as a research hypothesis has been repeatedly investigated
and repeatedly found wanting. It is simply not supported by the
evidence 11 She calls for a shift in research emphasis to the
questions dealing with poss"ble interactions between ethnic groups
and perceptions of job performance (Bass & Turner, 1973; Rock,
Campbell & Evans, 1970 }, the legitimacy of the entire selection
enterprise (Wallace, 1972), the trainability of the constructs
reasoned by many of the predictors used (Brown, 1972), and the social
utility of various models of test fairness (Gross & Su, 1975).
Looking for a possible explanation of moderator effects rather
than proving or disproving differential validity, Locke, Mento and
Latcher (1978) found ability predicted performance better in groups
hich were homogeneous with respect to motivation.
gain studying single-group and differential validity, Hunter and
Sc midt (1978) critically reviewed the methodology of three studies and
this time concluded the evidence overwhelming to lay to rest the notion
of single group validity and regarded the concept of differential
validity as highly questionable.
Boehm (1978) refused their criticisms and defended earlier
conclusions by accusing Hunter and Schmidt of reflecting an idealized
mathematical world far removed from one in which industrial-or1ganiza
tional psychologists practice. Additionally, Katzell and Dyer {1978)
reasserted that it was premature to dismiss the issue of differential
validity.
Hunter et al. (1979) examined 866 black-white employment test
validity pairs from 39 studies and once again disconfirmed the
differential validity hypothesis.
6
Linn (1978) contends that differential validity is too narrow a
focus and that differential prediction and considerations of bias in
selection procedures are more critical issues and require more than a
comparison of correlation coefficients. He continues by stating
differential validity undoubtedly exists under the restrictive
definition Population1 is precisely equal to Papulation2. However,
evidence is strong to suggest the magnitude of the difference is very
smal He argues that due to restrictions of range, correlation
coefficients should not be the statistics of primary interest.
He points out that differential validity has been used to refer
to differences in correlation coeff"cients, differences in standard
errors of estimate differences in slopes, and/or differences in
intercepts and regression lines. These multiple uses of the term have
certainly confused matters.
Linn acknowledges that questions of bias are central to
considerations of equal employment opportunity and, therefore,, standard
errors, intercepts, cutoffs, and utilities are of importance. He
supports Bobko and Bart ett (1977) who call for attention to be placed
on test fairness and differential prediction. In this light, they
site Bartlett, Babka, Hannan and Mosier's examination of 1 ,190 racial
group comparisons and found that 5.21% contained significant slope
differences and 17.98% contained significant intercept differences.
They found some kind of differential prediction for 23.19% of the
7
comparisons.
A key point raised by Boehm (1 978} is "neither validity nor its
lack is a property of a test but rather of the correctness of the
inference made about its utility. 11 Or as stated by the Standards for
Educational and Psychological Tests {APA, 1974), "validity refers
to the appropriateness of inferences from test scores or other forms
of assessment" (p. 25).
Katzell and Dyer (1978) also disputed Hunter and Schmidt's
er.tic sms of underestimating Type 1 error, that is the increased
"kelihood of rejecting the nu11 hypothesis that differences 1n
val ·di ty bet een the two ethnic groups do n,ot occur more often than
chance, and overestimating Type II error, that is exaggeration of -
nonsignificance of results, in their analysis and maintained
differential validity is "not a pseudo problem 11 since the inequivocal
and strong evidence eeded to sustain a null hypothesis has not been
mastered. Even those who would dismiss it by pointing to test
fairness as the real issue are inadvertently supporting its salience;
differential val "dity defined in terms of different regression slopes,
is after a l a major factor in differential prediction, which in turn
ies at the heart of unfairness.
In addition to the debate over the existence or frequency with
which differential validity occurs in the field, numerous methological
problems or limitations have been discussed in the literature. Arvey
and Mussio (1973) suggested that multiple regression techniques can
lead to different conclusions concerning unfair test discrimination
than if bivariate procedures are used.
8
Schdmit et al. (1978) raise yet another factor to consider,
namely .. instances of differential validity are not produced by subgroup
differences in predictor variance, but that differences in subgroup
criterion variance may have a sizeable impact on comparisons of male
fema e subgroup validities.u They point out the most frequently used
criterion in the studies that were the source of data analyzed in
their article was "a rating of some type. 11
Clearly this question of differential validity remains unanswered
as a result of unclear definitions, weak theoretical framework and
methodological problems such as, but not limited to, inadequate sample
sizes, ype errors, and Type II errors.
Distenfano et al. ( 980) address one of these problems by
suggesting a content validity method could be applied to criteria as
wel as selection tests to help solve the problem of criterion
relevance in validation research by providing quantitative evidence
of the job relatedness of criteria.
Per ormance Aporaisals
The Off'ce of Federal Contract Compliance Programs (OFCCP) has
recognized this content validity approach as a neeiessary step in
developing what must surely be the most wide1y used instrument in an
industrial setting -- the performance appraisal.
Un 1 i ke the previously covered top;· c ,, the performance appraisal
issue is rather straightforward. - Some observations become apparent:
a) performance appraisals have varied uses in the industrial
environment, b) performance appraisals are viewed by the Uniform
Guidelines (1978) as 11 testsu or 11 selection devices 11 as they are
frequently utilized in promotion decisions and other selection
decisions in industry, c) performance appraisals are, therefore,
subject to validation procedures, and d) over the years court
interpretations have created a body of do's and don'ts regarding
performance appraisals.
9
One interesting fact re,gardingi performance appraisals is they are
frequently used as both a predictor (as is the use in promotional
decisions and developmental decisions) and as a criterion (as is the
case of merit increases and test validation studies).
The w·despread use of performance appraisals or performance
evaluations is evidenced ·n a national study of 139 companies by the
Bureau of ational ffairs where over 901 of the companies surveyed
has fonnal perfonnance evaluation programs for their supervisors,
middle managers and professional/technical personnel (Holley &
Field 1974).
Ho 11 ey et a 1 . ( 1976) ana 1 yzed comparable performanc1e appra i sa 1
systems in 39 organizations and found 58% of them used the evaluation
data in promotions, demotions, and/or layoffs. 46% used the informa
tion in manpower planning and utilization activities. Additionally,
nearly 40% of the organizations utilized the performance appraisals
as the basis for communications between supervisors and subordinates
and for determining management development needs. Locher and Teel
(1977) state that 89% of the companies in their study conducted
performance appraisals on a regular basis.
Several authors have reviewed Title VII cases and extracted
10
their versions of an acceptable appraisal (Ashe, 1980; Klassen et al.,
1980; Winstanley, 1980; among others). Most recently, Kleiman et al.
(1981) reviewed twenty-three Title VII court cases and summarized the
courts• decisions regarding performance appraisal systems used as the
basis for promotional decisions. These findings are outlined below.
As cited by numerous authors, the performance appraisal when used
in promotional decisions is clearly within the purview of Title VII of
the 1964 Civil R·ghts Act and later the government guidelines on
employee selection.
dverse impact is today defined (Uniform Guidelines, 1978)
according to the ''four-fifths '' rule. Disproportionate now refers to a
situation where the selection rate of the protected group, i.e.,
females, minorities, Vietnam Era Veterans, and employees ages 40
through 65, is less than four-fifths of that of the majority group or
he roup with the highest rate.
According to the guidelines, validation of selection instruments
s not required unless adverse impact is determined to exist. If
adverse impact ·s established, the burden shifts to the employer to
defend or justify the selection procedure. This is typically the
situation.
However in the cases of Clinton v. Adams, Friend v. Leidinge,
Kelly v. Westinghouse, Movement for Opportunity v. Detroit Diesel, Rich
v. Martin Marietta, and Thompson v. McDonnell Douglas Corp., the
plaintiffs failed to establish adverse impact yet the courts assessed
the appraisals anyway (Kleiman & Durham, 1981). In the Saracini case,
the court bypassed the issue of adverse impact and directly examined
the performance appraisa 'l. Finally, in the case of Fisher, only
the adverse impact issue was addressed. The judge dismissed the
appraisal as an invalid method of evaluation in a situation where
adverse impact exi1sts (Kleiman & Durham, 1981 )1.
11
When reviewing evidence required to meet the employer's burden
of ·us ti fyi n,g their promotion procedure's, the emp1 oyer' s task is to show
a clear rel1ationship betwe,en performance on the selection procedure and
performance on the job~ i.e, validate the selection procedure (Griggs).
The Unifonn Guidelines (1978) state that the appropri ,ateness of
a validat"on strategy depends upon the situation. Content validation
is appropriate if the ''test" covers a representative sampl 1e of the job
content. If the selection procedure measures traits or constructs,
then either construct or criterion-related studies are most appropriate.
These standards, however, can be most easily applied to initial
selection decisions. Their application to other employment decisions,
such as promotion, is less straightforward. For example, how can a
performance appraisal measure be empirically validated when used in
making a promotion decision? W~at measure would serve as the criterion
in such a study? The most appropriate approach would appear to be
that of establishing the relevance or job relatedness of the apprai'sal
instrument ..
Such an approach would, at a minimum, examine the deficiency and
contamination (e.g., rating errors such as leniency, halo, and central
tendency) as well as the interrater reliability of the instrument. A
number of psychologists (e.g., Kavanagh, MacKinney~ and Wo1ins, 1971)
have called for a construct 'validity approach to assess relevance using
12
factor analysis or the multi-trait multi-method technique. James
(1973), however, states that such methods provide only the starting
point -- that of determining whether the appraisal instrument is
measuring the characteristics it was intended to measure. It is just
as important to determine if the characteristics are measured in the
same manner for all groups of empl 1oye,es. The rel:ationship between
these characteri'stics and the goals of the 01rganization must al1 so be
establl ished.
Cascio and Bernardin (1981) discussed the implications of
performance appraisa1 1itigatfon for personnel decisions when they
revie ed case law at the Federal Supreme and Appeals Court level.
n effect~ t ey highlight the standards for performance appraisals
set by case la Ex,amples of these 0 rules 111 for performance appraisals
are given below.,
Pat ersan v. merican Tobacco Co. (1976, 1978), Sledge v.
J.P. Stevens & Co. (1978), Robinson v. Union Carbide Corp. (1976),
Rowe v General otors Corp. (1972), EEOC v. Radiator Specialty Co.
(1979), and Myer v., Missouri State Highway Cammi ss ion (1977)
underlined the need for conducting Job analysis and establishing
performance standards.
Employers are required to communicate their performance standards
to empl oye,es as noted in 1Dona 1 dsoin v. Pi 11 l sbury Co. (1977) and Weahkee
v. Perry ( 1978) .
In two landmark easies, Albemarle Paper Co. v. Moody (1975) and
Watkins v. Scott Paper Co. (1976), the use of global, undifferentiated
paired comparisons were rejected. Employers are expected to define the
dimensions being judged and to rate individual performance
dimensions.
13
P1erforrnance dimensions shoiuld be behaviorally based .. Abstract
trait names in graphic rating scales should be avoided, and graphic
rating scale anchors should be brief and logically consistent (James
v. Stockholm Valves & Fitting Co., 1977; Gilmore v. Kansas City
Tenninal Railway Co., 1975; U.S.A. v. City of Chicaigo~ 1978; Marquez
v. Omaha District Sales Office, Ford Division of the Ford Motor
Company, 197 ; and Cleverly v. Western Electric Co., 1979).
Performance appraisals and their raters require validation. In
Brito v Zia Co. {1978) ~ the appraisal system was struck down because
the company ould not provide ''empirical data demonstrating that the
appraisal system was significantly correlated with important elements
of work behavior relevant to the jobs for which the appellants were
eing evaluated .. "
hen the performance appraisal s used as a predictor, as in
promotional decisions, validation evidence must be presented to show
first that the ratings of past performances are in fact valid, and
second y that the ratings and past performances are related to future
performance in another job (U.S .A. v·. City of Chicago, 1978). If the
performance appraisal is to be used in decisions such as merit pay,
layoffs, or demotions, then companies must prove the performance
appraisal provides a valid measure of past performance. The "ratings
should be examined for evidence of racial, ethnic or sex bias. All
criteria needs to be examined for freedom from factors which would
unfairly alter scores of numbers of any group. The relevance of
14
criteria and their freedom from bias are of particular concern when
there are si 1gntificant differences in measur1e of job performance for
different groups (Uni form Gui de 1 i nes,, 1978) . 11
A n1e1ed to 11 val idate the raters as w1el 1 as the appraisal systems"
was established in Brita1 v .. Zia and U.S.A. v. City of Chiicago where the
courts focused on particular raters and their rating behavior, work
habits and attitudes.
In reviewing the topics of differential validity .and the legal
issues of performance apprais(!ls, the following becomes apparent:
( ) The issue of differential validity is unresolved.
(2) Differential val i'dity has been usually reviewed in
predictive val1dity paradigms dealing with traditional
employment tests.
(3) Differential validity has not been addressed in the
context of performance appraisals.
(4) Performance appraisals are widely used and continue
to be in the limel "ght of discrimination questions.
Once the employment process begins, the performance appraisal
is probably the single most important instrument utilized by companies
as it influences many personnel decisions affecting millions of
employees.
Performance appraisals are as susceptible to issues of differential
validity as any other 5,election device and, therefore, should be
subjected to the same criteria of differ·ential validity. The predictive
validity mode1 makes the underlying assumptions that an instrument
measures the same constructs for all groups and the rat~rs make
15
judgments about these constructs i n the same manner for all groups.
These assumptions ignore some basically important questions. Are
the same constructs or dimensions being measured for all subgroups?
Are they being evaluated in the same manner by the raters? Are raters
distinguishing the constructs equally for all relevant subgroups or are
they perhaps allowing halo to operate for some and not for others?
This is an issue of factora l s imil a ri ty across groups for which the
instrument is being applied.
This is, in fact, a form of differential va idity which will be
ca 11 ed 11 construct different i a 1 va i di ty. 11 Construct differential
validity is a question of factoral invariance which is really a
bas1cal y important issue for any device utilized in employment
decisions, espec"ally the widely applied performance appraisal.
T erefore, the purpose of this study will be to statistically
analyze a performance appraisal system as it functioned in one
industrial plant in 1978 and 1979 to determine if differential
val "dity exists as defined in a construct paradigm between (a)
ema es and males and (b) minorities and non-minorities.
The null hypotheses are:
(1) The construct validity for the female groups is not
significantly different from the construct validity
of the male groups.
(,2) The construct validity for the minority groups is not
significantly different from the construct validity
of the non-minority group&.
METHOD
Subjects
The data for this research study wa.s obtained from a 1 arge
aerospace firm as a part of the normal personnel administration and
performance appraisal system. The groups of subjects were randomly
selected for analysis. Ratios of males to females and minorities to
non-minorities represent actual population differences. Their
character·st·cs are as follows:
1978 Group 1 - 2273 male (exempt salaried) employees including
mostly engineers; others perform professional
administrative functions in Contracts, Finance
and ateriel divisions.
Group 2 - 101 female (exempt salaried) employees including
mostly engineers; others perform professional
administrative functions in Contracts, Finance
and Materiel divisions.
Group 3 - 2282 non-minority (exempt salaried) employees
including mostly engineers· others perform
professiona1 administrative functions in
Contracts, Finance and Materiel divisions.
Group 4 - 92 minority (exempt salaried) employees
including mostly engineers; others perform
professional administrative functions in
Contracts, Finance and Materiel divisions.
1979 Group 1 - 2441 male (exempt salaried) employees including
mostly engineers; others perform professional
administrative functions in Contracts, Finance
and Materiel divisions.
Group 2 - 113 fema 1 e (exempt sa 1 ari ed) employees including
mostly engineers; others perform professional
administrative functions in Contracts, Finance
and Materiel divisions.
Group 3 - 2436 non-minority (exempt salaried) employees
including mostly engineers; others perform
profes~ional administrative functions in
Contracts, Finance and Materiel divisions.
Group 4 - 1 8 minority {exempt salaried) employees
including mostly engineers; others perform
professional administrative functions in
Contracts, Finance and Materiel divisions.
17
The women and minorities have been on the job approximately five
years where the males and non-minorities have been on the job
approximately fifteen years. This does not represent time on the
job, rather it is average length of service. Employees are scattered
among var ·ous job levels. However, females and minorities are not
found in the extremely high levels.
Materials
The global overall performance ratings were made for each
employee by his/her supervisor(s) as a normal process 0f annual
performance appraisals. Few, if any, raters were females or minorities.
18
These ratings were used for several personnel objectives such as
determining each employee's annual merit increase in salary,
identifying candidaties fo,r promotion, providing feedback for
perfonnance,, identifying developmental needs, and ide,ntifying training
needs.
Non-supervisory employees are rated on nine dimensions and
supervisors or managers are rated on those nine, plus three additional
dimensions dealing with supervisory behavior (see Figure 1). This
study covers non-supervisory emp1oyees only.
Th s system is known as the EBPA ( Evi denc1e Based Perfonnance
Appraisal) and was developed in a participatory style in 1976. In its
original form, ther'e were 20 dimensions or variables whi'ch were
reduced to nin1e and twelve after a factor analyti1c review. They
include 1) job performance, 2) vers,atil ity, 3) or1ganizational
effectiveness, 4) effectiv1ty with external groups, S) communications,
6) stability, 7) iniitiative, 8) interpersonal relations, 9) work
related self-development, 10)* managerial self-development, 11)*
developing subordinates., and 12)* Equal Employment Opportunity (* for
supervisors/managers only).
ach dimension is rated on a five point scale defined as
follows:
1 - Development needed - accomplishments and effectiveness need
improvement to meet the standards for the job.
2 - 1Eff,e1ctive performance - accomplishments and effectiveness
meet the standards for the job.
19
3 - Highly effective performance - accomp1ishments and effectiveness
~ exceed the standards for the job.
4 - Excellent performance - accomplishments and effectiveness far
exceed the standards for the job.
5 - Outstanding performance - superior accomplishments and
effectiveness; seldom obtained performance.
The dimension ratings are mathematically combined to give the
employee an overall or global rating.
Procedure
Upon col l ect "on of the data, both an orthogonal factor analysis
method i t h varimax rotations and an oblique factor analysis method
wi 11 be ut il ·zed to detenni ne factor patterns for each subgroup
(Statistical Package for the Social Sciences~ 1975 ) .
Factor patterns of subgroups will be analyzed in light of the
null hypotheses.
RESULTS
Means and standard deviations are contained in Table 1. Although
no statistical analysis of means was a part of this study, it can be
seen that scores for each dimension or variable are lower for
minorities and females. Intercorrelations of the variables for all
groups are included on Table 2.
A, test of factor invariance or construct va 1 i di ty includes a
comparison of the number of factors that emerge from the factor
analysis for each subgroup across years. A principal factoring
procedure was utilized with both orthogonal and oblique solutions ~
for each subgroup. he mineigen value was set at .8, which means
that more factors ere allowed to emerge than if the 1 .0 mineigen
as utilized. The orthogona and oblique factoring procedures
resu ted in the same number of factors for each subgroup for the same
time period. Specifically, the nine appraisal dimensions reduced to
two factors for the females in 1978, three factors for the males in
1978, four factors for the minorities in 1978, and three factors for
the non-minorities in 1978. Only two factors emerged for each sub-
group in 1979.
The eigenvalues and total percent of variance accounted for
by each of the factors are presented in Table 3. This data refers to
the first iteration of the correlation coefficients. Also in Table 3
are the eigenvalues and percent of variance accounted for on the
second iteration of factors after the mineigen was set at .8. Percent
21
of variance is reported as 100% in this Table.
Examination of the factor patterns can be seen for both the
oblique and orthogonal solutions in Table 4. Appraisal dimensions or
variables are listed vertically, while the extracted factors are
listed horizontally.
Rummell 's (1970) recorrmended approach for reordering the
var·ab es by size of high factor loadinq better displays the
factor saturation as seen in able 5. Factor loadings greater than
or equal o .5 were utilized in this review, and when there were no
factor loadi gs greater than or equal to .5, the actual highest
loading as recorded
Oblique Factors
A review of the emerging Ob ique Factors reveals that Factor 1
for the Females in 1978 was made up of the following dimensions in
order of their factor loadings: Versatility, Communications,
Organizational Effectiveness, Stability, Interpersonal Relations, and
Effectivity with External Groups. Due to the many dimensions
contributing to this factor, it is not possible to name it~ However,
Factor 2 can be ca 11 ed 11Work-Re 1 a ted Se 1f-Dev1e1opment 11 as is the name
of the singular d·mension contributing to Factor 2 for the females
in 1978.
For the males in 1978, Factor 1 can be called "Organizational
Effectiveness" and Factor 2 is '11 Work-Related · Self-01evelopment 111, while
Fact.or 3 is made up of negative 1 oadings on 11 Versati1 ity'i and
11 Initiative 11•
22
A review of the four oblique factors which emerged in the
minority group in 1978 reveals the dimensions of 11 0rS'anizationa1
Effectiv1eniess 11,
11 Versati 1 ·i ty 11, and "Communi cations 11 make up Factor 1 ,
while Factors 2, 3 and 4 can be easily named as only one dimension
has a factor loading of .5 or greater, specifically, Factor 2 is
"Work-Related Self-Oevelopment 11, Factor 3 is "Interpersonal' Relations",
and Factor 4 is nKey Elements".
Three obliqu1e factors appear for the n1on-minorities. Factor 1
will be cal ed 11 Interpersonal,Effecti'veness 11 as it is comprised of
11 0rganizational Effectiv1eness 11 and "Interpersonal Relations 111• Factor
2 is 11 ork-Related Self-Development" and Factor 3 is 11 V1ersatility 11•
Only two oblique factors emerged for each subgroup in the 1979
data Factor l in each group is very global a.nd many dimensions
contribute to this factor. aming this factor would not prove
ogical; however, it is important to note in order of their factor
oadings, the dimensions which have a factor loading of .5 or greater.
"Key Elements", 11 Initiative·111 "Communications,", "Versatility",,
"Organizational Effectiveness" 7 11 Interpersonal R1elations 11 and 11 Work-
Related Self-Development 111 made up Factor 1 for the females in 1979,
while "Key E1ements 11,
11 Initative 11, "Versatility 111
, 11 0rganizational
Effectiveness 111,
1 Effectivity with External Groups 111,
11 Stabil' ity 11,
11 Conmunicatiansu, 11 Interpersonal Relations", and "'Work-Relat1ed Self
Deve 1 opment u comprised Factor 1 for the ma 1 es in 1979. Factor 1:
for the minorities in 1979 was made up of 11 Key Elements", "Initiative,.,
'*Communications, "Organizational Effe1cti1veness 11,
1'1Eff1ectivity with
External Groups", and 11 Versatility 11• Factor 1 for the non-minorities
23
in 1979 included 11 Key Elements", "Initiative", 11 0rgani1zationa1
Effectiveness", "Effectivity with External Groups", "Interpersonal
Relations", 11 Versatility 11, "Stability", and 11 Communications 11
•
A negative factor loading on 11 Effectivity with Externa1 Groups"
produced Factor 2 in the females in 1979, whi1e a negative factor
loading on "Interpersonal Relations" produced Factor 2 for the males.
"Interpersonal Relations", "Stability", and 'Work-Related Self
Development11 comprised Factor 2 for the minorities in 1979. 11 Work
Related Self-Oeveloprne t" and a negative factor loading on "Inter
personal Relat·ons" contributed to Factor 2 for the non-minorities in
in 1979.
Orthogonal Factors
A very global factor made up of "Versatility", "Initiative",
11 Cornmunications 11, "Organizational Effect ·veness", 11 Stabi 1 ity 11
,
11 Interpersonal Relations' "Key Elements', and "Effectivity with
Extern a 1 Groups 11 emerged as Factor l for the fema 1 es in 1978, while
a singular dimension, "Organizational Effect·veness", produced
Factor 1 for the males in 1978. For minoriti es in 1978, Factor 1
included the variabl 1eS 110rganizati'onal Effectiveness 11,
111 Versatil ity 1',
and 11 Communicationsu. 110rganizational Effectiveness", alone, produced
Factor 1 for the non-minorities in 1978, just as was the case for the
males in 1978.
"Work-Re 1 ated Se 1f-De,ve1opment 11 was Factor 2 for both the females
and the non-minorities in 1978, and it was Factor 3 for the males and
minorities. "Versatility" was Factor 2 for the males and non-minorities
in 1978, while it is found in Factor 1 for the females and
minorities. A fourth factor, 11 Key Elements 0, emerged for the 1978
minorities.
24
In 1979, two orthogonal factors emerged for each subgroup.. For
the females, Factor 1 was comprised of "Key Elements", "Work-Related
Self-Development 0,
0 Initiative 11,
11 Communications 11,
111 Versatil i'ty 1, and
0 0rgani zati ona l Effectiveness 11• For the ma 1 es it wa.s made up of
11 Interpersonal Relations", 11 Effectivity with External Groups", and
"Organizational Effectiveness 111• For the minorities, Factor 1
·ncluded "Key Elements 11,
11 Initiative 11,
111 Comrnunications 111, "Effectivity
with External Groups 0,
11 V1e·rsatility 11, and 11 0rganizational Effectiveness 11
•
u nterpersonal Relations", 111 Effectivity with External Groups 11, ,
"Organizationall Effectiveness 11, and 11 Stability 11 contributed to Factor
1 for the non-minorities in 1979.
11 Eff1ect1vity with External Groups 11 produced the second factor
for females in 1979, while '1Versatility 11,
11 Key Elements", and
"Initiative" combined to form Factor ·2 for the males. 11 Stability 111,
11·Interpersonal Relations", and "Work-Related Self-Development 11
combined to form Factor 2 for the minorities, while "'Versatiliityu,
11 Key Elements" and 11 Inititative 11 formed the second orthogonal factor
for the non-minorities in 1979.
DISCUSSION
The results of this study generally did not support the null
hypotheses 1 and 2 for the 1978 data, that is the construct validity
for females was significantly different from the construct validity
for the malles and the construct validity for the minorities was
s1gn"ficantly different from the construct validity for non
minor'ties. T at is, construct differential validity did exist
bet een the subgroups studies
u 1 hypotheses l and 2 for the year 1979 were generally
supported in that the construct validity as defined by the numbe·r of
emerging factors was no diffe ent for females and males or minorities
and non-minorit"es.
Several implications can be drawn about the 1979 results. One
is that although construct differential validity does not exist, the
overall validity of the performance appraisal is somewhat questionable
as a result of the overwhelming existence of halo error as can be seen
by the large percent of variance accounted for by the first factor in
each group and by the fact that only two factors emerged for each group
from the original nine dimensions. At least, however, the raters were
consistent in their treatment of the ratees, i.e., halo existed for all
groups.
In the 1978 data, one can conclude that construct differential
validity occurred as a result of different emerging factors across the
groups. It is interesting to note that while only two factors emerged
26
for the femdles and three for the males, four factor emerged for the
minorities and only three for the non-minorities. It is desirable
to have a greater number of factors as they represent greater variance
among the rated scores.
When reviewing differences in mean scores, one may assume these
differences are a) true differences, i.e~, resulting from true
differences in performance on the same traits or dimensions, or b)
the differences represent error as a result of rater bias, either
intentional or unintentional . There is no reason to expect the
underlying factors to be different for the various subgroups. There
fore when both ean d.fferences occur and factor pattern differences
occur, it m'ght imply that there is something suspect about the way
the raters make their judgments. It implies that the ·nstrument is
somehow more or 1ess valid for one group than another. If mean
differences occur and there is no significant difference in the
emerging factor patterns, it is logical to imply that mean differences
represent true differences in performance rather than differences due
to rater bias.
When supervisors rate performance, it is normally assumed that
they view the dimensions or variables 1n the same manner for the
various subgroups. In the case of this particular appraisal system,
it is assumed employee performance is being rated on nine independent
dimensions or variables. It is also implied that the raters are
viewing these independent dimensions in the same manner for all groups.
However, the factor analysis reveals the supervisors do not rate
the dimensions independently and, furthermore, the lack of independence
27
among variables is greater for some groups in the 1978 data. When
reviewing the females and males in 1978, it becomes apparent that only
two factors emerged for the fema 1 es while three factors were found for
the males. Concurrently, the analysis reveals four factors for the
minorities and three for the non-minorities. Additionally, the results
show not just a different number of factors but that the composition
of the factors is varied for the subgroups.
What ay be surprising is that the two protected classes are
judged differently by the raters .. It appe_ars that gr1eater halo is in
operation for the females producing the fe'West factors, while the
greatest number of factors appears for the minorities.
hen studyrng the construct validity of an appraisal, it is
hoped t at the number of factors wil equal the number of dimensions
and that each dimension wil stand alone. The instrument is seen as
more valid because it implies the raters can conceptualize each
dimension 'ndependently
However, one possible implication of the results of this study
would be that the raters are hurried y judging the performance of the
females, that is at least more so than the males, while they are
carefully judging the performance of the minorities, perhaps unfairly
so when compared to the judgments made on the non-minorities'
perfonnancie as the 1 ower ratings might i ndi ca te.
The point which needs to be made is that one cannot use a
mathematical model to draw conclusions about the differential validity
of a performance appraisal instrument. There are nine potential
situations which might occur for each protected class group and its
28
antithesis when the means and factors are studied simultaneously.
These combinations and their potential implications as they relate to
those who design performance a ppra i sa 1 systems are i11 ustrated in
Figure 2.
There are several possible reasons for the disappointingly small
number of factors which emerged in both 1978 and 1979, including a)
eight of the nine dimensions are behaviors which the organization
values rather than dimensions based on thorough definition of the job
content and job analysis, b) there may have been a problem due to
ack of recency of training (training of raters was last done in 1976
and 1977), and c) due to a large increase in population in 1978 and
1979 many new supervisors did not receive any formal training on how
to ate per ormance. Since most training programs include material
on how to avoid corrmon rater errors such as halo and leniency, lack of
rain"ng may have contributed to the halo seen in these results.
Due to the direction of the ean score differences of appraisal
ratings, this instrument is vulnerable to crit"cism involving the
specific treatment of protected class groups studied, namely females
and minorities. The problem is more clear with the minority groups.
Since there is a direct relationship betwe,en appraisal ratings and
amount of merit increase, opportunity for promotions, selections for
new assignments, and development and training programs in this
particular situation, the organization would be advised to institute
some actions to remedy this disparity of mean scores. This might
include a forced distribution where the protected class group scores
would not be adversely impacted, a redesign of the appraisal where
29
dimensions or variables rated dre based upon job content for each
specific job, and training raters/supervisors on how to appropriately
rate performance.
When reviewing the application of any performance appraisal
system, industrial psychologists should give consideration to a) mean
differences, b) the number of emerging factors, and c) the composition
of those emerging factors as they relate to the protected class sub
groups. As seen in the results of the 1978 data in this study, not
on y were there different numbers of factors across subgroups but the
factors themselves were dissimilar. For example, in the oblique
solution Factor for the fema 1 es was comprised of 11 Versat i1 i ty",
11 Co111Tiurlications 11, "Organizational Effectiveness", nstabil ity 11
,
1 Interpersonal elations 11, and 'Effectivity with External Groups",
hereas Factor 1 for the males was made up of only "Organizationa
ffectiveness''. Additionally, Factor 1 for the minorities included
"Organizational Effectiveness", "Versatility", and 11 Communicationsu,
hereas Factor l for the non-minorities was made up of "Organizational
Ef ectiveness" and "Interpersonal Re1ations 11• It becomes apparent
the supervisors are somehow cognitively treating these various
subgroups differently
Therefore, when ana 1 yz i nq a performance appra i sa 1 system, it i, s
important to determine if mean differences occur between subgroups
and to analyze the variable or dimension ratings for construct
differential validity in terms of both the number and composition of
emerging factors.
CONCLUSION
The notion of construct differential' validity appears to be a
valuable concept, especially as it applies to the widely utilized
instrument known as the performance appraisal. In this study, construct
validity did exist in 1978, but not in 1979.
This particular appraisal has serious halo error in both years.
A though the results of this study cannot be generalized to other
perfor ance appraisal systems, the methodology may prove beneficial
hen ana 1 yz i ng other performance appra i s,a 1 systems.
It is strongly recorT1111ended that industrial psychologists and
others ho design and utilize performance appraisals focus on both the
means of the rat·ngs as they relate to members of protected classes
and, as importantly, on the construct validity of the instruments as
defined in this paper. Due to the restrictive nature of differential
validity as "t is defined in the classical predictive validity paradigm,
it is recorrmended that the hypothetical construct of differential
validity also be defined in terms of the validity model being utilized,
i.e., we may now refer to 11 predictive differential validity 11 a~d
"construct differential validity 11 logical1y in the future others
may define "content differential validity."
Tab
le
Key
Var
iabl
e Nu
mbe
r =
A
ppra
isal
D
imen
sion
Name
Var
iabl
e 1
:::;
Key
Elem
ents
Var
iabl
e 2
=
Ver
sati
lity
Var
iabl
e 3
=
Org
aniz
atio
nal
Eff
ectiv
enes
s
Var
iabl
e 4
=
Eff
ecti
vity
with
Ext
erna
l G
roup
s
Var
iabl
e 5
=
CoJ
JJllu
nica
tions
Var
iabl
e 6
=
Sta
bil
ity
Var
iabl
e 7
=
Init
iati
ve
Var
iabl
e 8
=
Inte
rper
sona
l R
elat
ions
Var
iabl
e 9
=
Wor
k-R
elat
ed S
elf-
Dev
elop
men
t
w -
Var
iabl
e 1
Var
iabl
e 2
Var
iabl
e 3
Var
iabl
e 4
Var
iabl
e 5
Var
iabl
e 6
Var
iabl
e 7
Var
iabl
e 8
Var
iabl
e 9
Tab 1
e l
Var
iabl
e M
eans
and
Sta
ndar
d D
evia
tions
Fem
ales
-
1978
M
ales
-
1978
M
inor
itie
s -
1978
N
on-M
inor
ities
-19
78
Mea
n S
.ID.
Mea
n S.
D.
ean
s.o.
Mea
n S.
D.
3.62
1 ..
08
3.85
1.
17
3.54
L
39
3.
85
l.1
5
3.39
.8
4 3.
73
.81
3.17
.8
3 3.
74
.81
3.41
.8
6 3.
52
.85
2.98
.7
0 3.
53
.85
3.67
1.
43
3.69
1.
00
3.37
L
0.7
3.70
1.
02
3.33
.9
2 3.
37
.81
2. 91
• 7
2 3.
38
.Bl
3.51
.8
2 3.
70
.80
1 3.
43
.92
.3.7
0 .8
0
3.54
.6
7 3.
70
.87
3.30
1.
04
3.71
.8
6
3.58
.7
9 3.
55
.84
3.32
.7
4 3
.57
.84
3.46
1.
30
3.46
.9
5 3.
42
.96
3.46
.9
7
w
N
Var
iabl
e l
Var
iabl
e 2
Var
iabl
e 3
Var
iabl
e 4
Var
iabl
e 5
Var
iabl
e 6
Var
iabl
e 7
Var
iabl
e 8
Var
iabl
e 9
Tabl
e 1
(con
tinue
d)
Var
iabl
e M
eans
and
Sta
ndar
d D
evia
tions
Fem
ales
-19
79
Mal
es -
1979
M
inor
ities
-19
79
Non
-Min
oriti
es
-19
79
Mea
n s.o
. M
ean
S.D
. M
ean
S.D
. M
ean
S.D
.
3.75
.6
9 3.
79
.66
3.45
. 7
1 3.
81
.65
3.70
. 7
7 3.
74
.74
3.32
.8
2 3.
75
.73
3.68
.7
2 3.
53
.72
3.26
.7
3 3.
55
• 72
3.68
.7
2 3.
63
.70
3.32
.7
0 3.
65
.70
3 .6
1 .7
3 3.
39
• 71
3 .1
6 • 7
2 3.
41
.71
3.71
.7
4 3.
67
.73
3.43
• 7
2 3.
68
973
3.90
.8
4 3.
73
.80
3.39
.8
2 3.
76
.79
3.81
• 7
7 3.
58
.69
3.43
.7
0 3.
60
.69
3.63
.7
3 3.
37
.. 70
3.34
.8
1 3.
39
. 70
w
w
Tab
le 2
Raw
Corr
elat
ion
Coef
fici
ent
Mat
rice
s
Fem
ales
-
1978
V
aria
ble
1 V
aria
ble
2 V
aria
ble
3 V
aria
ble
4 V
aria
ble
5
Var
iabl
e 11
1.
0000
0 0.
4593
0 0.
4428
1 0.
3133
7 0
.446
11
Var
iabl
e 2
0.4
5930
1.
0000
0 0.
5431
4 0.
4915
2 0.
4896
9 V
aria
ble
3 0.
4428
1 0
.543
14
l .0
0000
0.
5063
5 0.
5257
4 V
aria
ble
4 0.
3133
7 0.
4915
2 0.
5063
5 1.
0000
0 0.
4863
5 V
aria
ble
5 0.
4461
1 0.
5896
9 0.
5257
4 0.
4863
5 l.0
0000
V
aria
ble
6 0.
3323
5 0
.494
68
0.53
592
0.35
836
0.54
502
Var
iabl
e 7
o. 37
977
0.65
925
0.55
786
0.50
057
0.60
413
Var
iabl
e 8
0.3
7530
0.
5325
2 0.
5286
6 0.
4006
5 0.
4922
5 V
aria
ble
9 0
.101
43
0.1
1255
0.
2794
1 0.
2961
7 0.
2679
4
Fem
ales
-
1978
V
aria
ble
6 V
aria
ble
7 V
aria
ble
8 V
aria
ble
9 (c
ontin
ued}
Var
iabl
e 1
0.33
235
0. 3
7977
0.
3753
0 0.
1014
3 V
aria
ble
2 0.
4946
8 0.
6592
5 0.
5325
2 0.
1125
5 V
aria
ble
3 0.
5359
2 0.
5578
6 0.
5286
6 0.
2794
1 V
aria
ble
4 0.
3583
6 0.
5005
7 0.
4006
5 0.
296
17
Var
iabl
e 5
0.54
502
0.60
413
0.49
225
0.26
794
Var
iabl
e 6
1.00
000
0.48
828
0.58
026
O. l
9996
V
aria
ble
7 0.
4882
8 1.
0000
0 0.
5820
0 0.
3014
1 V
aria
ble
8 0.
5802
6 0.
5820
0 1.
0000
0 0.
3805
6 V
aria
ble
9 0 .
1999
6 0.
3014
1 0.
3805
6 1.
000
00
w
+::-
Tabl
e 2
(con
tinue
d)
Raw
Cor
rela
tion
Coe
ffic
ient
Mat
rice
s
Mal
es
-19
78
Var
iabl
e 1
Var
iabl
e 2
Var
iabl
e 3
Var
iabl
e 4
Var
iabl
e 5
Var
iabl
e 1
1 .0
0000
0
.310
76
0.25
627
0.18
909
0.25
444
Var
iabl
e 2
0.31
076
1 .00
000
0.39
551
0.28
787
0.42
012
Var
iabl
e 3
0.25
627
0. 3
9551
1.
0000
0 0.
3086
8 0.
4581
5 V
aria
ble
4 0.
1890
9 0.
2878
7 0.
3086
8 1.
0000
0 0.
3000
5 V
aria
ble
5 0.
2544
4 0
.420
12
0.45
815
0.30
005
l .00
000
Var
iabl
e 6
0.25
613
0.41
398
0.39
754
0.28
412
0.39
340
Var
iabl
e 7
0.29
070
0.45
059
0.43
922
0.27
678
0.35
539
Var
iabl
e 8
0.24
965
0.38
818
0.43
059
0.31
609
0.36
660
Var
iabl
e 9
0.15
163
0.28
343
0.20
406
0 .17
508
0.27
015
Mal
es
-19
78
Var
iabl
e 6
Var
iabl
e 7
Var
iabl
e 8
Var
iabl
e 9
(con
tinu
ed)
Var
iabl
e l
0.25
613
0.29
070
0.24
965
0.15
163
Var
iabl
e 2
0.41
398
0.45
059
0.38
818
0.28
343
Var
iabl
e 3
0.39
754
0.43
922
0.43
059
0.20
406
Var
iabl
e 4
0.28
412
0.27
678
0.31
609
0.17
508
Var
iabl
e 5
0.39
340
0.35
539
0.36
660
0.27
015
Var
iabl
e 6
1.00
000
0.42
138
0.43
516
0. 2
1764
V
aria
ble
7 0.
4213
8 1.
0000
0 0.
3888
4 0.
2631
3 V
aria
ble
8 0.
4351
6 0.
3888
4 1.
0000
0 0.
2795
3 V
aria
ble
9 0.
2176
4 0.
2631
3 0.
2795
3 1.
0000
0 w
U
'l
Tabl
e 2
(con
tinu
ed)
Raw
Cor
rela
tion
Coe
ffic
ient
Mat
rice
s
Min
orit
ies
-19
78
Var
iabl
e l
Var
iabl
e 2
Var
iabl
e 3
Var
iabl
e 4
Var
iabl
e 5
Var
iabl
e 1
l.000
00
0.29
688
0.16
780
0.07
217
0.29
720
Var
iabl
e 2
0.29
688
1.00
000
0.61
364
0.23
604
0.51
918
Var
iabl
e 3
0 .1
6780
0.
6136
4 1.
0000
0 0.
2927
4 0.
5225
0 V
aria
ble
4 0.
0721
7 0.
2360
4 0.
2927
4 1.
0000
0 0.
213
82
Var
iabl
e 5
0.29
720
0.51
920
0.52
250
0.21
382
1 .00
000
Var
iabl
e 6
0 .1
7703
0
.331
12
0 .4
1133
0
.182
18
0.25
712
Var
iabl
e 7
0.22
903
0.36
792
0.51
521
0 .11
8967
0.
2360
1 V
aria
ble
B
0. 3
0451
0.
3554
6 0.
4834
0 0.
2128
1 0 .
4019
7 V
aria
ble
9 0.
2107
0 0.
3040
4 0
.358
55
0.41
282
0.11
693
Min
orit
ies
-19
78
Var
iabl
e 6
Var
iabl
e 7
Var
iabl
e 8
Var
iabl
e 9
(con
tinue
d)
Var
iabl
e 1
0.17
713
0.22
903
0. 3
0451
0.
210
70
Var
iabl
e 2
0. 3
3112
0.
3679
2 0.
3554
6 0.
3040
4 V
aria
ble
3 0
.411
33
0.51
521
0.48
340
0.35
855
Var
iabl
e 4
0.18
218
0 .1
8967
0.
2128
1 0.
4128
2 V
aria
ble
5 0.
2571
2 0.
2360
1 0.
4019
7 0.
1169
3 V
aria
ble
6 1.
0000
0 0.
1824
4 0.
4593
8 0.
2243
0 V
aria
ble
7 0.
1824
4 1
.000
00
0.27
178
0 .1
8895
V
aria
ble
8 0.
4593
8 0.
2717
8 1.
0000
0 0.
1649
8 V
aria
ble
9 0.
2243
0 0.
1889
5 0.
1649
8 1
.000
00
w
m
Tab
le
2 (c
onti
nued
)
Raw
Cor
rela
tion
Coe
ffic
ient
Mat
rice
s
Non
-Min
oriti
es -
1978
Va
ria
ble
1 V
aria
ble
2 V
aria
ble
3 V
aria
ble
4 V
aria
ble
5
Var
iabl
e l
1.00
000
0.31
609
0.26
475
0 .1
9852
0.
2580
5 V
aria
ble
2 0.
3160
9 l.0
0000
0.
3853
1 0.
2957
6 0.
4155
51
Var
iabl
e 3
0.26
475
0.38
531
1 .0
0000
0.
3156
9 0.
4517
7 V
aria
ble
4 0.
1985
2 0.
2957
6 0.
3156
9 1.
0000
0 0.
3110
4 V
aria
ble
5 0.
25i8
05
0.41
555
0.45
177
0.31
104
1.00
000
Var
i,abl
e 6
0.26
258
0.41
968
0.40
070
0.28
968
0.40
306
Var
iabl
e 7
0. 2
9633
0.
4609
2 0.
4386
0 0.
2905
4 0.
3682
8 V
aria
ble
6 0.
2498
9 0.
389
72'
0.42
954
0.32
086
0.36
731
Var
iabl
e 9
0.14
523
0.27
217
0.20
384
0. l7
507
0.27
600
Non
-Min
oriti
es -
1978
V
aria
ble
6 V
aria
ble
7 V
aria
ble
8 V
aria
ble
9 (c
onti
nued
)
Var
iabl
e 1
0.26
258
0.29
633
0.24
989
0.14
523
Var
iabl
e 2
0.41
968
0.46
092
0.38
972
0.27
217
Var
iabl
e 3
0.40
070
0.43
860
0.42
954
0.20
384
Var
iabl
e 4
0.28
968
0.29
054
0.32
086
0.17
507
Var
iabl
e 5
0.40
306
0.36
828
0 i 36
731
0.27
600
Var
iabl
e 6
1.00
000
0.43
582
0.43
791
0.21
562
Var
iabl
e 7
0.43
582
1 .0
0000
0.
3993
9 0.
2684
4 V
aria
ble
8 0.
437
91
0.3
9939
1.
0000
0 0.
2882
4 V
aria
ble
9 0.
2156
2 0.
2684
4 0.
2882
4 1.
0000
0 w
'-
I
Tab
le
2 (c
ontin
ued)
Raw
Cor
rela
tion
Coe
ffic
ient
Mat
rice
s
fem
ales
-
1979
V
aria
ble
l V
aria
ble
2 V
aria
ble
3 V
aria
ble
4 V
aria
ble
5
Var
iabl
e 1
1.00
000
0.52
630
0.62
289
0.48
392
0.61
808
Var
iabl
e 2
0.5
26
30
l
.000
00
0.44
510
0.47
468
0.48
454
Var
iabl
e 3
0.62
289
0.44
510
1 .0
0000
0.
4185
1 0.
4746
2 V
aria
ble
4 0.
4839
2 0.
4746
8 0.
4185
1 1.
0000
0 0.
4569
1 V
aria
ble
5 0.
6180
8 0.
4845
4 0.
4746
2 0.
4569
1 1
.000
00
Var
iabl
e 6
0.48
164
0 .4
7284
0.
4009
0 0.
5188
3 0.
3711
02
Var
iabl
e 7
0.47
549
0.54
768
0.48
603
0.44
707
0.46
873
Var
iabl
e 8
0. 5
0491
0.
3708
6 0.
4050
9 0.
4816
8 0.
4902
1 V
aria
ble
9 0.
4264
8 0.
3846
4 0.
2658
2 0
.117
95
0.33
220
Fem
ales
-19
79
Var
iabl
e 6
Var
iabl
e 7
Var
iabl
e 8
Var
iabl
e 9
(con
tinue
d)
Var
iabl
e 1
0.48
164
0.47
549
0. 5
0491
0.
4264
8 V
adab
le 2
0
.472
84
0.54
768
0.37
086
0.38
464
Var
iiabl
e 3
0.40
090
0.48
603
0.40
509
0.26
582
Var
iabl
e 4
0.51
883
0.44
707
0.48
168
0.11
795
Var
iabl
e 5
0.37
102
0.46
873
0.49
021
0.33
220
Var
iabl
e 6
l.000
00
0.45
758
0.47
82'9
0.
3008
7 V
aria
ble
7 0.
4575
8 1
.000
00
0.47
008
0.39
807
Var
iabl
e 8
0.47
829
0.47
008
1.00
000
0.35
021
Var
iabl
e 9
0.30
087
0.39
807
0.35
021
1.00
000
w
CX>
Tabl
e 2
(con
tinue
d)
Raw
Cor
rela
tion
Coe
ffic
ient
Mat
rice
s
Mal
es
-19
79
Var
iabl
e 1
Var
iabl
e 2
Var
iabl
e 3
Var
iabl
e 4
Var
iabl
e 5
,
Var
iabl
e l
1.00
000
0.51
043
0.46
716
0.44
024
0.41
813
Var
iabl
e 2
0.51
043
1.00
000
0.40
867
0.37
941
0.,3
8083
V
aria
ble
3 0
.467
16
0.40
867
1.00
000
0.42
043
0.41
873
Var
iabl
e 4
0.44
024
0.37
941
0.42
043
1 .00
000
0.40
639
Var
i ab
le 5
0.
4181
3 0.
3808
3 0.
4187
3 0.
4063
9 1.
0000
0 V
aria
ble
6 0.
4542
1 0.
4041
5 0.
4090
2 0.
3769
3 0.
3128
2 V
aria
ble
7 0.
5264
1 0.
4734
5 0.
4590
4 0.
4201
1 0.
3968
8 V
aria
ble
8 0.
3986
4 0.
3550
7 0.
3989
5 0.
4404
8 0.
3425
3 V
aria
ble
9 0.
3668
2 0.
3851
7 0.
278
17
0.24
498
0.30
127
Ma 1
es,
-19
7 9
Var
iabl
e 6
Var
iabl
e 7
Var
iabl
e 8
Var
iabl
e 9
(con
tinue
d)
Var
iabl
e 1
0.45
421
0.52
641
0.39
864
0.36
682
Var
iabl
e 2
0.40
415
0.47
345
0.35
507
0.38
517
Var
iabl
e 3
0.40
902
0.45
904
0.39
895
0.27
817
Var
iabl
e 4
0.37
693
0. 4
2011
0.
4404
8 0.
2449
8 V
aria
ble
5 0.
3128
2 0.
3968
8 0.
3425
3 0.
3012
7 V
aria
ble
6 1.
0000
0 0.
4166
7 0.
4268
6 0.
2689
4 V
aria
ble
7 0.
4166
7 1.
0000
0 0.
3496
9 0.
3535
5 V
aria
ble
8 0.
4268
6 0.
3596
9 l.0
0000
0.
2204
6 V
aria
ble
9 0.
2689
4 0.
3535
5 0.
2204
6 1.
0000
0·
w
l..O
Tab
le
2 (c
onti
nued
)
Raw
Cor
rela
tion
Coe
ffic
ient
Mat
r1ce
s
Min
orit
ies
-19
79
Var
iabl
e 1
Var
iabl
e 2
Var
iabl
e 3
Var
iabl
e 4
Var
iabl
e 5
Var
iabl
e 1
l.000
00
0.62
156
0.53
035
0.52
583
0.51
552
Var
iabl
e 2
0.62
156
1.00
000
0.46
350
0.51
488
0 .4
7981
V
aria
ble
3 0.
5303
5 0.
4635
0 1.
0000
0 0.
520
21
0.43
013
Var
iabl
e 4
0.52
583
0.51
488
0.52
021
1.00
000
0.54
492
Var
iabl
e 5
0.51
552
0.46
981
0.43
013
0.54
492
1 .00
000
Var
iabl
e 6
0.47
313
0. 5
0621
0.
4408
1 0.
5111
6 0.
3900
1 V
aria
ble
7 0.
6041
3 0.
5500
9 0.
5009
5 0.
5092
5 0.
5445
9 V
aria
ble
8 0.
4013
1 0.
4374
7 0.
4157
5 0.
4802
7 0.
3533
0 V
aria
ble
9 0.
2534
7 0.
4346
4 0.
2414
2 0.
2403
0 0.
3245
2
Min
orit
ies
-19
79
Var
iabl
e 6
Var
iabl
e 7
Var
iabl
e 8
Var
iabl
e 9
(con
tinue
d)
Var
iabl
e 1
0.47
313
0 .. 6
0413
0
.401
31
0.25
347
Var
iabl
e 2
0.50
621
0.55
009
0.43
747
0.43
464
Var
iabl
e 3
0.44
081
0.50
095
0..4
1575
0.
2414
2 ·v
aria
ble
4 0.
5111
6 0.
5092
5 0.
4802
7 0.
2403
0 V
aria
ble
5 0.
3900
1 0.
5445
9 0.
3533
0 0.
3245
2 V
aria
ble
6.
1 .00
000
0.48
523
0.52
471
0.40
351
Var
iabl
e 7
0.48
523
1.00
000
0.37
218
0.36
397
Var
iabl
e 8
0.52
471
0.37
218
1.00
000
0.37
332
Var
iabl
e 9
0.40
351
0.36
397
0.37
332
1.00
000
~
0
Tab
le
2 (c
ontin
ued)
Raw
Cor
rela
tion
Coe
ffic
ient
Mat
rice
s
Non
-Min
oriti
es -
1979
V
aria
ble
1 V
aria
ble
2 V
aria
ble
3 V
aria
ble
4 V
aria
ble
5
Var
iabl
e 1
1. 0
0000
0.
4975
5 0.
4645
9 0 .
4317
5 0.
4173
2 V
aria
ble
2 0.
4975
5 1.
0000
0 0.
3999
8 0.
3694
4 0.
3754
7 V
aria
ble
3 0.
4645
9 0.
3999
8 1 .
0000
0 0.
4105
2 0.
4187
0 V
aria
ble
4 0.
4317
5 0.
3694
4 0.
4105
2 1.
0000
0 0.
3987
0 V
aria
ble
5 0.
4173
2 0.
3754
7 0.
4187
0 0.
3987
0 1.
0000
0 V
aria
ble
6 0.
4512
2 0.
3979
6 0.
4038
8 0.
373
71
0.30
892
Var
iabl
e 7
0.51
413
0.46
653
0.45
441
0.41
231
0.39
084
Var
iabl
e 8
0.39
932
0.34
653
0.39
692
0.43
805
0.34
949
Var
iabl
e 9
0.37
557
0.38
160
0.28
189
0.,23
967
0.30
434
Non
-Min
oriti
es
-19
79
Var
iabl
e 6
Var
iabl
e 7
Var
iabl
e 8
Var
iabl
e 9
(con
tinue
d)
Var
iabl
e 1
0.45
122
0.51
413
0.39
932
0.37
557
Var
iabl
e 2
0.39
796
0.46
653
0.34
653
0.38
160
Var
iabl
e 3
0.40
388
0.45
441
0.39
692
0.28
189
Var
iabl
e 4
0.37
371
0.41
231
0.43
805
0.23
967
Var
iabl
e 5
0.30
892
0.39
084
0.34
949
0.30
434
Var
iabl
e 6
l.000
00
0.41
214
0.42
231
0.26
252
Var
iabl
e 7
0.41
214
1.00
000
0.35
319
0. 3
5720
V
aria
ble
8 0.
4223
1 0.
3531
9 l .
0000
0 0.
2220
1 V
aria
ble
9 0.
2625
2 0.
357
20
0.22
201
1 .00
000
~
........
.
Fac
tor
1
Fact
or 2
Fac
tor
3
Fact
or 4
Fact
or 1
Fact
or 2
Tab
le 3
* E
igen
valu
es a
nd V
aria
nce
Acc
ount
ed f
or 1
978-
1979
Fem
ales
-19
78
Eig
enva
lue
Var
ianc
e
4 .1
4 87
.9
.57
12. l
Fem
ales
-
1979
Eig
enva
lue
Var
ianc
e
4.10
88
.6
.53
11.4
Mal
es
-19
78
Eiig
enva
lue
Va'ri
anc
e
3.04
91
.2
.18
5.4
.12
3.5
Mal
es
-19
79
Eig
enva
lue
Var
ianc
e
3.59
.28
92.9
7. l
Min
orit
ies
-19
78
Eig
enva
lue
Var
ianc
e
3.10
66
.0
. 77
16
.3
.48
10.2
.35
7.5
Min
orit
ies
-19
79
Eig
enva
lue
Var
ianc
e
4 .1
8
.37
92.0
8.0
Non
-Min
oriti
es -
1978
Eig
enva
lue
Var
ianc
e
3.08
88
.9
.23
6.8
.14
4.3
Non
-Min
oriti
es
-19
79
Eig
enva
lue
Var
ianc
e
3.55
.28
92.7
7.3
*T
his
rep
rese
nts
the
eige
nval
ues
whe
n th
e m
inei
gm i
s se
t at
.8
rat
her
than
1.0
. V
aria
nces
are
rep
rese
nted
~
as
a pe
rcen
t of
100
.
Fact
or l
Fact
or 2
Fact
or 3
Fact
or 4
(Tot
a 1)
fact
or
1
Fac
tor
2'
(Tot
al)
Fem
ales
-
1978
Eig
enva
lue
Var
ianc
e
4.60
1.00
51. l
11. l
62.2
Fem
ales
-
1979
Eig
enva
lue
Var
ianc
e
4.58
50
.8
.92
10.2
61.0
Tabl
e 3
(Con
tinue
d)
Eig
enva
lues
and
Var
ianc
e A
ccou
nted
for
197
8-19
79
Mal
es
-19
78
Eig
enva
lue
Var
ianc
e
3.64
.86
.83
40.4
9.6
9.3
59.3
Mal
es
-19
79
Eig
enva
lue
Var
ianc
e
4 .1
5 46
. l
.88
9.8
55.9
Min
oriti
es -
1978
Eig
enva
lue
Var
ianc
e
3.50
1.13
.90 89
38.9
12.6
10.0
9.9
71.4
Min
orit
ies
-19
79
Eig
enva
lue
Var
ianc
e
4.66
51
.8
.92
10.2
62.0
Non
-Min
oriti
es -
1978
Eig
enva
lue
Var
ianc
e
3.67
~87
.82
40.8
9.7
9.1
59.6
Non~Minorities
-19
79
Eig
enva
lue
Var
ianc
e
4. l
l 45
.7
.88
9.8
55.5
.p
i.
w
Var
iabl
e 1
Var
iabl
e 2
Var
iabl
e 3
Var
iabl
e 4
Var
iabl
e 5
Var
iabl
e 6
Var
iabl
e 7
Var
iabl
e 8
Var
iabl
e 9
Tab
le
4
Obl
ique
Fac
tor
Pat
tern
Mat
rix
-19
78
Fem
ales
M
ales
fact
or l
F
acto
r 2
Fac
tor
11
Fac
tor
2 I
.59
-.1
1 .0
0 -
.03
.87
-.2
0 ,_
.04
.0
8
. 71
.08
. 77
~ .1
3
.57
.12
.37
.10
.74
.02
.45
.10
.66
.03
.35
.07
.76
.07
. la
.0
1
.63
.22
.51
.22
.06
.75
.04
.42
Fact
or 3
-.4
6
-.6
9
-.0
2
-.0
3
-.1
3
~ .2
6
-.5
0
.01
-.0
7
..r;:::. ~
Fact
or l
Var
iabl
e 1
.08
Var
iabl
e 2
.68
Var
iabl
e 3
.93
Var
iabl
e 4
.10
Var
iabl
1e 5
.5
4
Var
iabl
e 6
.07
Var
i1abl
e 7
.48
Var
iabl
e 8
-.0
5
Var
iabl
e 9
-.0
6
Tab
le 4
(C
ontin
ued}
Obl
ique
Fac
tor
Pat
tern
Mat
rix
-19
78
Min
oriti
es
Non ..
. Min
oriti
es
Fact
or 2
Fa
ctor
3
Fact
or 4
Fa
ctor
1
Fact
or 2
.07
.09
.55
.04
-.0
3
.06
-.0
5 .1
7 -
.07
.06
.06
.10
-.3
0 .7
8 -
.06
.39
.09
-.0
4 .3
9 .0
4
-.1
1 .1
0 .1
9 .4
7 .1
1
.09
.48
-.0
3 .3
6 .0
0
.05
-.O
D .0
2 .2
0 .0
4
-.0
5 .8
8 .0
7 .so
.1
5
.93
-.0
5 .0
8 .0
2 .6
0
Fact
or 3
-.4
3
-,. 7
3
,.051
-.0
6
-. l
0
-. 3
1
-.4
6
-.0
6
.00
~
(..J
l
Tabl
e 4
(Con
tinue
d)
Obl
ique
Fac
tor
Pat
tern
Mat
rix -
1979
emal
es
Fact
or l
Fa
ctor
2
Fact
or 1
Var
iabl
e l
.80
.02
.74
Var
iabl
e 2
.68
-.0
5 .6
9
Var
iabl
e 3
.65
-.0
4 .6
4
Var
iabl
e 4
.44
-. 7
1 .6
0
Var
iabl
e 5
.69
-.0
3 .5
8
Var
iabl
e 6
.58
-.1
9 .5
9
Var
iabl
e 7
. 70
-.0
2 .7
0
Var
iabl
e 8
.62
-.1
1 .5
5
Var
iabl
e 9
.61
.30
.51
Mal
1es
Fact
or 2
.07
.16
-.1
0
-.2
2
-.0
3
-.1
2
.10
-• 3
4
.23
.p.
O"I
Var
iabl
e 1
Var
iabl
e 2'
Var
iabl
e 3
Var
iabl
e 4
Var
iabl
e 5i
Var
iiabl
e 6
Var
iabl
e 7'
Var
iabl
e 8
Var
iabl
e 9
Tabl
e 4
(Con
tinue
d)
Obl
ique
Fa
ctor
Pat
tern
Mat
rix
-19
79
Min
orit
ies
Non
-Min
oriti
es
Fac
tor
1 Fa
ctor
.2
Fac
tor
1 F
acto
r 2
.89
-. 1
3 .7
1 . 1
2
.. 52
.26
.63
. 21
.62
.06
.66
-.0
4
I 62
.1
4 .6
5 -
.18
.68
.00
.59
.01
.15
.63
.62
-.0
7
.73
.03
,67
• 14
.05
.65
.64
-.2
9
-.0
4 .5
9 .4
5 .2
'9
..p..
-.....J
I
Var
iabl
e l
Var
iabl
e 2
Var
iabl
e 3
Var
iabl
e 4
Var
iabl
e 5
Var
iabl
e 6
Var
iabl
e 7
Var
iabl
e 8
Var
iabl
e 9
Tab
le 4
(C
ontin
ued)
Var
imax
Rot
ated
Fa
ctor
Mat
rix -
1978
Fem
ales
Fact
or l
Fa
ctor
2
Fact
or 1
.56
.04
.20
.82
.02
.29
.69
.26
.64
.56
.27
.36
.72
.21
.46
.64
.20
.43
.73
.27
.38
.63
.39
.48
. 12
.76
.14
Mal
es
Fact
or 2
.37
.56
. 31
.20
. 31
.38
.48
.26
.18
Fact
or 3
.14
.32
. 12
.22
.28
.25
.23
.36
.45
~
00
fact
or l
Var
iabl
e 1
.15
Var
iabl
e 2
.63
Var
iabl
e 3
.88
Var
iabl
e 4
.19
Var
iabl
e 5
.52
Var
iabl
e 6
.24
Var
iabl
e 7
.45
Var
iabl
e 8
.25
Var
iabl
e 9
.12
ble
4 (c
ontin
ued)
Var
imax
Rot
ated
fac
tor
Mat
rix -
197B
Min
orit
ies
Non
-Min
oriti
es
!Fac
tor
2 Fa
ctor
3
Fact
or 4
F
acto
r l
Fact
or 2
.10
.16
.59
.23
.37
.20
.17
.31
.30
.60
.27
.33
~ .0
7 .6
3 .2
9
.41
.15
.03
.38
.24
.02
.25
.32
.48
.32
.18
.46
.09
.45
.42
.. 15
,.14
. 12
.40
.49
.08
.78
.23
.49
.31
.89
.05
.14
.15
.15
Fac
tor
3
.10
.24
.12
.15
.25
.17
.21
.28
.57
..po ~
Var
iabl
e 1
Var
iabl
e 2
Var
iabl
e 3
Var
iabl
e 4
Var
iabl
e 5
Var
iabl
e 6
Var
iabl
e 7
Var
iabl
e 8
Var
iabl
e 9
Tab
le 4
(co
ntin
ued)
Var
imax
Rot
ated
Fa
ctor
Mat
rix
-19
79
Fem
ales
M
ales
acto
r 1
Fact
or 2
F
acto
r 1
Fac
tor
2
.69
.41
.44
.59
.56
.41
.34
.60
.54
.39
. 51
.41
.16
.91
.58
.31
.57
.40
.42
.41
.43
.49
.50
.37
.59
.39
.40
.58
.49
.44
.64
.20
.61
.03
.16
.51
Ul
0
Var
iabl
e l
Var
iabl
e 2
Var
iabl
e 3
Var
iabl
e 4
Var
iabl
e 5
Var
iabl
e 6
Var
iabl
e 7
Var
iabl
e 8
Var
iabl
e 9
Tab
le 4
(co
ntin
ued)
Var
imax
Rot
ated
Fac
tor
Mat
rix
-19
79
Min
orit
ies
Non
-Min
oriti
es
fact
or
1 Fa
ctor
2
Fact
or
1 Fa
ctor
2
.76
.26
.45
.58
.58
.47
.34
.59
.58
.32
.51
.41
.61
.38
.59
.29
.62
.29
.43
.40
.39
.64
.50
.36
.67
.34
.41
.56
. 31
.61
.64
.20
.20
.53
.16
.53
U1
2.
Ver
sati
lity
5.
Con
vnun
icat
ions
3.
Org
aniz
atio
nal
Eff
ecti
vene
ss
6.
Sta
bil
ity
8.
Inte
rper
sona
l Re
l, at i
ons
4.
Eff
ecti
vity
with
E
xter
nal
Gro
ups
9.
Wor
k-R
elat
ed
Self-
Dev
elop
men
t
Tab
le
5
Hig
hest
Loa
ding
s of
Nin
e A
ppra
isal
V
aria
bles
on
Obl
ique
Fa
ctor
s
Obl
ique
Fa
ctor
s
Fem
ales
-
1978
M
ales
-
1978
Fact
or 1
Fa
ctor
2
Fact
or 1
Fa
ctor
2
.87
3.
Org
aniz
atio
nal
Eff
ecti
vene
ss
• 77
.74
9.
Wor
k-R
elat
ed
Self
-Dev
elop
men
t .4
2 .7
1 2.
V
ersa
ti 1
ity
.6
6 7.
In
itia
tiv
e
.63
.57
.75
Fact
or 3
-.6
9
-.5
0
U1
I"\.
)
3.
Org
aniz
atio
nal
Eff
ecti
vene
ss
2.
Ver
sati
lity
5.
Com
mun
icat
ions
9.
Wor
k-R
elat
ed
Sel
f-D
evel
opm
ent
8.
Inte
rper
sona
l R
elat
ions
l. K
ey E
lem
ents
able
5 (
cont
inue
d)
Hig
hest
Loa
ding
s o·f
N
ine
App
rais
al
Vari
able
s on
Obl
ique
Fac
tors
Obl
ique
F
acto
rs
Min
orit
ies
-19
78
Non
-Min
oriti
es
-19
78
Fact
or 1
Fa
ctor
2
Fact
or 3
F
acto
r 4
Fact
or 1
Fa
ctor
2
Fact
or 3
3.
Org
aniz
atio
nal
.93
Eff
ecti
vene
ss
.78
.68
8.
Inte
rper
sona
1
Rel
atio
ns
.50
.54
9. Work~Related
Self-
Dev
elop
men
t .6
0 .9
3 2.
V
ersa
tili
ty
-.7
3
.88
.55
Ul
w
Tab
le 5
(co
ntin
ued)
Hig
hest
Loa
ding
s of
N
ine
App
rais
al
Var
iabl
es o
n O
bliq
ue F
acto
rs
Obl
ique
Fa
ctor
s
Fem
ales
-
1979
M
ales
-
1979
Fact
or l
Fa
ctor
2
Fac
tor
l Fa
ctor
2
l. K
ey E
lem
ents
.8
0 1.
Ke
y El
emen
ts
.74
7.
Init
iati
ve
.70
7.
Init
iati
ve
.70
5.
Com
mun
icat
ions
.6
9 2.
V
ersa
tili
ty
.69
2.
Ver
sati
lity
.6
8 3.
Org
aniz
atio
nal
Eff
ecti
vene
ss
.64
3.
Org
aniz
atio
nal
Eff
ecti
vene
ss
.65
4.
Eff
ecti
vity
with
E
xter
nal
Gro
ups
.60
8.
Inte
rper
sona
l R
elat
ions
.6
2 6.
S
tab
ilit
y
.59
9.
Wor
k-R
elat
ed
5.
Com
mun
icat
ions
.5
8 Se
lf-D
evel
opm
ent
.61
8.
Inte
rper
sona
l 4.
E
ffec
tivi
ty w
ith
Rel
atio
ns
.55
Ext
erna
l G
roup
s -·
. 71
9.
Wor
k-R
elat
ed
Self-
Dev
elop
men
t . 5
1 U
l +::
-
8.
Inte
rper
sona
l -
.34
Rel
atio
ns
1.
Key
Elem
ents
7.
Init
iati
ve
5.
CofT
ITlu
nica
tions
3.
Org
aniz
atio
nal
Eff
ecti
vene
ss
4.
Eff
ecti
vity
with
E
xter
nal
Gro
ups
2.
Ver
sati
lity
8.
Inte
rper
sona
l Re
1 at
ions
6.
Sta
bil
ity
9.
Wor
k-R
elat
ed
Tab
le
5 (c
ontin
ued)
Hig
hest
Loa
ding
s of
Nin
e A
ppra
isal
V
aria
bles
on
Obl
ique
Fa
ctor
s
Obl
ique
Fac
tors
Min
orit
ies
-19
79
Fac
tor
l F
acto
r 2
.89
1 .
Key
Ele
men
ts
.73
7.
Init
iati
ve
.68
3.
Org
aniz
atio
nal
Eff
ecti
vene
ss
.62
4.
Eff
ecti
vity
with
E
xter
nal
Gro
ups
.62
8.
Inte
rper
sona
l R
elat
ions
.5
3 2.
V
ersa
tili
ty
.65
6.
Sta b
i 1 it
y
.63
5.
Com
mun
icat
ions
8.
Inte
rper
sona
l Se
lf-D
evel
opm
ent
.59
Rel
atio
ns
9.
Wor
k-R
elat
ed
Self-
Dev
elop
men
t
Non
-Min
oriti
es
-19
79
Fact
or 1
Fa
ctor
2
. 71
.67
.66
.65
.64
.63
.62
.59
-.2
9 U
1
Ul
.29
Tab
le
5 (c
onti
nued
)
Hig
hest
Loa
ding
s of
Nin
e A
ppra
isal
V
aria
bles
on
Orth
ogon
al
Fac
tors
Ort
hogo
nally
Rot
ated
Fac
tors
Fem
ales
-
1978
M
ales
-
1978
Fac
tor
1 Fa
ctor
2
Fac
tor
1 F
acto
r 2
2.
Ver
sati
lity
.8
2 3.
O
rgan
izat
iona
l E
ffec
tive
ness
.6
4 7.
In
itia
tiv
e .7
3 2.
V
ersa
tili
ty
.56
5.
Cor
rmun
icat
ions
. 7
2 9.
W
ork-
Rel
ated
3.
O
rgan
izat
iona
l Se
lf-D
evel
opm
ent
Eff
ecti
vene
ss
.69
6.
Sta
bi 1
i ty
.6
4
B.
I nte
rper
sona
1
Rel
atio
ns
.63
1 . K
ey
Ele
men
ts
.56
4.
Eff
ecti
vity
with
E
xter
nal
Gro
ups
.50
9.
Wor
k-R
elat
ed
Self-
Dev
elop
men
t .7
6
Fact
or 3
.45
U1
C
J)
3.
Org
aniz
atio
nal
Eff
ecti
vene
ss
2.
Ver
sati
lity
5.
Com
mun
icat
ions
9.
Wor
k-R
elat
ed
Tab
le 5
(co
ntin
ued)
Hig
hest
Loa
ding
s of
Nin
e A
ppra
isal
V
aria
bles
on
Orth
ogon
al
Fact
ors
Ort
hogo
nally
Rot
ated
Fac
tors
Min
orit
ies
-19
78
Non
-Min
oriti
es -
1978
Fac
tor
1 F
acto
r 2
Fact
or 3
Fa
ctor
4
Fac
tor
1 Fa
ctor
2
Fact
or 3
3.
Org
aniz
atio
nal
.88
Eff
ecti
vene
ss
.63
.63
2.
Ver
sati
lity
.6
0
.52
9.
Wor
k-R
elat
ed
Self-
Dev
elop
men
t .5
7
Self
-Dev
elop
men
t ~B
9
8.
Inte
rper
sona
1
Rel
atio
ns
.78
1.
Key
Ele
men
ts
.59
U1
.....
..,,
able
5 (
cont
inue
d)
Hig
hest
Loa
din
gs
of N
ine
App
rais
al
Var
iabl
es o
n O
rthog
onal
F
acto
rs
Ort
hogo
nally
Rot
ated
Fac
tors
Fem
ales
-
1979
Fact
or l
Fa
ctor
2
1 .
Key
Ele
men
ts
.69
8.
I nte
rper
sona
1
Rel
atio
ns
9.
Wor
k-R
elat
ed
Self
-Dev
elop
men
t .6
1 5.
E
ffec
tivi
ty w
ith
Ext
erna
l G
roup
s 7.
In
itia
tiv
e
.59
3.
Org
aniz
atio
nal
5.
Com
mun
icat
ions
.5
7 E
ffec
tive
ness
2.
Ver
sa ti
1 it
y
.56
2.
Ver
sati
lity
3.
Org
aniz
atio
nal
1.
Key
Ele
men
ts
Eff
ecti
vene
ss
.54
7.
Init
iati
ve
4.
Eff
ecti
vity
with
E
xter
nal
Gro
ups
• 91
Mal
es
-19
79
Fact
or 1
Fa
ctor
2
.64
.'58
.51
.60
.59
.58
U1
co
l. K
ey E
lem
ents
7.
Init
iati
ve
5.
Com
nuni
catio
ns
4.
Eff
ecti
vity
with
E
xter
nal
Gro
ups
2.
V'e
rsat
tl it
y
3.
Org
aniz
atio
nal
Eff
ecti
ven
ess
6.
Sta
bil
ity
8.
Inte
rper
sona
l R
elat
ions
Tab
le 5
(co
ntin
ued)
Hig
hest
Loa
ding
s of
Nin
e A
ppra
isal
V
aria
bles
on
Orth
ogon
al
Fact
ors
Ort
hoao
nally
Rot
ated
Fac
tors
Min
orit
ies
-19
79
Non
-Min
oriti
es
-19
79
Fact
or 1
Fa
ctor
2
Fact
or 1
Fa
ctor
2
.76
8.
I nte
rper
sona
1
Rel
atio
ns
.64
.67
4.
Eff
ecti
vity
with
.6
2 E
xter
nal
Gro
ups
.59
3.
Org
aniz
atio
nal
.61
Eff
ecti
vene
ss
. 51
.58
6.
Sta
bil
ity
.5
0
2.
Ver
sati
lity
.5
9 .5
8 1
. K
ey
Ele
men
ts
.58
.64
7.
Init
iati
ve
.56
.61
9.
Wor
k-R
elat
ed S
elf-
Dev
elop
men
t .5
3 U
l l.D
Dim
ensi
on
1.
Key
Elem
ents
2.
Ver
s,at i
l i t
y
3.
Org
aniz
atio
nal
Eff
ecti
vene
ss
4.
Eff
ecti
vity
with
E
xter
nal
Gro
ups
5.
Cor
rmun
icat
ions
6.
Sta
bil
ity
7.
Init
iati
ve
8.
Inte
rper
sona
l R
elat
ions
9.
Wor
k-R
elat
ed
Self-
Dev
elop
men
t
igur
e 1
App
rais
al
Dim
ensi
ons
and
Def
init
ions
efin
itio
n
Key
job
elem
ents
; th
e m
ajor
out
put
area
s of
you
r jo
b.
Perf
orm
ing
dif
fere
nt
task
s w
ithi
n yo
ur f
ield
; br
eadt
h an
d de
pth
in
a va
riet
y of
dif
fere
nt
area
s ou
tsid
e yo
ur a
rea.
Plan
ning
and
con
trol
ling
ava
ilab
le r
esou
rces
to
ach
ieve
re
quir
ed
resu
lts;
est
abli
shin
g ef
fect
ive
and
appr
opri
ate
follo
w-u
p;
usin
g th
e 11sy
stem
11
to a
chie
ve
resu
lts.
Inte
ract
ing
with
cus
tom
ers,
vendor~
, su
bcon
trac
tors
, ,an
d ot
her
outs
ide
grou
ps
to
achi
eve
requ
ired
res
ult
s.
Exp
ress
ion
in b
oth
oral
an
d w
ritt
en c
omm
unic
atio
n;
appr
opri
ate
use
of
lang
uage
.
Perf
orm
ing
effe
ctiv
ely
unde
r pr
essu
re.
Rec
ogni
zing
th
e ne
ed f
or a
ctio
ns a
nd
proc
eedi
ng
in a
res
pons
ible
m
anne
r; co
ntin
uing
to
purs
ue n
ew s
olut
ions
in
a pr
oble
m s
itua
tion
.
Ach
ievi
ng
team
wor
k w
ith a
gro
up;
gett
ing
alon
g w
ith a
nd u
nder
stan
ding
1 ot
hers
; aw
aren
ess
and
cons
ider
atio
n of
oth
er v
iew
poin
ts.
Sig
nifi
cant
tec
hnic
al
lear
ning
thr
ough
se
lf s
tu~y
an
d at
tend
ance
at
cour
ses
and
conf
eren
ces.
m
0
Figu
re 2
Pot
enti
al
Impl
icat
ions
of
Nin
e P
oten
tial
S
itua
tion
s In
volv
inq
Perf
orm
ance
App
rais
al
Mea
n Sc
ores
an
d Em
ergi
ng F
acto
rs
Mea
n Sc
ore
Dif
fere
nces
N
umbe
r of
Em
ergi
ng F
acto
rs
Equa
l
Low
er p
rote
cted
cl
ass
rati
ngs
Hig
her
prot
ecte
d cl
ass
rati
ngs
Equa
l
A -
ok
B -
ok
A -
not
ok
B -
ok
A -
ok
B -
ok
A =
Ant
icip
ated
rea
ctio
n of
EEO
C
Few
er i
n Mo
~e
in
rotect~d
Cla
ss G
roup
P
rote
cted
Cla
ss G
roup
A -
ok
A -
ok
B -
not
ok
B -
not
ok
A -
not
ok
-no
t ok
A -
ok
B -
not
ok
A -
not
ok
B -
not
ok
A -
ok
B -
not
ok
B =
Ant
icip
ated
rea
ctio
n of
Per
form
ance
App
rais
al
Des
igne
r an
d A
dmin
istr
ator
s
0)
__.
62
REFERENCES
Albermarle Paper Co v. Mo,ody. 422, U .. S. 405 (1975).
American Psychological Association, American Educationa1 Rese·arch Association, National Council on Me,asurement in Education. Standards for educational and psychological tests. Washington, 0 C.: American Psychological Association, 1974.
she, l . R .• Jr. How do your performance appraisals perform?' EEO oday, 1980, L (3), 216-222.
Bartlett, C. J., Bobko, P., Hannan, R. L., & Mosier, S. B. Testin~g for test fairness with a moderated multiple regression strategry:
n alternative to differential analysis. Paper presented at Pe,rsonnel Selection Research Conference, Unive,rsity of Maryland, Co 11 ege Park, aryl and, March 3-4, 1977.
Bass, A. R., & urner, J. Ethnic group differences in relat·onships among criteria of job perfonnance. Journal of Applied Psychology,
973' _E, 101-109
Bobko, P., & Bartlett, C. J. Subgroup validities: differential definitions and differential prediction. Journal of Applied Psycholoay, 1978, 63 (1),, 12-1.
Boehmt V. R. egro-white differences in validity of employment and training selection procedures: sU11mnary of research evidence. Journa 1 of App 111 ed Psycho l 09'Y, 1972, i§. ( 1); 33-39.
Boehm, v. R. Differential prediction: a methodological artifact? Journal of Appl1i'ed Psychology,, 1977, 62 (2), 1:46-154.
Boehm, V. R. Populations, preselect1on, and practicalities: a reply t ,o Hunter and Schm11dt.. Journal of Applied Psychology, 1978, 63 (11 ), 15-18.
Bray, O. W. & Moses, J. L. Personnel selection. Psychology, 1972, 23, 545-576.
Brito v. Zia Co 478 F.2d 1200 (10th Cir. 1973). App 1 i ed Ts ye ho 11 ogy in pe·rsonne 11 management., Reston~ 978.
Annual Review of
In Cascio, W.F. Res ton, Va.:
Brown, R. C. Sources of bias in the prediction of job performance: Implications fo1r future research. In L. A. ~ro~ks (Ed:), An investiaation of sources of bias in the red1ct1on of Job perfonnance: si£ year study Proceedin,gs of an invitational
conference). Princeton, N.J.: Educational Testing Service, 1972.
63
Campbell, J. T. A regression analysis of test bias: predicting job knowledge scores from an aptitude battery. Symposium on Test and Job Performance of Negros and Whites, American Psychological Association Convention, September 2, 1969.
Casc·o, . F., & Bernardin, H.J. Personnel Psychology, 1981, 34, 211-227.
Cleverly v. Western Electric Co. 594 F.2d 638 (8th Cir. 1979).
Distefano . K., Jr., Pryer, M. • , & Craig, S. H. Job relatedness of posttraining job knowledge criterion used to assess validity and test fairness. Persanne 1 Psycho 1 oay, 1980, 33 ,, 785- 793.
Dona 1 dson v. Pi 11 sbury Co. 554 F. 2d 825 (8th Cir. 1977).
Ounnette, . 0. A modified model for test validation and selecti on research . Journal of Applied Psychology, 1963, 47, 317-323.
Dunnette, . D. Personnel selection and job placement of disadvantaged and minor1ty persons: Problems, issues, and suggestions. H. L. Fromkin and J . J. Sherwood (Eds.) Integrating the, Organization.
ew York: The Free Press, 1974.
E.E . O.C. v. Radiator Spec"alty Co. 610 F.2d 178 (4th Cir. 1979).
F·ncher, C. Differential validity and test bias. Personnel Psychology, 1975, 28, 81-500.
Flaugher, R. L., & orris, L. Ethnic group membership as a moderator of supervisory ratings. Symposium on Test and Job Performance of Negros and Whites, American Psychological Association Convention, September 2, 969.
Fox, H., & Lefkowitz. Differential validity: moderator in predicting job performance. 1974, 27 209-223.
ethnic group as a Personnel Psychology,
Frederiksen, N., & Melville, S D. Differential predictability in the use of test scores. Educational and Psychological Measurement, 1954, ]i, 647-656
Gaylord, R. H., & Carrol, J. B. A general approach to the problem of the population control variable. American Psychologist, 1948, 1_, 310 (Astract).
Ghiselli, E. E. Differentiation of individuals in terms of their pr1edictability. Journal of Applied Psychology, 1956, 40,. 374-377.
Gi1rnor 1e v. Kansas City Terminal Railway Co. 509 F.2d 48 (8th Cir. 197 5) .
64
Gorsuch, R. L. Factor Analysis. Philadelphia: W. B. Saunders, 1974.
Griggs v. Duke Power Co. 401 U.S. 424 (1970).
Grooms, R. R .. , & Endler, '.. S. The effect of anxiety of achievement. Journal of Educational Psychology, 1960, .§]_, 299-304.
Gross, . l , & Su, W. Defining a 1 fair 11 or "unbiased" selection model: A question of utilities. Journal of Applied Psychology, 1976, 60, 345-351
Hannon, H. H. odern factor ana1~1s. Chicago: Univer·sity of Chicago Pres,s, 196
Holley . H , & F·eld H. S Performa nce appraisal and the law. Labor Law Journal, July 1975, 423-429.
Holley • H., Field, H.S., & Barnett, N. J. Personnel Journal, September 1976, 457-463
unter, J. E., & Schmidt, F L ~ Differential and single-group validity of employment tests by race: a critical analysis of three recent studies. Journal of Applied Psychology, 1978, g (1), 1-11
unter J . E., Schmidt, F. L., & Hunter, R. Differential validity of employment tests by race: A comprehensive review and analysis. Psychological Bulletin 1979, 86 (4}, 721-7'35.
James, L. R. Criterion models and construct validity for criteria. Psychologica Bulletin, 1973, ~O, 75-83.
James v. Stockholm Valves & Fittings Co. 559 F.2d 310 (5th Cir. 1977).
Johnson, C. a. The population control variable or moderator variable in personnel research. In Tri-Service Conference on Selection Research. ashington, D.C.: Office of Naval Research, 1960, 125-134.
Katzell, R. A., & Dyer, F. J. Differential validity revived. Journal of Applied Psychology, 1977, 62 (2), 137-145.
Katzell, R. A., & Dyer, F. J. On differential validity and bias. Journal of Appl,ied Psycholo1gy ., 1978, §1 (1 ), 19-21.
Kavanagh, M. J., MacKinney, A. C., & Wolins, l. Issues in managerial performance: Multitrait - multimethod analyses of ratings. Psychological Bulletin, 1971, 2§_, 34-49. ·
Klassen, C. R., Thompson, D. E., & Luben, G. L. How defensible is your performance appraisal system? Personnel Administrator, December 1980, 77-83.
65
Kleiman, L. S., & Durham, R. L. Performance appraisal, promotion and the courts: A critical r eview. Personnel Psychology, 1981, 34, 103-121.
Kirchner, W. K. Some questions about "differential validity": Ethnic group as a moderator in predicting job performance. Personnel Psycho 1 ogy, 1975, 28, 341-343.
Kirpatrick, J. J., Ewen, R. 8., Barrett, R. S., & Katzell, R. A. Testing and fair employment; Fairness and validity of personnel tests for different ethnic groups. New York: New York University Free Press, 1968.
Lefkowitz, J. Differential validity: Ethnic group as a moderator in predicting tenure. Personne 1 Psychology,, 1972, 25, 223-240.
Lefkowitz, J ., & Fox, H Some answers to "some questions about 'differential validity: Ethnic group as a moderator in predicting job performance. I II Personnel Psychology' 1975' r 28, 345-349.
L nn, R. l. Single group validity, differential validity, and differential prediction. Journal of A~plied Psychology, 1978, 63 ( ), SQJ ... 512
Locher A.H., & Teel, K current practices.
S. Performance appraisal - A survey of Personnel Journal, 1977, 56, 245-247.
Locke, E. A., ento, A.~., & Katcher, B. L. The interaction of ability and motivation in perfonnance: an exploration of the meaning of moder a tors. Personnel Psycho 1 oqy, 1978, ll_, 269-280.
Lopez, F~ M. Current problems in test performance of job applicants. Personnel Psychology, 1966 ., ~, l 0-17.
Management Performance Appraisal Programs. Washington: The Bureau of National Affairs, 1974, 2.
Marquez v. Omaha District Sales Office, Ford Division of Ford Motor Co. 440 F.2d 1157 (8th Cir. 1971).
Meyer v. Missouri State Highway Commission. 567 F.2d 804 (8th Cir. 11 977) .
Nie, N. H. Hull, C. H., Jenkins, J. G., Steinbrenner, K. & Bent, D. H. Statistical Package for the Social Sciences (2nd Ed.), New York New York: ~cGraw Hill, Inc.
66
O'Leary, B., S. ~ Farr, J. L, & Bartlett, C. J. Ethnic qr,oup membership as a moderator of job performance. Technical Report No. l. Washington, D.C.: American Institute for Research, April 1970.
Patterson v. American Tobacco Co. 535 F.2d 275 (4th Cir. 1976). Also 586 F.2d 300 (4th Cir. 1978).
Robinson v. Union Carbide Corp. 538 F.2d 652 (5th Cir. 1976).
Rock D. A., Campbell, J. T., & Evans, F. R. Prediction of job performance for negro and white medical technicians (Technical R~port PR-70-71). Princeton, N.J.: Educational Testing Service, 1970.
Rowe v. General Motors Corp. 457 F.2d 348 (5th Cir. 1972).
Ruda E. & Albright, L. E. Racial difference on selection instruments related to subsequent job performance. Personnel Psychology, 1 968 ,, .?.J.., 31 - 1 .,
Rummell, R. J. Appl1 i'ed factor analysis. Evanston: Northwestern University Press, 1970.
Sauf\ders D R. The 11moderator variabl 1e 11 as a useful tool in prediction. Proceedings of the 1954 ·nvi'tational conference on testing problems. Princeton, N.J.: Educational Testing Service, 1955.
Schmidt, F. L , Berner, J . C., & Hunter, J. E. Racial differences in val 1di y in employment tests: Rea 1 i ty or i 11 usi on? Journa 1 of Applied Psychology, 1973,, 58, 5,_g_
Schmidt , Coyle, B. W & Mellon, P. M. Subgroup differences in predictor and criterion variances and differential validity. Journal of Applied Psychology, 1978, 63 (6), 667-672.
Sledge v. J. P. Stevens & Co. 585 F.2d 625 (4th Cir. 1978).
Toops H. A. A research utopia in industrial psychology. Personnel Psychology, 1959 ]1, 189-225.
Thurstone, L Multiple factor analysis. Chicago: University of Chicago Press, 1947.
Uniform Gu1de1 ines on Employee Selection Procedur1es. 43, Federal Register. 38290-38315 (1978).
U.S.A. v. City of Chicago. 573 F.2d 416 (7th Cir. 1978).
Watkins v. Scott Paper Co. 503 F.2d 159 (5th Cir. 1976).
Weahkee v. Perry. 587 F.2d 1256 (D.C. 1978).
67
Winstanley, . B. Legal and ethical issues in performance appraisals. Harvard Business Review, November/December 1980.
Wallace, S. R. Sources of bias in the prediction of job performance : Implications for future research. In L.A. Crooks (Ed.) An investi ation of sources of bias in the rediction of job erfonnance: A six ear stud' Proceedings of an invHational
rinceton, Educational Testing Service, 1972 .
ollowick, H. B , Greenwood, J. M., & McNamara, W. J. Psychological testing with a minority group population. Symposium on Test and Job Perfonnance of Negros and Whites, American Psychological Association Conference, September 2, 1969.
