The Effects of Behaviorally Anchored Rating Scales as ...

Western Michigan University Western Michigan University

ScholarWorks at WMU ScholarWorks at WMU

Master's Theses Graduate College

12-1982

The Effects of Behaviorally Anchored Rating Scales as The Effects of Behaviorally Anchored Rating Scales as

Antecedent and Consequent Stimuli upon Instructional Behaviors Antecedent and Consequent Stimuli upon Instructional Behaviors

in a Special Education Setting in a Special Education Setting

Dennis L. Van Hartesvelt

Follow this and additional works at: https://scholarworks.wmich.edu/masters_theses

Part of the Industrial and Organizational Psychology Commons

Recommended Citation Recommended Citation Van Hartesvelt, Dennis L., "The Effects of Behaviorally Anchored Rating Scales as Antecedent and Consequent Stimuli upon Instructional Behaviors in a Special Education Setting" (1982). Master's Theses. 1748. https://scholarworks.wmich.edu/masters_theses/1748

This Masters Thesis-Open Access is brought to you for free and open access by the Graduate College at ScholarWorks at WMU. It has been accepted for inclusion in Master's Theses by an authorized administrator of ScholarWorks at WMU. For more information, please contact [email protected].

http://scholarworks.wmich.edu/


https://scholarworks.wmich.edu/

https://scholarworks.wmich.edu/masters_theses

https://scholarworks.wmich.edu/grad

https://scholarworks.wmich.edu/masters_theses?utm_source=scholarworks.wmich.edu%2Fmasters_theses%2F1748&utm_medium=PDF&utm_campaign=PDFCoverPages

http://network.bepress.com/hgg/discipline/412?utm_source=scholarworks.wmich.edu%2Fmasters_theses%2F1748&utm_medium=PDF&utm_campaign=PDFCoverPages

https://scholarworks.wmich.edu/masters_theses/1748?utm_source=scholarworks.wmich.edu%2Fmasters_theses%2F1748&utm_medium=PDF&utm_campaign=PDFCoverPages

mailto:[email protected]



THE EFFECTS OF BEHAVIORALLY ANCHORED RATING SCALES AS ANTECEDENT AND CONSEQUENT STIMULI UPON INSTRUCTIONAL

BEHAVIORS IN A SPECIAL EDUCATION SETTING

by

Dennis L. Van H a rte sv e lt

A Thesis Submitted to the

F acu lty o f The Graduate C ollege in p a r t i a l fu lf i l lm e n t o f the

requirem ents fo r the Degree of M aster o f A rts Department o f Psychology

W estern Michigan U niversity Kalamazoo, Michigan

December 1982

Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.

THE EFFECTS OF BEHAVIORALLY ANCHORED RATING SCALES AS ANTECEDENT AND CONSEQUENT STIMULI UPON INSTRUCTIONAL

BEHAVIORS IN A SPECIAL EDUCATION SETTING

Dennis L. Van H a rte sv e lt, M. A.

Western Michigan U n iv ers ity , 1982

The p resen t study was an attem pt to determ ine the e f fe c ts of the

B ehaviorally Anchored R ating Scale (BARS) upon th ree d i s t in c t in s tru c

t io n a l behaviors when p resen ted befo re In s tru c tio n a l sessio n s and when

p resen ted a f te r In s tru c t io n a l se ss io n s . A m u ltip le b a se lin e design

across behaviors was used w ith fou r su b jec ts w ith a re v e rs a l element

used w ith one of those su b je c ts . A s l ig h t p o s it iv e e f f e c t of BARS

p re sen ta tio n upon in s tru c t io n a l behaviors occurred. No s ig n if ic a n t

d iffe ren ces were observed between p re in s tru e tio n a l and p o s tin s tru c tio n a l

BARS p re se n ta tio n . The re levance of th e p resen t study i s explored in

r e la t io n to previous BARS s tu d ie s , along w ith the im p lica tions of the

p re sen t study.


ACKNOWLEDGEMENTS

There a re sev e ra l persons whose a id and cooperation were e s s e n tia l

to the completion of t h i s th e s is . I would f i r s t l ik e to thank the

ad m in is tra tio n a t the Croyden Avenue School o f the Kalamazoo V alley

In term ed ia te School D is t r ic t fo r providing th e research f a c i l i t i e s for

th i s p ro je c t. I would a lso l ik e to thank Joan Lundahl, Michele

Polver-Rueber, and Karin Uhlman fo r providing s ta f f and s tu den ts as

w e ll as Wayne Fuqua, Ph. D ., and Norman Pe te rson , Ph. D ., fo r providing

me w ith academic guidance which s e t the s tag e fo r th is th e s is . And

f in a l ly , my most s in ce re thanks a re o ffe red to Dale Brethower, Ph. D .,

who led me through the confusing and d i f f i c u l t ta sk of tu rn ing a few

id eas find academic preparedness in to a major l i t e r a r y work. W ithout

th i s h e lp , th i s ta sk would no t have been accomplished.

Dennis L. Van H arte sv e lt

i i


INFORMATION TO USERS

This reproduction was made from a copy of a document sent to us for microfilming. While the most advanced technology has been used to photograph and reproduce this document, the quality of the reproduction is heavily dependent upon the quality o f the material submitted.

The following explanation o f techniques is provided to help clarify markings or notations which may appear on this reproduction.

1. The sign or “ target” for pages apparently lacking from the document photographed is “Missing Page(s)” . If it was possible to obtain the missing page(s) or section, they are spliced into the film along with adjacent pages. This may have necessitated cutting through an image and duplicating adjacent pages to assure complete continuity.

2. When an image on the film is obliterated with a round black mark, it is an indication o f either blurred copy because of movement during exposure, duplicate copy, or copyrighted materials that should not have been filmed. For blurred pages, a good image o f the page can be found in the adjacent frame. If copyrighted materials were deleted, a target note will appear listing the pages in the adjacent frame.

3. When a map, drawing or chart, etc., is part of the material being photographed, a definite method o f “sectioning” the material has been followed. It is customary to begin filming at the upper left hand comer o f a large sheet and to continue from left to right in equal sections with small overlaps. If necessary, sectioning is continued again—beginning below the first row and continuing on until complete.

4. For illustrations that cannot be satisfactorily reproduced by xerographic means, photographic prints can be purchased at additional cost and inserted into your xerographic copy. These prints are available upon request from the Dissertations Customer Services Department.

5. Some pages in any document may have indistinct print. In all cases the best available copy has been filmed.

UniversityMicrofilms

Internationa)300 N. Zeeb Road Ann Arbor. Ml 48106


1320047

VAN HARTESVELT, DENNIS LEE

THE EFFECTS OF BEHAVIORALLY ANCHORED RATING SCALES AS ANTECEDENT AND CONSEQUENT STIMULI UPON INSTRUCTIONAL BEHAVIORS IN A SPECIAL EDUCATION SETTING

WESTERN MICHIGAN UNIVERSITY M.A. 1982

University Microfilms

International 300 N. Zeeb Road. Ann Arbor. MI 48106


TABLE OF CONTENTS

ACKNOWLEDGEMENTS . . . 11

LIST OF FIGURES . ................................. . iv

INTRODUCTION . . . . . . . . . .................... 1

METHOD . . . . . . . . . . 16

S u b j e c t s ........................ . . . . . • . . . . . . . 16

S e ttin g . . .................... * . . . . • • • • • • * 1 6

Scale D evelopm ent....................... . . . . . . . . . 18

Experim ental D e s ig n ......................... . . . • ......................... ' . .. 19

Data C o llec tio n . . ....................................... > . • • • ....................19

BARS P resen ta tio n . . . . . ......................... . . . . . . . . . 2 0

R e l ia b il i ty . . . . . . . . . . . . . . . • . . . . . . . • • 21

RESULTS . ..................... .... . . . . . . . . . . . . . . . 22

DISCUSSION . ......................... .... • .......................... • * * * 33

APPENDIX A. SD P resen ta tio n . . . . . . . . . . ........................ 43

APPENDIX B. Prompting . ....................... . . . . . . . 44

APPENDIX C. Consequation . .......................... . ............................ 45

APPENDIX D. Session M onitoring Form . . . . . . . . . . . . . . . 46

APPENDIX E. In te ra c tio n s between Observer and Subject . . . . . * 47

BIBLIOGRAPHY . . . . . . . . 49

i l l


LIST OF FIGURES

1. In s tru c to r Consequation, Prompting, and S® P resen ta tio n and T heir E ffec ts upon S tudent Responses as a Function o f Antecedent BARS P re se n ta tio n fo r Subject A . . . . . . . . . . 23

2. In s tru c to r Consequation, Prompting, and S® P resen ta tio n and Their E ffec ts upon S tudent Responses as a Function o f Antecedent BARS P re se n ta tio n fo r Subject B . . . . . . . . . . 24

3. In s tru c to r Consequation, Prompting, and S& P resen ta tio n and Their E ffe c ts upon Student Responses as a Function of Antecedent BARS P re se n ta tio n fo r Subject C .. . . . . . . . . . 2 5

4 . In s tru c to r Consequation, Prompting, and S® P resen ta tio n and T heir E ffec ts upon Student Responses as a Function o f Antecedent BARS P re se n ta tio n fo r Subject D ............................. 26

iv


INTRODUCTION

The purpose o f th is study was to determ ine on a b a s ic and em piri

c a l le v e l whether the B ehaviorally Anchored Rating Scale (BARS), which

i s norm ally used as a performance ev alu a tio n to o l, could a lso serve as

a t ra in in g to o l using p re sen ta tio n o f the sc a le s as feedback fo r appro

p r ia te in s tru c t io n a l behaviors.

Performance ev a lu a tio n o f employees w ith in o rgan iza tio n s in the

w estern world has become a prominent to p ic in contemporary management

and psychology jo u rn a ls (A llan & Rosenberg, 1978; Borman, 1978; Keeley,

1978) and in v i r tu a l ly every textbook dealing w ith in d u s tr ia l psychology

(Kane & Law ler, 1979; McCormick & T if f in , 1974; Wexley & Yuki, 1977).

An o u tlin e o f common su b jec tiv e performance ev a lu a tio n to o ls and tech

niques has been provided by Wexley and Yuki (1977).

An essay evalua tion i s the most open-ended o f the techniques. Here,

the su perv iso r simply w r ite s , in essay form, a summary o f the perform

ance o f each subord inate. This form may be more personal than o th e rs ,

b u t i t i s time consuming and lack s q u a n tif ic a tio n fo r employee com

parison . In a group a p p ra is a l , the su perv iso r o f the employees to

be evaluated as w ell as th re e o r four o th e r su p erv iso rs fa m ilia r w ith

them meet to d iscu ss th e i r performance. This method tends to elim

in a te much e r ro r in the essay ev a lua tion ; but again , employees can

not be compared q u a n tita tiv e ly ; and i t i s very , very c o s tly to the

o rg an iza tio n . In a f i e ld review , a tra in e d personnel manager goes

in to various sec tio n s o r d iv is io n s o f the o rg an iza tio n to d iscu ss the

performance o f in d iv id u a l employees w ith su p erv iso rs and sometimes

1


w ith the employees them selves. With the inform ation gained, the per

sonnel s p e c ia l i s t then w rite s an ev a lu a tio n o f the in d iv id u a ls ' per

formance which i s amended by th e i r s u p e rv iso r (s ) . This method cu ts

down on paperwork by sup erv iso rs and i s probably more c o n s is te n t, i f

n o t more accurate , than essay evalu a tio n s and group a p p ra is a ls . But

once again , employees cannot be q u a n tita tiv e ly compared; and i t i s pro

bably no t as accu ra te as observation-based ev a lu a tio n procedures.

Employee comparisons involve weighing up one employee a g a in s t

another to determ ine performance rank ings. A lte rn a tio n ranking i s the

sim plest o f the th ree comparison techniques, bu t i t may lack accuracy.

Here, the su p erv iso r determ ines the h ig h est and low est perform ing

employees in a p a r t ic u la r group and p laces th e i r names a t the h ig h est

and low est p o in ts on the sca le re sp e c tiv e ly . Then, he takes the next

h ig h est and low est perform ers and does the same. This i s repeated

u n t i l the l i s t is complete. In the p a ired comparison technique, each

employee is compared to every o th e r . This can be done on a m atrix

where the name of the b e t te r perform er i s in se r te d in the in te rs e c t io n

fo r the names o f each p a ir of employees. Rank can then be determined

by simply counting the number of times each employee was p re fe rre d .

While n o t performance s p e c i f ic , th is i s a f a i r ly accu ra te means of

ranking employees and can be used i f only rankings a re necessary . A

forced d is t r ib u t io n can be used fo r la rg e groups o f employees where

rough d isc rim in a tio n s between them w il l s u f f ic e and when i t can be

assumed th a t job performance conforms to a normal curve. Five spaces

a re provided in which the su p erv iso r p laces the names o f employees who

perform: (a) in the low est 10%; (b) in the next 20%; (c) in the middle


40%; (d) in the nex t h ig h es t 20%; and (e) in the h ig h es t 10%. The prob

lem here is th a t job performance o ften does no t conform to norm ality ,

e sp e c ia lly where th e re a re e f fe c t iv e s e le c tio n and/or tra in in g techn i

ques ( th is would skew th e d is t r ib u t io n to the l e f t ) . And w hile employees

f i t in to n ea t l i t t l e c a te g o rie s , no inform ation i s gained about perform

ance le v e ls so sa la ry and promotion dec is io n s a re d i f f i c u l t to make

w ith th is comparative procedure.

The c r i t i c a l in c id e n t technique f i r s t described by Flanagan (1949)

i s an observation-based technique. The su perv iso r keeps a running log

fo r .each employee in which he records s p e c if ic in s tan ces o f c r i t i c a l

co n trib u to ry and o b s tru c tiv e work in c id e n ts . This log then i s re fe rre d

to d ire c t ly during the performance ev a lu a tio n . While th is i s an

observation-based technique, i t i s not q u a n tif ia b le and does no t take

in to account the normal ongoing behaviors by employees v i t a l l y impor

ta n t in a tta in in g e f fe c t iv e p roduction le v e ls .

The weighted c h e c k lis t i s one where s tatem ents about poor to

e x ce lle n t performance a re l i s t e d in random o rd er. The score values

a re undisclosed to the r a t e r who checks every statem ent on each l i s t

th a t rep re se n ts the em ployee's performance (various dimensions o f per

formance a reas and a t t i tu d e s a re o fte n in c lu d ed ), and the l i s t s a re

scored by a judge.. The s ta tem en ts t hemselves, however, o f te n give

away th e i r v a lues s in ce the r a te r s recognize various performance le v e ls

and those r a te r s may avoid checking negative s ta tem en ts . In a forced

choice c h e c k lis t , blocks o f four s tatem ents about poor to ex ce lle n t

performance a re l i s t e d where the r a te r checks one statem ent which i s

most l ik e the employee and one statem ent which i s l e a s t l ik e the


employee. The statem ents a re then scored and a performance ra t in g can

be made. While p o ssib ly reducing r a t e r e r ro r , l i t t l e in form ation i s

gained about sp e c if ic performance to apply to personnel development.

Three ra t in g sca le s were o u tlin e d , two o f which were very s im ila r .

In the graphic ra tin g s c a le , the subheadings "poor," "average ," and

"ex c e lle n t" a re placed a t th e i r re sp ec tiv e p laces from l e f t to r ig h t

on the s c a le s , each of which has some performance o r a t t i tu d e heading.

The r a te r can then mark any p o in t on the sca le ( i . e . , between average

and e x ce llen t) to re p re sen t the em ployee's perform ance. The m ultis tep

sca le i s very s im ila r , except th a t the r a te r can mark only in the

th ree to f iv e evenly spaced boxes provided to re p re sen t from poor to

ex ce lle n t performance o r a t t i tu d e . The em ployee's dimension ra tin g s

can be compared w ith o th e r employees' ra tin g s o r can be averaged and

compared. Wexley and Yuki contend th a t both of these sc a le s a re q u ite

su scep tib le to r a te r e r ro r s .

The B ehaviorally Anchored S a ting Scale has probably been the most

widely d iscussed performance ev a lu a tio n technique in the l a s t two

decades. In th is Scale, a dimension o f performance i s ra te d on a

sca le which rep resen ts poor to e x c e lle n t performance by s p e c if ic

examples o f behavior. This i s sometimes c a lle d a Behavior Expectation

Scale because the anchors (behav ioral examples) a re o ften preceded by

"The employee can be expected to . . . . " The dimensions a re u su a lly

rep resen ted by th ree to ten anchors spread o u t over the continuum of

the s c a le . Each Anchor i s assigned a p a r t ic u la r num erical value th a t

i s meaningful only in re fe ren ce to the e n t i r e sca le which may be sca led

by ten th s o r by wholes and whose upper l im it may be from 2.5 to 10.


At the time o f the performance ev a lu a tio n , the r a t e r checks the anchor

in each o f about f iv e to ten dimensions which most rep resen ts the

employee's ty p ic a l work perform ance. The performance ev alu a tio n can

be made num erically so th a t comparisons between employees can be done

along dimensions and along o v e ra ll averages. The corresponding

anchors can a lso add valuab le performance inform ation fo r personnel

development. As a r e s u l t o f th is f l e x i b i l i t y , the B ehaviorally

Anchored Rating Scale may be a b e t te r ev a lu a tio n to o l than the o th e rs .

However, w ith a l l these ev a lu a tio n to o ls a t hand, W illiams (1972)

claim s th a t th e re i s su b s ta n tia l evidence around the world to suggest

th a t many performance a p p ra isa ls a re a w aste of time. He provides

evidence th a t many a p p ra isa ls a re in accu ra te ; and in many cases,

whether the ap p ra isa ls a re accu ra te o r n o t, the inform ation i s no t

used as a b a s is fo r s a la ry r a is e s , promotion, o r personnel development.

W illiam s, however, has n o t given up on personnel ev a lu a tio n ; and he

promotes fu r th e r development o f e f fe c t iv e performance ev a lu a tio n .

The B ehaviorally Anchored Rating S ca le , which i s the focus of

th is study , has been tou ted as a prom ising new to o l in s t a f f and per

sonnel ev a lu a tio n , a t l e a s t on a s t a t i s t i c a l le v e l (Kearney, 1976).

Bernardin and Smith (1981) m ain tain th a t the B ehaviorally Anchored

Rating Scale (BARS) i s d i f f e r e n t from o th e r methods o f ev alu a tio n

q u a li ta t iv e ly in th a t i t i s designed to s tan d a rd ize the ra tin g and

the observation o f employees. They p o in t o u t th a t w ith standard ized

o b serv atio n , th e re i s a common ground fo r r e l ia b le ra tin g o f perform

ance.

The s p e c if ic methods fo r BARS development vary somewhat, b u t the


6

author has encountered from f iv e to e ig h t s p e c if ic s tep s o f BARS

development in a nearly exhaustive skimming o f the BARS l i t e r a tu r e .

Blood (1974) has o u tlin e d th ree general s tep s which cover the ty p ic a l

BARS development. F i r s t , behav ioral in c id e n ts o f various degrees o f

job performance dimensions a re generated by a pool of people knowledgea

b le in the job fo r which the BARS i s being developed. Secondly, the

in c id en ts a re re tra n s la te d . This involves re a s so r tin g the in c id en ts

from a randomized pool in to th e i r proper dimensions. In o rder fo r

in c id en ts o r anchors to be re ta in e d , they must be r e l ia b ly p laced in

th e ir dimensions by a sound m ajo rity of the judges. F in a lly , the

anchors a re assigned s c a le values according to the le v e l o f perform

ance they rep re se n t. The mean of a l l values assigned i s the f in a l

sca le value o f the anchor, and in c id en ts w ith high variances a re d is

carded. O ccasionally , th e job dimensions themselves a re developed

through a generation and re tr a n s la t io n procedure where the dimensions

are reassigned from a random pool to groups o f anchors. There a re

numerous cookbook approaches in the l i t e r a t u r e which deal w ith BARS

development and which may be consu lted fo r fu r th e r d e ta i l s (B ernardin,

1977; B ernardin , A lvarez, & Cranny, 1976; B ernardin & Smith, 1981;

Borman & Dunnette, 1975; Borman & V allon, 1974; Campbell, Dunnette,

Arvey, & H e lle rv ik , 1973; Das, F ro s t, & Barnowe, 1979; D eC otiis, 1977;

D eCotiis, 1978; Dickinson & Z e llin g e r , 1980; Ivancevich , 1980b; Landy

& Guion, 1970; Kearney, 1976; Keavney & McGann, 1975; M illa rd , Luthans,

& Ottemann, 1976; Shapira & Shirom, 1980; Smith & K endall, 1963;

S tap les & Locander, 1972; Zedick & Baker, 1972; Zedeck, K afry, &

Jacobs, 1978).


The ty p ic a l performance ev alu a tio n w ithout a BARS u su a lly involves

a few sim ple s te p s . For hourly employees, th e i r sup erv iso rs decide how

w ell they perform th e i r s p e c if ic tasks and what th e i r work a t t i tu d e is

l ik e , and check th e ir a ttendance and p u n c tu a lity . For s a la r ie d

employees, a few o b je c tiv e s a re s e t a t the beginning o f the year such

as in an MBO program; and a t the annual ev a lu a tio n , some evalua tion

to o l may be used to determ ine o v e ra ll job performance, and o b jec tiv es

a re examined to see i f they have been met. The BARS appears to be

a su p erio r procedure to these on four p o in ts . F i r s t , th e superv iso rs

who use i t a re , in most c ase s , deeply involved in i t s development

which may provide some face o r con ten t v a l id i ty as w e ll as concurrent

v a l id i ty . Secondly, those sup erv iso rs a re thoroughly fa m ilia r w ith

the BARS' conceptions and param eters so they a re ex p erts in i t s use

which may provide some r e l i a b i l i t y in observation and ev a lu a tio n .

T hird , the sc a le dimensions a re broken down in to behav io ra l d esc rip

tio n s o f most o r a l l performance le v e ls (on a continuum o f poor to

ex ce llen t) as opposed to a one to f iv e r a t in g o f how the r a t e r be liev es

the ra te e i s doing in a g re a t number of a t t i tu d e s and performance

requirem ents. This provides o b je c tiv e s fo r the employee and d e f in i

tio n s of performance which should help them to avoid undesirab le

behaviors, th e re fo re ,le av in g room fo r d e s ired performance. F in a lly ,

the s p e c i f ic i ty of the to o l may make ev a lu a tio n e a s ie r thus promoting

more frequen t evalu a tio n . Wexley and Yuki (1977) p o in t out th a t f r e

quent evalua tions o f performance may help o rg an iza tio n s meet th e ir

s h o r t- and long-term o b je c tiv e s .

There i s evidence in psychometric terms to support the su p e r io r ity


of BARS over ocher ra t in g sc a le s , such as the Graphic Rating Scale,

the Mixed Standard Scale, and the M ultistep S cale . The BARS has been

shown to in c rease in te r r a te r r e l i a b i l i t y and to reduce r a te r b ia s

(Landy &Guion, 1970; M illard e t a l . , 1976; Zedeck & Baker, 1972) and

has decreased len iency and ra t in g v a r ia b i l i ty (Das e t a l . , 1979;

Shapira & Shirom, 1980; Vance, Kuhnert, & F a rr , 1978). Ivancevich

(1980b) re p o rts th a t w ith the im plem entation o f BARS fo r p ro fess io n a l

employees, th e re was an improvement in a t t i tu d e about equ ity in job

requirem ents and c la r i ty in feedback.

BARS has been used in a v a r ie ty o f jobs inc lud ing nurses (Smith

& K endall, 1963; Zedeck & Baker, 1972), engineers (Landy & Guion,

1970), r e t a i l personnel (Campbell e t a l . , 1973; S tap les & Locander,

1972), tank crews in the I s r a e l i army (Shapira & Shirom, 1980), co lleg e

p ro fesso rs (Das e t a l . , 1979; Dickinson & Z e llin g e r , 1980; Vance e t

a l . , 1978), bank managers (Schwind, 1978), and s a le s personnel

(Ivancevich, 1980a).

There a re , however, In c reasin g numbers o f s tu d ie s which s t a t e

th a t the BARS i s no b e t te r b u t no worse than o th e r form ats, such as

graphic ra tin g sca le s (Keaveny & McGann, 1975), behavior observation

Scales (Latham, Fay, & S a a ri, 1979) and mixed standard sca le s (Dickinson &

Z e llin g e r, 1980). Keeley (1978) claim s th a t the BARS i s lim ited in

a p p lic a tio n to ro u tin e ta sk s s in ce i t i s anchored to s p e c if ic in s tan ces

o f behavior and i t may no t be as f le x ib le as a g raphic ra tin g s c a le .

A tkin and Conlon (1978) o u tlin e o th e r p o ss ib le problems w ith the

BARS. F i r s t , th e re may be overlap o r nonexclusiveness o f item s w ith in

each dimension. Second, c r i t i c a l in c id en ts o f behavior (extrem ely


good o r poor performance) may stand o u t and be re f le c te d In the ra tin g

process w hile ro u tin e performance I s overlooked. And f in a l ly , th e pre

d isp o s it io n o f the r a te r toward ra te e s may a f f e c t h is ra tin g s o f those

people. But these problems would be common to a l l r a t in g procedures

and n e ith e r unique to the BARS, nor solved by the BARS.

Kingstrom and Bass (1981) c ite d sev e ra l s tu d ie s which c o n tra d ic t

the conclusions o f h e re to fo re popular pro-BARS s tu d ie s . In th e i r

review, Kingstrom and Bass show approxim ately equal numbers o f s tu d ie s

find ing su p e r io r ity w ith the BARS and w ith the a l te rn a te form at sca le s

(g raph ic , stimulated, mixed standard) in len iency , in te r r a t e r r e l i a b i l i t y ,

ra te e d is c r im in a b ili ty , dimension independence (h a lo ), and v a l id i ty .

A few s tu d ie s showed no d iffe ren c es in th e se . The a u th o rs ' conclusions

were th a t th e re were n e g lig ib le d iffe ren c es between BARS and a l te rn a te

form at s c a le s . This was q u a lif ie d , however, w ith th e f a c t th a t most

o f the a l te rn a te form ats used fo r comparison w ith BARS in th ese s tu d ie s

were developed from inform ation derived from BARS developmental proce

dures. This may in d ic a te th a t the s c a le development procedure and not

the s c a le i t s e l f i s the c r i t i c a l v a r ia b le .

Jacobs, K afry, and Zedeck (1980) concur w ith the find ings of

Kingstrom and Bass (1981). In a review o f a t l e a s t 39 s tu d ie s dealing

d ire c t ly w ith BARS comparisons, i t was th e i r conclusion th a t i t was

n e ith e r su p erio r nor in f e r io r to any o th e r form at i t has been compared

to . I t seems th a t the problems w ith BARS as a ra t in g to o l a re the

same as w ith any o th e r r a t in g to o l and th a t each may be j u s t as good

o r bad as the o th e r.

In shedding new l i g h t on the advantages o f and problems w ith BARS,


10

the author p o in ts out th a t a l l these s tu d ie s lacked any raw an a ly s is

of the e f fe c ts o f BARS a p p lic a tio n upon employee performance. Wexley

and Yuki (1977) contend th a t " c u rre n t resea rch suggests th a t the

ex tra time and e f f o r t req u ired to b u ild behavior-based ra tin g sc a le s

may n o t be w arranted i f used so le ly fo r performance ra tin g s" (p . 207).

Blood (1974) has o u tlin ed fou r sp in o ffs fo r BARS development. F i r s t ,

as a lluded to e a r l i e r , the o rg an iza tio n can b e n e f it from a s c a le which

can be developed by re p re se n ta tiv e s o f va rio u s o rg an iza tio n a l le v e ls ,

th e re fo re , ga in ing im portant p e rsp ec tiv es o f the job fo r which the

BARS i s being developed. Secondly, the inform ation about the job

obtained in BARS development can be used to c re a te a v a lid tra in in g pro

gram. The tra in e e s would le a rn what i s expected o f them and how they

would be evaluated . T h ird ly , when th e re i s disagreem ent in the BARS

development over the v a lue o f anchors o r in what dimension they belong,

th e re i s a s ig n a l th a t o rg a n iz a tio n a l p o licy does not e x is t fo r these

performance requirem ents o r i t i s n o t being conveyed. I f th e re a re

problems w ith p o licy , they can then be ironed ou t according to the

inform ation gained in th e BARS development p rocess . Where p o licy

e x is ts but disagreem ents occur, the fo u rth sp in o ff comes in to p lay .

The o rg an iza tio n can determ ine where the breakdown i s in communication

and deal w ith i t .

In an endeavor to t e s t the param eters o f BARS a p p lic a tio n , the

p re sen t study was an a ttem pt on a sim ple, b a s ic , and em pirica l le v e l

to determ ine whether the BARS method, used in a very lim ited co n tex t,

would be fu n c tio n a lly r e la te d to performance improvements in terms of

a c tu a l behavior change as a r e s u l t o f BARS p re sen ta tio n s a lone. The


11

BARS inform ation provided by ra tin g s accompanied by sp e c if ic examples

o f performance can be likened to feedback. Feedback i s defined as

inform ation about performance, compared to a s tandard , and used to

guide performance. The inform ation about performance here would be

the performance ra t in g . The standard would be the h ig h e s t anchor, and

an a tta inm ent o f th is would dem onstrate a guidance o f performance.

Prue and Fairbank (1981) o u tl in e param eters of feedback im portant

in a t ta in in g optimum e ffe c tiv en e ss in the p a r t ic u la r s i tu a t io n where

i t i s to be used. They p o in t ou t th a t performance feedback i s popular

in attem pting to c re a te performance improvements and th a t an ex tensive

da ta base has proven th a t provid ing inform ation to groups and to

in d iv id u a ls about the q u an tity and q u a lity of th e i r performance does

improve i t . Feedback has c e r ta in advantages which enhance i t s popu

l a r i t y . I t i s inexpensive to d e liv e r ; i t i s easy to implement s in ce

e lab o ra te tra in in g i s n o t a p re re q u is ite to accu ra te ad m in is tra tio n

( in the case o f BARS, lim ited a p p lic a tio n may make i t s ex tensive

development p ro h ib i t iv e ) ; i t cu ts back the use o f nonpolicy av ersiv e

co n tro l procedures; and i t i s u se fu l where monetary rewards cannot be

delivered , as in a union environment.

Four feedback mechanisms have been o u tlin e d : m echanical,

se lf-re co rd ed , v e rb a l, and w r it te n . Mechanical feedback i s th a t pro

vided by an apparatus, such as a cummulative ta l le y odometer o r a

v ideotape o f w idget assembly. This i s an inexpensive form of feedback

sin ce i t has moderate i n i t i a l and l i t t l e ongoing c o s t and i t w i l l never

adm in ister an em otional reprimand. S elf-reco rded feedback i s achieved

by the employee recording h is own performance on an odometer o r by


w ritte n t a l ly . This i s advantageous when the employee has l i t t l e

superv ision o r when th e re i s no permanent p roduct. Verbal feedback

is th a t provided by a superv iso r to an employee on a d i r e c t in te r

personal le v e l . This i s an easy method o f feedback d e liv e ry ; b u t i t

may lack consistency , s in c e r i ty may no t be ev id en t, and i f an adverse

re la tio n sh ip e x is ts between a su p erv iso r and employee, i t may no t be

a t a l l e f fe c t iv e . W ritten feedback can e a s ily be made c o n s is te n t.

I t i s a permanent p roduct, and i t can e a s ily f i t in w ith any informa

tio n p rocessing system the o rg an iza tio n p re sen tly u t i l i z e s . The

inform ation gained from BARS as a feedback to o l a lso has these q u a li

t ie s .

The con ten t o f the feedback can inc lude in d iv id u a l performance

as a percentage o f the g roup 's performance, comparison of a g roup 's

performance w ith a s tandard , comparison o f a g roup 's performance w ith

previous performance, comparison o f a in d iv id u a l 's performance w ith

a standard , and comparison o f an in d iv id u a l 's performance w ith previous

performance. The inform ation contained in an in d iv id u a l BARS p resen ta

tio n can be compared w ith a s tandard (th e h ig h es t anchor) and w ith

previous performance (previous ran g e ).

Prue and Fairbank a lso d iscussed the e f fe c ts o f feedback upon

performance. They p o in t o u t th a t feedback e f fe c ts have in the p a s t

alm ost b lin d ly been ch a rac te rized as re in fo rc in g . S ince response

ra te s were increased , th is could be tru e ; b u t they a lso p o in t o u t th a t

o th e r co n sid era tio n s must be made. One of these would be the stim ulus

co n tro l which feedback ex e rts over behavior, such as when an organiza

tio n s e ts new requirem ents fo r re inforcem ent. Performance could be


13ru le governed where feedback i s used fo r the f i r s t time in the o rgan i

za tio n so th e re would be no h is to ry o f con tingen t reinforcem ent.

Feedback could a lso a l t e r the o rg an iza tio n a l environment such th a t

inform al d iscu ssio n s would cen te r around performance improvement and

nonprogrammed so c ia l re in fo rc e rs would be a v a ila b le . I t could a lso

s e t up an atmosphere of com petition . F in a lly , ' they p o in t out th a t

w ith consequation based on o b jec tiv e d a ta , employees are more s a t i s f ie d

w ith th e i r work and surroundings which could enhance the feedback

system.

The e ffe c tiv en e ss o f BABS in improving performance was evaluated

in the p re sen t study by examining i t s e f fe c ts upon s t a f f in s tru c t io n a l

behaviors in a behavior m odifica tion o rien ted s p e c ia l education f a c i l i t y .

In the same f a c i l i t y , Shook, Johnson, and Uhlman (1978), in two sep ara te

experim ents, s tud ied graphing by s t a f f using (a) response e f f o r t

red u c tio n , in s tru c t io n s , and group feedback (a n o tic e board w ith a p er

centage of u p -to -d a te graphs l i s t e d ) , followed by (b) in s tru c tio n s ,

group feedback, in d iv id u a l feedback (n o tice o f percen t of d a ily graphs

which were c u r re n t) , and reinforcem ent (s o c ia l p r a is e ) . The l a t t e r

procedure appeared to be most e f fe c t iv e and most c o s tly , w hile

in d iv id u a l feedback alone appeared to be most co st e f fe c t iv e in terms

of response e f f o r t by experim enters to d e liv e r d e sc r ip tiv e p ra ise to

each su b jec t compared w ith one e a s i ly assembled c h a rt.

In an attem pt to analyze the feedback e f fe c ts o f the BARS, the

sca le in the p resen t study was presented as both an an tecedent and as

a consequence. Krumhus and M alott (1980) s tud ied th e e f fe c ts o f model

l in g , feedback before a se ss io n , and feedback a f t e r a session upon


14

d e sc rip tiv e so c ia l reinforcem ent of s tu den ts by tu to r s . No d iffe ren c e

was observed between immediate ( ju s t a f te r session ) and delayed (p r io r

to next session) feedback, but th e re were marked in c rease s from base

l in e to in te rv e n tio n in both p re - and p o s tse ss io n feedback and some

in c rease from p o s tse ss io n m odelling to p re - and p o s tse ss io n feedback

in tu to r responses. Feedback, both befo re and a f t e r se ss io n s , con

s is te d of a review o f a v ideotape o f the s u b je c t 's most re cen t sessio n

by the observer and the su b je c t, w ith the su b jec t c i t in g th re e in s tan ces

and noninstances of ap p ro p ria te d e sc r ip tiv e so c ia l reinforcem ent.

In a study by Sepler and Meyers (1978), " . . . verba l in s tru c

t io n alone was shown to be in s u f f ic ie n t fo r teach ing the f u l l range

o f behavior m od ifica tion s k i l l s requ ired to fu n c tio n e f fe c t iv e ly in

app lied s e t t in g s " (p. 198). In th is s tudy , p a rap ro fess io n a ls were

given a w ritte n p r e te s t , in s tru c te d in behavior m odifica tion s k i l l s ,

and then given a w ritte n p o s t te s t . They were a lso observed fo r appro

p r ia te responses throughout the study.

T rain ing was found to be e s s e n t ia l in in c reas in g the p roportion

o f s p e c if ic tra in in g behaviors by p a rap ro fe ss io n a ls during therapy

periods (Fabry & Reid, 1978) and in in c reas in g ap p ro p ria te S ^ 's

w hile reducing the elapsed time of each ta sk (Carnine & Fink, 1978).

As suggested by Blood (1974), the BARS could serve f i r s t as an

ev alua tion to o l to determ ine what s k i l l s a th e ra p is t dem onstrates and

lacks and secondly as a t ra in in g to o l by showing the th e ra p is t in

what way he o r she i s in e f fe c t iv e and what must be done to a t ta in

e ffe c t iv e n e s s . A t r a in e r could a lso use th is inform ation to d if fe re n

t i a l l y re in fo rce behaviors o r ap p ro p ria te approxim ations th e reo f


' 15

which c o n trib u te to e ffe c tiv e n e s s . As poin ted out above, such t r a in

ing was found to be im portant by Fabry and Reid (1978) and Carnine and

Fink (1978) in developing behavior m odifica tion s k i l l s . By surpassing

in s tru c t io n a lone , which Sepler and Myers (1978) found in s u f f ic ie n t

in tra in in g behavior m od ifica tion s k i l l s , the BARS could w ell be a

very e f f ic ie n t to o l fo r personnel development.


METHOD

Subjects

The su b jec ts were a l l employees o f a sp e c ia l education f a c i l i t y

and were a l l fem ale. Subject A was a 24-year-o ld c e r t i f ie d teacher

who, in th is study, worked w ith th ree s tu d en ts from her classroom on

D istar Math. She had previous experience in D is ta r Math. Subject B

was a 21-year-o ld in s tru c t io n a l a id e working w ith the same th ree s tu

dents as Subject A in D is ta r Math. Subject B had no previous experience

in D ls ta r Math. Subjects A and B a lte rn a te d p ro g ressive le sso n s so

th a t n e ith e r ran the same le sso n , although le sso n s o ften repeated

d r i l l s from previous le sso n s . Subject C was a 28-year-old c e r t i f ie d

teacher who was working w ith the same th ree s tu d en ts as above whom

she did no t o therw ise normally work w ith . In a d d itio n , she worked

w ith one o f h e r own studen ts who jo in ed the group o f the o th e r th ree

in D is ta r Language. Subject C had minimal prev ious experience in

D is tar Language. Subject D was a 21-year-o ld p a rt- tim e parap ro fes-

s io n a l working w ith a d if f e r e n t s tu d en t than those above on D ista r

Math. She had no previous experience in D is ta r Math. A ll o f the

math lessons fo r Subjects A, B, and D were conducted from D is ta r Math

Book I . A. A ll o f the language le sso n s fo r S ub jec t C were conducted

from D is ta r Language Book I . A.

S e ttin g

Subjects A, B, and C a l l worked in the same a rea a t d if f e r e n t

16


17

tidies. The area included a long ta b le , on e i th e r s id e of which s a t

the s tu d en ts and the su b jec ts ( in s tru c to r s ) ; a blackboard behind the

in s tru c to r ; and D is ta r Language o r Math m a te ria ls on the ta b le .

S ubject D worked in an in d iv id u a l sess io n bo th , which included a

s tu d e n t 's desk, a c h a ir fac ing the s tu d en t in which the su b jec t s a t

w ith D is ta r Math m a te ria ls on the desk. The observer always s a t in

a p o s itio n somewhere behind the s tu d en t o r s tu d en ts so as not to

in te r f e re v is u a lly w ith the su b je c ts .

The s tu d en ts a l l a ttended classrooms where behavior m odifica tion

was the main mode of s tu d en t m otivation . This behavior o r ie n ta t io n

requ ired c o n s is te n t behaviors by the in s tru c to rs in p re ta sk o r an te

cedent stim ulus p re sen ta tio n , c o rre c tio n , o r prompting and p o s tta sk

o r consequent stim ulus p re se n ta tio n . P a rap ro fessio n a ls and in s tru c

tio n a l a ides such as Subjects B and D norm ally received in -s e rv ic e

tra in in g in classroom behavior management inc lud ing behavior m odifica

tio n in in s tru c tio n a l se ss io n s . They were u su a lly monitored bi-m onthly

in in s tru c t io n a l behavior m od ifica tion . C e r tif ie d teachers such as

Subjects A and C re g u la r ly p a r tic ip a te d in classroom behavior manage

ment conferences; bu t in the f a c i l i t y where they were employed during

th is study , Subjects A and C received no formal tra in in g nor were

they monitored in in s tru c t io n a l behavior m o d ifica tio n . In the p a s t,

however, they had both received in te n s iv e tra in in g in behavior modifi

ca tio n in in s tru c t io n a l sessio n s concurren t w ith previous p o s itio n s

which requ ired th is t ra in in g .


18

S cale Development

The th ree BABS sc a le s were developed by the au thor (see Appendices A-C)

from job models p rev iously w r it te n by the p re sen t author and by o th e r

experts w ith in th e f a c i l i t y in job model development a s w e ll as from

the Session M onitoring Form (see Appendix D) which was developed by

employees of the f a c i l i t y who were experts in programmed in s tru c tio n

and in s tru c t io n a l behavior m od ifica tion as w ell a s in sess io n m onitor

ing . The job models and the Session M onitoring Form both encompassed

behaviors s p e c if ic a l ly addressed by the BABS in th e p re sen t study.

The sc a le dimensions co n sis ted o f:

1. S® p re se n ta tio n req u irin g a ttend ing by the s tu d en t, a 1-second pause fo r s tu d en t a tten d in g by the in s tru c to r b e fo re p resen ting a c le a r and c o rre c t S as described w ith in the D is ta r le sso n .

2. Prompting re q u ir in g tim elines (not b e fo re 5 seconds and not a f te r 10 seconds o f no responding o r immediately a f t e r a wrong response) and achievement of a ta rg e t response as s t ip u la te d w ith in th e D is ta r le sso n .

3. Consequation in th re e modes o r dimensions w ith ap p rop ria te reinforcem ent w ith in 1 second o f a response, inc lud ing desc rip tiv e p ra is e and a tan g ib le reward, such as a token, ed ib le ,o r p h y sica l co n tac t.

I t should be noted th a t the sca le s developed fo r the p re sen t

study were much more s p e c if ic and lim ite d in scope than the BABS

developed in previous s tu d ie s fo r psychometric ev a lu a tio n and q u a rte r ly

to annually general a p p lic a tio n . A ctually , the sc a le s in th e p re sen t

study were no t intended fo r general performance ev a lu a tio n b u t were

intended: (a) to determ ine i f BABS p re sen ta tio n in a c o n tro lle d s e t

tin g could d i r e c t ly a f f e c t performance in the above ca teg o rie s which

were c ru c ia l to e f fe c t iv e teaching; and (b) to ev a lu a te th a t performance


19

fo r ln trad a ss ro o m management d ec is io n s . Because of th e s p e c if ic

content o f th e sc a le s and the value o f the anchors, th e p resen t BARS

may be even more o b je c tiv e than those seen in previous s tu d ie s so the

sca les may be considered B ehaviorally Anchored Rating S ca les . But

because of the lim ited scope o f the sca le s and the method of th e i r

a p p lic a tio n in the p re sen t study, the BARS presented h ere could not

su ff ic e as a to ta l BARS ev a lu a tio n system.

Experimental Design

A sim ple m u ltip le b a se lin e design across behaviors was u t i l iz e d

w ith Subjects A, B, C, and D. A re v e rsa l was a lso conducted w ith

Subject C. The o rders o f p re sen ta tio n o f sca le s fo r each su b jec t

a re p resen ted : S ubject A—consequation, prompting, and S® presen ta

tio n ; Subject B—consequation, S® p re sen ta tio n , and prompting; Sub

je c t C—consequation, prompting, and S® p re sen ta tio n ; and Subject D--

prompting, S® p re se n ta tio n , and consequation. The o rd er of in te rv en

tio n was determined by d e f ic i t s in the in s t ru c to r s ' re p e r to ire s and

re f le c te d an a ttem pt to provide the b e s t in s tru c tio n p o ss ib le fo r the

s tu d en ts .

Data C o llec tion

A standard sess io n m onitoring form was used to record da ta about

the in s t ru c to r s ' teach ing behav iors. As mentioned b e fo re , the observer

( in th is case , the experim enter and p resen t au thor) s a t behind the

s tuden ts in a p o s it io n fac ing the in s tru c to r . For each t r i a l which

began w ith each new in s tru c t io n , a p lus o r a minus was marked to


20

in d ic a te whether o r n o t (a) the s tu d en t was a tten d in g ; (b) th e re was

a l-second pause by the in s tru c to r fo r s tu d en t a tten d in g ; (c) a cor

r e c t and c le a r S® was p resen ted ; (d) a c o rre c t response occurred and

(e) i f n o t, d id a prompt occur; (f) i f the prompt was n o t c o rre c t , was

i t l a t e o r did no ta rg e t besponse occur; and (g) was the consequation

co rrec t (ap p ro p ria te , immediate, d e sc r ip tiv e , and in c lu s iv e o f tan g ib le

rewards). A ppropriateness of consequation included nonreinforcem ent o f

in c o rre c t responses and reinforcem ent o f some form fo r c o rre c t

responses. In ap p ro p ria te reinforcem ent rendered consequation in c o rre c t

in a l l th re e dimensions fo r the re sp e c tiv e t r i a l .

BARS P re se n ta tio n

Subjects A and B were exposed to th e BARS immediately p r io r to

(w ith in 10 m inutes o f) the sess io n fo llow ing the one in which they

were observed and ra te d . This was done to a sse ss any antecedent

e ffe c ts o f the BARS upon s t a f f behav io rs. Subjects C and D were

exposed to the BARS immediately a f t e r (w ith in 10 seconds o f) the

session in which they were observed and ra te d . This was done to

assess any consequent e f fe c ts o f the BARS upon s t a f f behav io rs. A

summary o f experim enter-sub ject in te ra c t io n i s provided in Appendix E.

The BARS ra tin g s were always s ta te d in terms o f lower l im its s e t

down w ith in each s c a le . For example, i f 85% of re in fo rc e rs occurred

w ith in 1 second o f c o rre c t responses, th is was s ta te d as 70% as repre

sented by the lower l im i t on the s c a le . The num erical r a t in g always

represented the anchor d escrib ing the low est percentage v a lue fo r each

segment o f the dimension. S im ila r ly , i f S®'s were exact a t le s s than


50% occurrance w hile a tten d in g , 1-second pauses, and c le a r S®'s

occurred a t 50%, the s c a le value o f 2 was c irc le d to rep resen t "wrong

SD"; and then the remaining d e sc r ip tiv e performance anchors were

underlined to a v a il the su b jec ts to the most s p e c if ic feedback possi

b le w ith th is form.

R e l ia b i l i ty

R e l ia b i l i ty of the observer was assessed ac ro ss sev e ra l exposures

to the same s e t of behav ioral events s in ce "assess in g the s t a b i l i t y

of observer judgments i s most e a s ily accomplished i f a constan t behav

io ra l event can be repeated , p resen ted , and observed" (Johnston &

Pennypacker, 1980, p. 195). Two videotaped sessio n s fo r each su b jec t

served as the repeated behav ioral even ts. The tap ings were i n i t i a l l y

made in the th ird sessio n s fo r Subjects B and D and in the fo u rth

sessions fo r Subjects A and C. This was done to allow the s tuden ts

and in s tru c to rs to become d esen s itiz ed to the new system before in tro

ducing a new and in tr ig u in g stim ulus and to allow the use o f the only

a v a ilab le videotape w ith a l l su b jec ts . The second tap ings were made

in Session 9 w ith Subjects B and D and in Session 10 w ith Subjects A

and C. This was done a t th i s p o in t simply to r e c a l ib r a te the observer

and to allow the use of the one tap e . The v ideo tape equipment was

placed to the r ig h t s id e of a l l o f the s tu d en ts to allow a view of

each o f them. R e l ia b il i ty was computed by d iv id in g a l l the agreements

by a l l the agreements p lu s disagreem ents fo r the sev e ra l observations

o f each in d iv id u a l sess io n .


RESULTS

The r e s u l ts o f the p re sen t study a re p resen ted g rap h ica lly in

F igures 1 - 4 . Three in te rv e n tio n elements a re d isp layed in each

m u ltip le b a se lin e graph. S® p re sen ta tio n i s d isp layed in terms of

s tuden t a tten d in g (rep resen ted by la rg e so lid do ts connected by so lid

l in e s ) , a 1-second pause fo r a tten d in g by th e in s tru c to r (rep resen ted

by open c i r c le s connected by dashed l in e s ) , and c o rre c t SD,s (rep re

sented by open squares connected by d o tted l i n e s ) .

The prompting element simply c o n s is ts o f the percentage of cor

r e c t prompts by each su b je c t. An arrow p o in ts down from the upper

l im it o f th is element in some cases to in d ic a te th a t one in c o rre c t

prompt occurred in le s s than 10 o p p o rtu n itie s to prompt. This i s

im portant to no te because th is a f fe c ts more g re a tly the v a r ia b i l i ty

in the percentages here than in the o th e r elem ents, s in ce in the l a t t e r ,

th e re were always a t l e a s t 10 o p p o rtu n itie s fo r responses, and in most

cases, 20 o r more. And, th e re was no c o n tro l o f o p p o rtu n itie s to

prompt, s in ce th is depended on in c o rre c t responses by the s tu d en ts .

Consequation i s d isp layed as percentage c o rre c t in th re e dimen

sions o f behavior (p ra is e , d e sc r ip tio n o f the behav ior, and a tan g ib le

reward) and in two dimensions (any two o f the previous dim ensions).

I t r e fe rs to reinforcem ent o f c o rre c t behaviors w ith nonreinforcem ent

o f c o rre c t behaviors o r reinforcem ent o f in c o rre c t behaviors counting

as in c o rre c t in a l l th ree dim ensions. In the graph fo r S ubject A, an

arrow p o in ts down from th e upper l im it o f consequation a t Session 10

to in d ic a te th a t the in s tru c to r changed the procedure to a VR2

22


Reproduced

with perm

ission of the

copyright ow

ner. Further

reproduction prohibited

without

permission.

-O ' ,Q“o-

vtui«/)•o« mq...........

CL • •y».. •O’ -O-oc

ect* MIMNIAIIOMW M

U i

-O.-O-.CL. -O o-

INSTRUCTIONAl SESSIONS

Figure 1: Instructor Consequation, Prompting, and SD Presentation and Their E ffectsupon Student Responses as a Function o f Antecedent BARS Presentation forSubject A.

roto

Reproduced

with perm

ission of the

copyright ow

ner. Further


without

permission.

M i l « l A N t a C l t l N !

■-cf

CONIIO*<tflOM

I/Ian

otot *•o ••UL

u- - a .

ItV M M t M l90*1 M l

INSTRUCTIONAL SE S S IO N S

Figure 2: Instructor Consequation, Prompting, and S Presentation andTheir E ffects upon Student Responses as a Function o f Antecedent BARS Presentation for Subject B.

Reproduced

with perm

ission of the

copyright ow

ner. Further


without

permission.

-o-

-o'

uiM2i..O ..

....O■•o-

-Q-u

0£oc ••f l ••

O- .O-O•O' O' o-o o-

INSTRUCTIONAL SESSIONS

Figure 3: Instructor Consequation, Prompting, and SD Presentation and Their E ffectsupon Student Responses as a Function o f Consequent BARS Presentation forSubject C.

rocn

Reproduced

with perm

ission of the

copyright ow

ner. Further


without

permission.

■O'•o<

V)

a.

>-

y i n

-Oo-

•o-

INSTRUCTIONAL SESSIONS

Figure 4: Ins truc to r Consequation, Prompting, and Presentation and Their Effects uponStudent Responses as a Function o f Antecedent BARS Presentation fo r Subject D.

I\

r

27

schedule o f token consequation to fade i t o u t as was customary in

many procedures in her classroom . Here the sca le on consequation

served as feedback to in d ic a te to the in s tru c to r w hether token o r

tan g ib le consequation was occu rring a t , above, o r below 50%. This

simply in d ic a te s th a t th e c r i te r io n was 50% in the th ree dimension

component, w hile the experim enter s t i l l sought p e rfe c tio n in the

remaining two dim ensions.

Student responses a re a lso d isp layed w ith each s u b je c t 's da ta

to dem onstrate any in d ir e c t e f fe c t o f the BARS on these through the

su b je c ts ' in s tru c t io n a l behav io rs. The percentage o f c o r re c t responses

i s d isp layed by open c i r c le s connected by dashed l in e s , and the per

centage of c o rre c t p lus co rrec ted responses ( to t a l percentage of

evoked responses) i s d isp layed by la rg e s o lid dots connected by s o lid

l in e s .

With Subject A, where BARS was used as an an teceden t, i t s pre

se n ta tio n apparen tly had an i n i t i a l l y d e le te r io u s e f fe c t upon her

consequation of s tuden t behav io rs. Consequation in b a se lin e ranged

from 77% to 84% in th ree dimensions and from 93% to 100% in two

dim ensions. With the advent o f an tecedent BARS p re se n ta tio n , conse

quation in th ree dimensions f e l l to 64% and in two dimensions to 82%.

Over the f iv e follow ing se ss io n s , these seem to recover somewhat.

The da ta begin to follow a p re d ic ta b le p a tte rn a t Session 10, where

the VR2 schedule w ith tokens was implemented. The VR2 token schedule

seems to have n eg ativ e ly a ffe c ted the o th e r two dimensions o f re in

forcement in Session , b u t reinforcem ent in two dimensions shows gradual

improvement th e re a f te r . Reinforcement in th ree dimensions a f te r


28Session 10 stayed w ith in 10 percentage p o in ts o f c r i te r io n in f iv e

out o f e ig h t se ss io n s . In Session 17, S ubject A wanted to d iscon tinue

the use of tokens, so th e experim enter reminded h e r to use physica l

co n tac t in s te ad . Here consequation in th re e dimensions f e l l to 33%.

Reinforcement in th ree dimensions never dev ia ted more than 20 percen

tage p o in ts from the 50% c r i te r io n .

Percentage of c o r re c t prompting w ith S ubject A in the i n i t i a l

b a se lin e hovered around 50%, w ith an o v e ra ll in c rease in in te rv e n tio n

to about 80%. The graph sp ikes to 100% four times during the in te r

ven tio n , about 35 percentage p o in ts h igher than th e h ig h est percentage

during b a se lin e . In SD p re se n ta tio n , the experim enter w aited u n t i l

the d a ta showed some v a r ia b i l i ty b efo re in te rv en in g w ith the BARS

th e re , s in ce a l l the components o f SD p re se n ta tio n (a tten d in g ,

1-second pause, and c o rre c t S®) were hovering a t o r around c e i l in g .

There i s an upward trend in s tu d en t a tten d in g and in the 1-second

pause from 74% to 100% in the BARS in te rv e n tio n . However, due to the

end o f the school year, no fu r th e r d a ta were o b ta in ab le ; and given

more d a ta , perhaps a more d e f in i te trend would have appeared.

Student responses throughout th e study appear to improve in

terms o f c o rre c t responses and responses co rrec ted by prompts. The

l a t t e r n e c e s sa r ily co-varied to some ex ten t w ith the percentage o f

c o rre c t prompts by the in s t ru c to r . But the percentage o f co rre c t

s tu d en t responses a lso appears to co-vary w ith the percentage o f cor

r e c t prompts by Subject A. This i s p a r t ic u la r ly ev iden t in Sessions 9

through 14, where in every in s tan ce except Session 12, the percentage

o f c o r re c t s tuden t responses shows a d ire c t re la tio n s h ip w ith the


29

percentage o f c o rre c t prompts by the in s tru c to r . C orrect s tu d en t

responses in c rease from 50% in Session 9 to 73% and 90% in Sessions 10

and 11 re sp ec tiv e ly w ith the advent o f a VR2 schedule o f token re in

forcement. But a f t e r Session 11, s tu d en t response r a te s f e l l to a

lower b u t more c o n s is te n t r a te o f about 65% fo r the rem ainder o f the

study.

Consequation o f s tu d en t responses by Subject B shows a substan

t i a l amount o f v a r ia b i l i ty in b a se lin e . The range of percentage

p o in ts i s from 20 to 87 in th ree dimensions o f consequation and 68

to 100 in two dimensions o f consequation w ith no apparent general

tren d . With the in tro d u c tio n of an tecedent BARS p re se n ta tio n , th is

v a r ia b i l i ty i s reduced w ith a su b s ta n tia l in c rease in c o rre c t conse

quation in two and th ree dim ensions. The range of scores in BARS

in te rv e n tio n i s 60% to 100% in th re e dimensions o f consequation and

77% to 100% in two dimensions o f consequation.

SD p re sen ta tio n in a l l th ree ca teg o rie s (s tuden t a tten d in g ,

1-second pause, and c o rre c t S^) w ith Subject B shows some i n i t i a l

v a r ia b i l i ty but then approaches c e i l in g w ith some v a r ia b i l i ty w hile

s t i l l in b a se lin e . In te rv e n tio n was req u ired a t Session 11 because

o f the nearing o f the c lo se o f the school y ear. However, even w ith a

lack of room fo r improvement, the data do show some in c rease and

narrowing o f v a r ia b i l i ty in percentage p o in ts fo r a ttend ing and

c o rre c t S ^ 's and fo r the 1-second pause except fo r Session 13.

The b a se lin e da ta fo r S ubject B in prompting may be deceiv ing .

In Sessions 5, 7, 8, and 11, only one in c o rre c t prompt occurred .

Because o f th i s , the d a ta could have been very much d if f e r e n t given


30

more o p p o rtu n itie s to prompt. Conceivably, le s s v a r ia b i l i ty could

have occurred i f only a few more c o rre c t prompts were d e liv e red in

Sessions 5 and 8 and even in Session 6, where only two o p p o rtu n itie s

fo r prompting occurred bu t which y ie ld ed 0%. On the o th e r hand,

given more in c o rre c t prompts in Sessions 7 and 11, the da ta again may

have looked very d i f f e r e n t . T herefore, due to the i n s ta b i l i ty and

v a r ia b i l i ty in the b a se lin e prompting d a ta , the experim enter w aited

as long as p o ss ib le to implement an tecedent BARS p re sen ta tio n fo r

in s tru c to r prompting. Upon BARS in te rv e n tio n , no narrowing o f v a r ia

b i l i t y from the l a s t fou r b a se lin e d a ta days i s ev iden t.

The r a te o f c o rre c t s tu d en t responses shows l i t t l e re la tio n sh ip

w ith the percentage o f c o rre c t prompts by S ubject B. However, i t

does show a d i r e c t re la tio n s h ip w ith s tu d en t a tten d in g in Sessions 1

through 8 and some c o -v a ria tio n in the remainder of the se ss io n s . In

a d d itio n , c o rre c t s tuden t responses co-vary w ith the 1-second pause

throughout the study except in Sessions 1 , 6, 10, and 13. And they

co-vary w ith consequation in th ree dimensions in Sessions 2 through 9

and in Session 14.

With Subject C, one f a c t i s re a d ily apparen t. That i s , th a t no

re v e rsa l occurred in any elem ent. And in every elem ent, the same

general statem ent can be made about the e f fe c ts of consequent BARS

p re se n ta tio n . There was moderate v a r ia b i l i ty in b a se lin e w ith some

apparent improvement in In s tru c to r responses given the in tro d u c tio n

o f the BARS. There i s s t i l l fu r th e r improvement in the re v e rsa l and

in BARS re in tro d u c tio n re sp e c tiv e ly . Percentage o f c o r re c t s tu d en t

responses does no t apparen tly co-vary w ith any element o r any segment


31

th e reo f. And th e re Is no general Increase over time in the percentage

of c o rre c t s tu d en t responses.

The b a se lin e prompting da ta fo r S ubject D show a downward trend

from about 80% to about 40%. With the in tro d u c tio n of consequent BARS

p re se n ta tio n , a su b s ta n tia l in c rease in percentage of c o rre c t prompts

i s observed from Session 5 to 7, w ith a gradual decrease from Session 7

to 10. Again, a s u b s ta n tia l in c rease from 50% to 100% occurs in the

remaining four se ss io n s .

SD p re sen ta tio n by Subject D shows a moderate amount o f v a r ia b i l i ty

in b a se lin e , e sp e c ia lly fo r a tten d in g and the 1-second pause. With

the in tro d u c tio n o f the BARS, th e re i s both a red u c tio n in v a r ia b i l i ty

o f and a general in c rease in the percentage of c o rre c t SD,s , s tu d en t

a tten d in g , and the 1-second pause. The percentage of c o rre c t conse

quation in two and th re e dimensions in b a se lin e i s very high and shows

l i t t l e v a r ia b i l i ty . With the in tro d u c tio n o f BARS p re se n ta tio n , how

ever, consequation in th ree dimensions f a l l s f r a s t i c a l l y , w hile conse

quation in th ree dimensions remains very h ig h . C orrected studen t

responses appear to co-vary w ith consequation in two dim ensions, and

th e re appears to be a s l ig h t Improvement in both c a teg o rie s of s tuden t

responses over the course o f the study.

The r e l i a b i l i t y d a ta fo r the observer were c a lc u la te d as the

to ta l number of agreements over the to ta l number of agreements p lus

disagreem ents in th ree repeated observations and d a ta record ings o f

two d if f e r e n t s e ts o f s t im u li (two d i f f e r e n t v ideo tapes o f a c tu a l

experim ental sessions fo r each su b jec t) between su b je c ts over a l l the

dimensions. The r e l i a b i l i t y d a ta f o r Viewing A a re : S ubjec t A, 92%;


32

Subject B, 98%; Subject C, 94%; and Subject D, 89%. The r e l i a b i l i t y

d a ta fo r viewing B a re : Subject A, 95%; S ubject B, 91%; S ubject C,

97%; and Subject D, 90%.


DISCUSSION

I t i s apparent a f t e r a genera l overview of the r e s u l ts of th is

study th a t the in tro d u c tio n o f the BARS had a s l ig h t fu n c tio n a l re la

tio n sh ip w ith an in c rease in the percentage o f c o rre c t in s tru c to r

responses. This i s e sp e c ia lly tru e in the case of Subject A, where

consequation in th ree dimensions conforms c lo se to the 50% c r i te r io n

l in e when nowhere e a r l i e r in her case d id th is occur. BARS e f f e c t on

in s tru c to r responses was a lso apparent w ith S ub jec t B in h er consequa

tio n of s tu d en t behaviors and to a le s s e r ex ten t in her SD p resen ta

tio n . With Subject C, the BARS appear to have had a s l ig h t e f f e c t

upon in s tru c to r responses in a l l the elem ents; b u t the continued

gradual in c rease in these responses during re v e rsa l i s puzzling . And

f in a l ly , except fo r consequation, Subject D’ s in s tru c tio n a l responses

appear to improve w ith the in tro d u c tio n o f BARS p re se n ta tio n .

Before an an a ly s is o f the success o f BARS i s made, a d iscu ssio n

o f i t s shortcomings i s in o rd e r. The f i r s t major apparent f a i lu r e of

BARS was in i t s e f fe c ts upon consequation fo r Subject D. The v a riab le

th a t most d ram atically occurs along w ith the dip in consequation i s a

break in sessio n s o f 5 d a ta days between Sessions 9 and 10 due to a

long Memorial Day c lo sin g of the f a c i l i t y where th is study was con

ducted. Another v a r ia b le th a t occurred here was th a t S ubject D was

tra n s fe rre d to a d if f e r e n t classroom a f te r th a t break than the one

she worked in before i t . Due to th is t r a n s fe r , h e r schedule c o n flic te d

w ith the schedule normally m aintained fo r th is study; and she engaged

in a g re a t deal of behavior to enable h e r s e lf to p a r t ic ip a te in th is

33


34

study . These v a ria b le s may have had some e f f e c t on her performance in

the in s tru c tio n a l se ss io n s . Whether the cause o f the drop in perform

ance was due to these o r o th e r unsuspected v a r ia b le s , i t apparen tly

a lso had a negative e f f e c t on prompting by Subject D, although i t was

no t such a profound e f f e c t . The p o s s ib i l i ty a lso e x is ts th a t the

observer was le n ie n t o r la x in the record ing o f consequation in th ree

dimensions during b a se lin e and then became more observant o f re in fo rc e

ment w ith the In tro d u ctio n o f BARS fo r in s t ru c to r consequation. But

th is i s no t c o n s is te n t w ith the d a ta fo r the o th e r su b je c ts , nor w ith

the r e l i a b i l i t y d a ta .

Another problem w ith the consistency of BARS e f fe c ts on in s tru c to r

behaviors i s the lack of a re v e rsa l in any element fo r S ubject C.

There a re sev e ra l p o ssib le exp lana tions. F i r s t , the BARS may have had

l i t t l e o r no e f f e c t on h e r behavior w ith the improvement in her responses

being due to simple p ra c tic e e f f e c ts . Conceivably, reinforcem ent fo r

her responses could have been programming by growing fa m ilia r i ty w ith

the m a te ria ls or by success w ith s tuden t compliance as a fu n c tio n o f

follow ing the w r it te n D is ta r m a te r ia ls . Another p o s s ib i l i ty i s th a t

the feedback contained in the BARS evoked rule-governed behavior and

th a t th is p e rs is te d from the in tro d u c tio n of BARS. This l a t t e r explana

tio n seems the more parsim onious s in ce th e i n i t i a l in tro d u c tio n of

BARS in consequation and in S® p re se n ta tio n very quickly reduced

v a r ia b i l i ty in the d a ta and improved the in s t r u c to r 's responses. This

i s a lso tru e to a le s s e r ex ten t w ith prompting. In o th e r words, the

m u ltip le b a se lin e appears to have shown an e f fe c t o f BARS in tro d u c tio n .

The major e f fe c t o f BARS in th is study seems to have been due to


35i t s feedback c h a r a c te r is t ic s . The sca le s d id no t con ta in p a r t ic u la r ly

p o s itiv e o r negative la b e ls to serve as re in fo rc e rs o r pun ishers. The

su b jec ts d id , however, vo ice d isp leasu re w ith low rankings and p leasu re

w ith high rankings. The observer who presen ted the sca le s would, upon

the achievement o f a 10 (the h ig h est ra tin g ) , show the sc a le o r sca le s

to the su b jec t and announce, "You're a 1 0 ." The su b jec ts a l l seemed

to enjoy th i s . But o th e r than th i s , th e re was no in h eren t re in fo rc in g

o r punishing stim ulus a sso c ia ted w ith the s c a le s . The su b jec ts a l l

vo lunteered the inform ation th a t the sc a le s served two fu n c tions fo r

them. F i r s t , they showed the in s tru c to r ex ac tly what she d id in

conjunction w ith a d iscu ss io n of her performance w ith the observer.

Secondly, they showed h e r e x ac tly what i t was she had to do to a t t a in

the h ig h est ranking. F in a lly , th e observer was acquainted so c ia lly

w ith a l l the su b jec ts . This may have rendered low scores s o c ia lly

punishing and high sco res s o c ia l ly re in fo rc in g .

C onsisten t w ith the fin d ings of Krumhus and M alott (1980), th e re

was no s ig n if ic a n t d iffe re n c e between the e f fe c ts of BARS p re se n ta tio n

as an an tecedent and as a consequence. However, the d a ta fo r an tece

dent BARS p re sen ta tio n follow a gen era lly more p re d ic ta b le p a tte rn

than the da ta fo r consequent BARS p re sen ta tio n . I f the e f fe c ts of BARS

a re due mainly to th e i r feedback c h a r a c te r is t ic s , then th a t feedback

may be more e f fe c tiv e in evoking rule-governed behav ior when i t i s

presen ted immediately p r io r to the ta rg e ted behav io rs, w hile i t s

e f fe c ts in evoking ru le-governed behavior may dim inish over tim e.

However, th is is an is su e th a t should be d e lib e ra ted in the feedback

l i t e r a tu r e . The d a ta in the p resen t study provide nothing but


36sp ecu la tio n as to the d iffe ren c es between an tecedent and consequent

BARS p re sen ta tio n .

The v a lid a tio n fo r the study o f the e f fe c ts o f using BARS as a

tra in in g to o l i s ev iden t in the da ta fo r s tu d en t responses. The f i r s t

question personnel managers might have about BARS i s "W ill i t improve

performance?" That has been shown to have occurred to some ex ten t.

The second question i s "W ill i t have a p o s it iv e e f f e c t on the rece iv in g

system, the environment, o r the c l ie n te l le ? " In o ther words, w i l l i t

r e s u l t in re a l o rg an iza tio n a l improvement and b e n e f its? P o s itiv e

e f fe c ts in the rece iv in g system a re demonstrated to a moderate bu t

f a i r l y c o n s is te n t e x ten t w ith the improvement in s tu d en t responses.

In D is ta r Math and Language programs, an o v e ra ll success r a te o f 75%

c o rre c t s tuden t responses i s the goal. In the case of S ubjects A and

D, co rrec ted s tuden t responses went from under 75% to 100%; and they

went from under 90% to 100% fo r Subjects B and C. A 25% e r ro r r a te

i s supposed to allow fo r a s l ig h t amount o f programmed f a i lu r e to give

the s tu den ts some inform ation as to what no t to do and to make re in

fo rc e rs more e f fe c t iv e . But in th is study , the high response ra te s

served to v a lid a te the use o f BARS as a tra in in g to o l.

The p re sen t study d i f f e r s experim entally from previous s tu d ie s

in BARS ap p lic a tio n s in ce those, alm ost ex c lu s iv e ly , have d e a l t w ith

psychometric comparisons o f BARS w ith o th e r ra tin g form ats and have

no t d e a lt w ith e f fe c ts o f BARS on behavior o r o th e r o rg an iza tio n a l

r e s u l t s . Ivancevich (1980b) rep o rted an improvement in scheduling

performance by p ro fess io n a l employees, bu t th is was no t q u a n tif ie d .

This study was an a ttem pt on a b a sic and em pirica l le v e l to determ ine


37

whether BARS would have any tru e e f f e c t upon performance. The impor

tance o f th is study may no t l i e in the re s u l ts o f the experiment (or

the lack th e re o f) , b u t i t may w ell l i e in the precedent i t has s e t in

seeking and e s ta b lish in g beh av io ra l goals fo r and e f fe c ts o f BARS

a p p lic a tio n .

Landy and Barnes (1979) have s e t down sp e c if ic p o in ts of lo g ic in

using BARS. The people who use the sca le s have to some ex ten t in most

cases been i t s developers as w e ll. More inform ation i s provided w ith

behav io ral anchors than w ith sim ple heading d e sc r ip to rs . These anchors

have s im ila r meanings to a l l r a t e r s , and the performance dimensions in

BARS a re e a s ily d is tin g u ish ab le from one another.

And a r is in g need fo r accu ra te performance a p p ra isa ls makes BARS

perhaps the b e s t choice o f sev e ra l e v i l s s in ce , as C hurch ill sa id of

democracy, " I t ' s a t e r r i b l e system b u t i t ' s the b e s t th e re i s . "

Keaveny and McGann (1975) p o in t ou t th a t accu ra te performance a p p ra isa ls

allow fo r proper reward and promotion fo r e f fe c t iv e and high perform ing

personnel. This would be a break fo r employers who a re co n stan tly

seeking b e t te r q u a lif ie d persons to f i l l im portant p o s itio n s . On the

o th e r end o f the spectrum, w hite c o l la r employees a re d isp u tin g d is

charges more freq u en tly ; and an a ccu ra te a p p ra isa l of poor performance

would f a i r ly support such d isch arg es . So in terms of r e a l organiza

tio n a l c o s ts , accu ra te and e f fe c t iv e ap p ra isa ls o f performance a re a

grave n ecess ity .

The v a r ia t io n o f the sc a le s used in the p re sen t study from the

ty p ic a l BARS approaches a s im ila r i ty to the Behavior O bservation

S cale, o r BOS. The in te n t o f the BOS i s to equalize the value o f


38re sp e c tiv e le v e ls of performance across behavior item s (Latham, Fay, &

S a a ri, 1979). The BOS c o n s is ts o f performance dimensions named

" c r i t e r i a . " w ith subsets of "behav iora l item s." The c r i t e r i a and

item s a re generated in a manner s im ila r to the development o f per

formance dimensions and b ehav io ra l anchors fo r a BABS. The b ehav io ra l

item s in a BOS number from 5 to 10 in each of 5 to 10 c r i t e r i a . The

item s a re underscored by groups o f o b serv a tio n a l frequencies ( i . e . , 0 -

20%, 21 - 40%, 41 - 60%, 61 - 80%, and 81 - 100% frequency o f observa

tio n o f the behav io ral item ), the ap p ro p ria te category of which i s

checked a t the time of the performance ev a lu a tio n (Kane & B ernardin ,

1982).

As w ith every performance ev alu a tio n to o l reviewed by the pre

sen t au tho r, problems w ith the BOS have been p resen ted . Kane and

B ernardin (1982) claim th a t th e goal of the BOS, which i s to equalize

re sp ec tiv e le v e ls o f frequency, i s f a l l i b l e in th a t eq u a lity o f f r e

quency o f observation does no t n e c e s s ita te e q u a lity o f the w eight of

the frequency le v e ls between item s. For In s tan ce , they p o in t out th a t

equal frequencies o f o b ta in in g homicide a r r e s t w arrants and being

c lea red by the In te rn a l Review Board fo r the use of f a t a l fo rce by a

p a r t ic u la r p o lic e o f f ic e r i s unacceptable. E ighty percen t frequency

of o b ta in in g homicide a r r e s t w arrants may be su p e r io r , but being

c leared by the In te rv a l Review Board 80% of the time i s horrendous.

Latham, S aa ri, and Fay (1980) claim th a t th is e q u a li ty of frequency

le v e ls does e x is t but no d e f in i t iv e re so lu tio n s have been achieved

and the debate has become heated (Bernardin & Kane, 1980; Latham,

S a a ri, & Fay, 1980).


39In review , the BARS system c o n s is ts of performance dimensions

underscored by a sca le which re p re sen ts poor to e x ce lle n t performance

w ith b ehav io ra l anchors a ttach ed to the sca le values d esc rib in g

examples of poor to ex ce lle n t perform ance. The sca le s used in the

p resen t study c o n s is t of performance dimensions underscored by sca le s

rep resen tin g poor to e x ce lle n t performance w ith behav io ral anchors

c o n sis tin g of frequencies of observation of s e ts of behaviors a ttached

to the sca le values to d esc rib e poor to e x c e lle n t performance. I t i s

these frequencies of performance which resemble the frequencies of

observation o f behaviors in the BOS. While these frequencies do

resem ble, and indeed serve a s im ila r purpose a s , th e BOS, th e re a re

im portant d iffe ren c es to make c le a r . F i r s t , the frequencies provided

w ith in the sca le s in the p resen t study a re a departu re from a ty p ic a l

BARS which c o n s is t o f a d je c t iv a l ra th e r than num erical d e sc rip tio n s

of poor to e x ce lle n t performance comprising a performance dimension.

This d epartu re was u t i l i z e d to maximize the o b je c tiv e observation and

ev alu a tio n of su b jec t behavior and was conceived to make an em pirical

assessment o f the BARS as a t ra in in g to o l ra th e r than to c re a te a new

performance ev alu a tio n to o l . Second, the re sp ec tiv e frequency le v e ls

o f performance a re no t and were not intended to be p a r a l le l across

performance dimensions; and the ob serv a tio n a l frequency le v e ls must

be p a ra l le l across behavior items to q u a lify as a BOS.

D espite the fa c t th a t the p resen t study concentrated on assessin g

the BARS as a tra in in g to o l , i t does not mean th a t i t may not provide

im portant inform ation ap p licab le to the BOS and o th e r ev a lu a tio n to o ls .

As has been pointed out e a r l i e r , i t may w ell be the case th a t the


40

method o f o b ta in ing the inform ation p e r tin e n t to performance ev a lu a tio n ,

the inform ation a c tu a lly used, and the way i t i s used ( i . e . , w i l l any

change in the environment occur as a r e s u l t of the evalua tion ) a re

more im portant than the name of the to o l used or the way the informa

t io n i s d isp layed .

P eterson (1982) provides some suggestions fo r the use of term in

ology in feedback s tu d ie s . He claim s th a t the term "feedback" has

become "p ro fess io n a l s lang" and th a t i t should not be used as a des

c r ip t io n o f the in te rv e n tio n a l procedure. He suggests th a t au thors in

o rg an iza tio n a l behavior management l i t e r a tu r e do not s u f f ic ie n t ly

d e fine the s p e c if ic feedback procedures used and th a t these sp e c if ic

d e sc rip tio n s must be provided to in te rp re t and analyze the procedures

and th e i r r e s u l t s . T h is , he says, w i l l f a c i l i t a t e the a n a ly s is of the

e f fe c ts of feedback and th a t th is i s o f paramount importance in o rder

to move away from unfounded assum ptions in the l i t e r a tu r e about the

e f fe c ts feedback has on performance so th a t feedback e f fe c ts can be

assessed . When d e sc rip tio n s of s p e c if ic feedback procedures a re not

provided in the l i t e r a t u r e , the behav io ra l e f fe c t of feedback ( i . e . ,

re in forcem ent, punishment, stim ulus c o n tro l) cannot be assumed; and the

resea rch cannot be in te rp re te d in any meaningful way. S pecifiy ing

experim ental procedures and th e i r r e s u l t s should be, in th is a u th o r 's

op in ion , the focus o f subsequent s tu d ie s in using feedback to a l t e r

behavior and improve perform ance. These p o in ts can a lso be applied

to s tu d ie s involving ra t in g sc a le evalu a tio n s o f performance since

adequate d e sc rip tio n s o f the way in which the ra t in g sc a le s a re used

has freq u en tly not been sp e c if ie d in performance ev alu a tio n s tu d ie s .


41Remedies to these problems w ith feedback and ev alu a tio n s tu d ie s were

sought in the p resen t study by providing the sca le s used and d e ta ile d

d e sc rip tio n s o f the procedures.

More s tu d ie s l ik e the p re sen t one should be conducted in the

fu tu re to determ ine the ex ten t o f the e f f e c t BARS can have on an

o rg an iza tio n . The au thor would no te th a t i t has been encouraging to

see so many s tu d ie s devoted to the development of d e a r and o b jec tiv e

statem ents about o rg a n iz a tio n a l perform ance. As Drucker (1973) has

sa id :

An employer has no business w ith a man's p e rso n a lity . Employment i s a s p e c if ic c o n tra c t c a l l in g fo r s p e c if ic perform ance, and fo r nothing e ls e . Any a ttem pt o f an employer to go beyond th is i s u su rpa tion . I t i s immoral as w e ll a s i l l e g a l in tru s io n in to p rivacy . I t i s abuse of power. An employee owes no " lo y a lty "— he owes performance and nothing e ls e . . . . Management and manager development should concern themselves w ith changes in behav* io r l ik e ly to make a man more e f fe c t iv e , (p . 424-425)

I t i s the conclusion o f the au thor th a t the BARS may serve as an

e f fe c t iv e tra in in g to o l. .This may be accomplished by using the sca le

i t s e l f as a feedback mechanism as was done in th is study , o r by using

the inform ation gained in t r a d i t io n a l BARS development procedures to

develop a programmed in s tru c t io n o r o th e r behavior-based tra in in g pro

gram. But before th is can be a c e r ta in ty , more evalu a tio n s of BARS

as a tra in in g to o l must be made. The au thor suggests a move away from

psychometric evalua tions o f BARS un less new developments a re m anifest

in BARS and o th e r performance ev alu a tio n procedures. As B em ardin and

Kane (1980) p o in t o u t, " I t i s our view th a t another round of em pirica l com

parisons between form ats th a t do no t d ep art appreciab ly from prev iously

te s ted methods i s unwarranted" (p . 810). A behav io ral ev a lu a tio n of


42

BARS by experienced developers would be paramount. With the advent of

BARS th a t a re developed through the more t r a d i t io n a l means y ie ld in g

psychom etrically sound s c a le s , more profound tra in in g e f fe c ts o f BARS

than in th is study may become apparent.


APPENDIX A: SD PRESENTATION

90%—A ttending 90%—1 -sec . pause 90%~SD, s ig n a l exact 90%—C lear p re sen ta tio n

i—lO

9 -

90%—A ttending 90%—1 -sec . pause 70%—S°, s ig n a l exact 70%—C lear p re sen ta tio n

- 8

7 -

70%—A ttending 70%—1 -sec . pause 90%—SD, s ig n a l exact 90%—C lear p re sen ta tio n

- 6

50%—A ttending 50%—1 -sec . pause 50%—Unclear SD 50%—S®, s ig n a l inexact

3 -

Hrong S®

1 -

Wrong S®No a ttend ing

90%—A ttending 90%—SD, s ig n a l exact 90%—C lear p re sen ta tio n 70%—1 -sec . pause

90%—A ttending 70%—1 -sec . pause 70%—SD, s ig n a l exact 70%—C lear p re sen ta tio n

70%—A ttending 50%—1 -se c . pause 50%—SD, s ig n a l inexac t 50%—SD un c lea r

- 4

No S1

- 2

No a tten d in g

1— 0

43


APPENDIX B: PROMPTING

90%—A fte r 3 sec . of no responding

90%---A fter wrong response 90%—Target response achieved

9 -

70%—A fte r 3 sec . of no responding

70%—A fte r wrong response 90%—Target response achieved

7 -

50%-—Sometime a f t e r 3 sec . o f no responding

50%~More than 3 sec. a f t e r wrong response

70%—Target response achieved5 -

50%—-Sometime a f te r 3 sec . of no responding

50%—More than 3 sec. a f t e r wrong response

50%—-Target response achieved3 -

70%—No ta rg e t response 50%—In c o rre c t prompt

1 -

50%—No prompt a f te r no o r wrong response

r lO

90%—A fte r 3 sec . o f no responding 90%—A fte r wrong response .70%—T arget response achieved

- 8

70%-—A fte r 3 sec . o f no responding 70%—A fte r wrong response 70%—Target response achieved

- 6

-4

50%—Before 3 sec . o f no responding 50%—Before wrong response

- 2

50%—A fte r c o rre c t response

u 0

44


APPENDIX C: CONSEQUATION

90%~Sr+ w /in 1 sec. o f response 90%—Tangible Sr +, d e sc r ip tio n ,

inm ediate p ra is e ( th ree dimensions)

9 -

70%—Sr+ w /in 1 sec. o f response 90%—-Sr+ in th ree dimensions

50%—Sr + up to 5 sec. a f te r response

90%—Sr + in th ree dimensions

a f te r50%—Sr + up to 5 sec response

90%—Sr + in one dimension

30%—Sr + fo r wrong response

50%—Sr+ fo r wrong response

7 -

5 -

3 -

1 -

r 10

90%—Sr+ w /in 1 sec . o f response 90%—Sr+ in two dimensions

- 8

70%—Sr+ w /in 1 sec . o f response 90%—Sr in two dimensions

- 6

50%—Sr+ up to 5 sec . a f t e r response 90%—Sr + in two dimensions

- 4

30%—No Sr + fo r response

- 2

50%—no Sr+ fo r response

- 0

45


APPENDIX D: SESSION MONITORING FORM

Tutor:___________________ __________ _ P u p il:Date: Classroom:Session Time: ______________ ____________ Monitor:

Monitor da te:

I s the procedure a f fe c tin g behavior change in the d esired d ire c tio n ? YES NO

Number o f days on th is s te p / phase:

General comments:

Tutor s ig n a tu re :

M onitor da te:

I s the procedure a ffe c t in g behavior change in the d e sired d irec tio n ? YES NO

Number o f days on th is s te p / phase: _____

General comments:

Tutor s ig n a tu re :

46


TRIA

L/IN

TERV

AL

PRETASK T PROMPTS POSTTASK

INAP

P RE

SP

CONS

EQUE

NCE

A.

Atte

ndin

g?

B. 1

-sec

ond

paus

e?C.

SD

corr

ect?

D.

R co

rrec

t?E.

Prom

pt

corr

ect?

1. L

ate

2. No

targ

et

R

F.

Con

seq.

co

rrec

t?

1. In

appr

opri

ate

2. No

t Im

med

iate

3. No

des

crip

tio

i

123456789

10

TRIA

L/IN

TERV

AL

PRETASK T PROMPTS POSTTASK

INAP

P RE

SP

CONS

EQUE

NCE

A.

Atte

ndin

g?B.

1-

seco

nd

paus

e?C.

SD

corr

ect?

D.

R co

rrec

t?E.

Prom

pt

corr

ect?

1. L

ate

2. No

targ

et

R

F.

Con

seq.

co

rrec

t?

1. In

appr

opri

ate

2. No

t im

med

iate

3. No

desc

ript

ion

123456789

10

APPENDIX E: INTERACTIONS BETWEEN OBSERVER AND SUBJECT

When the BARS were p resen ted as an teceden ts and as consequences,

the experim enter in te ra c te d w ith the su b jec ts to exp lain exactly what

the s c a le feedback rep resen ted . Examples of th is in te ra c t io n follow .

The experim enter in te ra c te d w ith each su b je c t in d iv id u a lly .

F i r s t , he would p re sen t the s c a le (s ) and sim ply r e c i te the inform ation

contained in the s c a le . Then, he would r e la te th a t inform ation to the

previous d ay 's performance on each s c a le . In o th e r words, "You scored

a 4 today, and th a t i s worse than y este rd ay ," o r "You scored a 9 today,

and th a t i s b e t te r than y e s te rd a y ." This was done to give the sub

je c t s some id ea o f the p rogress o f th e i r performance. A fte r these

d a ily in te ra c t io n s , the experim enter asked the su b jec ts i f th e re were

any questions about the s c a le , th e i r perform ance, o r the experim ent.

Almost ex c lu s iv e ly , they would ask fo r an a n a ly s is of th e i r performance,

such a s , "How should I p re sen t th is new task?" o r "What should I do

about a c h i ld 's behavior problem?" Or they would ask what they should

do s p e c if ic a l ly to improve th e i r r a t in g . O ccasionally , they would make

suggestions fo r amendment o f the w r it te n procedure. And every su b jec t

asked a t l e a s t once how the experiment in general was going, but th is

was ju s t out o f c u r io s i ty about how an experiment i s run.

Regarding questions asked about how to deal w ith the s tu d en ts ,

because the s tu d e n t 's b e s t in te r e s t was o f g re a te s t im portance, the

observer would do whatever was necessary to help the su b jec t deal w ith

the s tu d en t, e sp e c ia lly in the case of the behavior problem. I f the

question d e a lt w ith the s p e c if ic D is ta r procedures involved, the sub

je c t was f i r s t d ire c ted to read the m anual's in s tru c tio n s ; and nex t,

i f necessary , the su b jec t was d ire c ted to read over the ra tin g sca le s

47


48

s in ce these included a l l the inform ation necessary to conducting an

In s tru c tio n a l sessio n w ith in the f a c i l i t y . And regard ing questions

about the resea rch p ro je c t , they were answered as honestly as p o ssib le

s in ce th is was a p o licy o f the experim enter.


BIBLIOGRAPHY

A llan , P . , & Rosenberg, S. The development of a ta sk o r ie n te d approach to performance ev a lu a tio n in the c i ty o f New York. P ublic Personnel Management, 1978, 1 , 26-32.

A tkin, R. S . , & Conlon, E. J . B ehaviorally anchored ra tin g s c a le s :Some th e o re tic a l is su e s . Academy of Management Review, 1978, 3., 119-128.

B ernardin, H. J . B ehavioral ex p ec ta tion s c a le s : A f a i r e r comparison.Journal of Applied Psychology, 1977, 62, 422-427.

B ernardin, H. J . , A lvarez, K. M., & Cranny, C. J . A recomparison of behav ioral ex p ec ta tion sca le s to summated s c a le s . Jou rnal o f Applied Psychology, 1976, 61, 564-570.

B ernardin, H. J . , & Kane, J . S. A second look a t b ehav io ra l observatio n s c a le s . Personnel Psychology, 1980, 33, 809-814.

B ernardin, H. J . , & Smith, P. C. A c la r i f i c a t io n o f some is su e s regarding the development and use o f b eh av io ra lly anchored ra t in g sca le s (BARS). Journal o f Applied Psychology, 1981, 66, 458-463.

Blood, M. R. S p in -o ffs from b ehav io ra l ex p ec ta tion s c a le procedures.Journal o f Applied Psychology, 1974, 59, 513-515.

Borman, W. C. Exploring upper l im i ts o f r e l i a b i l i t y and v a l id i ty in job performance r a t in g . Jou rna l o f Applied Psychology, 1978, 63, 135-144.

Borman, W. C ., & Dunnette, M. D. Behavior based versus t r a i t o rien ted performance ra tin g s : An em pirica l study . Jou rna l o f AppliedPsychology, 1975, 60, 561-565.

Borman, W. C ., & V allon, W. R. A view o f what can happen when behavio r a l expec ta tion sc a le s a re developed in one s e t t in g and used in ano ther. Journal of Applied Psychology, 1974, 50, 197-201.

Campbell, J . P . , Dunnette, M. D ., Arvey, R. D ., & H e lle rv ik , C. V.The development and ev a lu a tio n o f beh av io ra lly based ra tin g sca le s . Journal o f Applied Psychology, 1973, 57, 15-22.

C araine, C. W., & Fink, W. T. Increasing the r a te o f p re sen ta tio n and use o f s ig n a ls in elem entary classroom teach e rs . Journal o f Applied Behavior A nalysis, 1978, 11, 35-46.

Das, H ., F ro s t, P. J . , & Barnowe, T. B ehaviorally anchored ra t in g sca le s fo r assessing behav io ral sc ience teach ing . Canadian Journal o f Behavioural Science, 1979, 11, 79-88.

49


50

D eC otiis, T. A. An a n a ly s is o f the ex te rn a l v a l id i ty and applied re levance of th ree r a t in g form ats. O rgan izational Behavior and Human Performance. 1977. 19, 247-266.

D eC otiis, T. A. A c r i t iq u e and suggested re v is io n o f beh av io ra lly anchored ra t in g sc a le s development procedures. Educational and Psychological Measurement, 1978, 38, 681-690.

D ickinscn, T. L . , & Z e llin g e r , P. M. A comparison o f the b eh av io ra lly anchored ra t in g and mixed standard sc a le form ats. Journal of Applied Psychology, 1980, 65, 147-154.

Drucker, P. Management: Tasks, r e s p o n s ib i l i t ie s , and p ra c t ic e s .New York: Harper & Row, 1973, 424-425.

Fabry, P. L . , & Reid, D. H. Teaching fo s te r grandparents to t r a insevere ly handicapped persons. Journal o f Applied Behavior A nalysis,1978, 11, 111-123.

Flanagan, J . C. C r i t ic a l requirem ents: A new approach to employee ev a lu a tio n . Personnel Psychology, 1949, 2 , 419-425.

Ivancevich, J . M. B ehavioral exp ec ta tio n sca les versus nonanchored and t r a i t ra tin g system s: A s a le s a p p lic a tio n . Applied Psycholog ic a l Measurement, 1980a, _4, 131-133.

Ivancevich , J . M. A lo n g itu d in a l study o f b eh av io ra l expec ta tionsc a le s : A ttitu d e s and performance. Journal o f Applied Psychology,1980b, 65, 139-146.

Jacobs, R ., Kafry, D ., & Zedeck, S. Expectations o f behav io ra lly anchored s c a le s . Personnel Psychology, 1980, 33. 594-640.

Johnston, J . M., & Pennypacker, H. S. S tra te g ie s and ta c t ic s of human behav io ra l re se a rc h . H ills d a le , N. J . : Erlbaum, 1980.

Kafry, D ., Jacobs, R ., & Zedeck, S. D isc r im in a b ility in multidimens io n a l performance ev a lu a tio n s . Applied Psychological Measurement,1979, 3, 187-192.

Kane, J . S . , & B ernardin, H. J . B ehavioral observation sc a le s and the ev a lu a tio n o f performance a p p ra isa l e f fe c tiv e n e s s . Personnel Psychology, 1982, 35, 635-642.

Kane, J . S ., & Lawler, E. E ., I I I . Performance a p p ra isa l e f fe c t iv e ness: I t s assessm ents and determ inan ts. Research in O rganizationalBehavior. 1979, 1, 425-478.

Kearney, J . W. The value o f b eh av io ra lly based performance a p p ra isa ls . Business Horizons, 1976, 3 , 75-83.


51

Keaveny, T. J . , & McGann, A. F. A comparison o f behav io ral expec ta tion sca le s and graphic r a t in g s c a le s . Jo u rn a l of Applied Psychology, 1975, 60, 695-703.

Keeley, M. A contingency framework fo r performance ev a lu a tio n . Academy o f Management Review, 1978, 3̂ , 428-438.

Kingstrom, P. D ., & Bass, A. R. A c r i t i c a l a n a ly s is o f s tu d ie s comparing beh av io ra lly anchored ra t in g sc a le s (BARS) and o th e r ra tin g form ats. Personnel Psychology. 1981, 34, 263-289.

Krumhus, K. M., & M alo tt, R. W. The e f fe c ts of m odelling and immediate and delayed feedback in s t a f f t ra in in g . Journal of O rganizational Behavior Management, 1980, 2 , 279-293.

Landy, F. J . , & Barnes, J . L. Scaling behav io ral anchors. Applied Psychological Measurement, 1979, _3, 193-200.

Landy, F. J . , & Guion, R. M. Development of sc a le s fo r the measurement of work m otivation . O rgan izational Behavior and Human Performance, 1970, _5, 93-103.

Landy, F. J . , & Trumbo, D. A. Psychology o f Work Behavior. Homewood, 111.: Dorsey, 1976.

Latham, G. P . , Fay, C. H ., & S a a ri, L. M. The development o f behavior observation sca le s fo r ap p ra isin g th e performance o f foremen. Personnel Psychology, 1979, 32, 299-311.

Latham, G. P . , S aa ri, L. M., & Fay, C. H. BOS, BES and baloney:Raising Kane w ith B ernardin . Personnel Psychology, 1980, 33,815-821.

McCormick, E. J . , & T if f in , J . T. In d u s tr ia l Psychology. Englewood C l i f f s , N. J . : P ren tic e -H a ll, 1974.

M i l l a r d , C. W., Luthans, F . , & Ottemann, R. L. A new breakthrough fo r performance a p p ra isa ls . Business H orizons, 1976, 19, 66-73.

P e terson , N. Feedback i s n o t a new p r in c ip le of behavior. The Behavior A nalyst, 1982, _5, 101-102.

Prue, D. M., & Fairbank, J . A. Performance feedback in o rg an iza tio n a l behavior management: A review . Journal o f O rgan izational BehaviorManagement, 1981, J3, 1-16.

Schwind, H. F. An a l te rn a t iv e to beh av io ra lly anchored ra tin g sca le s : The behavior d e sc r ip tio n index. Academy of Management Proceedings, 1978, 38-42.


52

S epler, H. J . , & Myers, S. L. The e ffe c tiv en e ss o f v e rb a l in s tru c tio n on teaching behavior m od ifica tio n s k i l l s to nonpro fessionals . Journal of Applied Behavior A nalysis, 1978, 11, 198.

Shapira, Z ., & Shirom, A. New is su e s in the use o f b eh av io ra llyanchored ra tin g s c a le s : Level o f a n a ly s is , the e f fe c ts o f in c id en tfrequency, and ex te rn a l v a lid a tio n . Jou rna l of Applied Psychology, 1980, 65, 517-523.

Shook, G. L ., Johnson, L. M., & Uhlman, W. F. The e f fe c t o f response e f f o r t red u c tio n , in s tru c t io n s , group and in d iv id u a l feedback, and reinforcem ent on s t a f f perform ance. Jou rna l of O rgan izational Behavior Management, 1978, 206-215.

Smith, P. C ., & K endall, L. M. R e tran s la tio n o f ex p ec ta tio n s: Anapproach to the c o n stru c tio n of unambiguous anchors fo r ra tin g sc a le s . Journal o f Applied Psychology, 1963, 47, 149-155.

S tap le s , W. A ., & Locander, W. B. B ehav io ra lly anchored sc a le s : Anew to o l fo r r e t a i l management ev a lu a tio n and c o n tro l. Journal o f R e ta il in g . 1972, 52, 39-48, 94-95.

Vance, R. J . , Kuhnert, K. W., & F a rr , J . L. In terv iew judgements: Using ex te rn a l c r i t e r i a to compare behav io ral and graphic sca le r a t in g s . O rgan izational Behavior and Human Performance, 1978, 22, 279-294.

Wexley, K. N ., & Yukl, G. A. O rgan izational behavior and personnel psychology. Homewood, 111: R. D. Irw in , 1977.

W illiam s, M. R. Performance ap p ra isa l in management. London: Heinemann, 1972.

Zedeck, S . , & Baker, H. T. Nursing performance as measured by behavio r a l expec ta tion sc a le s : Some th e o re t ic a l is su e s . Academy ofManagement Review. 1972, _3, 119-128.

Zedeck, S . , Kafry, D ., & Jacobs, R. Format and scoring v a r ia tio n s in behav io ral expec ta tions ev a lu a tio n s . O rgan izational Behavior and Human Performance, 1978, 7 , 457-466.


The Effects of Behaviorally Anchored Rating Scales as ...

Documents

Transcript of The Effects of Behaviorally Anchored Rating Scales as ...