Post on 31-Oct-2019
International Journal of Disability, Development and EducationVol. 54, No. 1, March 2007, pp. 51–89
ISSN 1034-912X (print)/ISSN 1465-346X (online)/07/010051–39© 2007 Taylor & FrancisDOI: 10.1080/10349120601149755
Inside the Black Box of School Reform: Explaining the how and why of change at Getting Results schools
Dennis McDougalla*, William M. Saundersb and Claude GoldenbergcaUniversity of Hawai’i, USA; bPearson Achievement Solutions, USA; cCalifornia State University Long Beach, USATaylor and Francis LtdCIJD_A_214904.sgm10.1080/10349120601149755International Journal of Disability, Development and Education1034-912X (print)/1465-346X (online)Original Article2007Taylor & Francis541000000March 2007DennisMcDougallmcdougal@hawaii.edu
This article reports key findings from a process-focused external evaluation that compared a subsetof Getting Results project schools and comparison schools in order to understand the dynamics ofschool-wide reform efforts at these primary schools. Findings shed light on the “black box”of school reform and illuminate the limited empirical basis for understanding the inner workings ofmost reform efforts. We describe how Getting Results Model elements—goals, indicators, assis-tance, leadership, and settings—worked in concert to improve teaching and learning at projectschools. We also describe factors that inhibited and promoted change, as well as implications forhow whole-school reform might be accomplished through purposeful manipulation of these essen-tial change elements.
Keywords: Change; Evaluation; Process; Reform; Schools
Introduction
Whole-school reform proliferated in the United States (U.S.) during the 1990s andreached a high point with passage of Public Law 105-78, commonly know as “Obey-Porter”, in 1997. The bill authorised the Comprehensive School Reform Demon-stration Program and triggered hundreds of whole-school reform efforts nationwide.Many promising approaches to whole-school reform have been developed over thepast 15 years (American Institutes for Research, 1999; Borman, Hewes, Overman,& Brown, 2003; Desimone, 2002; Slavin & Fashola, 1998). Even before this recent
*Corresponding author. University of Hawai’i, Department of Special Education, 124 Wist Hall,1776 University Avenue, Honolulu, HI 96822, USA. Email: mcdougal@hawaii.edu
52 D. McDougall et al.
wave of whole-school reform models, educators have for years attempted to findways to improve entire schools (see, e.g., the “effective schools” research of the1970s and 1980s; Bliss, Fireston, & Richards, 1991). However, the empirical basisfor understanding the actual process of school reform is very limited. Few studieshave examined the effects of reform models within experimental or quasi-experi-mental research designs that would permit clear conclusions about the effects ofreform models on student outcomes. Even fewer studies have looked directly at theprocess of reform to examine prospectively the dynamics leading to school improve-ment. Fullan (2000) reviewed research on schools with effective collaborative schoolcultures and noted, “We know nothing about how these particular schools got thatway, let alone how to go about producing more of them” (p. 582). Although theliterature systematically examining the processes and outcomes of school improve-ment efforts has grown enormously over the past decade (e.g., Berends, Kirby,Naftel, & McKelvey, 2001; Borman et al., 2003; Desimone, 2002), the problemidentified by Sarason (1972) 35 years ago is still largely with us: We have very littledirect observational data to document how schools change from being less to moreeffective in educating their students. The problem is significant because withoutdetailed knowledge of how schools change, we know little about why school reformefforts obtain the results they do (Desimone). This is the “black box” problem weaddress in this article.
The study we report here contributes to the literature on school change for diverseand traditionally low-achieving populations of students. We report findings from aqualitative, prospective, external evaluation (McDougall, 2002) that illuminated theinternal workings of a whole-school reform project. The Getting Results (GR) Modelthat guides this school change project is comprised of five interdependentelements—goals, indicators, assistance, leadership, and settings. The evaluationstudy answers a set of five questions about model implementation, processes, andoutcomes. We discuss findings in relation to school reform and the importance ofunderstanding its inner workings.
Background and Basis of the GR School Change Model
In a previous project, we developed a school change model that was instrumental inproducing substantial changes in teaching and learning at one pilot site—a primaryschool that served primarily Latino children and families, in Southern California(Goldenberg, 2004; Goldenberg, Saunders, & Gallimore, 1996; Goldenberg &Sullivan, 1994). We refer to this model as the School Change/Getting Results Model,or, more simply, the GR Model. Over a 6-year period, the pilot school shifted frombeing the lowest achieving school in the school district to surpassing district averageson both standardised tests and performance-based assessments. Fashola, Slavin,Calderón, and Durán (1996) identified the GR Model as one of only three schoolchange models with demonstrated effectiveness in majority Latino schools.
The GR Model utilises five elements to leverage changes in educators’ instructionalbehaviours and attitudes, and student outcomes. These elements include: (a) goals
Inside the Black Box of School Reform 53
that are set, shared, and articulated explicitly by the school community; (b) meaning-ful indicators that measure progress toward goal attainment; (c) assistance by capableothers from within and outside the school; and (d) focused leadership that supportsand pressures goal attainment. “Settings” is a fifth element and a super-ordinateconcept in the model. Implementing the GR Model requires establishing new settingsand reformulating existing settings for educators to meet, collaborate, receive assis-tance, and do important, instructional tasks (Sarason, 1972; Tharp & Gallimore,1988). Within the context of these settings, the other change elements work inconcert to improve teaching, learning, and achievement in any targeted curriculumor subject. In working with schools, we discovered a need to focus specifically andconsistently on creating and sustaining concrete settings, at each school, wherechange elements are operationalised in effective ways—ways that permit educators todo the very important work of improving teaching and learning.
Research provides strong support for GR Model elements. Classic, and morerecent, educational research documents the importance of explicit academic goals(Carter & Chatfield, 1986; Peterson & Lezotte, 1991). Goals are important forachieving long-term change and substantive improvements because they are vital formaintaining a coherent and stable student-centered vision. Indicators that measureprogress toward agreed-upon goals reinforce the importance of the learning goals andhelp teachers and administrators gauge their goal-directed efforts. Consistent use ofachievement indicators is related to improvements in student outcomes (Brophy &Good, 1986; Edmonds, 1979). Assistance from fellow professionals (Lieberman,1988a, b; Little, 1982; Rosenholtz, 1991), including training from consultants (Joyce& Showers, 1983), is essential for successful reform (Saunders et al., 2001). Success-ful professional development models go well beyond “one-shot” workshops andpresentations that attempt to train teachers in short order (Goldenberg & Gallimore,1991). Professional development must be seen as long-term assistance to help educa-tors acquire knowledge and skills essential for accomplishing agreed-upon goals(Darling-Hammond, 1997). Finally, leadership must both support and pressurechanges necessary for goal attainment (Fullan, 1993). These two leadership dimen-sions complement each other, producing a creative tension that is perhaps the mostelusive but important aspect of leadership (Blase, 1987; Bliss et al., 1991; Miles,1983). Effective school leadership has long been recognised as fundamental to creat-ing more effective schools, regardless of the cultural or linguistic background ofstudents (August & Hakuta, 1997). The GR Model treats all five elements, includingleadership, as highly interdependent. Strong leadership artfully combines pressureand support in a way that moves schools relentlessly toward accomplishing studentachievement goals, utilising indicators, cultivating assistance and collaboration, andbuilding productive school settings.
Current “Scale-up” Phase of the GR Project
The most current GR Model project “scaled up” our previous school reform effortsand achieved simultaneously, at nine schools, improvements in teaching and
54 D. McDougall et al.
learning similar to those achieved at the original pilot school. We examined prospec-tively a model of school change within a quasi-experimental research design. Fifteenelementary schools participated during this scale-up phase—nine schools that usedthe GR Model, all of whom entered the study and initiated GR Model implementa-tion voluntarily, and six demographically similar schools in the same district thatagreed to be comparison sites. At the study’s inception, GR schools and comparisonschools, as two groups, had nearly identical mean achievement scores on annual,statewide, standardised tests. These K-5 schools served predominantly poor andworking class Hispanic communities. GR replication sites utilised modest localfunding—US$100,000 per school over a 3-year period—to support implementation.A research grant supported GR staff’s assistance to schools, as well as data collectionand analyses.
As members of the same, large, urban school district, all schools (both GR andcomparison) functioned under the same reform umbrella set forth by Californiastate legislation and local school district mandates. Four major reforms wereunderway in this local school district when the scale-up project began in 1997: (a)class size reduction (20 students to one teacher) at grades K-3; (b) state-established and district-implemented content standards for language arts andmathematics; (c) annual achievement accountabilities and reporting based on state-mandated, standardised tests; and (d) common curriculum, mandated training,and school-based coaches for reading/language arts and mathematics. Primaryschools in this district showed steady increases in student achievement since thelate 1990s, very probably related to these major district and state reform efforts.Our quantitative evaluation of achievement impact (Saunders, 2003), focused onthe additive effects of the GR Model, and the extent to which outcomes andincreases over time at GR schools surpassed those of other, comparable, schools inthis large, urban district.
Putting GR Model elements into practice. In brief, GR Model implementation estab-lished several settings and processes designed to ensure effective application ofmodel elements—goals, indicators, assistance, and leadership. Settings includedAcademic Achievement Leadership Teams (AALTs or ALTs), Grade Level Team(GLT) meetings, and GR Principals’ meetings. GR also established beginning,middle, and end-of-year (BME) assessments in reading, writing, and oral languageproficiency. GR staff provided on-site assistance to support school efforts to estab-lish and maintain these settings. GR staff also provided annual, 3-day, beginning-of-year leadership training institutes for school teams, and a 1-day, mid-year follow-up.GR staff spent approximately one day every two weeks at each GR school, metmonthly with principals, and facilitated many of the aforementioned GR settings.Schools also used materials developed and tested by GR staff, including rubrics andchecklists to evaluate GR implementation; beginning and advanced training manu-als for ALTs; a series of three videotapes that described the GR Model and illus-trated various settings and processes; reading, writing, and oral proficiency
Inside the Black Box of School Reform 55
assessments for schools that did not already have such assessments; worksheets andguidelines for analysing achievement data and student work samples; and guidelinesand training modules for GR coaches.
Academic achievement at GR and comparison schools. During this scale-up phase, GRschools (N = 9) showed significantly greater gains in academic achievement thancomparison schools (N = 6) across 5 years. The following data are based on testscores averaged across Grades 2–5 and across reading, mathematics, language, andspelling subtests on the Stanford Achievement Test, 9th edition (Stanford 9;Harcourt Educational Measurement, 1996). Mean achievement for GR schools, inNormal Curve Equivalent (NCE) units, increased from 32.8 (SD = 4.8) in 1997 to48.3 (SD = 4.2) in 2002, a net increase of 15.5 NCE units. Corresponding NationalPercentile Ranks increased from 21st to 47th. In contrast, NCE means for the sixcomparison schools increased from 32.0 (SD = 6.3) in 1997 to 43.0 (SD = 6.0) in2002, a net increase of 11 NCE units. Corresponding National Percentile Ranksincreased from 20th to 37th. Whereas achievement, as measured by NCE means,was nearly identical at GR and comparison schools in 1997 (32.8 versus 32.0), by2002, achievement at GR schools exceeded that of comparable schools by more than5 NCEs (48.3 versus 43.0)—an adjusted effect size of 0.75. An effect size of 0.75, inthis case, means that GR schools scored, on average, 0.75 SD units higher than theaverage achieved by schools in the comparison group. This magnitude of effect isconsidered in the high-moderate to large range (Cohen, 1988), and it compares veryfavorably with effect sizes of other school reform efforts, most of which are below0.40 (Borman et al., 2003).
Achievement also increased more rapidly for GR schools than for the total popula-tion of elementary schools (N = 600+) in the district. Mean NCEs district-wide forGrades 2–5 increased from 36.5 in 1997 (nearly four points higher than the GRschools) to 47.3 in 2002 (one point lower than GR schools). In sum, GR schoolsincreased an average of 15.5 points, six comparison schools increased by an averageof 11.0 points, and schools district-wide increased by an average of 10.8 points. SeeSaunders (2003) for a more comprehensive and technical presentation of achieve-ment results.
Methods
GR project leaders hired an external evaluator who conducted an independent, on-site, process evaluation, throughout the 2001–02 academic year (McDougall, 2002).The evaluator investigated qualitatively the how and why of school change processesand achievement gains at seven, purposefully selected, demographically similar,case-study schools—four of nine GR schools plus three of six comparison schools(see Table 1). The evaluator used qualitative methods, a comparative case-studyresearch design, and rubric-based coding and ratings to answer the followingresearch questions (RQs):
56 D. McDougall et al.
● RQ 1: To what extent is the GR Model implemented in GR project schools?● RQ 2: To what extent does the GR Model establish processes discernable at GR,
but not comparison, schools?● RQ 3: What does implementation of the GR Model do to impact student
achievement?● RQ 4: What helps and hinders schools’ implementation of the GR Model?● RQ 5: What has changed, as a result of GR Model implementation, from the
perspectives of participants and the external evaluator?
Data Collection and Analyses
Data collection included audio-taped and transcribed teacher focus groups and prin-cipal interviews; observations and field notes of GLT meetings, ALT meetings,school-wide faculty meetings, professional development sessions, and principals’meetings; and document retrieval. The evaluator collected additional informationfrom spontaneous contacts with participants. In addition, the evaluator collectedfunctionally similar types and amounts of information from each case-study schoolby observing and participating in over 100 events, in formal and informal settings, atGR and comparison schools.
Data analyses included qualitative analysis, plus GR Model, rubric-based, codingand rating of data from the aforementioned sources. The evaluator analysed dataduring and after a 9-month data collection period, and used: (a) the constantcomparative method to formulate and refine findings (Glaser & Strauss, 1967); (b)triangulation to corroborate findings from multiple data sources, across individuals,time, and settings (Denzin, 1978; Miles & Huberman, 1994); and (c) memberchecks whereby participants provided feedback on emerging findings (Goffman,1959; Kvale, 1996; Lincoln & Guba, 1985; Manning, 1997; Taylor & Bogden,1998). The evaluator also used a transformative strategy (Creswell, 2003) to codeand rate data.
Coding and rating episodes. The evaluator used the GR Model Rubric (Table 2) andmore detailed operational charts to code and rate episodes from all data sources.Episodes were narrative data that contained information pertaining to GR elements.Episodes ranged in length from a single sentence to multiple paragraphs. The evalu-ator used applicable elements and corresponding descriptive criteria of the GRModel, in Table 2, to code all episodes. For example, episodes coded L-A containedinformation that pertained to descriptive criterion A (i.e., leadership’s consistency infocusing on school-wide, academic achievement goals) of the leadership (L)element. Next, the evaluator rated the coded episodes 4, 3, 2, or 1 to indicate therubric level that applied to the information contained in the episode. Some episodesincluded information that pertained to adjacent rubric levels. Such episodes required“in-between” ratings, such as 2 to 3. Thus, the evaluator used seven rubric-basedoptions to rate episodes: 1, 1 to 2, 2, 2 to 3, 3, 3 to 4, and 4. Initial coding and rating
Inside the Black Box of School Reform 57T
able
1.
Dem
ogra
phic
and
des
crip
tive
info
rmat
ion
for
case
-stu
dy s
choo
ls
GR
Pro
ject
sch
ools
(N
= 4
)C
ompa
riso
n sc
hool
s (N
= 3
)
Oak
Pin
eE
lmF
irK
amH
all
Lot
Sch
edul
eY
ear-
roun
d m
ulti
-tra
ckT
radi
tion
al
sing
le-t
rack
Yea
r-ro
und
mul
ti-t
rack
Yea
r-ro
und
mul
ti-t
rack
Yea
r-ro
und
mul
ti-t
rack
Yea
r-ro
und
mul
ti-t
rack
Yea
r-ro
und
mul
ti-t
rack
Stu
dent
s en
rolle
d1,
204
557
1,38
81,
211
1,25
91,
123
1,24
9T
itle
I s
choo
laY
esY
esY
esY
esY
esY
esY
esS
tude
nts/
clas
s (M
)20
.717
.421
.420
.521
.421
.420
.8S
tude
nts/
com
pute
r55
1015
1411
2357
Cla
sses
on
Inte
rnet
00
02
886
0E
nglis
h L
earn
erb
(%)
7944
7276
6172
76L
angu
ages
of
EL
S
= 7
7S
= 3
5S
= 6
6S
= 6
9S
= 6
1S
= 7
2S
= 7
6S
tude
ntsc
(%)
Ar
= 0
.6A
r =
5K
h =
3A
r =
6A
ll ot
her
= 0
P =
0.1
All
othe
r =
0P
= 0
.5R
= 1
Ar
= 2
Ko
= 0
.3A
r =
0.1
F =
0.2
Ko
= 1
T =
0.5
T =
0.2
Fre
e/re
duce
d m
eals
(%
)95
7392
9793
9598
Cal
Wor
ksd
(%)
2219
2926
1316
13C
ompe
nsat
ory
Ed.
e (%
)91
104
9997
9597
98S
tude
nt e
thni
city
f (%
)H
= 9
1H
= 5
5H
= 8
3 H
= 8
4H
= 9
8H
= 9
8H
= 9
9W
= 4
W =
27
W =
5
Af
= 5
W =
11
W =
2W
= 1
Af
= 0
.2
Af
= 3
Af
= 1
4A
s =
5A
f =
3F
= 0
.2A
f =
0.5
W =
0.1
58 D. McDougall et al.T
able
1.
(Con
tinue
d)
GR
Pro
ject
sch
ools
(N
= 4
)C
ompa
riso
n sc
hool
s (N
= 3
)
Oak
Pin
eE
lmF
irK
amH
all
Lot
As
= 1
A
s =
2A
s =
1A
s =
0.1
As
= 0
.1A
s =
0.3
F =
0.1
Tea
cher
eth
nici
tyf (
%)
W =
52
W =
63
W =
57
W =
62
H =
60
H =
62
H =
45
H =
41
H =
20
H =
30
H =
25
W =
25
W =
28
W =
39
Af
= 5
Af
= 9
Af
= 7
As
= 6
F =
5A
f =
5
As
= 1
1A
s =
2A
s =
9A
s =
1A
f =
3A
s, A
f =
3F
= 3
Af
= 3
Tea
cher
s w
/o f
ull c
rede
ntia
lg26
/57
= 4
6%13
/33
= 3
9%18
/67
= 2
7%19
/63
= 3
0%22
/62
= 3
5%19
/56
= 3
4%25
/65
= 3
1%
Not
es: P
erce
ntag
e ca
lcul
ated
usi
ng t
he t
otal
num
ber
of s
tude
nts
enro
lled
(or
teac
hers
em
ploy
ed)
at t
he s
choo
l, ro
unde
d to
the
nea
rest
who
le
num
ber
unle
ss o
ther
wis
e in
dica
ted.
a Qua
lifies
for
fede
ral f
unds
bas
ed o
n a
high
-pov
erty
stu
dent
bod
y. b “
Eng
lish
Lea
rner
s” (
EL
) (i
.e.,
form
er t
erm
used
was
“L
EP
” or
stu
dent
s w
ith
limit
ed E
nglis
h pr
ofici
ency
). c L
angu
age:
Ar
= A
rmen
ian,
F =
Far
si, K
h =
Khm
er, K
o =
Kor
ean,
P =
Pun
jabi
, R
= R
ussi
an, S
= S
pani
sh, T
= T
agal
og. d C
alW
orks
= C
alif
orni
a W
ork
Opp
ortu
nity
and
Res
pons
ibili
ty t
o K
ids;
for
mer
ly A
id t
o F
amili
es w
ith
Dep
ende
nt C
hild
ren.
e Pro
gram
me
for
educ
atio
nally
dis
adva
ntag
ed s
tude
nts.
f Eth
nici
ty: A
f =
Afr
ican
Am
eric
an, A
s =
Asi
an, F
= F
ilipi
no, H
=
His
pani
c, W
= W
hite
. g Wit
hout
(w
/o)
full
cred
enti
al =
em
erge
ncy
cred
enti
al/h
ire,
wai
ver,
pre
-int
ern,
Dis
tric
t in
tern
, Uni
vers
ity
inte
rn; D
ata
repo
rted
or
deri
ved
from
ww
w.e
d-da
ta.k
12.c
a.us
/sch
ool/s
choo
lpro
file;
199
9–20
00 =
mos
t re
cent
ava
ilabl
e da
ta d
urin
g ev
alua
tion
stu
dy.
Inside the Black Box of School Reform 59T
able
2.
GR
Mod
el R
ubri
c
Ele
men
tsL
evel
1L
evel
2L
evel
3L
evel
4
Goa
ls th
at a
re s
et
and
shar
edN
o sc
hool
-wid
e ac
adem
ic
achi
evem
ent
goal
s ar
e id
enti
fied
.
Sch
ool-
wid
e ac
adem
ic
achi
evem
ent
goal
s id
enti
fied
; go
als
are
fair
ly g
ener
al.
Sch
ool-
wid
e ac
adem
ic
achi
evem
ent
goal
s id
enti
fied
; goa
ls a
re
spec
ific
and
con
cret
e.
Sch
ool-
wid
e ac
adem
ic
achi
evem
ent
goal
s id
enti
fied
; go
als
are
spec
ific
, con
cret
e,
and
publ
ishe
d. A
All/
near
ly a
ll te
ache
rs a
re
unaw
are
of g
oals
or
assu
me
goal
s ar
e un
rela
ted
to
teac
hing
.
Mos
t te
ache
rs m
ay b
e un
awar
e of
the
goa
ls o
r as
sum
e go
als
are
unre
late
d to
th
eir
teac
hing
.
Mos
t tea
cher
s ar
e aw
are
of
the
goal
s an
d ne
w te
ache
rs
are
usua
lly t
old
abou
t go
als.
Tea
cher
s th
ink
the
goal
s ar
e im
port
ant;
new
tea
cher
s ar
e ex
plic
itly
intr
oduc
ed t
o th
e go
als.
BT
each
ers
did
not
play
som
e ro
le in
iden
tify
ing
goal
s.F
ew t
each
ers
(and
oth
ers)
pl
ayed
som
e ro
le in
id
enti
fyin
g go
als.
Tea
cher
s (a
nd o
ther
s)
play
ed s
ome
role
in
iden
tify
ing
goal
s.
Tea
cher
s (a
nd o
ther
s) h
elpe
d de
velo
p or
ada
pt t
he g
oals
. C
Indi
vidu
al t
each
ers
do n
ot
use
the
goal
s in
pla
nnin
g in
stru
ctio
n.
Few
indi
vidu
al t
each
ers
use
the
goal
s in
pla
nnin
g in
stru
ctio
n.
Som
e in
divi
dual
tea
cher
s us
e th
e go
als
in p
lann
ing
inst
ruct
ion.
Tea
cher
s us
e th
e go
als
indi
vidu
ally
and
col
lect
ivel
y in
pl
anni
ng in
stru
ctio
n. D
Indi
cato
rs th
at
mea
sure
suc
cess
No
scho
ol-w
ide
indi
cato
rs o
f st
uden
t ac
hiev
emen
t ar
e us
ed.
Som
e sc
hool
-wid
e in
dica
tors
us
ed t
o ex
amin
e ge
nera
l tr
ends
(up
/dow
n) in
stu
dent
ac
hiev
emen
t.
Spe
cifi
c sc
hool
-wid
e in
dica
tors
use
d to
ass
ess
whe
ther
stu
dent
s ar
e m
eeti
ng id
enti
fied
goa
ls.
Spe
cifi
c sc
hool
-wid
e in
dica
tors
use
d to
ass
ess
whe
ther
stu
dent
s ar
e m
eeti
ng
iden
tifi
ed g
oals
. AM
anda
ted
test
s ar
e gi
ven,
but
no
t us
ed t
o re
view
in
form
atio
n ab
out
stud
ent
achi
evem
ent.
Mos
t te
ache
rs m
ay b
e un
awar
e of
indi
cato
rs o
r do
no
t vi
ew t
hem
as
valid
and
us
eful
sou
rces
of
info
rmat
ion.
Mos
t tea
cher
s ar
e aw
are
of
the
indi
cato
rs a
nd th
ey a
re
som
ewha
t in
tere
sted
in
the
info
rmat
ion.
Mos
t te
ache
rs t
hink
the
in
dica
tors
are
impo
rtan
t an
d th
ey v
iew
the
info
rmat
ion
as
valid
and
use
ful.
BN
one/
near
ly n
o in
divi
dual
te
ache
rs u
se in
form
atio
n fr
om
indi
cato
rs t
o re
fine
the
ir
inst
ruct
ion
and
curr
icul
um.
Few
indi
vidu
al t
each
ers
use
info
rmat
ion
from
indi
cato
rs
to r
efin
e th
eir
inst
ruct
ion
and
curr
icul
um.
Som
e in
divi
dual
tea
cher
s us
e th
e in
form
atio
n to
re
fine
thei
r ins
truc
tion
and
cu
rric
ulum
.
Mos
t te
ache
rs u
se t
he
indi
cato
rs in
divi
dual
ly a
nd
colle
ctiv
ely
to r
efin
e in
stru
ctio
n an
d cu
rric
ulum
. C
60 D. McDougall et al.T
able
2.
(Con
tinue
d)
Ele
men
tsL
evel
1L
evel
2L
evel
3L
evel
4
Ass
ista
nce
from
ca
pabl
e ot
hers
No/
very
few
opp
ortu
niti
es fo
r te
ache
rs t
o re
ceiv
e as
sist
ance
.In
term
itte
nt o
ppor
tuni
ties
for
assi
stan
ce.
Reg
ular
, ong
oing
, sch
ool-
wid
e op
port
unit
ies
for
assi
stan
ce.
Reg
ular
, ong
oing
, sch
ool-
wid
e op
port
unit
ies
for
assi
stan
ce. A
Pro
vide
d by
som
eone
who
do
es n
ot h
ave
rele
vant
ex
peri
ence
, exp
erti
se, r
espe
ct.
Pro
vide
d by
som
eone
who
m
ay o
r m
ay n
ot h
ave
rele
vant
ex
peri
ence
and
exp
erti
se.
Pro
vide
d by
som
eone
wit
h re
leva
nt e
xper
ienc
e an
d ex
pert
ise.
Pro
vide
d by
som
eone
wit
h re
leva
nt e
xper
ienc
e an
d ex
pert
ise
who
has
or
gain
s th
e re
spec
t of
mos
t te
ache
rs. B
Unr
elat
ed t
o m
eeti
ng s
choo
l-w
ide
achi
evem
ent
goal
s.M
ostl
y un
rela
ted
to m
eeti
ng
scho
ol-w
ide
achi
evem
ent
goal
s.
Rel
ated
to
mee
ting
id
enti
fied
sch
ool-
wid
e ac
hiev
emen
t go
als.
Foc
used
dir
ectl
y on
mee
ting
id
enti
fied
sch
ool-
wid
e ac
hiev
emen
t go
als.
CU
nava
ilabl
e/av
aila
ble
brie
fly.
Ava
ilabl
e ov
er s
hort
tim
e pe
riod
.A
vaila
ble
over
ext
ende
d pe
riod
of
tim
e.A
vaila
ble
over
ext
ende
d pe
riod
of
tim
e. D
Lea
ders
hip
that
su
ppor
ts a
nd
pres
sure
s
Lea
ders
hip
ofte
n di
stra
cted
or
doe
s no
t fo
cus
on
acad
emic
ach
ieve
men
t.
Lea
ders
hip
gene
rally
foc
uses
on
aca
dem
ic a
chie
vem
ent
scho
ol-w
ide.
Lea
ders
hip
focu
ses
on
mee
ting
iden
tifi
ed
achi
evem
ent
goal
s sc
hool
-w
ide.
Lea
ders
hip
focu
ses
cons
iste
ntly
on
mee
ting
id
enti
fied
ach
. goa
ls s
choo
l-w
ide.
AD
oes
not
part
icip
ate
in
esta
blis
hing
and
rev
isin
g pl
ans.
Iden
tifi
es g
ener
al a
reas
of
need
, rec
omm
ends
po
ssib
iliti
es.
Hel
ps e
stab
lish
a co
ncre
te
plan
to
achi
eve
goal
s.H
elps
est
ablis
h co
ncre
te p
lan,
an
d ov
er t
ime
cont
inue
s to
id
enti
fy n
ext
step
s. B
Pro
vide
s lit
tle
if a
ny
enco
urag
emen
t an
d re
sour
ces.
Pro
vide
s so
me
enco
urag
emen
t an
d av
aila
ble
reso
urce
s.
Pro
vide
s en
cour
agem
ent
and
need
ed r
esou
rces
.P
rovi
des
ampl
e en
cour
agem
ent
and
reso
urce
s. C
Rar
ely
if e
ver
enco
urag
es
indi
vidu
al a
nd c
olle
ctiv
e re
spon
sibi
lity
Usu
ally
doe
s no
t en
cour
age
indi
vidu
al a
nd c
olle
ctiv
e re
spon
sibi
lity.
Enc
oura
ges
indi
vidu
al a
nd
colle
ctiv
e re
spon
sibi
lity.
Cla
rifi
es a
nd h
olds
peo
ple
acco
unta
ble
for
thei
r in
divi
dual
and
col
lect
ive
resp
onsi
bilit
ies.
D
Inside the Black Box of School Reform 61T
able
2.
(Con
tinue
d)
Ele
men
tsL
evel
1L
evel
2L
evel
3L
evel
4
Set
ting
s th
at a
llow
st
aff t
o ge
t im
port
ant t
hing
s do
ne
Tim
e ou
tsid
e of
cla
ssro
om is
us
ed to
cla
rify
pro
cedu
res
and
polic
ies,
and
del
iver
req
uire
d tr
aini
ngs.
Tim
e ou
tsid
e of
the
cl
assr
oom
is r
egul
arly
use
d to
de
liver
nec
essa
ry in
form
atio
n,
and
disc
uss
acad
emic
ac
hiev
emen
ts.
Tim
e ou
tsid
e of
the
cl
assr
oom
is r
egul
arly
sc
hedu
led
and
gene
rally
us
ed t
o w
ork
on t
hing
s re
late
d to
ach
ievi
ng
acad
emic
goa
ls.
Tim
e ou
tsid
e of
the
clas
sroo
m
is d
epen
dabl
y sc
hedu
led
and
cons
iste
ntly
use
d to
get
don
e im
port
ant
thin
gs r
elat
ed t
o ac
hiev
ing
acad
emic
goa
ls. A
Tea
cher
s an
d ad
min
istr
ator
s vi
ew m
eeti
ngs
as a
n in
trus
ion
on t
heir
tim
e.
Mos
t te
ache
rs/a
dmin
istr
ator
s vi
ew m
eeti
ngs
as p
art
of t
heir
w
ork,
but
do
not
valu
e it
as
impo
rtan
t.
Mos
t te
ache
rs a
nd
adm
inis
trat
ors
valu
e ti
me
to m
eet
and
colla
bora
te a
s a
wor
thw
hile
use
of
tim
e.
Tea
cher
s an
d ad
min
istr
ator
s va
lue
tim
e to
mee
t,
colla
bora
te a
nd g
et t
hing
s do
ne a
s an
impo
rtan
t pa
rt o
f th
eir
wor
k. B
Mee
ting
s ar
e ra
rely
eff
ecti
ve
and
alm
ost
neve
r ef
fici
ent.
Som
e m
eeti
ngs
are
effi
cien
t an
d ef
fect
ive.
Mos
t m
eeti
ngs
are
sche
dule
d, p
lann
ed, a
nd
carr
ied
out
effi
cien
tly
and
effe
ctiv
ely.
Nea
rly
all m
eeti
ngs
cons
iste
ntly
sch
edul
ed,
plan
ned,
and
car
ried
out
ef
fici
entl
y an
d ef
fect
ivel
y. C
Not
e: B
old
lett
ers
A, B
, C, a
nd D
iden
tify
des
crip
tive
cri
teri
a (s
can
hori
zont
ally
) fo
r ea
ch e
lem
ent
of t
he r
ubri
c.
62 D. McDougall et al.
of all data sources produced 2,940 episodes. The evaluator subsequently reviewedall episodes, eliminated episodes that lacked sufficient content or clarity, andidentified episodes that were overlooked during the initial coding and rating. Thissystematic review resulted in a net reduction of 309 episodes for a final total of 2,631coded-rated episodes.
Assigning ratings to GR elements and schools. As was the case for individual episodes,the evaluator utilised seven rubric-based options to rate GR elements: 1, 1 to 2, 2, 2to 3, 3, 3 to 4, and 4. The evaluator assigned element ratings, for each school, basedon the mean and median ratings for that element’s distribution of episode ratings(see Table 3). The evaluator assigned a single rating to an element if its episoderating distribution had a similar mean and median. However, if these measures ofcentral tendency were not consistent (e.g., if the mean suggested a rating of 3 andthe median suggested a rating of 3 to 4), then the evaluator assigned a dual rating forthe element. The evaluator also assigned an overall rating of school change, to eachschool, based on the pattern of element ratings for each of the five GR Modelelements. The bold numbers in Table 3 highlights rating patterns.
Reliability of episode ratings. Secondary evaluators conducted reliability checks ofrubric-based episode ratings. Overall inter-rater agreement for exact agreementsequaled 88%, for 104 episodes selected at random from 2,631 total episodes, usingthe formula “agreements divided by agreements plus disagreements multiplied by100%”. Overall inter-rater agreement for exact plus adjacent agreements equaled100% using the same formula.
Results
RQ 1: To what extent is the GR Model implemented at GR schools?
GR Model implementation was fairly strong at three of the four GR schools, includ-ing Oak, Pine, and Elm, with overall school ratings of 3 to 4, and comparativelyweaker at Fir, which was rated 2 to 3. As seen in Table 3, ratings for each of the fiveGR elements at GR schools varied somewhat between and within schools. In addi-tion, ratings for the settings element, based on direct observations of GLT meetingsat GR schools, indicated that K-3 teachers tended to implement GR proceduresmore effectively than their colleagues in Grades 4 and 5.
RQ 2: To what extent does the GR Model establish processes observable at GR schools but not at comparison schools?
Finding 1. GR schools utilised processes associated with the five elements of the GRModel more frequently and more effectively than comparison schools. Ratings for thefive change elements at GR schools, except Fir, exceeded ratings for the comparison
Inside the Black Box of School Reform 63
Table 3. Frequency distribution of episode ratings for GR elements at GR and comparison schools (ratings based on GR Model Rubric levels)
Rating
School Element 1 1 to 2 2 2 to 3 3 3 to 4 4
Oak (GR) Goals 1 2 9 5 11 4 15Indicators 0 2 5 0 9 33 20Assistance 1 0 17 21 14 100 101Leadership 4 11 2 18 16 45 60Settings 0 1 0 6 7 16 6
Pine (GR) Goals 0 0 4 8 12 56 22Indicators 2 4 8 9 32 29 13Assistance 0 3 7 5 26 61 18Leadership 0 12 7 22 27 56 41Settings 0 0 0 3 4 14 9
Elm (GR) Goals 1 0 3 5 8 18 16Indicators 2 7 2 4 1 11 16Assistance 0 2 4 6 21 39 24Leadership 15 24 6 13 17 5 12Settings 0 2 0 10 6 20 3
Fir (GR) Goals 2 3 12 21 16 0 2Indicators 0 7 12 14 15 1 3Assistance 0 31 12 31 15 21 1Leadership 13 39 28 33 16 8 0Settings 1 2 3 14 7 3 0
Kam (Comparison) Goals 0 9 4 26 17 15 0Indicators 1 6 4 11 14 21 0Assistance 2 3 11 23 24 7 1Leadership 1 14 6 1 18 32 5Settings 0 11 2 13 5 2 0
Hall (Comparison) Goals 0 30 17 10 9 1 0Indicators 12 4 5 10 10 4 0Assistance 8 9 11 30 19 0 0Leadership 24 19 5 2 24 17 3Settings 0 19 6 8 0 0 0
Lot (Comparison) Goals 1 16 5 3 11 6 0Indicators 9 13 1 7 10 8 1Assistance 0 6 13 10 11 0 1Leadership 1 22 0 8 11 17 7Settings 1 2 2 13 3 1 0
Notes: Bold numbers indicate the element rating based on mean and median values for distribution of episode ratings for that element. One bold cell per row indicates the evaluator applied a single rating for the element based on similar mean and median values for that distribution of episode ratings. Two bold cells per row indicate the evaluator applied a dual rating for the element based on dissimilar mean and median values for that distribution of episode ratings
64 D. McDougall et al.
schools (see Table 3). Likewise, the overall school rating for each GR school (3 to 4)except Fir (2 to 3) exceeded that of comparison schools Kam and Lot (2 to 3) andHall (2). With few exceptions, GLTs at GR schools used more effective processes toconduct meetings than GLTs at comparison schools. Comparative ratings for thesettings element of the GR Model indicated that teachers at GR schools accom-plished important instructional tasks more frequently and more effectively, duringtheir GLT meetings, than teachers at comparisons schools (see Table 4).
Qualitative analyses indicated that teachers’ behaviours and discourse, as well asmeeting procedures and outcomes, differed appreciably during most team meetingsat GR versus comparison schools (see Table 5). Notable examples, all suggestingmore positive environments existed in GR team meetings, included: (a) more consis-tent focus, planning, and time for academic topics, goals, and indicators and lesstime discussing topics or doing tasks of a non-academic nature, such as duplicatingor collating materials, and planning field trips; (b) analysis of students’ productsabove and beyond state or district-mandated assessments (e.g., BME writing assess-ments at GR schools versus district-mandated, once-per-year, writing assessments atcomparison schools); (c) discussing the relation between instruction, studentoutcomes, and the need for instructional changes; (d) modeling instructional meth-ods for colleagues; (e) assigning and completing goal-related assignments, and usingacademic data with follow-up at subsequent meetings; (f) preparing and evaluatingmutually agreed upon teaching strategies; (g) teachers’ consistent versus sporadicattendance and participation at the meetings; (h) teachers’ punctual arrivals anddepartures versus late arrivals and early departures; (i) principals’ participationversus non-attendance at meetings; (j) teachers’ use of typed agendas and priorawareness of meeting topics versus no agenda and limited, or last-minute, awareness
Table 4. Ratings for the setting element for grade-level team meetings at GR and comparison schools
Rubric rating
1 1 to 2 2 2 to 3 3 3 to 4 4
GR schools FFFFFFFF FF FFF FFPPPEE PPPOO EEEEEE PPPOOO
EOOOO OO(1) (2) (3) (20) (7) (6) (6)
Comparison schools HHHHH HHKKKHHHHH HKL KKKKL KKLHKKLL LLLLLL
L(15) (3) (17) (3) (0) (0)
Note: Each letter represents one episode rating, for one school, for one descriptive criterion from the settings element of the GR Model Rubric. Numbers in parentheses indicate the total number of settings ratings assigned at rubric level. Schools: E = Elm, F = Fir, H = Hall, K = Kam, L = Lot, O = Oak, P = Pine.
Inside the Black Box of School Reform 65T
able
5.
Gra
de-l
evel
tea
m m
eeti
ngs
at G
R s
choo
ls v
ersu
s co
mpa
riso
n sc
hool
s
GR
sch
ools
Com
pari
son
scho
ols
TC
init
iate
s/te
lls o
ther
tea
cher
s to
pul
l out
Ope
n C
ourt
re
adin
g bo
ok fr
om t
he s
tude
nt d
esks
at
whi
ch t
hey
are
sitt
ing,
op
en t
o st
ory,
“M
ake
Way
for
Duc
klin
gs”.
Pag
e 17
6 is
fir
st
page
of s
tory
. In
desc
ribi
ng t
o ot
her
teac
hers
how
she
del
iver
s in
stru
ctio
n on
the
ski
ll of
new
par
agra
phs,
TC
ask
s th
e te
ache
rs, a
nd s
ays
that
she
ask
s/te
ache
s he
r st
uden
ts t
o id
enti
fy w
hy d
id t
he a
utho
r m
ake
a ne
w p
arag
raph
… w
hat
is
the
reas
on f
or t
he c
hang
e …
wha
t ch
ange
d? T
C h
as t
he
teac
hers
rea
d si
lent
ly, a
ppro
xim
atel
y th
e fi
rst
five
par
agra
phs
and
asks
the
m t
o id
enti
fy W
HY
the
par
agra
ph c
hang
ed.
Var
ious
rea
sons
em
erge
: cha
nge
in lo
cati
ons,
cha
nge
in t
ime,
ch
ange
in s
peak
er. T
C m
akes
tha
t po
int
that
we
need
to
inst
ruct
exp
licit
ly k
ids
abou
t pa
ragr
aph
chan
ges
and
reas
ons
why
par
agra
phs
chan
ge, v
ia e
xam
ples
. She
als
o m
akes
the
po
int
that
we
need
to
be c
aref
ul a
bout
wha
t st
orie
s/m
ater
ials
w
e se
lect
to
use
for
this
inst
ruct
ion
beca
use
of p
oor
proo
fing
an
d in
corr
ect,
om
itte
d pa
ragr
aphi
ng in
som
e bo
oks.
L t
each
er w
rite
s a
post
-it
note
to
self
. [I
thin
k th
at t
his
note
, al
ong
wit
h on
e or
tw
o ot
hers
, is
his
way
of
reco
rdin
g
D o
ff-t
ask
diad
dis
cuss
ion
wit
h F
nex
t to
her
re:
I n
eed
to g
o to
the
off
ice.
R
espo
nse:
for
wha
t? T
o dr
op o
ff m
oney
… w
hile
M a
nd c
oach
dis
cuss
issu
e M
ra
ised
. Tha
t is
, M is
get
ting
at
wha
t dr
ives
inst
ruct
ion.
Coa
ch s
ays
trus
t th
e fr
amew
ork
(sta
ndar
ds).
Coa
ch r
eite
rate
s th
at t
he D
istr
ict
is d
evel
opin
g a
paci
ng/
inst
ruct
iona
l pla
n fo
r m
ath
for
the
com
ing
year
, thu
s, n
ext y
ear
will
be
easi
er. O
C:
I ob
serv
e th
at D
is s
itti
ng b
ack,
look
s at
wat
ch. C
oach
: So
does
eve
rybo
dy k
ind
of
know
wha
t th
ey n
eed
to w
ork
on …
Res
pons
es f
rom
tea
cher
s ar
e on
the
min
imal
si
de, s
ome
yes/
nods
. Coa
ch a
sks:
So
wha
t do
you
… (
i.e.,
the
coac
h at
tem
pt to
get
te
ache
rs, p
arti
cula
rly
D t
o vo
calis
e so
me
of t
he s
peci
fic
stan
dard
sub
-are
as/s
kills
th
at t
hey
high
light
ed, w
hich
the
y w
ill p
rior
itiz
e/in
stru
ct o
ver
the
next
few
wee
ks.
[OC
: The
coa
ch p
ress
es f
or m
ore
spec
ific
ity
in t
his
mee
ting
tha
n in
pri
or u
pper
G
L m
eeti
ng. G
ood
mov
e, b
ut …
] D
: Jus
t re
view
eve
ryth
ing
… I
kno
w t
hey’
ll ge
t it
[ag
ain,
a n
on-s
peci
fic
resp
onse
]. N
say
s th
at it
’s c
omfo
rtin
g to
kno
w t
hat
the
fram
ewor
k/pa
cing
pla
n is
bei
ng d
evel
oped
in t
ime
for
next
yea
r; t
hat
this
yea
r is
a
loss
and
I h
ave
good
, sm
art
kids
. It’
s re
ally
har
d be
caus
e of
the
spa
n …
N a
nd a
co
uple
of
othe
rs t
hey
shou
ld k
now
mul
tipl
icat
ion
tabl
es b
y G
rade
3. D
con
curs
an
d sa
ys if
the
y do
n’t
they
sho
uld
be k
icke
d ou
t of
sch
ool.
D s
tand
s, s
ayin
g “I
’ve
got
to s
tand
…”
Coa
ch a
nd o
ne t
each
er t
alk
abou
t an
acr
onym
. D t
o co
ach,
I’v
e go
t to
go
drop
off
$ t
o of
fice
, lea
ves.
(K
am,
grad
e-le
vel t
eam
mee
ting)
pers
onal
ly u
sefu
l inf
orm
atio
n fo
r hi
mse
lf, s
uch
as t
he
info
rmat
ion
that
TC
just
des
crib
ed a
bout
tea
chin
g pa
ragr
aphi
ng, r
easo
ns f
or p
arag
raph
cha
nges
, rel
ated
to
the
wri
ting
ski
ll of
inde
ntin
g] …
TC
sta
tes
[in
a su
mm
ing/
conf
irm
ator
y m
anne
r]: “
So
I w
ill m
ake
copi
es o
f …
the
run
-on
sen
tenc
es u
nit
for
you
[oth
er t
each
ers]
and
you
guy
s m
ake
…”
(Elm
, gr
ade-
leve
l tea
m m
eetin
g)
66 D. McDougall et al.T
able
5.
(Con
tinue
d)
GR
sch
ools
Com
pari
son
scho
ols
AL
T T
red
irec
ts th
e gr
oup
back
to ta
sk o
f try
ing
to d
eter
min
e w
hat
port
ions
of c
hapt
ers
to k
eep/
drop
… P
rinc
ipal
chi
mes
in
(in
rega
rd t
o po
ssib
le w
ays
to t
each
mat
h sk
ills,
par
ticu
larl
y ot
her
than
form
al s
ched
uled
mat
h ti
me)
: “ …
if y
ou c
an [
take
ti
me
to t
each
a p
arti
cula
r m
ath
skill
] do
at
8:20
[A
.M.]
, but
ev
eryo
ne h
as t
o do
it”.
MT
ref
ocus
es t
he g
roup
by
sum
mar
isin
g: S
o, h
ave
we
deci
ded
that
we
will
do
… A
LT
T
begi
ns t
o m
odel
a t
each
ing
acti
vity
/pro
cedu
re s
he u
ses
wit
h he
r st
uden
ts (
stan
ding
at
her
mat
h bu
lleti
n bo
ard,
coi
ns
acti
vity
). P
rinc
ipal
chi
mes
in: l
ast
wee
k w
hen
I w
as in
___
_’s
clas
s, s
he h
ad k
ids
mak
e up
dif
fere
nt w
ays
of m
akin
g 26
…
AL
T T
[fo
cusi
ng g
roup
by
conc
ludi
ng]:
“…
so
can
we
say
we’
re g
oing
to
do t
ime
and
mon
ey …
one
wee
k …
tha
t le
aves
__
_ w
eeks
for
rem
aini
ng …
(O
ak,
grad
e-le
vel t
eam
mee
ting)
F s
ays
that
nex
t w
eek’
s co
llabo
rati
ve p
lann
ing
mee
ting
will
hav
e so
met
hing
gu
ided
, but
tha
t th
is w
eek
we’
re “
mos
tly
on o
ur o
wn”
. F m
enti
ons
curr
ent/
upco
min
g ta
sk t
hat
teac
hers
nee
d to
do,
indi
cati
ng t
hat
this
tim
e ca
n be
use
d fo
r th
ese
task
s …
Ope
n H
ouse
, Rep
ort
Car
ds, [
Impl
emen
tati
on?
and/
or E
LD
?],
cum
ulat
ive
fold
ers.
I a
sk, s
o yo
u ha
ve r
epor
t ca
rds
to d
o? F
: Not
us
(not
F a
nd M
w
ho a
re o
n sa
me
trac
k), b
ut t
each
ers
on o
ther
tra
ck. M
and
F lo
okin
g at
stu
dent
pa
inti
ngs
… d
iscu
ss/ r
ecom
men
d us
e of
tem
pura
pai
nt v
ersu
s/or
wat
erco
lour
s.
The
tw
o te
ache
rs a
re w
orki
ng in
depe
nden
tly
on s
tude
nt c
umul
ativ
e fo
lder
s,
wri
ting
info
rmat
ion
and
chec
king
info
rmat
ion
in c
umul
ativ
e fo
lder
s …
the
y ch
at
abou
t M
’s w
ife
and
kids
… b
abys
itti
ng a
rran
gem
ents
. (H
all,
grad
e-le
vel t
eam
m
eetin
g)
Lea
der
dist
ribu
tes
“Wri
ting
Scr
ipt”
tha
t gr
ade
leve
l use
s. W
e al
l use
the
“B
erns
tein
Bea
rs”.
She
ask
s if
it’s
OK
if w
e us
e it
ag
ain
… it
’s w
hat w
e’ve
use
d in
the
past
. Lea
der
revi
ews
step
s th
at t
each
ers
will
be
doin
g le
adin
g up
to
asse
ssm
ent
(lea
der
and
othe
rs a
re lo
okin
g at
“W
riti
ng S
crip
t” w
hich
spe
cifi
es
thes
e st
eps)
. In
disc
ussi
ng h
ow t
o ad
min
iste
r th
e as
sess
men
t,
grou
p cl
arif
ies
wha
t to
do/
wha
t no
t to
do
and
gets
to
idea
tha
t th
is is
an
init
ial a
sses
smen
t. L
eade
r di
stri
bute
s “W
riti
ng
Rub
ric”
… E
des
crib
es a
n ex
ampl
e fr
om a
noth
er y
ear/
te
ache
r. O
C: E
see
ms
to p
rovi
de v
ery
solid
exa
mpl
es.
I no
tice
tha
t du
ring
the
init
ial p
orti
on o
f th
e m
eeti
ng t
hat
Mrs
. Z is
pre
pari
ng
som
e ty
pe o
f m
ater
ials
/cut
ting
con
stru
ctio
n pa
per
[i.e
., u
nrel
ated
to
mee
ting
to
pic/
inte
nded
tas
ks].
(K
am,
grad
e-le
vel t
eam
mee
ting)
Inside the Black Box of School Reform 67
Tab
le 5
.(C
ontin
ued)
GR
sch
ools
Com
pari
son
scho
ols
Dis
cuss
ion
of w
hat
“det
ail”
mea
ns in
thi
s gr
ade
leve
l’s
wri
ting
. Lea
der
revi
ewin
g “W
riti
ng R
ubri
c”. L
eade
r di
scus
ses
upco
min
g m
eeti
ng a
nd t
hat,
am
ong
othe
r ac
tivi
ties
, we’
ll sc
ore
wri
ting
ass
essm
ents
tha
t da
y. D
iscu
ssio
n of
sch
edul
ing
item
s. 2
:10
Pri
ncip
al d
epar
ts. …
Lea
der
shif
ts to
OC
R, w
hich
is
age
nda
item
2. A
ny c
once
rns?
Mal
e as
ks f
or c
lari
fica
tion
ab
out
som
e w
riti
ng a
sses
smen
t pr
oced
ures
tim
e =
30
min
utes
; will
mod
el s
till
be u
p =
no;
but
web
can
sta
y up
. E
show
s m
ater
ials
tha
t sh
e us
es w
ith
her
stud
ents
; oth
er
teac
hers
vie
w; f
avou
rabl
e re
acti
ons.
2:1
7 M
eeti
ng e
nds.
(P
ine,
gra
de-l
evel
team
mee
ting)
68 D. McDougall et al.T
able
5.
(Con
tinue
d)
GR
sch
ools
Com
pari
son
scho
ols
N (
AL
T r
ep f
or t
his
grad
e le
vel)
rev
iew
s la
st m
eeti
ng:
NO
TE
: Ini
tial
ly, I
don
’t s
ee o
r he
ar m
uch
acti
ve r
espo
nses
fro
m t
each
ers;
the
y’re
m
ostl
y ve
ry q
uiet
. The
faci
litat
or d
oes
utili
se th
e ap
proa
ch/a
ppea
l in
atte
mpt
ing
to
get
to g
et t
each
ers
to p
rovi
de in
put/
deci
sion
s, t
hat
she
does
n’t
know
the
ir
stud
ents
, you
kno
w y
our
stud
ents
and
wha
t th
ey s
houl
d/sh
ould
not
be
culle
d. F
or
the
firs
t fe
w w
eek-
by-w
eek
atte
mpt
s to
get
tea
cher
inpu
t, I
alm
ost
have
tha
t “p
ullin
g te
eth”
fee
ling
and
I w
onde
r if
par
t of
the
issu
e is
whe
ther
som
e of
the
te
ache
rs a
re im
plem
enti
ng a
ny o
f Mov
e It
Mat
h, o
r do
ing
so o
n a
limit
ed b
asis
. At
one
poin
t in
the
mee
ting
/bef
ore
or a
fter
I h
ear
gree
n sw
eate
r te
ache
r sa
y in
re
spon
se t
o ei
ther
O’s
(di
stri
ct/ n
atio
nal c
oach
) or
A’s
(sc
hool
-bas
ed c
oach
) qu
ery,
“A
re y
ou o
n a
set
wee
k?”
that
“ye
s” b
ut s
ubse
quen
tly
“We
wer
en’t
rea
lly
doin
g as
muc
h m
ath
as w
e’re
sup
pose
d to
… (
due
to, f
or e
xam
ple,
not
hav
ing
trai
ning
). O
late
r te
lls m
e w
hen
I am
alo
ne w
ith
her
that
the
y ha
d pu
rpos
eful
ly
done
thr
ee t
rain
ings
in t
he s
umm
er, o
ne p
er m
onth
, so
that
the
y co
uld
hit
all
trac
ks’ t
each
ers.
I a
lso
saw
a f
ax d
ated
11-
08 f
rom
O t
o L
ot/A
tha
t lis
ted
item
s ne
eded
for
thi
s m
eeti
ng, i
nclu
ding
list
/num
ber
of t
each
ers
who
had
not
gon
e to
/re
ceiv
ed t
he M
ove
It M
ath
trai
ning
. 11:
08 O
ne s
tude
nt e
nter
s ro
om, h
ands
pap
er
to o
ne t
each
er. T
he o
ne t
each
er d
epar
ts r
oom
for
abo
ut 3
0 se
cond
s. A
bout
one
m
inut
e la
ter,
ano
ther
tea
cher
dep
arts
roo
m f
or a
bout
15
seco
nds.
30
seco
nds
late
r, a
stu
dent
ope
ns d
oor
and
calls
one
tea
cher
’s n
ame.
The
tea
cher
say
s, “
Go
find
Mrs
. Gom
ez”
and
repe
ats
that
phr
ase.
11:
12 O
: “D
o yo
u fe
el c
omfo
rtab
le
…?”
(sk
ippi
ng)
C: R
egar
ding
Mon
ster
add
itio
n: “
You
’re
not
aski
ng t
hem
to
mas
ter
it. Y
ou’r
e ju
st in
trod
ucin
g it
.” A
tea
cher
had
com
men
ted
that
thi
s w
as a
to
uchy
one
, tha
t M
onst
er m
ath
is d
iffi
cult
for
the
kid
s. B
tea
cher
ask
s: “
How
im
port
ant
is S
cott
-For
esm
an?”
B t
each
er m
akes
poi
nt t
hat
afte
r do
ing
man
ipul
ativ
es a
nd …
tha
t ki
ds a
re t
ired
of
it a
nd t
here
fore
is it
nec
essa
ry t
o do
w
hole
Mov
e it
Mat
h pa
ge/a
ll pr
oble
ms.
R&
B t
each
er: “
In y
our
scho
ol, y
ou w
ere
man
agin
g tw
o m
ath
prob
lem
s as
wel
l?”
“Bec
ause
I a
m w
onde
ring
…”
O: N
o, it
ju
st h
appe
ned.
Inside the Black Box of School Reform 69T
able
5.
(Con
tinue
d)
GR
sch
ools
Com
pari
son
scho
ols
•D
aily
ora
l exe
rcis
es …
Coa
ch w
ill p
rovi
de•
Tw
o te
ache
rs s
hare
d st
rate
gies
for
seq
uenc
ing
Spa
rks
disc
ussi
on, L
B t
each
er s
ays
… “
feel
fru
stra
ted”
•Id
enti
fied
new
nee
d …
new
obj
ecti
ve …
•N
shi
fts
to w
hat
this
mee
ting
will
add
ress
. He
plac
es O
H•
Nee
d =
incr
ease
stu
dent
kno
wle
dge
of s
eque
ncin
g w
hen
wri
ting
sto
ry
O: A
s P
roje
ct G
rad
scho
ol, y
ou’r
e su
ppos
ed t
o fo
llow
Mov
e it
Mat
h an
d us
e S
cott
-For
esm
an a
s su
pple
men
t. W
e do
n’t
real
ly c
are
if y
ou u
se S
cott
-For
esm
an
or …
or …
as
long
as
you’
re a
ddre
ssin
g th
e M
ove
it M
ath
focu
s (i
.e.,
hav
ing
all
your
com
pone
nts,
par
ts o
f le
sson
incl
udin
g w
arm
-up,
etc
.) (
Lot
, gr
ade-
leve
l tea
m
mee
ting)
•O
bjec
tive
= B
y en
d of
sch
ool y
ear
75%
of s
tude
nts
WB
AT
to
wri
te 4
or
mor
e se
nten
ces
to d
escr
ibe
begi
nnin
g,
mid
dle,
end
.N
lead
s te
ache
rs t
o to
day’
s ta
sk o
f an
alys
ing
one
low
, one
m
ediu
m, o
ne h
igh
stud
ent
wor
k sa
mpl
es t
o de
term
ine
stre
ngth
s an
d w
eakn
esse
s. O
C: I
t ap
pear
s th
at e
ach
teac
her
has
sele
cted
one
hig
h, o
ne lo
w, a
nd o
ne m
iddl
e st
uden
t su
mm
ary
from
the
ir o
wn
clas
s. G
ood
sign
in t
erm
s of
peo
ple
actu
ally
bro
ught
the
sam
ples
as
agre
ed u
pon/
dire
cted
in p
rior
m
eeti
ng/c
omm
unic
atio
ns.
N s
how
s on
OH
a “
1” e
xam
ple
of s
tude
nt w
ritt
en s
umm
ary
of t
he T
hree
Litt
le P
igs.
Sho
ws
a “2
” ex
ampl
e. N
sho
ws
a hi
gh
(“3”
) R
ask
s T
s w
hat
are
som
e of
the
str
engt
hs t
hey
see
in t
he
sam
ples
. Ts
iden
tify
abo
ut 7
dif
fere
nt s
peci
fic
stre
ngth
s w
ith
mul
tipl
e te
ache
rs r
espo
ndin
g, o
ne a
t a
tim
e …
OC
: Mor
e go
od s
igns
= m
ulti
ple
teac
hers
par
tici
pati
ng a
nd t
each
ers
utili
se c
omm
on/e
ffic
ient
cou
rtes
ies
of s
peak
ing
one
pers
on a
t a
tim
e. S
ome
of t
he s
tren
gths
tha
t te
ache
rs v
ocal
ise–
and
N
wri
tes
thes
e on
the
OH
… (
Elm
, gr
ade-
leve
l tea
m m
eetin
g)
70 D. McDougall et al.
of meeting topics; (k) scheduled, weekly, “hands-off” (i.e., protected fromcompeting demands and conducted as scheduled) versus “more loosely” scheduledmeetings that were frequently cancelled, curtailed, rescheduled, changed at the lastminute, or otherwise disrupted.
Staff at GR and comparison schools reported that the Content Standards forCalifornia Public Schools (California State Department of Education, n.d.) consti-tuted their school-wide academic goals. They also stated that improving students’scores on annual standardised tests was a school-wide goal. However, only GLTs atGR schools actually formulated, wrote, disseminated, and evaluated “more specific”academic goals based on the Standards. Some of these GLTs also focused theirinstruction on these specific goals, in a “planful” manner, by utilising a systematicprocess they acquired via GR training (see Table 6). GLTs at comparison schoolsdid not have or execute such systematic processes during their meetings.
Finding 2. The GR Model established tighter linkages between teachers andadministrators in their efforts to focus on academic goals and improve students’academic achievement. First, teachers at GR schools were much more visible andregular participants in their schools’ academic leadership teams (termed ALTs atGR schools). Each of the four GR schools included at least one teacher from eachgrade level on their school-wide ALT. Of the three comparison schools, Hall hadzero teachers on its leadership team, Lot did not have a formal leadership team, andKam sometimes included one or two teachers on its team on an as-needed basis.Second, principals at GR schools attended and participated more consistently atGLT meetings and teachers’ professional development sessions than principals atcomparison schools. Principals at most GR schools demonstrated greater awareness,focus, and participation in the day-to-day academic plans and actions of teachers ateach grade level. The tighter academic linkages between teachers and administratorsat GR schools facilitated more effective execution of goal-directed plans than atcomparison schools, where the evaluator observed more frequent “slippage”between intended actions and actual implementation of academic initiatives.
RQ 3: What does implementation of the GR Model do to impact academic achievement?
Finding 1. GR Model implementation impacted students’ academic achievementby developing settings and processes whereby educators’ behaviours and instruc-tional processes became more focused and produced visible improvements instudents’ academic achievement and attainment of academic goals. teachers’ attri-butions for academic gains, teachers’ attitudes toward purposeful instructional tasks,teachers’ instructional efficacy, and teachers’ expectations for themselves and forstudents’ academic achievement changed when, and to the extent that, teachersexperienced—frequently, punctually, and directly—visible improvements inacademic achievement associated with their “results-producing” behaviours andinstructional processes.
Inside the Black Box of School Reform 71T
able
6.
GR
em
phas
is o
n sy
stem
atic
pla
nnin
g an
d in
stru
ctio
n (F
ocus
ing
on a
nd A
ddre
ssin
g C
omm
on S
tude
nt N
eeds
in G
rade
Lev
el M
eeti
ngs:
T
he 7
Im
port
ant T
asks
/Ste
ps …
in a
Nut
shel
l)
Tas
kG
oal
Thi
ngs
to r
emem
ber
1Id
enti
fy a
nd c
lari
fy s
peci
fic
and
com
mon
stu
dent
ne
eds
to w
ork
on t
oget
her
●O
ptio
ns f
or id
enti
fyin
g ne
eds:
Sta
ndar
ds, S
tanf
ord
9 co
nten
t cl
uste
rs,
Beg
inni
ng o
r M
idye
ar a
sses
smen
t re
sult
s, C
urri
culu
m (
read
ing
or
mat
hem
atic
s se
ries
)●
If y
ou h
ave
BM
Es,
use
tho
se t
o id
enti
fy n
eeds
●C
onne
ct/c
lari
fy n
eeds
usi
ng s
tand
ards
●C
lari
fy n
eeds
by
anal
ysin
g st
uden
t w
ork
●W
rite
a b
rief
des
crip
tion
of
the
need
2F
orm
ulat
e a
clea
r ob
ject
ive
for
each
com
mon
nee
d an
d id
enti
fy r
elat
ed s
tude
nt w
ork
to b
e an
alys
ed●
Rev
iew
the
com
pone
nts
of c
lear
obj
ecti
ves:
Wha
t do
you
wan
t st
uden
ts t
o do
, und
er w
hat
circ
umst
ance
s, a
nd w
ith
wha
t, if
any
, kin
d of
sup
port
or
set
up?
How
man
y st
uden
ts d
o yo
u ex
pect
to
mee
t th
e ob
ject
ive
by w
hen,
and
w
hat
do y
ou w
ant
to s
ee t
o ev
iden
ce s
ucce
ss o
r m
aste
ry (
desi
red
qual
itie
s)?
●F
ollo
w a
n ex
ampl
e w
hen
wri
ting
an
obje
ctiv
e:S
ampl
e cl
arifi
ed n
eed:
For
Gra
de 1
, wri
ting
flu
ency
is b
eing
abl
e to
thi
nk
abou
t a
topi
c, f
ind
thin
gs t
o sa
y ab
out
the
topi
c, a
nd e
ffec
tive
ly u
se le
tter
s to
wri
te w
hat
you
wan
t to
say
, suc
h th
at s
omeo
ne e
lse
can
read
and
un
ders
tand
it.
Sam
ple
obje
ctiv
e: G
iven
fam
iliar
topi
cs, b
y th
e en
d of
Jan
uary
, mos
t Gra
de 1
st
uden
ts (
say,
75%
) w
ill b
e ab
le t
o co
nsis
tent
ly p
rodu
ce jo
urna
l ent
ries
du
ring
dai
ly jo
urna
ling
tim
e of
fou
r or
mor
e re
late
d se
nten
ces
that
sta
y on
to
pic,
sho
w r
easo
nabl
e th
ough
t, a
nd a
re r
eada
ble.
72 D. McDougall et al.T
able
6.
(Con
tinue
d)
Tas
kG
oal
Thi
ngs
to r
emem
ber
3Id
enti
fy a
nd a
dopt
a p
rom
isin
g in
stru
ctio
nal f
ocus
to
add
ress
eac
h co
mm
on n
eed
•C
onsi
der
firs
t w
hat
you
alre
ady
have
•C
onsi
der
som
ethi
ng a
bout
whi
ch s
omeo
ne in
the
gro
up h
as e
xper
tise
•C
onsi
der
som
ethi
ng a
bout
whi
ch y
ou c
an g
et p
ublis
hed
info
rmat
ion
•C
onsi
der
som
ethi
ng a
bout
whi
ch y
ou c
an s
ecur
e ou
tsid
e as
sist
ance
•U
se t
hese
que
stio
ns t
o ga
uge
the
prom
ise
of t
he in
stru
ctio
nal f
ocus
:-
Can
you
gen
eral
ly e
xpla
in w
hat
it is
, and
how
you
do
it in
the
cl
assr
oom
?-
Can
you
fit
it in
to y
our
exis
ting
pro
gram
me?
-C
an y
ou g
ener
ally
exp
lain
why
it w
ill li
kely
hel
p ad
dres
s th
e ob
ject
ive
(ass
umin
g yo
ur t
eam
wor
ks h
ard
to im
plem
ent
it w
ell)
?-
Is it
som
ethi
ng t
hat
a re
ason
able
num
ber
of f
olks
on
your
tea
m w
ill t
ry?
-C
an y
ou s
ecur
e th
e re
sour
ces
nece
ssar
y to
impl
emen
t it
as
a te
am?
•U
se w
orks
heet
to
reco
rd t
asks
1, 2
, 3 (
page
39
of G
R M
anua
l 200
1)4
Pla
n an
d co
mpl
ete
nece
ssar
y pr
epar
atio
n to
try
the
in
stru
ctio
nal f
ocus
in t
he c
lass
room
•P
lann
ing
and
prep
arat
ion
are
ongo
ing
and
cycl
ical
•In
vent
ory
the
nece
ssar
y pl
anni
ng a
nd p
repa
rati
on a
nd s
ecur
e co
mm
itm
ents
•G
et c
oncr
ete—
deve
lop
actu
al w
ritt
en p
lans
and
nee
ded
mat
eria
ls•
Get
con
cret
e—do
moc
k or
rea
l dem
onst
rati
ons
so p
eopl
e ca
n ob
serv
e an
d di
scus
s T
HE
TE
AC
HIN
G
Inside the Black Box of School Reform 73T
able
6.
(Con
tinue
d 2)
Tas
kG
oal
Thi
ngs
to r
emem
ber
5D
eliv
er I
nstr
ucti
on: M
ake
cons
iste
nt a
nd g
enui
ne
effo
rts
to t
ry t
he t
eam
’s in
stru
ctio
nal f
ocus
in t
he
clas
sroo
m
•T
he c
halle
nge:
thi
s st
ep is
not
tak
en d
urin
g a
mee
ting
but
in e
ach
teac
her’
s cl
ass
•T
ry t
o ge
t a
publ
ic c
omm
itm
ent
duri
ng m
eeti
ngs
from
eac
h te
am m
embe
r to
try
the
inst
ruct
iona
l foc
us in
the
cla
ssro
om•
Iden
tify
indi
vidu
als
who
will
bri
ng s
tude
nt w
ork
back
to
the
subs
eque
nt
mee
ting
or
will
be
prep
ared
to
repo
rt o
n th
eir
clas
sroo
m e
ffor
ts•
Rem
ind,
rem
ind,
rem
ind
(i.e
., “
Don
’t f
orge
t w
e al
l agr
eed
this
wee
k to
try
…
”)6
Ana
lyse
stu
dent
wor
k to
: (a)
see
whe
ther
the
ob
ject
ive
is b
eing
met
, (b)
bet
ter
unde
rsta
nd t
he
need
, and
(c)
eva
luat
e in
stru
ctio
n
•W
ith
rega
rd t
o ge
ttin
g pe
ople
to
brin
g st
uden
t w
ork:
spe
cify
ahe
ad o
f ti
me
wha
t sp
ecif
ic w
ork
to b
ring
, fro
m w
hich
tea
cher
s an
d fo
r w
hich
stu
dent
s•
Whe
n an
alys
ing
stud
ent
wor
k: u
se o
verh
eads
, foc
us o
n th
e ob
ject
ive,
and
id
enti
fy s
tren
gths
and
wea
knes
ses
in t
he w
ork;
the
n co
nnec
t th
e st
reng
ths
to w
hat
wor
ked
in t
he in
stru
ctio
n an
d th
e w
eakn
esse
s to
wha
t ne
eds
to b
e ad
dres
sed
in s
ubse
quen
t le
sson
s•
Use
a c
hart
like
the
one
on
page
37
of G
R M
anua
l, 20
01 t
o gu
ide
the
anal
ysis
7R
eass
ess:
con
tinu
e an
d re
peat
cyc
le o
r m
ove
on t
o an
othe
r ar
ea o
f ne
ed?
•C
entr
al q
uest
ion:
Is
the
obje
ctiv
e be
ing
met
?•
Als
o: A
re t
eam
mem
bers
will
ing
to s
usta
in e
ffor
ts o
n th
is n
eed
and
obje
ctiv
e?•
Gui
delin
e: I
f a
maj
orit
y of
the
mem
bers
wan
t to
sta
y th
e co
urse
, the
n st
ay
the
cour
se
74 D. McDougall et al.
Finding 2. GR implementation shifted some educators’ attributions for studentachievement toward specific, teacher-implemented, instructional actions and plan-ning processes, and away from teacher and student traits, and non-instructionalexplanations (see Table 7). Teachers in schools and GLTs where GR implementa-tion was strong frequently attributed student achievement, or lack thereof, tospecific instructional actions or processes they had or had not instituted. Teachers inschools and GLTs where GR implementation was weak were more likely to attributestudent achievement to global factors or student traits, such as experience andknowledge, socio-economic conditions, inexperience with the English language,academic inability, lack of readiness, and inadequate parental involvement.
For some educators, GR’s emphasis on systematic instruction cycles (Table 6)altered their instructional habits and beliefs about what constituted good teaching.For other educators, GR implementation affirmed these habits and beliefs. In bothinstances, educators clarified and raised their expectations for performing essentialtasks needed to promote student achievement. In educational contexts fraught withcompeting and shifting demands, GR implementation helped many educators,individually and collectively, to prioritise goals and focus instructional effortsaccordingly. Some teachers shifted from strict adherence to timeline-driven or page-sequenced coverage of materials based on actual mandates or perceived pressures.They recognised increasingly the implications of purposeful planning and “teachingless, better” instead of “covering” material. Similarly, GR implementation crystal-lised, for most principals, the importance of focusing on academic goals and essen-tial tasks that most directly impact student achievement. GR “reminded” principalsnot to get distracted by the “operations side” of their job, or by the gauntlet ofnumerous, emerging, competing demands.
Finding 3. GR implementation fostered a group ethos among some teams of grade-level teachers and ALT representatives—a collective willingness and commitment toformulate, adapt, implement, and evaluate instructional processes that targetedstudent achievement. This group ethos was revealed during many, but not all, GLTand ALT meetings via participants’ focused academic discourse, systematic plan-ning, purposeful use of assessment data, and agreements to implement and evaluategoal-directed teaching strategies. By providing adequate time and assistance (e.g.,substitute teachers and class coverage) for teachers to analyse student work and eval-uate assessment data with their peers, GR implementation altered teachers’ attitudestoward such tasks. Teachers increased and improved their analysis of assessmentdata to better inform important instructional decisions that impacted studentachievement. Viewing and interpreting information—especially student work prod-ucts and test results that provided teachers with timely evidence of academicimprovements—increased teachers’ instructional efficacy and promoted attributionsthat their instructional decisions and actions improved student achievement. Dataindicated that GR Model implementation: (a) increased teacher’s willingness toshare assessment data with colleagues; (b) altered teachers’ understanding and
Inside the Black Box of School Reform 75T
able
7.
Com
pari
son
of t
each
ers’
att
ribu
tion
s fo
r st
uden
t ac
hiev
emen
t
Str
onge
r G
R M
odel
impl
emen
tati
on a
ssoc
iate
d w
ith
teac
her
belie
fs t
hat
inst
ruct
ion
acco
unte
d fo
r st
uden
t ac
hiev
emen
tW
eake
r/no
n-im
plem
enta
tion
of
GR
Mod
el a
ssoc
iate
d w
ith
teac
her
belie
fs t
hat
fact
ors
othe
r th
an in
stru
ctio
n ac
coun
ted
for
stud
ent
achi
evem
ent
A …
mos
t of
the
gra
de le
vels
wer
e do
ing
dem
os [
righ
t]. T
each
ers
wou
ld
volu
ntee
r if
som
ethi
ng w
as w
orki
ng w
ell t
o de
mo
it. S
o, p
art
of t
hese
gra
de
leve
l mee
ting
s st
arte
d be
com
ing
dem
onst
rati
on t
ime-
doin
g lit
tle
min
i-le
sson
s…W
e sh
ared
dif
fere
nt le
sson
s …
som
eone
wou
ld d
emon
stra
te a
le
sson
–wha
teve
r’s
wor
king
in t
he c
lass
room
. So,
I t
hink
it w
as a
lso
very
he
lpfu
l for
us
… a
nd t
hen,
whe
n yo
u fi
nd o
ut w
hat’
s w
orki
ng in
one
cla
ss,
then
you
can
try
it in
the
oth
er c
lass
room
, and
so
even
tual
ly e
very
body
’s
doin
g th
e sa
me
thin
g be
caus
e it
wor
ks.
Com
men
t abo
ut a
cade
mic
ach
ieve
men
t and
we
know
abo
ut th
e st
uden
ts’ a
cade
mic
ach
ieve
men
t (i
.e.,
how
stu
dent
s ar
e do
ing)
…
sta
tem
ents
abo
ut lo
ss o
f con
trol
, “w
e’re
pro
fess
iona
ls”,
don
’t
need
to
have
peo
ple
telli
ng u
s w
hat
to d
o, w
e kn
ow, w
e ha
ve
degr
ees,
mas
ter’
s de
gree
s …
(F
ield
not
es,
Fir
, gr
ade-
leve
l tea
m
mee
ting)
A …
it g
oes
back
to
som
ethi
ng t
hat
I sa
id e
arlie
r. U
m, y
eah,
wit
h w
hate
ver
pres
sure
and
wha
teve
r do
wns
ide,
we
are
show
ing
resu
lts.
(T
rans
crip
t, P
ine,
te
ache
r fo
cus
grou
p)
[Tea
cher
] …
Say
s th
ese
kids
are
n’t
read
y fo
r w
hat
we’
re a
skin
g th
em t
o do
… A
nd I
hav
e ei
ght
stud
ents
who
don
’t s
peak
E
nglis
h, d
on’t
kno
w a
nyth
ing
abou
t w
riti
ng …
the
y do
n’t
get
it.
[Pri
ncip
al a
sks
teac
her]
In
gene
ral o
r ju
st w
riti
ng?
Tea
cher
: In
gene
ral.
(Fie
ld n
otes
, F
ir,
grad
e-le
vel t
eam
mee
ting)
2:23
. T h
olds
up
som
e of
the
bar
cha
rt r
esul
ts s
choo
l wid
e. H
e sa
ys t
hat
he
did
not
mak
e co
pies
of
this
for
gro
up. H
e m
enti
ons
that
(st
uden
ts’
perf
orm
ance
on)
hig
h fr
eque
ncy
wor
ds a
ctua
lly d
ropp
ed fr
om b
egin
ning
to
mid
-yea
r as
sess
men
t, e
spec
ially
in G
rade
s 4
and
5.
Coa
ch d
istr
ibut
es f
irst
one
-pag
e do
cum
ent
wit
h st
anda
rdis
ed
test
res
ults
fro
m la
st y
ear.
Tea
cher
s no
te t
hat
3rd
Gra
de s
core
s fr
om la
st y
ear
wer
e hi
gh a
nd r
elat
e th
at t
his
year
’s g
roup
(i.e
.,
who
are
now
the
ir G
rade
4 s
tude
nts)
sho
uld
do w
ell o
n te
st t
his
year
. Mal
e T
sta
tes
reas
on w
hy t
he s
core
s w
ere
good
/his
st
uden
ts a
re d
oing
wel
l in
mat
h is
bec
ause
mos
t of
the
/his
st
uden
ts a
re m
ale.
He
also
sta
tes
that
tha
t’s
why
the
rea
ding
sc
ores
are
low
; gir
ls d
o be
tter
wit
h re
adin
g, m
ales
do
bett
er w
ith
mat
h …
(F
ield
not
es,
Kam
, gr
ade-
leve
l tea
m m
eetin
g)
76 D. McDougall et al.T
able
7.
(Con
tinue
d)
Str
onge
r G
R M
odel
impl
emen
tati
on a
ssoc
iate
d w
ith
teac
her
belie
fs t
hat
inst
ruct
ion
acco
unte
d fo
r st
uden
t ac
hiev
emen
tW
eake
r/no
n-im
plem
enta
tion
of
GR
Mod
el a
ssoc
iate
d w
ith
teac
her
belie
fs t
hat
fact
ors
othe
r th
an in
stru
ctio
n ac
coun
ted
for
stud
ent
achi
evem
ent
R a
sks:
Are
peo
ple
doin
g w
ord
wal
l eve
ryda
y?O
C: R
see
ms
to p
rovi
de o
n-ta
rget
que
stio
ns/ c
omm
ents
tha
t fo
cus
on
anal
ysis
of
the
issu
es. (
Fie
ld n
otes
, O
ak,
grad
e-le
vel t
eam
mee
ting)
T s
ays
nobo
dy is
goi
ng to
han
d yo
u a
blue
prin
t … it
has
to c
ome
from
us
…
one
reas
on w
hy w
e’re
doi
ng s
o m
uch
bett
er t
han
1995
. It’
s a
huge
cha
nge
sinc
e 19
95; i
t’s
not
just
per
sona
litie
s, it
’s h
ow w
e vi
ew t
he s
tude
nts’
wor
k/w
ork.
(F
ield
not
es,
Oak
, ac
adem
ic le
ader
ship
team
mee
ting)
B c
omm
ents
abo
ut w
hen
kids
are
/are
not
rea
dy
“dev
elop
men
tally
” fo
r w
riti
ng t
asks
… r
efer
ence
s to
the
ir (
lack
of
) E
nglis
h re
adin
g/E
nglis
h la
ngua
ge p
rofi
cien
cy …
B m
akes
re
fere
nce
to O
pen
Cou
rt s
core
s …
B s
ays
we
have
a g
oal/f
ocus
on
voc
abul
ary.
The
n di
scus
sion
of
com
preh
ensi
on s
core
s be
ing
not
too
good
. T s
ays
she
thin
ks it
was
the
sto
ry …
(F
ield
not
es,
Fir
, gr
ade-
leve
l tea
m m
eetin
g)
Inside the Black Box of School Reform 77
expectations about the purposes of assessment data; (c) fostered an “improvementover time” versus a “one-shot” orientation for collecting, analysing, and using data;and (d) shifted grade-level teams from talking about indicators to systematic analysisand actual use of indicators to plan and implement instruction. The following exam-ple illustrates aspects of this finding. A teacher at Pine explained how GR-initiated,BME writing assessments changed the GLT’s use of assessment tools.
Our grade-level meetings, which were supposed to be geared toward improving scores,really had no relation to what tests we were giving. But now this year, since we’re able tochoose a goal to pick with the writing assessment … Those [assessments] have becomea valuable tool, whereas previously, I was just given them and that was it. They had novalue to me at all. (Teacher focus group)
Finding 4. GR implementation, through collaborative goal setting, analysis of indi-cators, and reflection on teacher-controlled instructional variables, impacted teach-ers’ expectations for student achievement. Some teachers’ raised their expectationsfor student achievement after they viewed student work or analysed data thatprovided explicit evidence of students’ academic improvement. However, mostteachers’ expectations did not appear to rise dramatically. Modest expectations forstudent achievement were reflected in teachers’ initial specification and subsequentrevision of academic goals, objectives, and criteria. Document retrieval indicatedthat GLTs frequently set a criterion of “75% of the students” when formulatingacademic goals. More notably, the practice of collaborative goal setting brought tothe surface teacher expectations for student achievement, and prompted someteachers to examine implications of their individual assumptions and collectiveexpectations, as reflected in the following exchange between teachers at OakElementary:
T1: I give them a piece of writing almost every night, or some kind of poem.T2: And you’re expecting them now to be able to able to look at a rubric and get a
three or a four?T3: I see an improvement, you know, big time … But I think some kids will never
have the ability to be a four in my mind.T1: At the fifth-grade level?T3: Yeah, because, um, you know, whatever factors … there are; their families or, I
have a girl who’s only been here, you know, for a few months … let’s say fromArgentina … you know, that’s just the way it is.
T1: But I think kids that have been here … that have had exposure to this writingprocess, exposure, it’s only going to help them.
T2: … I think the first graders now, let’s say in five years when they come to fifthgrade, hopefully they’ll be, uh, able to achieve better because they would havebeen exposed to the process.
T1: Yeah. You know, I’m a little dismayed to hear teachers say that they can neverachieve that level.
Finding 5. GR implementation required teachers to assume academic leadershiproles and to chart the academic course and outcomes of their schools. As teachers
78 D. McDougall et al.
performed leadership tasks that impacted their colleagues and promoted academicachievement, this altered teachers’ professional responsibility, instructional efficacy,and collegial accountability, particularly among teachers who served as ALT repre-sentatives for their GLTs. This finding is supported strongly via stark contrastsbetween what constituted a GLT meeting before and after GR implementation, andat GR versus comparison schools (see Table 5). Data collection identified manyexamples associated with the transformation of GLT meetings from “chat sessions”to settings where teachers took responsibility to meet collaboratively and focus onimproving academic achievement. Germane examples appeared in ALT meetings,too. These examples reflected teachers’ attitudes toward performing important tasksthat impacted student achievement, including sharing expertise with colleagues,supporting and pressuring colleagues, preparing instruction, utilising meeting time,and completing professional tasks.
A heightened sense of professional responsibility was illustrated vividly in twocases in which staff confronted very challenging issues. In the first case, GR andschool staff formed and implemented a strategy designed to get “reluctant” teach-ers to participate more actively in GLT efforts to improve students’ writing. In thesecond case, GR and school staff formed and implemented an interventiondesigned to assist a “struggling” upper-grade GLT. In both cases, ongoing exter-nal assistance from GR staff was paired with internal expertise and leadership ofschool staff. This melding of assistance and leadership triggered a sense ofurgency, agency, and responsibility among staff (Earl & Lee, 1998). It enabledstaff to confront problematic issues that had inhibited necessary changes ininstruction.
RQ 4: What helps and hinders schools’ implementation of the GR Model?
Five factors facilitated or inhibited GR Model implementation. The first factor thataffected GR Model implementation was combined leadership of school administra-tors and teachers who represented their GLT on school-wide ALTs. Administratorsand ALT representatives at Oak and Pine focused more consistently on academicgoals than their colleagues at Elm and Fir, as evidenced in their respective actions,participation, and discourse during GLT and ALT meetings. Administrators at Oakand Pine attended GLT meetings more frequently than their colleagues at Pine andFir. Administrators at Elm were distracted frequently by the day-to-day operationaldemands of their jobs. Administrators at Fir and Elm more frequently used andresponded to “walkie-talkie” communications during GLT and ALT meetings.These off-topic communications interrupted the flow and academic focus of themeetings. Elm’s Principal reported being “overwhelmed” by the combined academicand operational demands of the job. Some ALT representatives (i.e., teachers whowere GLT leaders) at Elm tended to “pick up” some of the leadership roles andresponsibilities for implementing GR activities. The principal at Fir was not asdistracted as the principal at Elm, but the leadership provided by ALT representa-tives at Fir was the least effective of all GR schools. Consequently, the overall rating
Inside the Black Box of School Reform 79
for the leadership element at Fir was 2—slightly lower than Elm’s rating of 3, andlower still than the rating of 3 to 4 for both Oak and Pine.
The second factor that impacted GR implementation was the relative frequencyand continuity of expert assistance provided by GR staff. Direct observations of GRstaff’s assistance confirmed school staffs’ repeated assertions that GR staff membersprovided essential expertise and leadership. GR assistance was strong over time atElm, Pine, and Oak. At Fir, a GR staff member provided frequent assistance to theprincipal and staff during Year 1. However, that assistance was provided much lessfrequently during subsequent years. The principal and some teachers at Fir reportedthat they “missed” this external assistance. Ratings and direct observations of GLTand ALT meetings at Fir confirmed this finding (see Table 4).
A third factor that affected GR implementation was school staffs’ readiness forchange, which varied by individual teachers and GLTs. Most nascent and veteranteachers were optimistic about school change initiatives and programmes. Theyinvested energy to implement GR Model activities and other initiatives. However, afew veteran teachers were pessimistic. They limited their investment and participa-tion based on past experiences with short-lived initiatives and programmes that“come and go”. In addition, most staff at GR schools were relatively unencumberedby “baggage” or past experiences that negatively impacted current relationships andjob performance. However, lingering issues, differing expectations, and idiosyncratichabits among some GLTs and individuals diverted valuable energy and time fromschool change efforts.
A fourth factor that impacted GR implementation was concurrent job demands.As stated previously, K-3 GLTs implemented GR procedures more effectively thantheir colleagues in Grades 4 and 5. Unlike their K-3 colleagues who had imple-mented a new structured reading programme the preceding year, teachers in theupper grades, at three of four GR schools, were implementing those sameprogrammes for the first time. Upper-grade teachers reported consistently that theyexpended extraordinary time implementing this new reading programme. First-yearimplementation demands, combined with concurrent first-year implementation of anew mathematics programme, proliferation of assessments, and difficulties resolvingpacing plans, accounted, in part, for greater stress and less effective GR implementa-tion among upper versus lower GLTs.
A fifth factor that impacted GR implementation was the type of operating sched-ule used at each school. Year-round multi-track scheduling—a response to over-crowding and shortages of school buildings in this dense urban district—clearlycomplicated communication, planning, and scheduling at Elm, Fir, and Oak (andcomparison schools), but not at Pine, which used a traditional, 9-month single-trackschedule. Staff at Elm, Oak, and Fir expended considerable time during GLT meet-ings, ALT meetings, faculty meetings, and professional development sessions tryingto determine how teachers who were “off-track” could be informed, accommodated,or trained on GR and related activities. Teachers and administrators consistentlyexpressed concerns about the “hand-off” and continuity of initiatives betweenincoming and outgoing tracks. Multi-track scheduling made “teacher buy-in” more
80 D. McDougall et al.
challenging because one-third of the staff and their students were not present; theywere on their off-track vacation period. Consequently, only two-thirds of the staffwere present, at any given time, when important decisions were made, such as whenGLTs formulated academic goals and agreed to use corresponding instructionaltechniques.
RQ 5: What has changed as a result of GR Model implementation from the perspectives of participants and the external evaluator?
According to participants and the external evaluator, GR implementation increasededucators’ focus on academic learning and on getting academic results in fourimportant ways (see Table 8). First, GR transformed GLT meetings, from settingswhere non-academic topics and activities consumed time into settings where system-atic academic planning, instructional modeling, analysis of student work, and otherpurposeful, goal-directed tasks informed teachers’ instruction. Second, GR assistedand challenged teachers to provide leadership in charting the academic course of theschool through participation in newly created ALTs. Third, GR staff assisted andchallenged administrators and staff to prioritise and emphasise purposeful, goal-directed efforts aimed at improving teaching and learning. Fourth, GR fosteredsystematic collection and use of assessment data, particularly BME writingassessments, in ways that more punctually and more effectively informed teachers’instructional decisions.
Discussion
In this final section, we discuss limitations of this external evaluation study, as wellas implications of this study’s findings for school reform in general, and morespecifically for understanding what goes on inside the black box of reform.
Limitations of the Evaluation Study
Important limitations of this study relate to the case-study approach used in thisprocess-focused evaluation. Case studies, inherently, must be bounded (Merriam,1998). We purposefully selected seven cases—four GR schools and three demo-graphically similar comparison schools. Thus, our findings, discussion, and implica-tions must be placed into context. One obvious limitation is that our findingsemanate strictly from primary schools. We did not and have not investigated howGR processes might operate in middle schools or high schools. So, we are left towonder if and how GR processes might apply to such schools. Some researchershave reported that educational reform processes, and even evaluating suchprocesses, are more challenging in middle schools and high schools compared withprimary schools (Greene & Lee, 2006).
Another major limitation of this study is that the external evaluator conductedrelatively few observations of teachers instructing students in their classrooms.
Inside the Black Box of School Reform 81T
able
8.
Par
tici
pant
s’ p
ersp
ecti
ves
on c
hang
es r
esul
ting
fro
m G
R im
plem
enta
tion
Gra
de-
leve
l tea
m m
eeti
ngs
tra
nsf
orm
edA
dm
inis
trat
ors
pri
orit
ised
an
d f
ocu
sed
“We’
re u
sing
our
gra
de-l
evel
mee
ting
s to
foc
us o
n in
stru
ctio
n no
w…
the
grad
e-le
vel m
eeti
ngs
are
very
wel
l pla
nned
…ha
ve
agen
das…
grad
e-le
vel m
eeti
ngs
are
no lo
nger
[us
ed f
or]
plan
ning
tr
eats
and
ass
embl
y pe
rfor
man
ces,
unf
ortu
nate
ly [
som
e-on
e ch
uckl
es].
But
it’s
goo
d …
for
the
stud
ents
bec
ause
it’s
not
a lo
t of
do
wn
tim
e at
the
se m
eeti
ngs…
focu
s on
[st
uden
ts’ a
cade
mic
] ne
eds
… m
ost
of t
he g
rade
leve
ls w
ere
doin
g de
mos
. Tea
cher
s w
ould
vo
lunt
eer
if s
ome-
som
ethi
ng w
as w
orki
ng w
ell t
o de
mo
it. S
o pa
rt o
f th
ese
grad
e-le
vel m
eeti
ngs
star
ted
beco
min
g de
mon
stra
tion
tim
e—do
ing
littl
e le
sson
s.”
(Pin
e fo
cus
grou
p)
[GR
] “h
elpe
d m
e fo
cus”
giv
en a
ll of
the
man
y th
ings
that
dra
w p
rinc
ipal
s aw
ay f
rom
inst
ruct
ion.
She
say
s, “
I di
dn’t
kno
w h
ow t
o le
ad”
in t
he p
ast
whe
n sh
e ca
me
on a
s a
prin
cipa
l, bu
t th
at B
ruce
/[G
R]
show
ed h
er h
ow,
the
spec
ific
s of
lead
ing
peop
le a
nd f
ocus
ing
them
on
inst
ruct
ion.
She
sh
owed
the
AL
T (
how
to le
ad a
nd fo
cus)
, the
n th
ey le
ad a
nd d
id li
kew
ise
wit
h th
eir
colle
ague
s …
whe
n sh
e fi
rst
chan
ged
from
gra
de-l
evel
cha
irs
form
at t
o A
LT
, tha
t te
ache
rs s
till
used
the
mee
ting
s as
com
plai
nt
sess
ions
and
to
deal
wit
h th
e m
any
non-
inst
ruct
iona
l ite
ms
… m
anag
ed
to g
et t
hem
focu
sed
on in
stru
ctio
nal g
oals
… v
ia p
ress
ure.
(P
ine
prin
cipa
l in
terv
iew
)…
hav
ing
mee
ting
s th
at f
ocus
on
stud
ent
achi
evem
ent
and
impr
ovin
g it
, ver
sus
facu
lty
mee
ting
s th
at t
alk
abou
t co
pier
s an
d th
ings
like
tha
t al
l of
the
tim
e.”
(Elm
focu
s gr
oup)
“And
, Bru
ce h
as b
een,
rea
lly, a
god
send
. He
chal
leng
es m
e an
d he
pu
shes
me,
and
he
the
mon
thly
mee
ting
s th
at w
e ha
ve w
ith
him
her
e at
sc
hool
—th
e ad
min
istr
ativ
e te
am [
AL
T]
here
hav
e be
en w
onde
rful
. B
ecau
se h
e pu
shes
you
, and
he
chal
leng
es y
ou, a
nd h
e ju
st s
uppo
rts
you
all t
he w
ay. S
o, w
e—I
don’
t th
ink
we
wou
ld b
e as
far
as
we
are
if it
w
eren
’t f
or, f
or t
he G
etti
ng R
esul
ts t
eam
.” (
Oak
pri
ncip
al in
terv
iew
)…
whe
n sh
e to
ok o
ver
as P
rinc
ipal
GL
T m
eeti
ng ti
me
was
som
ethi
ng
that
tea
cher
s m
ight
or
mig
ht n
ot h
ave
atte
nded
, ser
ved
as c
ompl
aint
ti
me…
get
ting
tea
cher
s to
att
end
was
a f
irst
ste
p &
acc
ompl
ishm
ent.
G
etti
ng t
hem
to
star
t ta
lkin
g ab
out
inst
ruct
ion
was
/is a
noth
er s
tep.
(F
ir P
rinc
ipal
inte
rvie
w)
Tea
cher
s p
rovi
ded
lead
ersh
ip v
ia A
LT
Ass
essm
ents
use
d t
o in
form
in
stru
ctio
n“I
thi
nk t
hat
AA
LT
, the
tea
m …
the
y ha
d a
lot
of le
ader
ship
thi
s ye
ar. A
nd I
kno
w t
hey
felt
the
pre
ssur
e. A
nd I
thi
nk t
hey
wer
e th
e le
ader
s be
caus
e th
ey w
ere
the
ones
who
wer
e ke
pt p
ushi
ng u
s …
le
adin
g th
e m
eeti
ngs
… Y
es a
nd k
eepi
ng u
s fo
cuse
d …
” (P
ine
focu
s gr
oup)
“No.
Tha
t’s
neve
r ha
ppen
ed b
efor
e. A
t th
e be
ginn
ing,
tha
t w
as a
lso
a to
ugh
nut
to c
rack
bec
ause
peo
ple
didn
’t w
ant
to b
ring
stu
dent
—“I
’m
not
goin
g to
bri
ng m
y st
uden
t w
ork
beca
use
then
peo
ple
will
thi
nk I
’m
not a
goo
d te
ache
r”. …
it to
ok a
whi
le to
get
aw
ay fr
om th
at …
to c
hang
e th
e m
inds
et …
82 D. McDougall et al.T
able
8.
(Con
tinue
d)
We
star
ted
the
Aca
dem
ic A
chie
vem
ent
Tea
m. A
nd t
hat’
s be
com
e ou
r cu
rric
ular
lead
ersh
ip, u
h, t
ool …
It’
s be
en a
cha
lleng
e to
, um
, br
ing
teac
hers
act
ivel
y in
to a
lead
ersh
ip r
ole
beca
use,
um
, the
y’re
ve
ry u
ncom
fort
able
tel
ling
thei
r co
lleag
ues
som
ethi
ng t
hey
don’
t w
ant
to h
ear
… w
hen
it c
omes
to
som
ethi
ng t
hat
they
’ve
com
e to
a
conc
lusi
on n
eeds
to
be d
one
that
’s n
ot g
oing
to
go d
own
wel
l, th
ey
still
wan
t th
e bo
ss t
o te
ll th
em”
[lau
ghs]
. (F
ir p
rinc
ipal
inte
rvie
w)
Bec
ause
tea
cher
s w
ere
very
unc
omfo
rtab
le w
ith
shar
ing
and
brin
ging
w
ork
in. N
ow it
’s n
ot a
n is
sue
anym
ore
… W
e’ve
nev
er s
at d
own
wit
h co
lleag
ues
to a
naly
se s
tude
nt w
ork
… A
nd w
e ne
ver
got
toge
ther
to
talk
. A
nd w
e ne
ver
got
toge
ther
to,
to
plan
tog
ethe
r. I
mea
n, it
’s ju
st t
hese
last
tw
o to
thr
ee y
ears
has
rea
lly c
hang
ed t
hing
s …
(O
ak p
rinc
ipal
inte
rvie
w)
“I r
emem
ber
my
sub
days
wer
e ju
st t
o gr
ade
the
asse
ssm
ents
… a
ctua
lly
got
me
to g
rade
the
ass
essm
ent
… r
eally
, rea
lly h
elpe
d …
and
dis
cuss
ed
the
resu
lts
… m
ade
note
s of
stu
dent
nee
ds o
n th
e sp
ot, s
o th
en w
hen
we
had
our
next
mee
ting
we
… s
et g
oals
… A
nd n
ot o
nly
asse
ssin
g bu
t lo
okin
g at
tho
se …
hel
ping
gui
de o
ur in
stru
ctio
n w
here
our
wea
knes
ses
are
… w
here
we
can
focu
s …
and
how
we
can
impr
ove
the
wea
knes
ses
and
cont
inue
wit
h th
e st
reng
ths
… m
ost
usef
ul is
the
wri
ting
ass
essm
ent
that
we
do a
t the
beg
inni
ng o
f the
yea
r, m
iddl
e, li
ke s
he s
aid,
and
the
end
of t
he y
ear.
It
help
s us
fin
d ou
t w
hat
need
s, u
h, w
e ne
ed t
o ad
dres
s w
ith
the
stud
ents
. And
I t
hink
it r
eally
hel
ps. [
Rig
ht a
way
.] [
Yea
h.]
Whe
reas
no
t, “
Oh,
sch
ool’s
end
ing
and
whe
re a
re, w
hat
are
we
lack
ing”
. (P
ine
focu
s gr
oup)
Inside the Black Box of School Reform 83
Frequent classroom observations of teacher–student interactions would haveprovided greater insights about linkages between GR Model elements, GR-relatedprocesses, and teaching and learning at GR schools. Likewise, observing teachingand learning, in vivo, in classrooms at comparison schools would have further illumi-nated our findings. Additionally, findings reported here are subject to the benefitsand limitations of using a sole external evaluator, as opposed to an evaluation team.
Implications for School Reform
Findings indicated that although implementation levels varied somewhat across GRsites, the GR Model was successfully implemented at GR sites. Moreover, GRimplementation established apparently unique processes that contributed toimproved academic achievement at low-achieving schools. GR schools also wererated higher than comparison schools on key elements of school improvement.These findings have important implications for school reform.
First, findings from this external evaluation contribute to the body of literaturethat supports the efficacy of change elements in the GR Model—elements that arenot unique to the GR Model. These elements are, indeed, evident in other reformdesigns (Fullan, 2000; Marzano & Kendall, 1996; Schmoker, 1996), although wewould note the GR model stands in contrast to much more prescriptive reforms thatspecify curricula and instruction (Borman et al., 2003; Desimone, 2002) and thathave shown the strongest evidence of impact. Nonetheless, the evaluation findingsreported here and in Saunders (2003) support the premise that goals (or standards),indicators (or assessments), assistance (or collaboration and professional develop-ment), leadership, and settings are influential levers for change. We have not yetestablished empirically the individual contribution of each element to improvedachievement. In one sense, the individual contribution of each element is irrelevant.The premise of the model is that no one element is sufficient; all are necessary. Wedo know that, in combination, and given reasonable development (i.e., Level 3 onthe GR Model Rubric), these elements were associated with more rapid gains inacademic achievement at GR schools, compared with the gains at demographicallysimilar comparison schools, and the overall school district, over a 5-year period.
Second, data collected in this study indicated that GR change elements were, atleast, somewhat evident at comparison schools. We believe that the challenge is notinstalling these elements, but developing and utilising them effectively and at a highlevel (e.g., a 3 or 4 on the GR Model Rubric). Recall that most GR schools were ratedat rubric levels 3 and 3 to 4 for each element, and most comparison schools were ratedat 2 and 2 to 3. If this school district reflects reform efforts in the U.S., then manyschools, based solely on local, state, and national emphases (e.g., content standards,annual assessment, professional development, school governance), and without localassistance of an entity like the Getting Results Network, might function at Level 2 orslightly higher (see GR Model Rubric, Table 2). In other words, the following arelikely to occur: (a) achievement goals are probably identified in the form of state ordistrict content standards; (b) school-wide indicators, in the form of standardised
84 D. McDougall et al.
tests, are probably used, at a minimum, to examine annual achievement trends; (c)assistance or professional development is likely to be available, at least intermittently;(d) leadership is most probably focused on improving academic achievement, at leastin general terms; and (e) settings, such as leadership teams, grade-level meetings, andfaculty meetings, are available and used, to some degree, to foster school improve-ment efforts. The GR Model provides one example of how schools might move thesechange elements toward a higher level of development and utilisation. We would thenexpect this to result in achievement benefits beyond those currently observed inschools where standards, mandated tests, professional development, and otherreform features have been mandated from the top down and with less systematicattention to change elements.
Third, findings from this study, like countless other school reform studies, pointto the critical role of leadership. In particular, this study’s findings seem to demon-strate the promise of increasingly distributed leadership that involves both adminis-trators and teachers working together. Indeed, some prominent scholars have calledfor more creative forms of distributed leadership that not only improve studentachievement, but also do so in socially just ways (Hargreaves & Fink, 2006), and inways that transform the culture of learning and teaching in schools (Fullan, 2002).GR implementation required teachers to assume academic leadership roles (RQ 3,Finding 5), helped establish tighter linkages between administrators and teachers(RQ 2, Finding 2), and brought about change by challenging and assisting teachersto actually provide leadership to staff and GLTs (RQ 5, Finding 2). Not unexpect-edly, some teachers in GR and comparison schools expressed concerns about theirroles as change agents (e.g., worrying about the extent to which the focus onacademic achievement in reading and mathematics impacted other aspects ofschooling, including physical education and health, music, and art). Mostprominent among those factors that helped schools successfully implement the GRModel was the combined leadership of school administrators and teachers via theALT (RQ 4, Finding 1). We address, here, three implications of our findings forresearch on, and practice of, leadership as a lever for school change.
First, we conceptualise and focus on leadership as it relates to other elements ofreform and school effectiveness; that is, not leadership per se, but leadership as itbears on setting and sharing goals or standards, utilising indicators or assessmentinformation, seeking out and cultivating assistance and collaboration, and establish-ing and maintaining productive settings such as leadership teams, grade-level meet-ings, and faculty meetings. We could have written a very different version of thisarticle, one that might have isolated our findings on leadership. But it seems morerelevant and theoretically more important to examine and discuss leadership in thecontext of the full GR Model, wherein leadership functions as one of five highlyinterdependent change elements. At least as it is conceptualised in the GR Modeland enacted at GR schools, leadership operates as a necessary although notsufficient condition for productive change.
Second, related to our first point, the findings of this study provide one example ofleadership as it functions within the context of a specific, and fairly concrete,
Inside the Black Box of School Reform 85
approach to school change. Although the GR Model/Network is curriculum-free(unlike, e.g., Success for All) (Slavin & Madden, 2006) and process-oriented, it hasadvanced over the years to become increasingly “nuts-and-bolts” oriented in itsapproach to the process of change and to the administrator and teacher leadershipthat successful execution requires. For example, GR assistance and trainings forprincipals and teacher-leaders, for the most part, do not focus on leadership per se.We certainly highlight the concept of “leadership that supports and pressures”—oneway to distill the large body of empirical and theoretical work on leadership. And wework with teachers to help them understand principles of keeping meetings focusedand people on-task (e.g., by creating and adhering to agendas, and dealing withproblematic personalities). But, more commonly, GR assistance and training forprincipals and teacher-leaders focus very directly on leading and facilitating specificand concrete instructional tasks. These tasks include, for example, using a specificprotocol to review and interpret standards; applying detailed procedures to adminis-ter, score, tally, and analyse results of specific assessments at the BME of the schoolyear; and using a specific protocol to identify common student needs, formulateobjectives, and analyse student work in the context of grade-level meetings.
An ongoing debate in the reform literature weighs the comparative benefits anddrawbacks of more conceptual versus more “nuts and bolts” approaches to reform(Bodilly, 1998). As the GR Network has evolved, we have consistently found it valu-able to try to work on both planes with school staffs—on the concepts or principlesunderlying concrete procedures and the successful execution of those procedures.Without question, over time we have produced higher levels of implementation andeffectiveness, and, we would argue, stronger and more effective leadership from bothprincipals and teachers as the nuts and bolts of the work have been made increas-ingly clear. It is possible that our efforts have been myopic—that we have cultivatedhighly contextualised, task-specific leadership skills with limited generalisability. Wehave completed no studies to investigate this possibility. Perhaps this is one of manyissues that merit study. Nonetheless, while noting this caution, GR results to datesuggest that benefits do accrue when we approach leadership programs and inter-ventions from a more contextualised, task-specific orientation. In other words, whenwe help educators in the schools get things done. Moreover, findings from recentmulti-method studies of primary school reform are consistent with our experiences.For example, Lithwood, Jantzi, and McElheron-Hopkins (2006) found that schoolleadership and school improvement processes accounted for the largest proportionof variance in explaining modest yet significant differences in student achievementacross primary schools.
Third, it seems important to note what the GR Model and its implementationdoes and does not focus on in terms of leadership. At the centre of the GR radar isteaching and learning—not school governance per se. Inherent in our initial theoret-ical orientation of school change, this heart-of-the-matter focus has been furtherreinforced as our work in schools evolved. Originally, we non-descriptly named ourmodel the “School Change Model” because we saw it as a vehicle for makingfundamental changes in how teachers approach the work of teaching and learning.
86 D. McDougall et al.
As the focus on student outcomes became increasingly pronounced, and the educa-tional system as a whole bore down on tangible evidence of student achievement asan impetus of improvement (e.g., Schmoker, 1996), we started to use the name“Getting Results”. Throughout the years, we have tried consistently to stay as closeas possible to the major source of improved achievement over which schools haveenormous direct control—teaching and learning in the classroom (Wang, Haertel, &Walberg, 1993). Some GR schools, as well as non-GR comparison schools, havehad governance training (e.g., LEARN, a local school-district reform effort that waspopular in the 1980s and 1990s, and School-Based Management), and some havenot. But we have not found focusing on governance issues independent of the issuesthat most directly impact teaching and learning in classrooms to be an effectiveapproach to accomplishing our fundamental goal: improving student achievement.
We raise this issue of governance in light of the strong findings this study uncov-ered about distributed leadership. Perhaps teaching and learning, in general—andmodel elements such as academically focused goals, indicators, and assistance, morespecifically—provide a fruitful common ground upon which shared or distributedleadership among administrators and teachers can be cultivated efficiently andsuccessfully. This is not to negate the potential import of shared or distributed lead-ership that focuses on governance. However, governance has for many years receivedsubstantial attention and emphasis in the reform literature. GR results suggestsubstantial promise in shared leadership that focuses specifically on helping teachersget results in their classrooms.
In the GR project, strong external assistance provided by GR staff helped admin-istrators and teachers become more effective leaders and experts—professionals whofocused their efforts and who pressured, supported, and assisted colleagues towardachieving specific, measurable goals in ways that impacted teaching and learning ona daily basis. GR staff provided the expertise initially, to help implement key aspectsof the operational model—ALTs, grade-level teams, collection and scoring ofstudent papers—and subsequently, to help sustain these settings and foster expertiseand leadership among key educators at GR schools (e.g., teachers who served asGLT leaders and ALT representatives, principals, academic coaches who workedacross grade levels). In time, a critical mass of focused educators emerged at GRschools and teaching became more coherent across each school. One very consistentrelated theme was that GR staff provided assistance in ways that earned educators’respect and impacted their attitudes and teaching practices. This assistance, whichoriginally was perceived by educators to be from an outsider, morphed into a moreinternal form of assistance as educators began to interact with GR staff as “one ofour own”.
Paths that lead to getting results are not likely to be smooth. Indeed, GR schoolsand comparison schools experienced many bumps in their respective paths,including a gauntlet of emerging demands that often impeded their way. Externalexpertise, combined with leadership within the school, helped schools stay on track.Elements of change could wane—focus in the form of goals that are set and shared,indicators that measure progress toward achieving goals, and settings where the nuts
Inside the Black Box of School Reform 87
and bolts of change must occur in order to get results. Indeed, these elementsseemed to be in danger of continual derailment since so much effort had to beexpended to maintain them. The external assister played a crucial role in helpingschool staffs keep focused on their ultimate goals and maintain settings and activitiesthat were essential for achieving them.
References
American Institutes for Research. (1999). An educators’ guide to schoolwide reform. Washington,DC: Author.
August, D., & Hakuta, K. (Eds.). (1997). Improving school for language minority children: A researchagenda. Washington, DC: National Academy Press.
Berends, M., Kirby, S., Naftel, S., & McKelvey, C. (2001). Implementation and performance in NewAmerican schools: Three years into scale-up (MR-1145-EDU). Santa Monica, CA: Rand.
Blase, J. (1987). Dimensions of effective school leadership: The teacher’s perspective. AmericanEducational Research Journal, 24, 589–610.
Bliss, J., Firestone, W., & Richards, C. (Eds.). (1991). Rethinking effective schools: Research andpractice. Englewood Cliffs, NJ: Prentice Hall.
Bodilly, S. J. (1998). Lessons from the New American Schools’ scale-up phase: Prospects for bringingdesign to multiple schools. Santa Monica, CA: RAND.
Borman, G., Hewes, G., Overman, L., & Brown, S. (2003). Comprehensive school reform andachievement: A meta-analysis. Review of Educational Research, 73, 125–230.
Brophy, J., & Good, T. (1986). Teacher behavior and student achievement. In M. Wittrock (Ed.),Handbook of research on teaching (3rd ed., pp. 328–375). New York: Macmillan.
California State Department of Education (n.d.). Content standards for California public schools.Retrieved March 4, 2002, from http://www.cde.ca.gov/standards.
Carter, T., & Chatfield, M. (1986). Effective bilingual schools: Implications for policy andpractice. American Journal of Education, 95, 200–232.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.Creswell, J. W. (2003). Research design: Qualitative, quantitative, and mixed methods approaches.
Thousand Oaks, CA: Sage.Darling-Hammond, L. (1997). The right to learn: A blueprint for creating schools that work. San
Francisco, CA: Jossey-Bass.Denzin, N. (1978). The research act: A theoretical introduction to sociological methods. New York:
McGraw-Hill.Desimone, L. (2002). How can comprehensive school reform models be successfully
implemented? Review of Educational Research, 72, 433–479.Earl, L. M., & Lee, L. E. (1998). Evaluation of the Manitoba School Improvement Program.
International Centre for Educational Change at OISE/UT.Edmonds, R. (1979). Effective schools for the urban poor. Educational Leadership, 37(1), 15–27.Fashola, O., Slavin, R., Calderón, M., & Durán, R. (1996). Effective programs for Latino students in
elementary and middle schools. Unpublished manuscript. Baltimore, MA: The Johns HopkinsUniversity.
Fullan, M. (1993). Change forces: Probing the depths of educational reform. London: Falmer.Fullan, M. (2000). The three stories of educational reform. Phi Delta Kappan, 81, 581–584.Fullan, M. (2002). The change leader. Educational Leadership, 59(8), 16–20.Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory. Chicago: Aldine.Goffman, E. (1959). The presentation of self in everyday life. Garden City, NJ: Doubleday.Goldenberg, C. (2004). Successful school change: Creating settings to improve teaching and learning.
New York: Teachers College Press.
88 D. McDougall et al.
Goldenberg, C., & Gallimore, R. (1991). Changing teaching takes more than a one-shot work-shop. Educational Leadership, 49(3), 69–72.
Goldenberg, C., Saunders, W., & Gallimore, R. (1996). Settings for change: A practical model linkingrhetoric and action to improve achievement of diverse students. Proposal submitted to the SpencerFoundation.
Goldenberg, C., & Sullivan, J. (1994). Making change happen in a language minority school: A searchfor coherence (Educational Practice Report No. 13). Washington, DC: National Center forResearch on Cultural Diversity and Second Language Learning.
Greene, J. C., & Lee, J. H. (2006). Quieting educational reform … with educational reform.American Journal of Evaluation, 27, 337–352.
Harcourt Assessment. (1996). Harcourt educational measurement. Retrieved from http://harcour-tassessment.com.
Hargreaves, A., & Fink, D. (2006). The ripple effect. Educational Leadership, 63(8), 16–20.Joyce, B., & Showers, B. (1983). Power in staff development through research on training. Alexandria,
VA: Association for Supervision and Curriculum Development.Kvale, S. (1996). Interviews: An introduction to qualitative research interviewing. Thousand Oaks,
CA: Sage.Lieberman, A. (1988a). Expanding the leadership team. Educational Leadership, 45(5), 4–8.Lieberman, A. (1988b). Teachers and principals: Turf, tension, and new tasks. Phi Delta Kappan,
69, 648–653.Lincoln, E., & Guba, Y. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage.Lithwood, K., Jantzi, D., & McElheron-Hopkins, C. (2006). The development and testing of a
school improvement model. School Effectiveness and School Improvement, 17, 441–464.Little, J. (1982). Norms for collegiality and experimentation: Workplace conditions of school
success. American Educational Research Journal, 19, 325–340.Manning, K. (1997). Authenticity in constructivist inquiry: Methodological considerations
without prescription. Qualitative Inquiry, 3(1), 93–115.Marzano, R., & Kendall, J. (1996). Designing standards-based districts, schools, and classrooms.
Alexandria, VA: Association for Supervision and Curriculum Development.McDougall, D. (2002). Getting Results project external evaluation. Long Beach, CA: Getting Results
Network.Merriam, S. B. (1998). Qualitative research and case study applications in education. San Franciso,
CA: Josey-Bass.Miles, M. (1983). Unraveling the mystery of institutionalization. Educational Leadership, 41(3),
14–19.Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis. Thousand Oaks, CA: Sage.Peterson, K., & Lezotte, L. (1991). New directions in the effective schools movement. In J. Bliss,
W. Firestone, & C. Richards (Eds.), Rethinking effective schools (pp. 128–137). EnglewoodCliffs, NJ: Prenctice Hall.
Rosenholtz, S. (1991). Teachers’ workplace: The social organization of schools. New York: TeachersCollege Press.
Sarason, S. (1972). The creation of settings and the future societies. San Francisco, CA: Jossey-Bass.Saunders, W. (2003). “Getting Results”: Final report of achievement impact. Long Beach, CA:
California State University.Saunders, W., O’Brien, G., Marcelletti, D., Hassenstab, K., Saldivar, E., & Goldenberg, C.
(2001). Getting the most out of school-based professional development. In P. Schmidt & P.Mosenthal (Eds.), Reconceptualizing literacy in the new age of multiculturalism and pluralism(pp. 289–320). Greenwich, CT: IAP.
Schmoker, M. (1996). Results: The key to continuous school improvement. Alexandria, VA:Association for Supervision and Curriculum Development.
Slavin, R., & Fashola, O. (1998). Show me the evidence! Proven and promising programs for America’sschools. Thousand Oaks, CA: Corwin.
Inside the Black Box of School Reform 89
Slavin, R. E, & Madden, N. A. (2006). Success for all/roots and wings: 2006 summary of research onachievement outcomes. Baltimore, MD: John Hopkins University.
Taylor, S. J., & Bogden, R. (1998). Introduction to qualitative research methods. New York: JohnWiley.
Tharp, R., & Gallimore, R. (1988). Rousing minds to life. Cambridge, England: Cambridge Univer-sity Press.
Wang, M., Haertel, G., & Walberg, H. (1993). Toward a knowledge base for school learning.Review of Educational Research, 63, 249–294.