Post on 17-Dec-2015
11
Using assessment to support learning: why, what and how?
Dylan Wiliam,Institute of Education
www.dylanwiliam.net
22
Overview of presentation• Why raising achievement is
important• Why investing in teachers is the
answer• Why assessment for learning should
be the focus• Why teacher learning communities
should be the mechanism• How we can put this into practice
33
Raising achievement matters
• For individuals– Increased lifetime salary– Improved health
• For society– Lower criminal justice costs– Lower health-care costs– Increased economic growth
44
Where’s the solution?
• Structure– Small high schools– K-8 schools
• Alignment– Curriculum reform– Textbook replacement
• Governance– Charter schools– Vouchers
• Technology
55
School effectiveness
• 3 generations of effectiveness research– Raw results approaches
• Different schools get different results• Conclusion: Schools make a difference
– Demographic-based approaches• Demographic factors account for most of the variation• Conclusion: Schools don’t make a difference
– Value-added approaches• School-level differences in value-added are relatively
small• Classroom-level differences in value-added are large• Conclusion: An effective school is little more than a
school full of effective classrooms
66
It’s the classroom• Variability at the classroom level
is up to 4 times greater than at school level
• It’s not class size• It’s not the between-class
grouping strategy• It’s not the within-class grouping
strategy• It’s the teacher
77
Teacher quality• A labor force issue with 2 solutions
– Replace existing teachers with better ones?• No evidence that more pay brings in better teachers• No evidence that there are better teachers out there
deterred by certification requirements– Improve the effectiveness of existing teachers
• The “love the one you’re with” strategy• It can be done• We know how to do it, but at scale? Quickly?
Sustainably?
88
Cost/effect comparisonsIntervention Effect
(sd)Cost/yr
Class-size reduction (by 30%) 0.1 $20k
Increase teacher content knowledge by 2 sd
0.05 ?
Formative assessment/Assessment for learning
0.2 $2k
99
Learning power environments
• Key concept:– Teachers do not create learning– Learners create learning
• Teaching as engineering learning environments
• Key features:– Create student engagement (pedagogies of
engagement)– Well-regulated (pedagogies of contingency)
1010
Why pedagogies of engagement?
• Intelligence is partly inherited– So what?
• Intelligence is partly environmental– Environment creates intelligence– Intelligence creates environment
• Learning environments– High cognitive demand– Inclusive– Obligatory
1111
Why pedagogies of contingency?
• Several major reviews of the research– Natriello (1987)– Crooks (1988)– Kluger & DeNisi (1996)– Black & Wiliam (1998)– Nyquist (2003)
• All find consistent, substantial effects
1212
Types of formative assessment
• Long-cycle– Span: across units, terms– Length: four weeks to one year
• Medium-cycle– Span: within and between teaching units– Length: one to four weeks
• Short-cycle– Span: within and between lessons– Length:
• day-by-day: 24 to 48 hours• minute-by-minute: 5 seconds to 2 hours
1313
Effects of formative assessment
• Long-cycle– Student monitoring– Curriculum alignment
• Medium-cycle– Improved, student-involved, assessment– Improved teacher cognition about learning
• Short-cycle– Improved classroom practice– Improved student engagement
1414
Kinds of feedback (Nyquist, 2003)• Weaker feedback only
– Knowledge of results (KoR)
• Feedback only– KoR + clear goals or knowledge of correct results (KCR)
• Weak formative assessment– KCR+ explanation (KCR+e)
• Moderate formative assessment– (KCR+e) + specific actions for gap reduction
• Strong formative assessment– (KCR+e) + activity
1515
Effect of formative assessment (HE)N Effect
Weaker feedback only 31 0.16
Feedback only 48 0.23
Weaker formative assessment 49 0.30
Moderate formative assessment 41 0.33
Strong formative assessment 16 0.51
1616
Formative assessment
• Classroom assessment is not (necessarily) formative assessment
• Formative assessment is not (necessarily) classroom assessment
1717
Formative assessmentAssessment for learning is any assessment for which the first priority in its design and practice is to serve the purpose of promoting pupils’ learning. It thus differs from assessment designed primarily to serve the purposes of accountability, or of ranking, or of certifying competence. An assessment activity can help learning if it provides information to be used as feedback, by teachers, and by their pupils, in assessing themselves and each other, to modify the teaching and learning activities in which they are engaged.
Such assessment becomes ‘formative assessment’ when the evidence is actually used to adapt the teaching work to meet learning needs.
Black et al., 2002
1818
Feedback and formative assessment“Feedback is information about the gap between the actual level and the reference level of a system parameter which is used to alter the gap in some way” (Ramaprasad, 1983 p. 4)
• Three key instructional processes– Establishing where learners are in their learning– Establishing where they are going– Establishing how to get there
1919
Aspects of formative assessment
Where the learner is going
Where the learner is
How to get there
TeacherClarify learning
intentions
Engineering effective
discussions
Providing feedback that
moves learners on
PeerUnderstand/
clarify criteria for success
Activating students as instructional resources for one another
LearnerUnderstand
criteria for successActivating students as owners of their
own learning
2020
Five Key Strategies …
Questioning Engineering effective classroom discussions, questions, and learning tasks
Feedback Moving learners forward with feedback
Sharing LearningExpectations
Clarifying and sharing learning intentions and criteria for success
Self Assessment Activating students as the owners of their own learning
Peer Assessment Activating students as instructional resources for one another
2121
…and one big idea
• Use evidence about learning to adapt instruction to meet student needs
2222
Keeping Learning on Track (KLT)• A pilot guides a plane or boat toward its
destination by taking constant readings and making careful adjustments in response to wind, currents, weather, etc.
• A KLT teacher does the same:– Plans a carefully chosen route ahead of time
(in essence building the track)– Takes readings along the way – Changes course as conditions dictate
2323
Questioning
2424
Which fraction is the smallest?
a) 16
, b) 23
, c) 13
, d) 12
.
Success rate 88%
Which fraction is the largest?
Success rate 46%; 39% chose (b)
a) 45
, b) 34
, c) 58
, d) 7
10.
[Vinner, PME conference, Lahti, Finland, 1997]
Kinds of questions: Israel
2525
2626
Misconceptions
2727
Misconceptions
3a = 24
a + b = 16
2828
Molecular structure of water?
2929
Feedback
3030
Kinds of feedback: Israel• 264 low and high ability grade 6 students in 12 classes in
4 schools; analysis of 132 students at top and bottom of each class
• Same teaching, same aims, same teachers, same classwork
• Three kinds of feedback: scores, comments, scores+comments
[Butler(1988) Br. J. Educ. Psychol., 58 1-14]
Feedback Gain Attitudescores none top +ve
bottom -ve
comments 30% all +ve
3131
Responses
[Butler(1988) Br. J. Educ. Psychol., 58 1-14]
Feedback Gain Attitudescores none top +ve
bottom -ve
comments 30% all +ve
What do you think happened for the students given both scores and comments:
A: Gain: 30%; Attitude: all +ve
B: Gain: 30%; Attitude: top +ve, bottom -ve
C: Gain: 0%; Attitude: all +ve
D: Gain: 0%; Attitude: top +ve, bottom -ve
E: Something else
3232
Kinds of feedback: Israel (2)• 200 grade 5 and 6 Israeli students• Divergent thinking tasks• 4 matched groups
– experimental group 1 (EG1); comments– experimental group 2 (EG2); grades– experimental group 3 (EG3); praise– control group (CG); no feedback
• Achievement– EG1>(EG2≈EG3≈CG)
• Ego-involvement– (EG2≈EG3)>(EG1≈CG)
[Butler (1987) J. Educ. Psychol. 79 474-482]
3333
Effects of feedback• Kluger & DeNisi (1996)• Review of 3000 research reports• Excluding those:
– without adequate controls– with poor design– with fewer than 10 participants– where performance was not measured– without details of effect sizes
• left 131 reports, 607 effect sizes, involving 12652 individuals• Average effect size 0.4, but
– Effect sizes very variable– 40% of effect sizes were negative
3434
Feedback
• Formative assessment requires– data on the actual level of some measurable attribute;– data on the reference level of that attribute;– a mechanism for comparing the two levels and
generating information about the ‘gap’ between the two levels;
– a mechanism by which the information can be used to alter the gap.
• Feedback is therefore formative only if the information fed back is actually used in closing the gap.
3535
Formative assessment
• Frequent feedback is not necessarily formative
• Feedback that causes improvement is not necessarily formative
• Assessment is formative only if the information fed back to the learner is used by the learner in making improvements
• To be formative, assessment must include a recipe for future action
3636
How do students make sense of this?
• Attribution (Dweck, 2000)– Personalization (internal v external)– Permanence (stable v unstable)– Essential that students attribute both failures and
success to internal, unstable causes. (It’s down to you, and you can do something about it.)
• Views of ‘ability’– Fixed (IQ)– Incremental (untapped potential)– Essential that teachers inculcate in their students a
view that ‘ability’ is incremental rather than fixed(by working, you’re getting smarter)
3737
Sharing learning intentions
3838
Sharing criteria with learners• 3 teachers each teaching 4 grade 7 science
classes in two US schools• 14 week experiment• 7 two-week projects, scored 2-10• All teaching the same, except:• For a part of each week
– Two of each teacher’s classes discusses their likes and dislikes about the teaching (control)
– The other two classes discusses how their work will be assessed
[Frederiksen & White, AERA conference, Chicago, 1997]
3939
Sharing criteria with learners
7.47.26.7
6.65.94.6
Reflective assessment
Likes and dislikes
HighMiddleLowGroup
Iowa Test of Basic Skills
4040
Peer- and self-assessment
4141
Self-assessment: Portugal
• Teachers studying for MA in Education– Group 1 do regular programme– Group 2 work on self-assessment for 2 terms (20 weeks)– Teachers matched in age, qualifications and experience
using the same curriculum scheme for the same amount of time
• Pupils tested at beginning of year, and again after two terms– Group 1 pupils improve by 7.8 points– Group 2 pupils improve by 15
[Fontana & Fernandez, Br. J. Educ. Psychol. 64: 407-417]
4242
Putting it into practice
4343
Eliciting evidence of student achievement by engineering
effective classroom discussions, questions and learning tasks
4444
Practical techniques questioning• Key idea: questioning should
– cause thinking– provide data that informs teaching
• Improving teacher questioning– generating questions with colleagues – closed v open– low-order v high-order– appropriate wait-time
• Getting away from I-R-E– basketball rather than serial table-tennis– ‘No hands up’ (except to ask a question)– class polls to review current attitudes towards an issue– ‘Hot Seat’ questioning
• All-student response systems– ABCD cards, Mini white-boards, Exit passes
4545
Questioning in math: discussion
Look at the following sequence:3, 7, 11, 15, 19, ….
Which is the best rule to describe the sequence?A. n + 4B. 3 + nC. 4n - 1D. 4n + 3
4646
Questioning in math: diagnosis
In which of these right triangles is a2 + b2 = c2 ?
A a
c
b
C b
c
a
E c
b
a
B a
b
c
D b
a
c
F c
a
b
4848
Questioning in science: discussion
Ice-cubes are added to a glass of water. What happens to the level of the water as the ice-cubes melt?
A. The level of the water dropsB. The level of the water stays the sameC. The level of the water increasesD. You need more information to be sure
4949
Wilson & Draney, 2004
Questioning in science: diagnosis
The ball sitting on the table is not moving. It is not moving because:
A. no forces are pushing or pulling on the ball. B. gravity is pulling down, but the table is in the way.C. the table pushes up with the same force that gravity pulls downD. gravity is holding it onto the table. E. there is a force inside the ball keeping it from rolling off the table
5050
Dinosaurs extinction
Why did dinosaurs become extinct?A) Humans destroyed their habitatB) Humans killed them all for foodC) There was a major change in climate
5151
Save the ozone layer
What can we do to preserve the ozone layer?A. Reduce the amount of carbon dioxide produced
by cars and factoriesB. Reduce the greenhouse effectC. Stop cutting down the rainforestsD. Limit the numbers of cars that can be used
when the level of ozone is highE. Properly dispose of air-conditioners and fridges
5252
Questioning in English: discussion
• Macbeth: mad or bad?
5353
Questioning in English: diagnosis
Where is the verb in this sentence?
The dog ran across the road
A B C D
5454
A B C D
Questioning in English: diagnosis
Where does the subject end and the predicate begin in this sentence?
The dog ran across the road.
5555
Questioning in English: diagnosis
• Which of these is a good thesis statement?– The typical TV show has 9 violent incidents– There is a lot of violence on TV– The amount of violence on TV should be reduced– Some programs are more violent than others– Violence is included in programs to boost ratings– Violence on TV is interesting– I don’t like the violence on TV– The essay I am going to write is about violence on TV
5656
Questioning in history: discussion
• In which year did World War II begin?– 1919– 1937– 1938– 1939– 1941
5757
Questioning in History
Why are historians concerned with bias when analyzing sources?
A. People can never be trusted to tell the truthB. People deliberately leave out important detailsC. People are only able to provide meaningful information if
they experienced an event firsthandD. People interpret the same event in different ways,
according to their experienceE. People are unaware of the motivations for their actionsF. People get confused about sequences of events
5858
Hinge Questions
• A hinge question is based on the important concept in a lesson that is critical for students to understand before you move on in the lesson.
• The question should fall about midway during the lesson.
• Every student must respond to the question within two minutes.
• You must be able to collect and interpret the responses from all students in 30 seconds
5959
Figurative languageA. AlliterationB. HyperboleC. IronyD. MetaphorE. OnomatopoeiaF. PersonificationG. SimileH. None of the above
1. He was a bull in a china shop.2. May I have a drop of water?3. This backpack weighs a ton.4. The sweetly smiling sunshine…5. He honked his horn at the cyclist.6. I’ve told you a million times already.7. The Redcoats are coming!8. “They in the sea being burnt, they in
the burnt ship drown’d.”9. He was as tall as a house.
Triangle shirt waist factory fire, March 25th, 1911
6161
Triangle factory fire
Which of the following sources is biased?A. Photograph of the eventB. New York Times story on Mar 26, 1911C. Description of the fire in the textbookD. Transcript of talk by Frances Perkins, Sep 30 1964
6262
Practical techniques: feedback• Key idea: feedback should
– cause thinking– provide guidance on how to improve
• Comment-only grading• Focused grading• Explicit reference to rubrics• Suggestions on how to improve
– ‘Strategy cards’ ideas for improvement– Not giving complete solutions
• Re-timing assessment– (eg two-thirds-of-the-way-through-a-unit test)
6363
Practical techniques: sharing learning intentions
• Explaining learning intentions at start of lesson/unit– Learning intentions– Success criteria
• Intentions/criteria in students’ language• Posters of key words to talk about learning
– eg describe, explain, evaluate
• Planning/writing frames• Annotated examples of different standards to ‘flesh out’
assessment rubrics (e.g. lab reports)• Opportunities for students to design their own tests
6464
Practical techniques:peer and self-assessment
• Students assessing their own/peers’ work – with rubrics– with exemplars– “two stars and a wish”
• Training students to pose questions/identifying group weaknesses
• Self-assessment of understanding– Traffic lights– Red/green discs
• End-of-lesson students’ review
6565
Putting it into practice
6666
A model for teacher learning
• Content (what we want teachers to change)– Evidence– Ideas (strategies and techniques)
• Process (how to go about change)– Choice– Flexibility– Small steps– Accountability– Support
6767
Strategies and techniques
• Distinction between strategies and techniques– Strategies define the territory of AfL (no brainers)– Teachers are responsible for choice of techniques
• Allows for customization/ caters for local context• Creates ownership• Shares responsibility
• Key requirements of techniques– embodiment of deep cognitive/affective principles – relevance– feasibility– acceptability
6868
Design and interventionOur design process
Teachers’ implementation process
cognitive/affectiveinsights
synergy/comprehensiveness
set ofcomponents
set ofcomponents
synergy/comprehensiveness
cognitive/affectiveinsights
6969
Why research hasn’t changed teaching• The nature of expertise in teaching• Aristotle’s main intellectual virtues
– Episteme: knowledge of universal truths– Techne: ability to make things– Phronesis: practical wisdom
• What works is not the right question– Everything works somewhere– Nothing works everywhere– What’s interesting is “under what conditions” does
this work?
• Teaching is mainly a matter of phronesis, not episteme
7070
Knowledge ‘transfer’
aaa
Dialogue
Learning by doing
Socializationsympathised knowledge Externalizationconceptual knowledge
Internalizationoperational knowledge Combinationsystemic knowledge
Tacit knowledge Explicit knowledgeto
from
Tacit knowledge
Explicit knowledge
Sharing experience Networking
After Nonaka & Tageuchi, 1995
7171
Supporting Teachers and Schools to Change through Teacher
Learning Communities
7272
Implementing AfL requires changing teacher habits• Teachers “know” most of this already• So the problem is not a lack of knowledge• It’s a lack of understanding what it means to do AfL• That’s why telling teachers what to do doesn’t work• Experience alone is not enough—if it were, then the
most experienced teachers would be the best teachers—we know that’s not true (Hanushek, 2005)
• People need to reflect on their experiences in systematic ways that build their accessible knowledge base, learn from mistakes, etc. (Bransford, Brown & Cocking, 1999)
7373
That’s what TLCs are for:• TLCs contradict teacher isolation• TLCs reprofessionalize teaching by valuing teacher
expertise• TLCs deprivatize teaching so that teachers’
strengths and struggles become known• TLCs offer a steady source of support for struggling
teachers• They grow expertise by providing a regular space,
time, and structure for that kind of systematic reflecting on practice
• They facilitate sharing of untapped expertise residing in individual teachers
• They build the collective knowledge base in a school
7474
The synergy• Content: assessment for learning• Process: teacher learning communities• Components of a model
– Initial workshops– Support for TLC leaders– Monthly TLC meetings– Peer observations– ‘Drip-feed’ resources
• Web-site• Writings• New ideas
7575
Summary• Raising achievement is important• Raising achievement requires improving teacher
quality• Improving teacher quality requires teacher
professional development• To be effective, teacher professional development
must address– What teachers do in the classroom– How teachers change what they do in the classroom
• AfL + TLCs– A point of (uniquely?) high leverage– A “Trojan Horse” into wider issues of pedagogy,
psychology, and curriculum