When is assessment learning-oriented? Dylan Wiliam dylanwiliam

44
When is assessment learning-oriented? Dylan Wiliam www.dylanwiliam.net 4th Biennial EARLI/Northumbria Assessment Conference, Potsdam, Germany, August 2008

description

When is assessment learning-oriented? Dylan Wiliam www.dylanwiliam.net. 4th Biennial EARLI/Northumbria Assessment Conference, Potsdam, Germany, August 2008. Overview of presentation. Why do we need formative assessment? Clarifying assumptions and definitions about formative assessment - PowerPoint PPT Presentation

Transcript of When is assessment learning-oriented? Dylan Wiliam dylanwiliam

Page 1: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

When is assessment learning-oriented?Dylan Wiliam

www.dylanwiliam.net

4th Biennial EARLI/Northumbria Assessment Conference, Potsdam, Germany, August 2008

Page 2: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

Overview of presentationWhy do we need formative assessment?Clarifying assumptions and definitions about formative assessmentA theoretically-based definition of formative assessmentHow formative assessment relates to other aspects of education

Page 3: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

Which of the following categories of skill has disappeared from the work-place most rapidly over the last 40 years?

1. Routine manual2. Non-routine manual3. Routine cognitive4. Complex communication5. Expert thinking/problem-solving

Page 4: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

The changing demand for skill

Autor, Levy & Murnane, 2003

Page 5: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

There is only one 21st century skillSo the model that says learn while you’re at school, while you’re young, the skills that you will apply during your lifetime is no longer tenable. The skills that you can learn when you’re at school will not be applicable. They will be obsolete by the time you get into the workplace and need them, except for one skill. The one really competitive skill is the skill of being able to learn. It is the skill of being able not to give the right answer to questions about what you were taught in school, but to make the right response to situations that are outside the scope of what you were taught in school. We need to produce people who know how to act when they’re faced with situations for which they were not specifically prepared. (Papert, 1998)

Page 6: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

A convergence of interestsPhilosophies of education (Williams, 1966)…Transmission of culturePreparation for employementSelf-actualization

…all require preparation for future learning (PFL)Cannot be taught in isolation from other learningStudents still need the basic skills of literacy, numeracy, concepts and factsLearning power is developed primarily through pedagogy, not curriculumWe have to develop the way teachers teach, not what they teach

Page 7: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

Learning power environmentsKey concept:Teachers do not create learningLearners create learningTeaching is the engineering of effective learning environments

Key features of effective learning environments:Create student engagement (pedagogies of engagement)Well-regulated (pedagogies of contingency)

Page 8: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

Why pedagogies of engagement?Intelligence is partly inheritedSo what?Intelligence is partly environmentalEnvironment creates intelligence Intelligence creates environmentLearning environmentsHigh cognitive demand InclusiveObligatory

Page 9: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

Motivation: cause or effect?

competence

challenge

Flow

apathyboredom

relaxation

arousalanxiety

worry control

high

low

low high

(Csikszentmihalyi, 1990)

Page 10: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

Why pedagogies of contingency?Learners do not learn what they are taughtAssessment is the bridge between teaching and learning, and thus the central process of teaching (as opposed to lecturing).

Pedagogies of contingency Personalisation

Mass customization (rather than mass production or individualisation) Diversity

A valuable teaching resource (rather than a challenge to be minimized)

Page 11: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

What gets learnt?Denvir & Brown (1996)Understanding of basic number

in low-achieving 7-9 year oldsExtensive conceptual hierarchy

developedStudents assessedTeacher plans teaching

programmeStudents re-assessed

Page 12: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

The research evidenceSeveral major reviews of the research…Natriello (1987)Crooks (1988)Kluger & DeNisi (1996)Black & Wiliam (1998)Nyquist (2003)

… all find consistent, substantial effects

Page 13: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

It’s the cost-benefit ratio stupid…Intervention Extra months of

learning per yearCost/

classroom/yr

Class-size reduction (by 30%) 4 €25k

Increase teacher content knowledge from weak to strong

2 ?

Formative assessment/Assessment for learning

8 €2.5k

Page 14: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

Independent dimensions of assessmentScale Large-scale (nomothetic) versus small-scale (idiographic)Locus Classroom versus examination hallAuthority Teacher-produced versus expert-producedScope Continuous versus one-offFormat Multiple-choice versus constructed responseFunction Formative versus summative

Page 15: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

No such thing as formative assessmentPurposes of assessments Evaluative Summative Formative

Instruments Purposes Functions

Prospects for integration are bleak

Formative assessment involves the creation of, and capitalization upon, moments of contingency in instruction

“An assessment functions formatively when evidence about student achievement elicited by the assessment is interpreted and used to make decisions about the next steps in instruction that are likely to be better, or better founded, than the decisions that would have been made in the absence of that evidence.”

Black and Wiliam, 2009 (we hope!)

Page 16: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

Some principlesA commitment to formative assessmentDoes not entail any view of what is to be learnedDoes not entail any view of what happens when learning takes place

…although clarity on these is essential.

Page 17: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

Evolving conceptions of formative assessment“Feedback” metaphor

Components of a feedback system data on the actual level of some measurable attribute; data on the reference level of that attribute; a mechanism for comparing the two levels and generating information

about the ‘gap’ between the two levels; a mechanism by which the information can be used to alter the gap.

Feedback system Importance of eliciting the right data The role of the learner The role of the learning milieu (e.g., as activity system)

Page 18: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

Unpacking formative assessmentKey processesEstablishing where the learners are in their learningEstablishing where they are goingWorking out how to get there

ParticipantsTeachersPeersLearners

Page 19: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

Aspects of formative assessmentWhere the learner

is going Where the learner is How to get there

Teacher Clarify and share learning intentions

Engineering effective discussions, tasks and

activities that elicit evidence of learning

Providing feedback that moves learners

forward

PeerUnderstand and share learning

intentions

Activating students as learningresources for one another

Learner Understand learning intentions

Activating students as ownersof their own learning

Page 20: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

Five “key strategies”…Clarifying, understanding, and sharing learning intentionscurriculum philosophyEngineering effective classroom discussions, tasks and activities that elicit evidence of learningclassroom discourse, interactive whole-class teachingProviding feedback that moves learners forward feedbackActivating students as learning resources for one another collaborative learning, reciprocal teaching, peer-assessmentActivating students as owners of their own learningmetacognition, motivation, interest, attribution, self-regulated learning, self-

assessment

(Wiliam & Thompson, 2007)

Page 21: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

…and one big ideaUse evidence about achievement to adapt instruction to meet learner needs

Page 22: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

Keeping learning on track (KLT)A pilot guides a plane or boat toward its destination by taking constant readings and making careful adjustments in response to wind, currents, weather, etc.A KLT teacher does the same:Plans a carefully chosen route ahead of time (in essence building the track)Takes readings along the way Changes course as conditions dictate

Page 23: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

Effects of formative assessmentLong-cycle Span: across units, terms Length: four weeks to one year Impact: Student monitoring; curriculum alignmentMedium-cycle Span: within and between teaching units Length: one to four weeks Impact: Improved, student-involved, assessment; teacher cognition about learningShort-cycle Span: within and between lessons Length:

day-by-day: 24 to 48 hours minute-by-minute: 5 seconds to 2 hours

Impact: more responsive classroom practice; increased student engagement

Page 24: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

System responsiveness and time-framesIf evidence is to inform decision-making, the evidence needs to be available before the decision… Long-cycle: Are our professional development programmes well-aligned with the

needs of our teachers? Cycle-length: two years

Long-cycle: Does our curriculum adequately cover the state standards as operationalized in the annual state test? Cycle-length: one year

Medium-cycle: Is this student responding adequately to the tier 1 intervention for reading or do they require a tier 2 intervention? Cycle-length: one month

Short cycle: Does the class understand the generation of equivalent fractions well enough to move on to the addition of fractions? Cycle-length: five minutes

Page 25: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

The formative assessment hi-jack…“Statistical process control” models of learningUSA: “Formative tests”Tests administered at intervals of 6 to 10 weeksOften not even keyed to instructionEngland (5 to 16 year olds):“Assessment for learning strategy”Government policy focused on target-setting and level chasingFocus on “tracking achievement”England (Higher Education): Portfolio assessment New focus on formative e-assessment

Page 26: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

Ideas whose time has come…or gone…Diagnostic analysis of standardized tests is probably dead Lack of agreements about models Models make assumptions not about items, but how students answer them Dearth of assessment developers who know enough about learners Poor efficiency

More promising developments Use of Bayesian inference networks to build proficiency models But

Proficiency models are not necessarily developmental models Models need large amounts of data to run

Page 27: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

Getting the cycle right (and the right cycle)Within this view of formative assessment feedback is not whole of formative assessment It’s not even the most important component of formative assessment

Medium- and long-cycle formative assessments Are supported by existing psychometrics Are easy to manage, but Generally produce small effectsShort-cycle formative assessments Contradict important psychometric assumptions

Reliability Monotonicity of ICCs

Are difficult to establish, but Generally produce large effects

Page 28: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

The overlap between age-cohorts is large…

QuickTime™ and a decompressor

are needed to see this picture.

The spread of achievement within each cohort is greater than generally assumed

Page 29: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

…so individual progress is hard to trackOn typical standardized tests growth is slow…Average annual growth of achievement of individuals is around 0.4 sdSo monthly growth of individual achievement is 0.03 sd…and the reliability of the test is limited…A reliability of 0.90 corresponds to a standard error of measurement of 0.3 sd In other words, the SEM of a highly reliable test is ten times the monthly

growth in achievement.So standardized tests are completely useless for monitoring individual progress in achievement—they are insensitive to instruction.

Page 30: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

…and the data is no use when it arrives…Traditional testing deals with individuals, but teachers mostly deal with groupsData-Push vs. Decision-Pull “Data-push”

Quality control at end of an instructional sequence Monitoring assessment that dentifies that remediation is needed, but not what Requires new routines to utilize the information “a series of unwanted answers to unasked questions” (Popper)

Decision-Pull Starts with the decisions teacher make daily Supports teachers “on-the-fly” decisions

If a 30-item test provides useful information on an individual, then responses from 30 individuals on a single item might provide useful information on a class

Page 31: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

Characteristics of hinge-point questionsRelate to important learning outcomes necessary for progression in learningCan be used at any point in a learning sequenceBeginning (range-finding)Middle (mid-course correction)End (e.g., “exit pass”)When used in “real-time” teacher must be able to collect and interpret the response of all students in 30 seconds

Page 32: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

Low probability of correct guessingIn which of these right-angled triangles is a2 + b2 = c2 ?

A a

c

b

C b

c

a

E c

b

a

B a

b

c

D b

a

c

F c

a

b

Page 33: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

Build on key (mis-)conceptions…in mathWhat can you say about the means of the following two data sets?

Set 1: 10 12 13 15Set 2: 10 12 13 15 0

A. The two sets have the same mean.B. The two sets have different means.C. It depends on whether you choose to count the zero.

Page 34: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

Wilson & Draney, 2004

…in Science…

The ball sitting on the table is not moving. It is not moving because:

A. no forces are pushing or pulling on the ball.

B. gravity is pulling down, but the table is in the way.C. the table pushes up with the same force that gravity pulls downD. gravity is holding it onto the table. E. there is a force inside the ball keeping it from rolling off the table

Page 35: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

… and History.Why are historians concerned with bias when analyzing sources?

A. People can never be trusted to tell the truthB. People deliberately leave out important detailsC. People are only able to provide meaningful information if they

experienced an event firsthandD. People interpret the same event in different ways, according to their

experienceE. People are unaware of the motivations for their actionsF. People get confused about sequences of events

Page 36: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

Cognitive Rules Responses

A

B

C

D

Correct

Incorrect

Requirements for hinge-point questionsFor an item to support instructional decision-making, the key requirement is that in no case do incorrect and correct cognitive rules map on to the same response (Wylie & Wiliam, 2007)

Page 37: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

B C DA

B C DA

Item improvement

The discovery of new incorrect cognitive rules that interpret item keys leads to item improvement

In which of these figures is one-quarter shaded?

Page 38: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

FeedbackKinds of feedback in Higher Education (Nyquist, 2003)

Weaker feedback only Knowledge of results (KoR)

Feedback only KoR + clear goals or knowledge of correct results (KCR)

Weak formative assessment KCR+ explanation (KCR+e)

Moderate formative assessment (KCR+e) + specific actions for gap reduction

Strong formative assessment (KCR+e) + activity

Page 39: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

Effect of formative assessment (HE)N Effect*

Weaker feedback only 31 0.14

Feedback only 48 0.36

Weaker formative assessment 49 0.29

Moderate formative assessment 41 0.39

Strong formative assessment 16 0.56

*corrected values

Page 40: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

FeedbackFeedback should Cause thinking Provide guidance on how to improve Focus on what to take forward to the next assignment rather that what is deficient

about the last assignment Be used Techniques

Delayed scores/grades Learning portfolios “Five of these answers are wrong. Find them and fix them” ‘Three-quarters of the way through a unit” test

Page 41: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

Sharing learning intentionsEffective summative assessment:

Requires raters to share a construct of qualityEffective formative assessment

Requires learners to share the same construct of quality as the raters Requires teachers to have an anatomy of quality

Techniques Explaining learning intentions at start of lesson/unit

Learning intentions Success criteria

Intentions/criteria in students’ language Posters of key words to talk about learning Planning/writing frames Annotated examples ‘flesh out’ assessment standards (e.g. lab reports) Opportunities for students to design their own tests

Page 42: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

Students owning their learning and as learning resources for one anotherTechniquesStudents assessing their own/peers’ work

with rubrics with exemplars “two stars and a wish”

Training students to pose questions/identifying group weaknessesSelf-assessment of understanding

Traffic lights Red/green discs

Page 43: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

The learning milieuDual processing theory (Boekaerts, 1993)Self-regulated learning is both metacognitively governed and affectively

charged (Boekaerts, 2006 p. 348)Students form mental representations of the task-in-context and appraise:

current perceptions of the task and the physical, social, and instructional context within which it is embedded;

activated domain-specific knowledge and (meta)cognitive strategies related to the task;

beliefs about motivation (including domain-specific capacity), interest and the effects of effort

Resulting appraisal generates activity along one of two pathways: ‘well-being’ pathway ‘growth’ pathway

Page 44: When is assessment learning-oriented? Dylan Wiliam dylanwiliam

When is assessment learning oriented?Assessment is learning oriented when it Is integrated into instructional design so that it becomes invisibleCreates engagement in learningHelps learners to understand what successful performance looks likeGenerates information that can be interpreted in terms of a learning

progressionFocuses attention on growth rather than well-beingProvides a focus for supportive conversations between learners