Why teaching will never be a research-based profession and why that’s a Good Thing Dylan Wiliam (@dylanwiliam)

www.dylanwiliam.net1

Outline

What does it mean for a practice to be “research-based”?

Why educational research falls short What educational research should do, and how it

should do it The role of teachers in educational research

2

What does it mean to be research-based?3

In a ‘research-based’ profession: Professionals would, for the majority of decisions they

need to take, be able to find and access credible research studies that provided evidence that particular courses of action that would, implemented as directed, be substantially more likely to lead to better outcomes than others.

Important caveats about research findings4

Educational research can only tell us what was, not what might be.

Moreover, in education, “What works?” is rarely the right question, because everything works somewhere, and nothing works everywhere, which is why in education, the right question is, “Under what

conditions does this work?”

5

Causality: a tricky issue

Traditionally, causality has been defined in terms of a counter-factual argument “We may define a cause to be an object followed by

another, and where all the objects, similar to the first, are followed by objects similar to the second. Or, in other words, where, if the first object had not been, the second never had existed.” (Hume, 1748 Section VII)

“If c and e are two actual events such that e would not have occurred without c, then c is a cause of e.” (Lewis 1973 p. 563)

Research methods 101: causality

Does c cause e? Given c, e happened (factual)

Problem: post hoc ergo propter hoc If c had not happened, e would not have happened

(counterfactual) Problem: c did happen

So we need to create a parallel world where c did not happen Same group different time (baseline measurement)

Need to assume stability over time Different group same time (control group)

Need to assume groups are equivalent Randomized controlled trial

6

Problems with RCTs in education7

Clustering Power Implementation Context

Meta-analysis in education:“I think you’ll find it’s a bit more complicated than that” (Goldacre, 2008)

8

Educational Endowment Foundation toolkit9

Intervention Cost Quality of evidence

Extra months of learning

Feedback ££ ★★★ +8

Metacognition and self-regulation ££ ★★★★ +8

Peer tutoring ££ ★★★★ +6

Early years intervention £££££ ★★★★ +6

One to one tuition ££££ ★★★★ +5

Homework (secondary) £ ★★★ +5

Collaborative learning £ ★★★★ +5

Phonics £ ★★★★ +4

Small group tuition £££ ★★★★ +4

Behaviour interventions £££ ★★ +4

Digital technology ££££ ★★★★ +4

Social and emotional learning £ ★★★★ +4

Educational Endowment Foundation toolkit10

Intervention Cost Quality of evidence

Extra months of learning

Parental involvement £££ ★★★ +3

Reducing class size £££££ ★★★ +3

Summer schools £££ ★★ +3

Sports participation £££ ★★ +2

Arts participation ££ ★★★ +2

Extended school time £££ ★★ +2

Individualized instruction £ ★★★ +2

After school programmes ££££ ★★ +2

Learning styles £ ★★★ +2

Mentoring £££ ★★★ +1

Homework (primary) £ ★★★ +1

Educational Endowment Foundation toolkit11

Intervention Cost Quality of evidence

Extra months of learning

Teaching assistants ££££ ★★ 0

Performance pay ££ ★ 0

Aspiration interventions £££ ★ 0

Block scheduling £ ★★ 0

School uniform £ ★ 0

Physical environment ££ ★ 0

Ability grouping £ ★★★ -1

12

An illustrative example: feedback

Kluger and DeNisi (1996) review of 3000 research reports Excluding those:

without adequate controls with poor design with fewer than 10 participants where performance was not measured without details of effect sizes

left 131 reports, 607 effect sizes, involving 12652 individuals

On average, feedback increases achievement Effect sizes highly variable 38% (50 out of 131) of effect sizes were negative

Understanding meta-analysis13

A technique for aggregating results from different studies by converting empirical results to a common measure (usually effect size)

Standardized effect size is defined as:

Problems with meta-analysis The “file drawer” problem Variation in population variability Selection of studies Sensitivity of outcome measures

The “file drawer” problem

14

The importance of statistical power

The statistical power of an experiment is the probability that the experiment will yield an effect that is large enough to be statistically significant.

In single-level designs, power depends on significance level set magnitude of effect size of experiment

The power of most social studies experiments is low Psychology: 0.4 (Sedlmeier & Gigerenzer, 1989) Neuroscience: 0.2 (Button et al., 2013) Education: 0.4

Only lucky experiments get published…

15

Variation in variability

16

Annual growth in achievement, by age17

Bloom, Hill, Black, and Lipsey (2008)

5 6 7 8 9 10 11 12 13 14 15 160.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

Age

annu

al g

row

th (S

Ds)

A 50% increase in the rate of learning for six-year-olds is equivalent to an effect size of 0.76 A 50% increase in the

rate of learning for 15-year-olds is equivalent to an effect size of 0.1

Variation in variability18

Studies with younger children will produce larger effect size estimates

Studies with restricted populations (e.g., children with special needs, gifted students) will produce larger effect size estimates

Selection of studies

19

Feedback in STEM subjects20

Review of 9000 papers on feedback in mathematics, science and technology

Only 238 papers retained Background papers 24 Descriptive papers 79 Qualitative papers 24 Quantitative papers 111

Mathematics 60 Science 35 Technology 16

Ruiz-Primo and Li (2013)

Classification of feedback studies21

1. Who provided the feedback (teacher, peer, self, or technology-based)?2. How was the feedback delivered (individual, small group, or whole

class)?3. What was the role of the student in the feedback (provider or

receiver)?4. What was the focus of the feedback (e.g., product, process, self-

regulation for cognitive feedback; or goal orientation, self-efficacy for affective feedback)

5. On what was the feedback based (student product or process)?6. What type of feedback was provided (evaluative, descriptive, or

holistic)?7. How was feedback provided or presented (written, video, oral, or

video)?8. What was the referent of feedback (self, others, or mastery criteria)?9. How, and how often was feedback given in the study (one time or

multiple times; with or without pedagogical use)?

Main findings22

Characteristic of studies included Maths Science

Feedback treatment is a single event lasting minutes 85% 72%

Reliability of outcome measures 39% 63%

Validity of outcome measures 24% 3%

Dealing only or mainly with declarative knowledge 12% 36%

Schematic knowledge (e.g., knowing why) 9% 0%

Multiple feedback events in a week 14% 17%

Sensitivity to instruction

23

24

Sensitivity of outcome measures

Distance of assessment from the curriculum Immediate

e.g., science journals, notebooks, and classroom tests Close

e.g., where an immediate assessment asked about number of pendulum swings in 15 seconds, a close assessment asks about the time taken for 10 swings

Proximal e.g., if an immediate assessment asked students to construct boats out of paper

cups, the proximal assessment would ask for an explanation of what makes bottles float

Distal e.g., where the assessment task is sampled from a different domain and where

the problem, procedures, materials and measurement methods differed from those used in the original activities

Remote standardized national achievement tests.

Ruiz-Primo, Shavelson, Hamilton, and Klein (2002)

Impact of sensitivity to instruction25

Effect size

Close Proximal

Why research hasn’t changed teaching

Aristotle’s main intellectual virtues Episteme: knowledge of universal truths Techne: ability to make things Phronesis: practical wisdom

Flyvbjerg (2001) “By definition, phronetic researchers focus on values;

for example by taking their point of departure in the classic value-rational questions: Where are we going? Is it desirable? What should be done?” (p130)

26

Maxims and rules

“Maxims are rules, the correct application of which is part of the art which they govern. The true maxims of golfing or of poetry increase our insight into golfing or poetry and may even give valuable guidance to golfers and poets; but these maxims would instantly condemn themselves to absurdity if they tried to replace the golfer's skill or the poet's art. Maxims cannot be understood, still less applied by anyone not already possessing a good practical knowledge of the art. They derive their interest from our appreciation of the art and cannot themselves either replace or establish that appreciation.”Polanyi (1958 pp. 31-32)

27

The knowledge-creating spiral

Nonaka and Takeuchi (1995)

28

Inquiry systems

System Evidence

Leibnizian Rationality

Lockean Observation

Kantian Representation

Hegelian Dialectic

Singerian Values, ethics, practical consequences

Churchman (1971)

29

30

Inquiry systems

The Lockean inquirer displays the ‘fundamental’ data that

all experts agree are accurate and relevant, and then

builds a consistent story out of these. The Kantian inquirer

displays the same story from different points of view,

emphasising thereby that what is put into the story by the

internal mode of representation is not given from the

outside. But the Hegelian inquirer, using the same data,

tells two stories, one supporting the most prominent policy

on one side, the other supporting the most promising story

on the other side (Churchman, 1971 p. 177).

Singerian inquiry systems31

The ‘is taken to be’ is a self-imposed imperative of the community. Taken in

the context of the whole Singerian theory of inquiry and progress, the

imperative has the status of an ethical judgment. That is, the community

judges that to accept its instruction is to bring about a suitable tactic or

strategy [...]. The acceptance may lead to social actions outside of inquiry,

or to new kinds of inquiry, or whatever. Part of the community’s judgement

is concerned with the appropriateness of these actions from an ethical

point of view. Hence the linguistic puzzle which bothered some empiricists

—how the inquiring system can pass linguistically from “is” statements to

“ought” statements— is no puzzle at all in the Singerian inquirer: the

inquiring system speaks exclusively in the “ought,” the “is” being only a

convenient façon de parler when one wants to block out the uncertainty in

the discourse. (Churchman, 1971: 202).

32

Educational research…

…can be characterised as a never-ending process of assembling evidence that: particular inferences are warranted on the basis of the

available evidence; such inferences are more warranted than plausible rival

inferences; the consequences of such inferences are ethically

defensible. The basis for warrants, the other plausible

interpretations, and the ethical bases for defending the consequences, are themselves constantly open to scrutiny and question.

A way forward: in Pasteur’s quadrant

Considerations of use

No Yes

Quest for fundamental

understanding?

Yes Pure basic research (Bohr)

Use-inspired basic research (Pasteur)

NoApplied research unmotivated by

applications (Brahe)Pure applied

research (Edison)

Stokes (1997)

33

The roles of teachers and researchers34

The role of teachers All teachers should be seeking to improve their

practice through a process of ‘disciplined inquiry’ Some may wish to share their work with others Some may wish to write their work up for publication Some may wish to pursue research degrees Some may even wish to undertake research

The role of education researchers Abandoning “physics envy” Working with teachers to make their findings

applicable in contexts other than the context of data collection

References

Bloom, H. S., Hill, C. J., Black, A. R., & Lipsey, M. W. (2008). Performance trajectories and performance gaps as achievement effect-size benchmarks for educational interventions. Journal of Research on Educational Effectiveness, 1(4), 289–328.

Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafo, M. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, advance online publication. doi: 10.1038/nrn3475

Churchman, C. W. (1971). The design of inquiring systems: basic concepts of systems and organization. New York, NY: Basic Books.

Flyvbjerg, B. (2001). Making social science matter: why social inquiry fails and how it can succeed again. Cambridge, UK: Cambridge University Press.

Goldacre, B. (2008). Bad science. London, UK: Fourth Estate.

Hume, D. (1748). An enquiry concerning human understanding. London, UK: Andrew Millar.

Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: a historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119(2), 254-284.

Lewis, D. (1973). Causation. Journal of Philosophy, 70(17), 556-567.

Nonaka, I., & Takeuchi, H. (1995). The knowledge-creating company: how Japanese companies create the dynamics of innovation. New York, NY: Oxford University Press.

Polanyi, M. (1958). Personal knowledge. London, UK: Routledge & Kegan Paul.

Ruiz-Primo, M. A., & Li, M. (2013). Examining formative feedback in the classroom context: New research perspectives. In J. H. McMillan (Ed.), Sage handbook of research on classroom assessment (2 ed., pp. 215-232). Thousand Oaks, CA: Sage.

Ruiz-Primo, M. A., Shavelson, R. J., Hamilton, L., & Klein, S. (2002). On the evaluation of systemic science education reform: searching for instructional sensitivity. Journal of Research in Science Teaching, 39(5), 369-393.

Sedlmeier, P., & Gigerenzer, G. (1989). Do studies of statistical power have an effect on the power of studies? Psychological Bulletin, 105(2), 309-316. doi: 10.1037/0033-2909.105.2.309

Stokes, D. E. (1997). Pasteur's quadrant: basic science and technological innovation. Washington, DC: Brookings Institution Press.

35