NFER Cymru Seminar Supporting student engagement March 2014.
Research into research use: A ten-arm cluster RCT 10 th September 2014 Dr Ben Styles, Dr Anneka...
-
Upload
marsha-griffin -
Category
Documents
-
view
218 -
download
1
Transcript of Research into research use: A ten-arm cluster RCT 10 th September 2014 Dr Ben Styles, Dr Anneka...
![Page 1: Research into research use: A ten-arm cluster RCT 10 th September 2014 Dr Ben Styles, Dr Anneka Dawson (NFER), Dr Lyn Robinson and Dr Christine Merrell.](https://reader036.fdocuments.in/reader036/viewer/2022062407/56649ddc5503460f94ad3dda/html5/thumbnails/1.jpg)
Research into research use:A ten-arm cluster RCT
10th September 2014
Dr Ben Styles, Dr Anneka Dawson (NFER), Dr Lyn Robinson and Dr Christine Merrell (CEM)
![Page 2: Research into research use: A ten-arm cluster RCT 10 th September 2014 Dr Ben Styles, Dr Anneka Dawson (NFER), Dr Lyn Robinson and Dr Christine Merrell.](https://reader036.fdocuments.in/reader036/viewer/2022062407/56649ddc5503460f94ad3dda/html5/thumbnails/2.jpg)
Background on evidence- based education
What research use experts said:• How are you going to measure outcomes?• Can you expect to see any difference?
![Page 3: Research into research use: A ten-arm cluster RCT 10 th September 2014 Dr Ben Styles, Dr Anneka Dawson (NFER), Dr Lyn Robinson and Dr Christine Merrell.](https://reader036.fdocuments.in/reader036/viewer/2022062407/56649ddc5503460f94ad3dda/html5/thumbnails/3.jpg)
Research dissemination: competing ideas
Passive methods are rarely effective
Passive methods are cheap i.e. a tiny effect is still cost effective
Measurement of attainment is cheap
(National Pupil Database)
Measurement of research use is complex
and expensive
Passive methods may not require consent
Several interventions
trying to achieve the same goal
Active methods can be expensive
![Page 4: Research into research use: A ten-arm cluster RCT 10 th September 2014 Dr Ben Styles, Dr Anneka Dawson (NFER), Dr Lyn Robinson and Dr Christine Merrell.](https://reader036.fdocuments.in/reader036/viewer/2022062407/56649ddc5503460f94ad3dda/html5/thumbnails/4.jpg)
Trial design
Two RCTs focused on improving Key Stage 2 literacy through teachers’ research use:
• Active trial- 780 primary schools randomised to experience one of varied activities from CPD to online communities (or control)
• Passive trial- Most other English primary schools randomised to receive one of various research-based communications (or control)
![Page 5: Research into research use: A ten-arm cluster RCT 10 th September 2014 Dr Ben Styles, Dr Anneka Dawson (NFER), Dr Lyn Robinson and Dr Christine Merrell.](https://reader036.fdocuments.in/reader036/viewer/2022062407/56649ddc5503460f94ad3dda/html5/thumbnails/5.jpg)
Outcomes
ACTIVE TRIAL PASSIVE TRIAL
Primary outcome Attainment in literacy at age 11
Attainment in literacy age 11
Secondary outcomes Teachers’ awareness, understanding and action in relation to research use
n/a
![Page 6: Research into research use: A ten-arm cluster RCT 10 th September 2014 Dr Ben Styles, Dr Anneka Dawson (NFER), Dr Lyn Robinson and Dr Christine Merrell.](https://reader036.fdocuments.in/reader036/viewer/2022062407/56649ddc5503460f94ad3dda/html5/thumbnails/6.jpg)
How many arms in the active trial?
• Four providers of research evidence, each with an active and passive approach
• Control group• CEM need three arms
![Page 7: Research into research use: A ten-arm cluster RCT 10 th September 2014 Dr Ben Styles, Dr Anneka Dawson (NFER), Dr Lyn Robinson and Dr Christine Merrell.](https://reader036.fdocuments.in/reader036/viewer/2022062407/56649ddc5503460f94ad3dda/html5/thumbnails/7.jpg)
Active trial
IEE
Electronic/ paper
materials
Evidence fair
Campaign for Learning/ Train
Visual
Website
Support
CEM
Paper materials
Light CPD
Advanced CPD
ResearchEd/ Nat Cen
Conference
Online community
Control group
780 schools recruited
60
60
60
60
60
60
60
60
60
240
![Page 8: Research into research use: A ten-arm cluster RCT 10 th September 2014 Dr Ben Styles, Dr Anneka Dawson (NFER), Dr Lyn Robinson and Dr Christine Merrell.](https://reader036.fdocuments.in/reader036/viewer/2022062407/56649ddc5503460f94ad3dda/html5/thumbnails/8.jpg)
Passive trial
IEE
Electronic/ paper
materials
Campaign for Learning/ Train
Visual
Website
CEM
Paper materials
ResearchEd/ Nat Cen
Conference
Control group
12,500 schools
2500 2500 2500 2500 2500
![Page 9: Research into research use: A ten-arm cluster RCT 10 th September 2014 Dr Ben Styles, Dr Anneka Dawson (NFER), Dr Lyn Robinson and Dr Christine Merrell.](https://reader036.fdocuments.in/reader036/viewer/2022062407/56649ddc5503460f94ad3dda/html5/thumbnails/9.jpg)
Timeline
Recruitment and baseline research
use measure
sAutumn
2014
Randomisatio
n
December
2014
Research
dissemination
Spring 2015
New teaching methods
and future
planning
Summer 2015
New teaching methods
Autumn 2015-spring 2016
Research use outcom
es
Spring 2016
Attainment
outcome
Summer 2016
Attainment
outcome
Summer 2017
![Page 10: Research into research use: A ten-arm cluster RCT 10 th September 2014 Dr Ben Styles, Dr Anneka Dawson (NFER), Dr Lyn Robinson and Dr Christine Merrell.](https://reader036.fdocuments.in/reader036/viewer/2022062407/56649ddc5503460f94ad3dda/html5/thumbnails/10.jpg)
CEM intervention
LEVEL 3: Evidence-based materials, two CPD sessions, pupil diagnostic information (InCAS), peer observations
LEVEL 2: Evidenced-based materials, one CPD session
LEVEL 1: Evidence-based materials (includes advice booklet, ‘tips for teachers’ cards and monthly classroom posters)
![Page 11: Research into research use: A ten-arm cluster RCT 10 th September 2014 Dr Ben Styles, Dr Anneka Dawson (NFER), Dr Lyn Robinson and Dr Christine Merrell.](https://reader036.fdocuments.in/reader036/viewer/2022062407/56649ddc5503460f94ad3dda/html5/thumbnails/11.jpg)
Challenge 1-Recruitment
Starting point- why do schools want to take part in research?:
• Having their say• Doing something good for education• The process being easy and sounding easy• Trust in the organisation approaching them• Security of data • Clear concise communication • Offer of feedback
![Page 12: Research into research use: A ten-arm cluster RCT 10 th September 2014 Dr Ben Styles, Dr Anneka Dawson (NFER), Dr Lyn Robinson and Dr Christine Merrell.](https://reader036.fdocuments.in/reader036/viewer/2022062407/56649ddc5503460f94ad3dda/html5/thumbnails/12.jpg)
Our recruitment approach
• Promotion at Research Ed and other provider events
• Word of mouth between schools and providers, NFER, EEF
• Website with detailed information on the trial and what is expected of schools and form to express an interest
• Letters, phoning, emails targeted response• Helping inform schools about RCTs including
our guide for schools
![Page 13: Research into research use: A ten-arm cluster RCT 10 th September 2014 Dr Ben Styles, Dr Anneka Dawson (NFER), Dr Lyn Robinson and Dr Christine Merrell.](https://reader036.fdocuments.in/reader036/viewer/2022062407/56649ddc5503460f94ad3dda/html5/thumbnails/13.jpg)
Recruitment (continued)
![Page 14: Research into research use: A ten-arm cluster RCT 10 th September 2014 Dr Ben Styles, Dr Anneka Dawson (NFER), Dr Lyn Robinson and Dr Christine Merrell.](https://reader036.fdocuments.in/reader036/viewer/2022062407/56649ddc5503460f94ad3dda/html5/thumbnails/14.jpg)
Challenge 2- Secondary outcome measurement
• Secondary outcomes are ‘awareness’, ‘understanding’ and ‘action’
• Measure using a bespoke survey • Schools involved in other research projects
using this survey cannot be in this trial
![Page 15: Research into research use: A ten-arm cluster RCT 10 th September 2014 Dr Ben Styles, Dr Anneka Dawson (NFER), Dr Lyn Robinson and Dr Christine Merrell.](https://reader036.fdocuments.in/reader036/viewer/2022062407/56649ddc5503460f94ad3dda/html5/thumbnails/15.jpg)
Research use outcomes
• Baseline survey developed without pilot• ‘Awareness’, ‘understanding’ and ‘action’
assessed through a series of questions in a survey of teachers
• Follow-up survey will be piloted in autumn• Questions will be retained if they discriminate
well and contribute to reliable measures
![Page 16: Research into research use: A ten-arm cluster RCT 10 th September 2014 Dr Ben Styles, Dr Anneka Dawson (NFER), Dr Lyn Robinson and Dr Christine Merrell.](https://reader036.fdocuments.in/reader036/viewer/2022062407/56649ddc5503460f94ad3dda/html5/thumbnails/16.jpg)
Intention-to-treat analysis
• Attainment (multilevel model, attainment at age 7 as covariate)
• Research use is also hierarchical but low numbers per cluster (research use at baseline as a covariate)
![Page 17: Research into research use: A ten-arm cluster RCT 10 th September 2014 Dr Ben Styles, Dr Anneka Dawson (NFER), Dr Lyn Robinson and Dr Christine Merrell.](https://reader036.fdocuments.in/reader036/viewer/2022062407/56649ddc5503460f94ad3dda/html5/thumbnails/17.jpg)
Challenge 3- Coordination of the delivery organisations
With 6 delivery partners carrying out different interventions things to consider:
• Effective, timely communication (inc one main point of contact at NFER and one at each provider)
• Funder as mediator for more basic updates• Monitoring information planned in advance
![Page 18: Research into research use: A ten-arm cluster RCT 10 th September 2014 Dr Ben Styles, Dr Anneka Dawson (NFER), Dr Lyn Robinson and Dr Christine Merrell.](https://reader036.fdocuments.in/reader036/viewer/2022062407/56649ddc5503460f94ad3dda/html5/thumbnails/18.jpg)
Acknowledgements
• Education Endowment Foundation• NFER Education Trials Unit
![Page 19: Research into research use: A ten-arm cluster RCT 10 th September 2014 Dr Ben Styles, Dr Anneka Dawson (NFER), Dr Lyn Robinson and Dr Christine Merrell.](https://reader036.fdocuments.in/reader036/viewer/2022062407/56649ddc5503460f94ad3dda/html5/thumbnails/19.jpg)
For more information
For Literacy Octopus project information:
www.nfer.ac.uk/octopus
For our guide for schools:
http://www.nfer.ac.uk/publications/RESM07/RESM07_home.cfm
For information on RCTs for educational researchers:
http://www.nfer.ac.uk/publications/RCT01/RCT01_home.cfm
For the NFER Education Trials Unit with details of our other trials
http://www.nfer.ac.uk/research/trials-unit/
![Page 20: Research into research use: A ten-arm cluster RCT 10 th September 2014 Dr Ben Styles, Dr Anneka Dawson (NFER), Dr Lyn Robinson and Dr Christine Merrell.](https://reader036.fdocuments.in/reader036/viewer/2022062407/56649ddc5503460f94ad3dda/html5/thumbnails/20.jpg)
NFER provides evidence for excellence
through its independence and insights, the
breadth of its work, its connections, and a
focus on outcomes.
National Foundation for Educational Research
The Mere, Upton Park
Slough, Berks SL1 2DQ
T: 01753 574123
F: 01753 691632
www.nfer.ac.uk