2005(1)

Knowledge mobilization, setting the context: systematic research synthesis

David Gough

Social Science Research Unit, Institute of Education, London, UK

2005 CESC-SSHRC SymposiumOttawa, 25th May 2005

2005(2)

Social Science Research Unit

Childhood Studies Evaluation of Social Interventions Sexual Health, Reproduction and Social

Exclusion Evidence for Policy and Practice Information

and Co-ordinating (EPPI) Centre Perspectives, Participation and Research

2005(3)

EPPI vision (1)EPPI vision (1)

A process of accumulating knowledge about key policy and practice issues

Which involves citizens (different groups of stakeholders) at all stages (setting the question… interpreting and disseminating the findings)

Which is held in accessible formats

2005(4)

EPPI vision (2)EPPI vision (2)

Which is free from ‘technical’ language Which results in usable and useful

‘evidence’ Which is subject to constant reflection

and development Which enables citizens to become

experts

2005(5)

Need for synthesis

Brings together what we know – whatever you are studying

Contextualizes information from new studies

Involves explicit systematic methods and thus transparency

These may be lacking in non systematic reviews and expert opinion however excellent

2005(6)

Key features of a systematic review

Synthesises the results of primary research

Uses explicit and transparent method A piece of research, following standard

set of stages Accountable, replicable, updateable Need for user involvement

2005(7)

Question led synthesis

Questions looking for answers Make implicit assumptions explicit All types of question so all types of

research design Statistical, narrative empirical and

conceptual synthesis Mixed methods synthesis

2005(8)

Not just any reviews:

6 reviews of older people and accident prevention

Total studies included Total studies included 137 137Common to at least two reviews 33Common to at least two reviews 33Common to all six reviews 2Common to all six reviews 2Treated consistently in all reviews 1Treated consistently in all reviews 1

Oliver et al. 1999Oliver et al. 1999

2005(9)

User question led synthesis

What do we want to know? Who wants to know and why? What do we know and how do we know

it? What more do we need to know and

how can we know it? Can not be value free Involves intellectual work

2005(10)

Dimensions of difference in synthesis models

Review/research questions (impact, process, need, explanatory concepts)

Research designs considered relevant Types of data - numerical or textual Quality assessment of different designs Breadth of designs and study focus Variation in contribution of each study to

the systematic synthesis

2005(11)

Types of systematic synthesis

Numerical

Narrative empirical

Conceptual

2005(12)

Numerical synthesis

For example: Statistical meta analysis of effect sizes from experimental studies of effect of interventions

2005(13)

Does sex education improve the use of contraception amongst young people?

From: DiCenso A, Guyatt G, Willan A, Griffith L (2002) Interventions to reduce unintended pregnancies amongst adolescents: a systematic review of randomised controlled trials. BMJ 231: 1426-1434

2005(14)

Narrative empirical

For example, EPPI review of Personal Development Planning (for LTSN): types of PDP and types of outcome (indep. & depend. variables)

Map and in-depth synthesis stage Weight of evidence Synthesis from conceptual framework

within SR question

2005(15)

Meta ethnography

Develop new interpretative constructions Data are concepts not empirical findings Emphasis on relevance Quality assurance:

exclusion criteria on quality E.g. Campbell R, Pound P, Pope C, Britten N, Pill R, Morgan M, Donovan J (2003)

Evaluating meta-ethnography: a synthesis of qualitative research on lay

experiences of diabetes and diabetes care. Social Science and Medicine 56(4): 671-684 worth of studies emerging during synthesis E.g. Noblitt G, Hare R (1988) Meta-Ethnography: Synthesizing Qualitative Studies.

London: Sage.

application of weighting system (for e.g. EPPI’s procedures)

2005(16)

Stages of an EPPI-Centre review

Defining studies (inclusion and exclusion criteria)

Searching exhaustively (search strategy)

Describing the key features of studies

Assessing their quality/ weight of evidence

Synthesising findings across studies

User involvement, setting question and developing protocol

Communication and engagement

MAP

IN-DEPTH REVIEW

Possibly apply further inclusion criteria here

2005(17)

Systematic maps and systematic synthesis

Map: What has been done?

maps out research activity (e.g. broader question with multiple designs)

provides context for synthesis research designs in primary studies part of

that context

Synthesis: What is known from what has been done?

2005(18)

Systematic mapping of research Study type by country of study in PDP review

0

2

4

6

8

10

12

14

16

18

Descriptive Exp of Rel Eval Nat Occ Eval RM

USA

UK

Australia

Canada

Hong Kong

Netherlands

Finland

Israel

Japan

Spain

Belgium

China

Singapore

South Africa

Taiwan

Unknown

2005(19)

‘Weight of Evidence’*

A. methodological quality of execution of study (in its own terms)

B. appropriateness of study design to review question

C. relevance of focus of study to the review question

D. overall WoE provided by study to answering the review question

review authors (not EPPI) determine WoE of each study on dimensions A, B,and C and then their relative contribution to total weight D

*allows use of ‘best evidence’ (Slavin)

2005(20)

Searching and screening Bibliographic

Data extraction and quality / relevance assessmentEPPI-Reviewer data-extraction

Characterising studies EPPI-Reviewer keywording

Numerical synthesis

EPPI-Reviewer

Narrative ‘empirical’synthesis

EPPI-Reviewer

Thematic/ conceptual synthesis

EPPI-R to NVivoWeb based data input / coding and analysis; web based

access to EPPI bibliographic and data-extraction databases, and review reports and summaries

2005(21)

This flexible system allows us to…

Address any type of policy, practice or research question

Include more than one study type and more than one type of synthesis method in the same review

Use different approaches to quality assessment

2005(22)

Review questione.g. What is known about the barriers to, and facilitators of, fruit

and veg intake amongst children aged 4 to 10 years?

Trials1. Application of inclusion criteria

2. Quality assessment3. Data extraction

4. Statistical meta-analysis

‘Qualitative’ studies1. Application of inclusion criteria

2. Quality assessment3. Data extraction

4. Qualitative synthesis

Trials and ‘views’Mixed methods synthesis

An EPPI-Centre review including more than one synthesis method

2005(23)

National and international collaborations undertaking systematic reviews Cochrane Collaboration (C1) Campbell Collaboration (C2) NHS Centre for Reviews and Dissemination

(CRD) ESRC Centre at QMW and Nodes Network Evidence for Policy and Practice Information

and Coordinating (EPPI) Centre: Developing methods for education and the social sciences

2005(24)

Thank you

d.gough@ioe.ac.uk

http://eppi.ioe.ac.uk

2005(25)

2005(26)

2005 CESC-SSHRC SymposiumOttawa, 26th May 2005

Knowledge mobilization and policy making

David Gough

EPPI-Centre, Social Science Research Unit, Institute of Education, London, UK

2005(27)

Key features of a systematic review

Synthesises the results of primary research

Uses explicit and transparent method A piece of research, following standard

set of stages Accountable, replicable, updateable Need for user involvement

2005(28)

User question led synthesis

What do we want to know? Who wants to know and why? What do we know and how do we know

it? What more do we need to know and

how can we know it? Can not be value free Involves intellectual work

2005(29)

Stages of an EPPI-Centre review

Defining studies (inclusion and exclusion criteria)

Searching exhaustively (search strategy)

Describing the key features of studies

Assessing their quality/ weight of evidence

Synthesising findings across studies

User involvement, setting question and developing protocol

Communication and engagement

MAP

IN-DEPTH REVIEW

Possibly apply further inclusion criteria here

2005(30)

Factors Influencing Policy Making in Government Davies 2004

Professional Experience &

Expertise

Political Judgement

Resources

ValuesHabits & Tradition

Lobbyists & Pressure Groups

Pragmatics & Contingencies

ResearchEvidence

2005(31)

Is research relevant to policy?

Habits and tradition Experience and expertise Judgement Resources Values Lobbyists and pressure groups Evidence

2005(32)

Use of research evidence

Political –support prior view Tactical – other purpose Enlightenment –framing of issues Problem solving– inform decision

making Naïve rational – lead to decision making

From Weiss (1979)

2005(33)

Types of research evidence

Implementation Evidence

Organizational Evidence

Economic/ Financial Evidence

Ethics Evidence

Forecast Evidence

Attitudinal Evidence

Program or Intervention Effectiveness

ExperimentalQuasi-Experimental

Counterfactual

SurveysAdmin DataComparativeQualitative

Cost-BenefitCost-Effectiveness

Cost-UtilityEconometrics

ExperimentalQuasi-Experimental

QualitativeTheories of Change

Public ConsultationDistributional Data

MultivariateRegression

SurveysQualitative

Lomas 2005 adapted from Davies, 2004

2005(34)

A research impact continuum from Sandra Nutley 2005

CONCEPTUAL USE INSTRUMENTAL USE

Awareness Knowledge Attitudes Practice/policy /understanding Perception change

2005(35)

Accessibility to inform naïve use!

Access to papers on primary studies Completeness of information in papers Guidelines for authors:

- Consort Statement for Trials

- Draft education guidelines

2005(36)

Study of 489 published papers and survey of guidelines in 12 journals

incomplete reporting of primary empirical studies in education

little guidance for authors about the important information that they should report about their studies in some well known educational journals

Draft new guidance:Newman M, Elbourne D, Leask M (forthcoming) Improving the

usability of education research:guidelines for the reporting of primary empirical research studies in education (The REPOSE Guidelines). Evaluation and Research in Education

2005(37)

Knowledge transfer

Research literacy of policy makers Research summaries, e.g. CCKM, CHSRF

‘Myth Busters’ and ‘Evidence Boost’ Worked examples Intermediaries: champions, translators Intermediary organisations, e.g. NICE, SCIE,

NERF Interactive working

2005(38)

Models of transmissionfrom Sandra Nutley 2005

Research-based practitioner

Embedded research

Organisational excellence

2005(39)

Access to / transfer of what?

Quality assessment of recommended programmes

Vulnerability of individual studies Distillation through systematic reviews Quality assurance of reviews Research evidence applicable locally? Intermediaries to adapt but open to bias Dangers of experts and panels

2005(40)

Salience of the research

Timelines Anecdotes and clear messages Lack of research data (no strategic

focus to research activity) Research but not focused on policy

makers needs Users of research and agenda setting

2005(41)

PERSPECTIVES AND PARTICIPATION

Interpretation and application

Communication

What do we know? How do we know it?What don’t we know?

How could we know it?

What has been done?

What do we want to

know?

TYPES OF KNOWLEDGE

RESEARCH QUESTIONS

RESEARCH STUDIES AND

METHODS

RESEARCH EVIDENCE

(iv) Policy community

(i) Research

(iii) Practitioner

(v) Organisational

(ii) User of service/ public

How/processes?

Nature/extent/frequency?

Effects?

Perspectives/ concepts?

Multi method

Action research

Case study

Survey

Experimental

How/processes?

Nature/extent/frequency?

Effects?

Perspectives/ concepts?

2005(42)

Beyond knowledge transfer

Linkage and exchange Ongoing interaction, collaboration, and

exchange of ideas between researcher and decision-maker communities

Knowledge brokers and brokering Links researcher and decision makers,

facilitating their interaction so that they better understand each other’s goals and professional culture, influence each other’s work, forge new partnerships and use research based evidence

2005(43)

Communities of practice

“An activity system about which participants share understandings concerning what they are doing and what it means in their lives and for their community”

Lave & Wenger, 1991

2005(44)

Limits to evidence use

“Decisions are less about projected consequences and more about process and legitimation. Politics is about shaping interpretations and expressing preferences. … Research… clarifi(es) issues and informs the wider public debate”

Young et al (2002)

2005(45)

Don’t forget the supply side

Still need investment in research and researchers!

In UK 2-3% of health budgets are on research. Same for education?

2005(46)

Thank you

d.gough@ioe.ac.uk

http://eppi.ioe.ac.uk