Mixing different types of research in systematic reviews
Methodological Challenges for the Twenty First CenturyManchester, 22-23 November 2007
James Thomas
EPPI-Centre, Social Science Research Unit, Institute of Education, University of London
(2)
RESEARCH SYNTHESIS/ SYSTEMATIC REVIEWS
• A particular type or class of reviews
• Bridge between research and policy and practice
• Usually question-driven
• Use explicit methods, taking steps to increase trustworthiness
• Observational research or ‘Research on research’
(3)
Typical stages of a systematic review
Defining studies (inclusion and exclusion criteria)
Searching exhaustively (search strategy)
Describing the key features of studies
Assessing their quality/weight of evidence
Synthesising findings across studies
Setting question and developing protocol
Communication and engagement
(4)
Types of questions for systematic reviews*
• Effectiveness
• Screening and diagnosis
• Exploring risk or protective factors
• Observational associations between interventions and outcomes
• Questions about prevalence
• Questions about meanings and process
• Methodological questions
• Economic questions
* from Petticrew and Roberts (2005)
(5)
Examples of synthesis methods
• Statistical meta-analysis• Meta-ethnography• Grounded theory• Thematic analysis• Realist synthesis• Critical interpretive synthesis• Bayesian synthesis
Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A (2005) Synthesising qualitative and quantitative evidence: a review of possible methods. Journal of
Health Services Research and Policy 10: 45-53.
(6)
Why synthesise quantitative research?
Meta-analysis refers to the analysis of analyses . . . the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings. It connotes a rigorous alternative to the casual, narrative discussions of research studies which typify our attempts to make sense of the rapidly expanding research
literature.
Glass, 1976, p 3
(7)
From DiCenso et al. (2002) Interventions to reduce unintended pregnancies amongst adolescents: a systematic review of randomised controlled trials. British Medical Journal 231: 1426-1434
Does sex education increase the use of contraception amongst young people?
(8)
Why synthesise qualitative research?
“…the full contribution of qualitative research will not be realised if individual studies merely accumulate and some
kind of synthesis is not carried out…there are generalisations to be made across qualitative research
studies that do not supplant the detailed findings of individual studies, but add to them”
Britten et al. (2002)
(9)
Deriving Deriving descriptivedescriptive themesthemes
(10)
‘Mixed method’ systematic reviews (1/2)
• Policy and practice concerns often precede, or go beyond, questions of effectiveness
• Different types of questions likely to be answered by different types of findings
• Different types of findings may require different types of synthesis
(11)
‘Mixed method’ systematic reviews (1/2)
Single reviews with three syntheses
1) Effect sizes from trials pooled using statistical meta-analysis
2) Findings from qualitative studies synthesised using thematic analysis
3) Synthesis 2) used to interrogate synthesis 1)
(12)
An example of a ‘mixed method’ review
Children and healthy eating: a systematic review of barriers and facilitators*
*Thomas J, Sutcliffe K, Harden A, Oakley A, Oliver S, Rees R, Brunton G, Kavanagh J (2003a) Children and Healthy Eating: A systematic review of
barriers and facilitators. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London (The full report of this review is
available at the EPPI-Centre website http://eppi.ioe.ac.uk/EPPIWeb/home.aspx)
:.
(13)
REVIEW PROCESS
Searching, screening and mapping
Synthesis 1: Trials (n=33)1. Quality assessment
2. Data extraction3. Statistical meta-analysis
Synthesis 2: Qualitative studies (n=8)
1. Quality assessment2. Data extraction
3. Thematic synthesis
Synthesis 3: Trials and qualitative studies
Focus narrowed to ‘fruit &veg’
(14)
Synthesis 1: Statistical meta-analysis
-0.5
0
0.5
1
1.5
2
2.5
Fruit and vegetables
Fruit only
Vegetables only
(15)
Line-by-line coding(Synthesis 2)
(16)
1) Children don’t see it as their role to be interested in health.
2) Children do not see future health consequences as personally relevant or credible.
3) Fruit, vegetables and confectionary have very different meanings for children.
4) Children actively seek ways to exercise their own choices with regard to foods.
5) Children value eating as a social occasion.
6) Children recognise contradiction between what is promoted and what is provided.
Children consider taste, not health, to be a key influence on their food choice
Food labelled as healthy may lead children to reject them (‘I don’t like it so it must be healthy’)
Buying healthy foods not seen as a legitimate use of their pocket money
Synthesis 2: Thematic analysis
(17)
Brand fruit and vegetables as ‘tasty’ rather than ‘healthy’.
Reduce health emphasis of messages
Do not promote fruit and vegetables in the same way within the same intervention.
Create situations for children to have ownership over their food choices.
Ensure messages promoting fruit and vegetables are supported by appropriate access to fruit and vegetables
1) Children don’t see it as their role to be interested in health.
2) Children do not see future health consequences as personally relevant or credible.
3) Fruit, vegetables and confectionary have very different meanings for children.
4) Children actively seek ways to exercise their own choices with regard to foods.
5) Children value eating as a social occasion.
6) Children recognise contradiction between what is promoted and what is provided.
Preparing for synthesis 3
(18)
Children’s views Trials
Recommendation for interventions
Good quality Other
Do not promote fruit and vegetables in the same way 0 0
Brand fruit and vegetables as an ‘exciting’ or child-relevant product, as well as a ‘tasty’ one
5 5
Reduce health emphasis in messages to promote fruit and vegetables particularly those which concern future health
5 6
Synthesis 3: Across studies
(19)
Increase (standardised portions per day) in vegetable intake across trials
0
0.1
0.2
0.3
0.4
0.5
0.6
War
dle
Liquo
ri
Henry
Ander
son
Reyno
lds Auld
Auld (b
)
Baran
owsk
i
Perry
Study
Po
rtio
ns
Little or no emphasis on health messages
Synthesis 3: Across studies
(20)
‘Added value’ of mixing methods
• Operationally, the method is simple, but conceptually, it is strong
• Integrates ‘quantitative’ estimates of benefit and harm with ‘qualitative’ understanding from people’s lives
• Facilitates a critical analysis of intervention studies from the point of view of those targeted by interventions - and vice versa
• Preserves the integrity of the findings of the different types of studies
(21)
James ThomasEPPI-Centre
SSRU18 Woburn Square
London, WC1H 0NR
Email: [email protected]
The methods described here can be found in: Thomas et al (2004) Integrating qualitative research with trials
in systematic reviews. British Medical Journal 328:1010-1012
For full details of the systematic review discussed in this paper and other EPPI-Centre reviews please see
our website:http://eppi.ioe.ac.uk/
Top Related