Post on 24-Mar-2016
description
(1)
Systematic Reviews: The Potential of
Different Methods Steve HigginsHenry Potts
James ThomasGeoff Wong
ESRC Research Methods FestivalOxford, 5 July 2012
(2)
Introduction
• Aims– To outline major approaches to systematic
reviewing and research synthesis– To demystify what on first glance looks like a
bewildering set of overlapping approaches– To examine some of the assumptions that
underpin these approaches– To discuss how these approaches might help in
my (your!) situation
(3)
Introduction
• Session structure:– 20 Minute presentation + 10 minutes discussion
/ clarification of four approaches to synthesis– Coffee break after the 3rd / 4th presentation– Discussion in pairs– Questions + panel discussion
(4)
What is a systematic review?
• Systematic: ‘done or acting according to a fixed plan or system; methodical’
• Review: ‘a critical appraisal of a book, play, or other work’(OED)
(5)
What is systematic research synthesis?
• Synthesis: ‘The process or result of building up separate elements, especially ideas, into a connected whole, especially a theory or system’ (OED)
• Not just a report of the findings of the individual studies in a review
• Involves a transformation of the data from primary studies in some way
• Synthesis of the findings of all the included studies in order to answer the review question
(6)
Aims of synthesis reviews
• Answer questions: what do we want to know and how can we know it?
• Bring together and ‘pool’ the findings of primary research (i.e. clarify what we know)
• Any question so potentially any type of evidence
• Driven by review users to answer relevant questions in relevant ways
(7)
Introduction
• We will look at four main approaches to reviewing:– Meta-analysis– The synthesis of ‘qualitative’ research– Realist review– Meta narrative review
• Each have different disciplinary backgrounds and underlying assumptions
• They all bring together research using two main logics -
(8)
Aggregation in reviews
Aggregation refers to ‘adding up’ (aggregating) findings from primary studies to answer a review question…
… to indicate the direction or size of effect… and our degree of confidence in that finding
(9)
Configuration in reviews
Configuration involves the arrangement (configuration) of the findings of primary studies to answer the review question….
… to offer a meaningful picture of what research is telling us... across a potentially wide area of research
•
(10)
Most reviews contain elements of both, but some patterns are common
Aggregation Configuration
Approach to theory Test Generate
Research question Closed Open
Concepts Secure prior to review Emergent in review
Timing of methods decisions
A priori Iterative / emergent
Search for studies Representative / unbiased / exhaustive
Sufficient / purposive
Quality appraisal Avoiding bias Richness of data
Explore
(11)
The EPPI-Centre is part of the Social Science Research Unit at the Institute of Education, University of London, UK1
Methods for the synthesis of qualitative
researchJames Thomas1
ESRC Research Methods FestivalOxford, 5 July 2012
(12)
Synthesis of qualitative research: outline
• What can qualitative research tell us?
• Current developments/approaches in qualitative review methods
• Examples of syntheses of qualitative
research
(13)
What can qualitative research tell us?
(14)
Qualitative research…
• aims to provide an in-depth understanding of people’s experiences, perspectives and histories in the context of their personal circumstances or settings…*
• with the use of unstructured methods which are sensitive to the social context of the study*
• the capture of data which are detailed, rich and complex…*
• is generalisable on the basis of theory not statistical probability **
* Spencer et al (2003); ** Popay (2006)
(15)
Methods used include*…
• exploratory interviews• focus groups• observation• conversation• discourse and narrative analysis• documentary and video analysis
*Spencer et al (2003)
(16)
Is it simply the analysis of textual data?
“Increasingly, the terms ‘quantitative research’ and ‘qualitative research’ came to signify much more than ways of gathering data; they came to denote divergent assumptions about the nature and purposes of research in the social sciences”
Bryman (1988) Quantity and Quality in Social Research. London: Unwin Hyman. p.3.
(17)
Qualitative research can*
• help to define policy questions• look in detail at how a programme or trial
was actually implemented• help to determine appropriate outcome
measures by looking at ‘subjective’ outcomes
• help to clarify what counts as effective or successful
* Davies (2000) ‘Contributions from qualitative research’ in Davies et al (eds) What works? Evidence-based policy and practice in public services. Bristol: Policy Press. Cited in *Spencer et al (2003). p.36
(18)
…continued
• identify and explore unintended consequences• contribute to service delivery and policy
development by describing processes and contexts
• inform and illuminate quantitative studies, e.g. by contributing to the design of structured instruments, assessing the fairness of comparisons in experimental studies, or unpacking variation within aggregated data
Davies (2000) ‘Contributions from qualitative research’ in Davies et al (eds) What works? Evidence-based policy and practice in public services. Bristol: Policy Press. Cited in *Spencer et al (2003). p.36
(19)
Current developments in synthesising qualitative research
• Growing recognition of the value of qualitative research to inform policy & practice
• Methods developing to facilitate this
(20) Try: http://www.wordle.net/
(21)
Methodological study
• We searched for methodological papers concerning the synthesis of concepts or theories– Purposive, rather than systematic search– Reference ‘chasing’– Google scholar search– Handsearching key journals
• 203 papers retrieved• 9 distinct methods for synthesis
(22)
Methods for the synthesis of qualitative research identified
• Meta narrative synthesis• critical interpretive synthesis• meta-study• meta-ethnography• grounded theory• thematic synthesis• framework synthesis• ecological triangulation
(23)
Many methods: similar or different?
• Examined the methods across different dimensions:– Epistemology– Approach to quality assessment– Attitudes towards problematizing the literature– Use of review question– How similar / different the included studies were– Characteristics of the synthetic product
• Found they fell into two broad camps: ‘idealist’ and ‘realist’
(24)
‘Realist’ approaches
Purpose• To answer a policy relevant questionMethods• Qualitative/ quantitative data analysed with
qualitative/ quantitative methods• Searching linear or iterative• Quality assessment of study methodsProduct • Directly applicable to policy and practice
decisions
(25)
‘Idealist’ approaches
Purpose• To explore and construct concepts from the data• For generating theoryMethods• Qualitative data analysed with qualitative methods• Searching iterative• Qualitative assessment of study content > methodProduct • Complex, requiring further interpretation before being
used for policy or practice
(26)
Conclusions
• Operationally, many methods are very similar
• Underlying principles differ• Product differs in terms of the amount of
additional interpretation required. This may reflect…
• (Barnett-Page and Thomas (2009))
(27)
Perspectives on how evidence informs the policy process
Idealist Realist
Knowledge-driven model; Problem-solving (policy driven) model*
(either researcher disseminates research or policy-maker commissions / requests specific research)
Less suited More suited
Interactive model; Enlightenment model*
(ongoing interactions between researcher and policymaker; ‘gradual sedimentation of insights…’)
More suited Less suited
*Weiss (1979)
(28)
Two examples
• All this is a bit abstract… what do systematic reviews of qualitative research look like?
• Two examples:– The ‘fine grain’ of a synthesis– ‘Mixed methods’ synthesis
(29)
Example 1: Informing policy about the ‘obesity epidemic’
What are children’s views about:
• the meanings of obesity or body size, shape or weight (including their perceptions of their own body size)?
• influences on body size?• changes that may help
them to achieve or maintain a healthy weight?
Views = attitudes, opinions, beliefs, understanding or experiencesAs distinct from health/weight status, behaviour, factual knowledge
Rees et al 2009
(30)
‘Thematic synthesis’
• Similar to other methods of synthesising qualitative research (e.g. ‘meta-ethnography’)
• Source data = text (documents)• Source material = conceptual• Key method = translation• Final product = interpretation
• (Thomas and Harden 2008)
(31)
Stages of thematic synthesis
• Stages one and two: coding text and developing descriptive themes– Identifying the ‘findings’– Line-by-line coding– Developing descriptive themes
• Stage three: generating analytical themes– In the light of the review question
(32)
Screenshot – line-by-line coding
(33)
Screenshot – descriptive codes diagram
(34)
Example 2. adding insights to other existing reviews
• How might a systematic review of qualitative research support or add insight to a meta-analysis?
• Product of qualitative synthesis can form the conceptual framework within which heterogeneity can be explored in a meta-analysis
Searching, screening mapping and user involvement
e.g. Statistical meta-analysis of trials a) Data extraction
b) Quality assessmentc) Effect sizes pooledd) Narrative synthesis
Addresses sub RQ e.g. ‘which interventions are effective?’
e.g. Thematic synthesis of qualitative studies a) Data extraction
b) Quality assessmentc) Descriptive themesd) Analytical themes
Addresses sub RQ e.g. ‘what are people’s perspectives and experiences ?
Synthesis 1 Synthesis 2
Synthesis 3Driven by overall review question
Integration of separate syntheses e.g. a) Matches, mis-matches and gaps
b) Hypotheses generated in synthesis 2 tested amongst trials in synthesis 1
Review Questione.g. ‘What is known about the barriers to, and facilitators of, outcome X (e.g. physical activity)
amongst population A (e.g. young people
(36)
Cross study synthesis: FindingsChildren’s views Outcome evaluations
Recommendation for interventions
Good quality Other
Do not promote fruit and vegetables in the same way
No soundly evaluated
interventions
No other interventions
identified
Brand fruit and vegetables as an ‘exciting’ or child-relevant product, as well as a ‘tasty’ one
5 soundly evaluated
interventions identified
5 other interventions
Reduce health emphasis in messages to promote fruit and vegetables particularly those which concern future health
5 soundly evaluated
interventions identified
6 other interventions identified
(37)
Cross study synthesis: an example of sub-group analysisIncrease (standardised portions per day) in vegetable intake across trials
0
0.1
0.2
0.3
0.4
0.5
0.6
Wardle
Liquo
ri
Henry
Anders
on
Reyno
lds Auld
Auld (b
)
Barano
wski
Perry
Study
Port
ions
Little or no emphasis on health messages
(38)
This method of synthesis across study types:
• preserves the integrity of the findings of the different types of studies
• allows us to integrate ‘quantitative’ estimates of benefit and harm with ‘qualitative’ understanding from people’s lives
• allows the exploration of heterogeneity in ways in which it would be difficult to imagine in advance– protects against ‘data dredging’
(39)
Assumptions / theoretical models
• Mixed methods synthesis can draw on thinking relating to mixed methods in primary research
• In terms of a frequently used taxonomy of paradigm stances (Creswell 2011) e.g.– Incommensurability (cannot be mixed)– A-paradigmatic (can be mixed and matched in different ways)– Complementary strengths (not incompatible, but are different
and should be kept separate)– Dialectic (paradigms are important in different ways leading to
useful tensions & insights)– Alternative paradigm (‘mixed methods’ paradigm; foundation in
e.g. pragmatism)
(40)
Conclusions
• Qualitative research offers valuable insights that can inform policymaking
• Methods for synthesising qualitative research are still being developed
• Selection of a particular method depends on the type of answer required (and the means by which the review will inform policy)
(41)
References
• Barnett-Page E, Thomas J (2009) Methods for the synthesis of qualitative research: a critical review. BMC Medical Research Methodology, 9:59. doi:10.1186/1471-2288-9-59. (http://www.biomedcentral.com/1471-2288/9/59)
• Creswell J (2011) Mapping the Developing Landscape of Mixed Methods Research in Teddlie C, Tashakkori A: SAGE Handbook of Mixed Methods in Social & Behavioral Research. New York: Sage.
• Gough D, Oliver S, Thomas J (2012) An Introduction to Systematic Reviews. London: Sage
• Thomas J, Harden A (2008) Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Medical Research Methodology, 8:45 doi:10.1186/1471-2288-8-45 (http://www.biomedcentral.com/1471-2288/8/45)
• Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R, Brunton G, Kavanagh J (2004) Integrating qualitative research with trials in systematic reviews: an example from public health. British Medical Journal 328: 1010-1012. (http://www.bmj.com/cgi/content/full/328/7446/1010)
(42)
Systematic Reviews: The Potential of
Different Methods Steve HigginsHenry Potts
James ThomasGeoff Wong
ESRC Research Methods FestivalOxford, 5 July 2012
(43)
Summary
Heterogeneity Aggregation / configuration
Meta-analysis A problem in fixed effectAllowance made for in random effectsFor ‘exploration’
Fixed & random effects models are pure aggregationSub-group & meta-regression both aggregate & configure
Synthesis of qualitative research
Expected. Key challenge is judging when contexts / concepts are too dissimilar to be compared.
Similar concepts are ‘aggregated’; most synthesis involves configuration
Realist review Necessary to obtain different contexts etc
Aggregates within CMO configurations; configures between them
Meta narrative review Necessary to obtain diversity in paradigms
Aggregation within paradigms; configuration between them
(44)
Your turn
Research question
Heterogen-eity?
Meta-analysis
Qualitative synthesis
Realist review
Meta narrative review
Research question 1
Research question 2