ESN Presentation 050214

Quality Appraisal of Qualitative Studies

Alison CookeMidwife &

NIHR Doctoral Research Fellow

Why appraise quality?

• A systematic review is a scientific exercise which may influence healthcare provision

• The quality of a review will be influenced by the methods used to minimise error and bias

• There is a plethora of published work on how to appraise quality of research

• Whilst there are some differences in how to appraise, there is a high level of agreement on the key components of quality appraisal

What is metasynthesis?

“A systematic review of qualitative research”(Booth 2001)

“Seeks to develop and refine theories while retaining the uniqueness of individual studies”

(Jensen 2004)

“The science of summing up” (Light & Pillemer

1984)

Types of qualitative synthesis• Meta-ethnography (Noblit & Hare)• Grounded Theory (Kearney / Eaves / Finfgeld)• Thematic Synthesis (Thomas & Harden)• Textual Narrative Synthesis (Lucas et al.)• Meta-study (Paterson et al.)• Meta-narrative (Greenhalgh et al.)• Critical Interpretive Synthesis (Dixon-Woods et al.)• Ecological Triangulation / Ecological Sentence Synthesis

(Banning)• Framework Synthesis (Brunton et al. / Oliver et al.)

Noblit & Hare • Meta-ethnographic approach: to reveal

analogies between studies, making sense of what the collection of studies is saying

• Line-of-argument synthesis; the interpretive synthesis is concerned with inference – what can we say of the whole phenomenon based on selective studies of the parts?

• Repeated comparison between studies reveal similarities and differences put findings into new interpretive context

Quality Appraisal

• Best tool for assessment is often defined by type of metasynthesis being conducted, e.g. specific 10-point tool for Framework synthesisDIAD tool for ecological triangulation JBI specific 10-point checklist

• Over 100 tools and frameworks are available Are rigid checklists appropriate?Role of expert judgement?Insufficient evidence to inform judgement on rigour or

added value of various approaches (Noyes et al., 2008)

Which tool to use?

Rigid yes/no

checklist

Expert review

Advantages/DisadvantagesRigid checklist• Too restrictive for qualitative appraisal• More suited to quantitative appraisal; more objective• Validated tools; reader aware of robustness of quality

assessment process

Expert review• Experienced reviewer reads the paper as a whole and

provides assessment of quality• Not credible; reader not able to assess robustness of

quality assessment process• Highly subjective

Optimum tool

• Somewhere along the continuum between ‘rigid checklist’ and ‘expert opinion’

• Best ‘fits’ the type of metasynthesis being conducted

• Validated, or has been used many times in the topic area

• Provides an organised and systematic approach which can be replicated between reviewers, but also by the reader

Worked example

• Quality assessment tool developed by Walsh & Downe; 2005

• Grading tool developed by Downe et al. 2009, based on work by Lincoln & Guba, 1985.

Stages Essential Criteria Notes

Scope and Purpose Clear statement of, and rationale for, research questions/aims/purposes

Study thoroughly contextualised by existing literature

Design Method/design apparent, and consistent with research intent

Data collection strategy apparent and appropriate

Sampling Strategy Sample and sampling method appropriate

Analysis Analytic approach appropriate

Interpretation Context described and taken account of in interpretation

Clear audit trail given

Data used to support interpretation

Reflexivity Researcher reflexivity demonstrated

Ethical Dimensions Demonstration of sensitivity to ethical concerns

Relevance and Transferability

Relevance and transferability evident

A No, or few flaws. The study credibility, transferability, dependability and confirmability is high.

B Some flaws, unlikely to affect the credibility, transferability, dependability and/or confirmability of the study.

C Some flaws that may affect the credibility, transferability, dependability and/or confirmability of the study.

D Significant flaws that are very likely to affect the credibility, transferability, dependability and/or confirmability of the study.

Grading Systemcreated by Downe et al based on the work by Lincoln & Guba

Activity

• Read the extract from Dobrzykowski et al., 2003

• Look at the explanatory notes for use of relevant sections of assessment tool

• Complete your quality assessment of this extract

Completed assessment: 1Design Method/design

apparent, and consistent with research intent 

Grounded theory: rationale provided.Discussion of rationale for design choice.Setting appears appropriate.

Face to face interview, telephone conversation, email (first author).Tape recorded and transcribed verbatim.2-3 interviews; appears sufficient to capture complexity/diversity.

Data collection strategy apparent and appropriate 

Sampling Strategy

Sample and sampling method appropriate 

n=53Data and emergent categories dictated.Sample selection process, justification explained.Thickness of description likely.

Completed assessment: 2Design Method/design

apparent, and consistent with research intent 

Clear rationale given for the qualitative approach, used grounded theory in order to explore processes and symbolic meanings for study group.

In-depth interviews in person, by telephone and follow up interview by email.

Data collection strategy apparent and appropriate 

Sampling Strategy

Sample and sampling method appropriate 

Yes, fully explained.

Consensus

• All 3 reviewers gave a grade A/B for this study(Very few flaws. The study credibility, transferability, dependability and confirmability is high)

• Important to have a meeting following independent quality assessment to reach consensus over grading and an agreed decision over inclusion and exclusion of studies for synthesis

Flaws to determine grading

• “No evidence of systematic approach to literature review”

• “Reflexivity unclear: no discussion of researchers influence on participants etc.”

• “Limitations of study not discussed”• “No evidence of ethical approval”• “No evidence of member checking”• “Unlikely to capture thickness of data”• “?Bias: only one researcher involved in data collection

and analysis process”

Should studies be excluded?

Grade D = “Significant flaws that are very likely to affect the credibility, transferability,

dependability and/or confirmability of the study”

Should we exclude in a metasynthesis any studies found in answer to a research

question?

Argument to include all?• Edwards et al., 1998Advocate balancing an assessment of methodological quality against the weight of its message – the ‘signal to noise ratio’, rather than excluding studies that fall below a certain quality threshold• Booth, 2001If research is rejected on the basis of design alone there is a high risk of denying valuable insights that contribute to our interpretation of a phenomenon

Challenges of qualitative evidence synthesis

• Literature searching: poor indexing of qualitative studies in databases (use SPIDER! Cooke et al. 2012)

• Journal word count may mean that points appraised may be badly explained or not included

• Appraisal techniques are dominated by the quantitative paradigm

• Structured approaches may not provide greater consistency of judgements to include or exclude

• Subjective judgement• Interpretive, rather than aggregating, intent• Time consuming

In summary• There is no ‘right’ way of

conducting quality assessment of qualitative research

• Having a structured approach can provide the reader with clarity of the decision process used

• Multiple reviewers required to review papers independently, then meeting to reach a consensus of decision to include/exclude

Thank you

Any questions?

Alison.Cooke@manchester.ac.uk

Reference List Banning J. undated. Design and Implementation Assessment Device (DIAD) Version 0.3: A response from a

qualitative perspective [Website] Available from: http://mycahs.colostate.edu/James.H.Banning/PDFs/design%20and%20implementation%20assessment%20device.pdf [Accessed: 03/02/14]

Banning J. undated. Ecological Triangulation [Website] Available from: http://www.mychhs.colostate.edu/James.H.Banning/PDFs/Ecological%20Triangualtion.pdf [Accessed: 03/02/14]

Banning J. undated. Ecological Sentence Synthesis [Website] Available from: http://www.mychhs.colostate.edu/James.H.Banning/PDFs/Ecological%20Sentence%20Synthesis.pdf [Accessed: 03/02/14]

Booth A. 2001. Cochrane or cock-eyed? How should we conduct systematic reviews of qualitative research? Qualitative Evidence-based Practice Conference, Coventry University, May 14-16

Brunton G, Oliver S, et al. 2006. A synthesis of Research Addressing Children’s, Young People’s and Parents’ Views of Walking and Cycling for Transport EPPI-Centre, London

Cooke A, Smith D & Booth A. 2012. Beyond PICO: The SPIDER Tool for Qualitative Evidence Synthesis. Qualitative Health Research 22(10) p1435-43

Dixon-Woods M, Cavers D, et al. 2006. Conducting a critical interpretive synthesis of the literature on access to healthcare by vulnerable groups. MBC Medical Research Methods 6(35)

Downe S Finlayson K et al. 2009. ‘Weighing up and balancing out’: a meta-synthesis of barriers to antenatal care for marginalised women in high-income countries BJOG 116 p518-529

Eaves YD. 2001. A synthesis technique for grounded theory data analysis. Journal of Advanced Nursing 35 p654-663

Edwards AG, Russell LT & Stott NCH. 1998. Signal versus noise in the evidence base for medicine: an alternative to hierarchies of evidence? Family Practice 15(4) p319-322

Reference List Finfgeld D. 1999. Courage as a process of pushing beyond the struggle. Qualitative Health Research 9 p803-

814 Greenhalgh T, Robert G, et al. 2005. Storylines of research in diffusion of innovation: a meta-narrative

approach to systematic review. Social Science & Medicine 61 p417-430 Jensen LA. 2004. Extending meta-analysis. Qualitative Health Research 14(10) p1346-1347 Kearney MH. 2001. Enduring love: a grounded formal theory of women’s experience of domestic violence.

Research in Nursing & Health 24 p270-282 Light RJ & Pillemer DB. 1984. Summing up: the science of reviewing research. Havard University Press,

Cambridge, MA Lincoln YS & Guba EG. 1985. Naturalistic Inquiry. Sage Publications, California Lucas PJ, Arai L, et al. 2007. Worked examples of alternative methods for the synthesis of qualitative and

quantitative research in systematic reviews. BMC Medical Research Methods 7(4) Noblit GW & Hare RD. 1988. Meta-Ethnography: Synthesizing Qualitative Studies (Qualitative Research

Methods Series 11). Sage Publications, California Noyes J, Popay J, et al. 2008. Chapter 20: Qualitative research and Cochrane reviews. In: Higgins JPT &

Green S (Eds) Cochrane Handbook for Systematic Reviews of Interventions. John Wiley & Sons, Chichester Oliver S, Rees R, et al. 2008. A multidimensional conceptual framework for analysing public involvement in

health services research. Health Expectations 11 p72-84 Paterson BL, Thorne SE, et al. 2001. Meta-study of Qualitative Health Research. A Practical Guide to Meta-

Analysis and Meta-Synthesis. Sage Publications, California Thomas J & Harden A. 2008. Methods for the thematic analysis of qualitative research in systematic

reviews. BMC Medical Research Methods 8 p45 Walsh D & Downe S. 2006. Appraising the quality of qualitative research. Midwifery 22 p108-19