The Qualitative Assessment and Review Instrument (QARI)
Slide 2
PART 3: META-AGGREGATION Its history Its philosophy A worked
example
Slide 3
Meta-aggregation A structured, and process driven approach to
systematic review drawing on the classical understandings and
methods associated with systematic review of quantitative
literature as practiced by the Cochrane and Campbell Collaboration
Based on an a-priori protocol Established, answerable question
Explicit criteria for inclusion Documented review methods for
searching, appraisal, extraction and synthesis of data
Slide 4
Slide 5
Meta-aggregation: History Participative consensus project
involving Australian academics The outcome = an aggregative
approach to the synthesis of qualitative evidence that: emphasised
the complexity of interpretive and critical understandings of
phenomena; recognised the need to ensure that the process is
practical and usable; balanced utility of the outcomes with the
complexity of the material; was grounded in pragmatism.
Slide 6
Meta-aggregation: A pragmatic approach The philosophy of
pragmatism - Charles Peirce, William James, John Dewey the value of
any thought lies in its practical use and consequences. It is the
focus on practical consequences that characterizes meta-aggregation
as a synthesis approach communities of inquiry Central to
pragmatism were communities of inquiry or groups of people with a
shared interest, problem or issue trying to resolve doubt through
critical reasoning. Actions are assessed in the light of their
practical consequences. Aggregation concentrates on the original
researchers findings (processed data) and summarizes common and
competing findings to produce cross- study generalizations that
lead to recommendations for action The utility of an outcome is
important. Research articles
Slide 7
Meta-aggregation: a worked example Review Topic: Obstacles to
the implementation of Evidence-Based Practice in Belgium: a
context-specific study Question formulation: Setting: Belgium
(Belgian health care system) Population: Health Care Practitioners
(Phenomena of) Interest: Obstacles toward the implementation of
Evidence-Based Practice Comparison: (additional literature search
on strategies developed by countries with other systems)
Evaluation: Experiences and perceptions
Slide 8
Meta-aggregation: a worked example Search strategy: Major
databases: Medline, CINAHL, Psychinfo, Embase, Social Sciences
abstracts and ERIC (1990-May 2008). Where possible we used a
methodological filter for qualitative research. Other information
sources: Federal research actions-database from the Belgian
governmental department of science Consultation of Belgian experts
in qualitative research methods and/or Evidence-Based Practice to
check on any other published material that could be of use for the
synthesis. Screening of references Keywords: Evidence-Based
Practice (Medical Subject Heading (MeSH) term), including
Evidence-Based Dentistry, Nursing and Medicine. Additional
keywords: evidence-based combined with the geographical notion
Belgian, Flemish/Flanders or Walloon.
Slide 9
Meta-aggregation: a worked example Inclusion criteria: Study
type: Qualitative, empirical research papers. Opinion pieces and
descriptive articles were excluded. Study participants: All types
of health care practitioners e.g. physicians, dentists, nurses,
physiotherapists, psychologists Topic of interest: Obstacles to the
implementation of EBP. Context: the Belgian health care system.
Outcome of interest: Experiences and/or perceptions from
participants. 8 studies met the inclusion criteria
Slide 10
Meta-aggregation: a worked example Critical appraisal of
selected studies: No criteria based quality appraisal, although the
software QARI provides a standard critical appraisal checklist.
limited amount of studies found the majority of the selected
studies were written by the lead reviewer, who would also be one of
the appraisers. An overall judgment approach was used instead. This
approach has been proven to deliver the same outcome (Dixon-Woods,
2007). However, it tends to be less explicit about potential
reasons for exclusion. An evaluation based on the JBI-critical
appraisal instrument is recommended for other review teams opting
for a meta-aggregative approach to synthesis.
Slide 11
85 different findings were found The findings were classified
in 9 major categories: (1) evidence is hard to implement (2)
decision making processes are influenced by practitioner variables
(3) decision-making processes are influenced by practitioner
variables (4) commercial/financial interests affect EBP (5)
governmental regulations influence the process of implementation,
(6) EBP is more feasible for practitioners working in a scientific
setting compared to those working in other settings (7) physicians
display of power hampers other professionals to put evidence-based
practice into effect (8) a lack of knowledge and skills hinders the
implementation of evidence- based practice (9) attitudes of health
care practitioners hinder the implementation of EBP.
Meta-aggregation: a worked example
Slide 12
These categories were further analysed to produce 4 synthesized
statements Synthesis 1: Evidence might have a limited role in
decision-making processes in daily practice, if the importance of
the scientific component is not stressed (categories 1, 2 and 3).
Synthesis 2: Aspects other than quality of care will steer the EBP
agenda, if governmental regulations and economic interests are
contra-productive for delivering the best possible care (categories
4 and 5). Synthesis 3: Although EBP is intended to serve all
practitioners, some health care providers will benefit less from
EBP than others, if inequity issues between practitioners are not
solved and support for field workers is not established (categories
6 and 7). Synthesis 4: A lack of competences will hinder the
implementation of EBP, if gaps in knowledge and skills are not
being filled and efforts to change contra- productive attitudes are
not undertaken (categories 8 and 9). Meta-aggregation: a worked
example
Slide 13
Slide 14
Implications for practice and policy Consider educating
patients and exploring potential information channels that
influence patients opinions to create space for well-informed
decisions (synthesis 1). Provide easy and free access to
well-structured, compact and relevant information targeted to a
particular discipline and consider helpdesks. Screen information,
control its quality and translate it to the field (synthesis 1).
Consider updating the Belgian nomenclature and reimbursement system
to bring it in line with the latest evidence (synthesis 2).
Consider incentives for those practitioners who are engaging
themselves for the implementation of EBP in daily practice, to keep
them motivated and to prevent from creating a negative spiral in
which practitioners tend to believe that efforts to improve
practice automatically lead to a loss of income (synthesis 2).
Meta-aggregation: a worked example
Slide 15
Implications for practice and policy Enhance communication and
cooperation between physicians and other partners in health care,
via their professional groups and journals (synthesis 3) Consider
programs tailored to the needs of specific disciplines and
stimulate multidisciplinary education should be stimulated to
create mutual understanding across disciplines (synthesis 3, 4).
Consider direct access to allied health services to increase
autonomy (synthesis 3) Integrate EBP in the basic curricula
(synthesis 4). Meta-aggregation: a worked example
Slide 16
The Qualitative Assessment and Review Instrument (QARI)
Designed to provide a systematic process mirroring that taken for
systematic reviews of quantitative research, whilst being sensitive
to the nature of qualitative data.
Slide 17
Slide 18
Starting a synthesis project (or review of qualitative
evidence)
Slide 19
Slide 20
Slide 21
Slide 22
Slide 23
Critical Appraisal of Evidence arising out of qualitative
research
Slide 24
Slide 25
Slide 26
Data Extraction Reduce The findings of many studies into a
single document Summarise Methods Phenomena Findings
Slide 27
Slide 28
The units of extraction in this process are specific findings
and illustrations from the text that demonstrate the origins of the
findings. In QARI a finding is defined as: A conclusion reached by
the researcher(s) and often presented as themes or metaphors.
Slide 29
Levels of credibility Unequivocal - relates to evidence beyond
reasonable doubt which may include findings that are matter of
fact, directly reported/observed and not open to challenge Credible
- those that are, albeit interpretations, plausible in light of
data and theoretical framework. They can be logically inferred from
the data. Because the findings are interpretive they can be
challenged. Not Supported - when 1 nor 2 apply and when most
notably findings are not supported by the data
Slide 30
Meta-Synthesis The aim of meta-synthesis is to assemble
findings; categorise these findings into groups on the basis of
similarity in meaning; and to aggregate these to generate a set of
statements that adequately represent that aggregation. These
statements are referred to as synthesised findings - and they can
be used as a basis for evidence based practice.
Slide 31
Slide 32
Slide 33
Slide 34
Slide 35
Slide 36
In QARI, a synthesised finding is defined as an overarching
description of a group of categorised findings that allow for the
generation of recommendations for practice.
Slide 37
Slide 38
Can be stated propositionally as if-then statements - for
example: If students are advised reconsider their choice of study,
their relatives will sometimes feel as if they are not involved. (a
somewhat awkward and eccentric form) We prefer the declamatory form
that emphasizes the probability of the claim: Relatives of students
that are advised to reconsider their choice of study may feel as if
they are not involved if strategies to include them are not pursued
.