Www.kmin-vumc.nl Grading the evidence in systematic reviews of measurement properties 23 september...
-
Upload
shavonne-summers -
Category
Documents
-
view
218 -
download
0
Transcript of Www.kmin-vumc.nl Grading the evidence in systematic reviews of measurement properties 23 september...
www.kmin-vumc.nl Grading the evidence in systematic reviews of
measurement properties
23 september 2010
Caroline Terwee
Knowledgecenter Measurement InstrumentsDepartment of Epidemiology and Biostatistics
VU University Medical Center
www.kmin-vumc.nl What is a systematic review of measurement properties?
A review of the measurement properties of all available measurement instruments that aim to measure a particular construct in a particular population.
Aim is to select the best instrument for a specific purpose
www.kmin-vumc.nl What is special about a systematic review of measurement properties (SR-MP)?
• A SR-MP has more than one outcome measure, i.e. multiple measurement properties
• Different studies evaluate different measurement properties the number of studies in the analysis is different per measurement property
• The quality of the studies is evaluated per measurement property
• Data synthesis is different per measurement property
• Evidence for one measurement property may come from different studies
Therefore, a SR-MP is actually a collection of separate reviews per measurement property
www.kmin-vumc.nl Methodology of systematic reviews of measurement properties
1. formulating a research question
2. performing a literature search
3. formulating eligibility criteria
4. selecting abstracts and full-text articles
5. evaluating the methodological quality of the included studies
6. data extraction
7. content comparison
8. data synthesis - evaluating of quality of the instrument
9. overall conclusion of the systematic review
10. reporting a systematic review of measurement properties
10 steps
www.kmin-vumc.nl Methodology of systematic reviews of measurement properties
1. formulating a research question
2. performing a literature search
3. formulating eligibility criteria
4. selecting abstracts and full-text articles
5. evaluating the methodological quality of the included studies
6. data extraction
7. content comparison
8. data synthesis - evaluating of quality of the instruments
9. overall conclusion of the systematic review
10. reporting a systematic review of measurement properties
10 steps
www.kmin-vumc.nl Data synthesis
2 steps
1. Decide on combining studies
• Homogeneity of study characteristics study population, setting, (language) version of the instrument, mode of administration, design characterstics (time interval)
• Methodological quality• Consistency of the results of the measurement properties
www.kmin-vumc.nl
2 steps
1. Decide on combining studies
• Homogeneity of study characteristics study population, setting, (language) version of the instrument, mode of administration, design characterstics (time interval)
• Methodological quality• Consistency of the results of the measurement properties
2. Decide on the analysis:
• Quantitative analysis statistical pooling• Qualitative analysis best evidence synthesis
Data synthesis
www.kmin-vumc.nl ExampleReliability of the Quebec Pain Disability Scale (QBPDS)
Nine studies evaluated reliability; 8 included in the analysis
1. Kopec et al. J Clin Epidemiol 1996
2. Schoppink et al. Phys Ther 1996
3. Fritz et al. Phys Ther 2001
4. Davidson et al. Phys Ther 2002
5. Mousavi et al. Spine 2006
6. De Beer et al. S Afr J Physiother 2008
7. Melikoglu et al. Spine 2009
8. Hicks et al. Pain Med 2009
www.kmin-vumc.nl
1 2 3 4 5 6 7 8
Age 40 (10) 44 (12) 79 (5) 40 (12) 42 (9) 55 (17) 37(10) 45 (15)
% Male 70 32 29 45 5 64 48 26
Duration complaints (mo)
12 (0-128)
85%
>12
7 (6) 84 (108)
? 50% >6 <1 51 (50)
Setting General practice
Retirement
community
Orthop/ trauma clinic
Hospital - PT
Hospital Hospital + PT
practice + com.
serv
PT outpatient clinic
Hospital - PT
Country NL Brazil US Iran South Afrika
Australia
US Turkey
Language Dutch Portugese
English Persian Tswana English English Turkish
Study population, setting, country
www.kmin-vumc.nl
1 2 3 4 5 6 7 8
Age 40 (10) 44 (12) 79 (5) 40 (12) 42 (9) 55 (17) 37(10) 45 (15)
% Male 70 32 29 45 5 64 48 26
Duration complaints (mo)
12 (0-128)
85%
>12
7 (6) 84 (108)
? 50% >6 <1 51 (50)
Setting General practice
Orthop/ trauma clinic
Retirement
community
Hospital - PT
Hospital Hospital + PT
practice + com.
serv
PT outpatient clinic
Hospital - PT
Country NL Brazil US Iran South Afrika
Australia
US Turkey
Language Dutch Portugese
English Persian Tswana English English Turkish
Study population, setting, country
www.kmin-vumc.nl Methodological quality
COSMIN checklist
COnsensus-based Standards for theSelection of Health Measurement INstruments
Different boxes for each measurement property, with questions regarding quality aspects
Box reliability: 14 items to evaluate the quality of a reliability study
www.kmin-vumc.nl
1 2 3 4 5 6 7 8
Missing items
20% deleted
? ?% deleted
0 ? ? ? ?
Sample size 89 54 56 31 31 47 23 100
Time interval (d)
7 3-4 11 1 1 42 28 1
Stable patients
Assumable?
Treatment?
Assumable?
Treatment?
Based on
GPC.
Treatment?
Assumable
Assumable
Based on
GPC. PT
treatment
Based on
GPC. PT
treatment
Assumable
Test conditions
2x mail inter/intra-rater
2x mail ? 2x clinic 2x mail 2x clinic 2x clinic
Methodological quality of the studies
www.kmin-vumc.nl
1 2 3 4 5 6 7 8
Missing items
20% deleted
? ?% deleted
0 ? ? ? ?
Sample size 89 54 56 31 31 47 23 100
Time interval (d)
7 3-4 11 1 1 42 28 1
Stable patients
Assumable?
Treatment?
Assumable?
Treatment?
Based on
GPC.
Treatment?
Assumable
Assumable
Based on
GPC. PT
treatment
Based on
GPC. PT
treatment
Assumable
Test conditions
2x mail inter/intra-rater
2x mail ? 2x clinic 2x mail 2x clinic 2x clinic
COSMIN good good good fair fair fair poor fair
Methodological quality of the studies
www.kmin-vumc.nl
1 2 3 4 5 6 7 8
Missing items
20% deleted
? ?% deleted
0 ? ? ? ?
Sample size 89 54 56 31 31 47 23 100
Time interval (d)
7 3-4 11 1 1 42 28 1
Stable patients
Assumable?
Treatment?
Assumable?
Treatment?
Based on
GPC.
Treatment?
Assumable
Assumable
Based on
GPC. PT
treatment
Based on
GPC. PT
treatment
Assumable
Test conditions
2x mail inter/intra-rater
2x mail ? 2x clinic 2x mail 2x clinic 2x clinic
COSMIN good good good fair fair fair poor fair
Methodological quality of the studies
www.kmin-vumc.nl
1 2 3 4 5 6 7 8
ICC 0.90 0.89-0.93
0.94 0.86 0.91 0.84 0.55 0.92
Consistency of results
www.kmin-vumc.nl
1 2 3 4 5 6 7 8
ICC 0.90 0.89-0.93
0.94 0.86 0.91 0.84 0.55 0.92
Results of the studies
COSMIN good good good fair fair fair poor fair
www.kmin-vumc.nl
1 2 3 4 5 6 7 8
ICC 0.90 0.89-0.93
0.94 0.86 0.91 0.84 0.55 0.92
Results of the studies
COSMIN good good good fair fair fair poor fair
Duration complaints (mo)
12 (0-128)
85%
>12
7 (6) 84 (108) ? 50% >6 <1 51 (50)
www.kmin-vumc.nl Step 2: Best evidence synthesis
Data syntheses is different per measurement property
General guideline: levels of evidence, based on Cochrane Back Review group
Level Rating Criteria
strong +++ or ---
Consistent findings in multiple studies of good
methodological quality OR in one study of excellent
methodological quality
moderate ++ or --
Consistent findings in multiple studies of fair
methodological quality OR in one study of good
methodological quality
limited + or - One study of fair methodological quality
conflicting +/- Conflicting findings
unknown ? Only studies of poor methodological quality
www.kmin-vumc.nl ExampleReliability of the Quebec Pain Disability Scale (QBPDS)
Consistent findings of good reliability (ICC>0.70) in three studies of good methodological quality and in four studies of fair methodological quality
Strong evidence for a good reliability (+++)
1 2 3 4 5 6 7 8
ICC 0.90 0.89-0.93
0.94 0.86 0.91 0.84 0.55 0.92
COSMIN good good good fair fair fair poor fair
www.kmin-vumc.nl Other measurement propertiesInternal consistency
Consistent findings in mutiple studies of good methodological quality or one study of excellent methodological quality that (sub)scales are unidimensional
PLUS
Consistent findings in mutiple studies of good methodological quality or one study of excellent methodological quality that Cronbach’s alpha is > 0.70
www.kmin-vumc.nl Content validity
Strong evidence: all items are considered relevant for the construct, purpose, and target population and the instrument is considered comprehensive.
Moderate evidence: the items are considered relevant for the construct or target population and the instrument is considered comprehensive.
Limited evidence: only one aspect of content validity is assessed.
www.kmin-vumc.nl Construct validity and responsiveness
Levels of evidence as described in the Table are applied.
Challenges
• some studies examine more hypotheses than others
• some hypotheses are more challenging than others
• some comparison instruments are better than others
www.kmin-vumc.nl Summary
similarities and dissimilarities with GRADE
Study limitations (methodological quality) are taken into account, but not to downgrade the level of evidence but to exclude studies from the data analysis
Inconsistency is taken into account in applying levels of evidence
Indirectness (generalizibility) is usually not taken into account because there are too many differences in study characteristics and the influence on the measurement properties is unclear
Impresision (only one study) is very common. This is taken into account in applying levels of evidence
Publication bias is not considered, but might be a problem
www.kmin-vumc.nl Systematic reviews of measurement properties
www.kmin-vumc.nl
PhD defence Wieneke Mokkink
COSMIN checklist
COnsensus-based Standards for theSelection of Health Measurement INstruments
TODAY
13.45
AULA VU