Making sense of results - a workshop for healthcare librarians
description
Transcript of Making sense of results - a workshop for healthcare librarians
Making sense of results- a workshop for healthcare librarians
Dr Amanda Burls2nd UK Clinical Librarian Conference
Objectives
• To look at how results can be presented• To understand what a meta-analysis is• To be able to interpret a “blobbogram”• To be able to make sense of tests for “statistical significance” • To explore how uncertainty in results can be summarised and
understand:• P-values• Confidence intervals
• To have fun!
Making sense of results
• How are results summarised?
How many of you have attended a critical appraisal skills workshop?
• What sort of study design were you appraising?
• What are the key things you remember?
Critical appraisal of any study design must consider
• Validity– Can the study (results) be trusted?
• ResultsResults– What are the results and how are they (or can they
be) expressed?
• Relevance– Do these results apply to the local context?
Warning!• Everything I say from now onwards assumes
that the results being considered come from an unbiased study!
How are results summarised?
• Most useful studies compare at least two alternatives.
• How can the results of such comparisons be expressed?
Expressing results:What did the study show?
• Patients with backache:– 100 randomised to receive a firm mattress– 100 randomised to receive a medium mattress
• After 3 months:– 80 get better in the firm mattress group– 20 get better in the medium mattress group
• How would you summarise this for a friend?
Summarise
• 80 out of 100 (80%) better in firm mattress group
• 20 out of 100 (20%) better in the medium mattress group
• 4 times as likely to get better with a firm mattress
• An extra 60% of people get better with a firm mattress
How were the results summarised?
• There are two basic ways to summarise results of studies that compare two or more groups:
1. Difference (take them away)
2. Ratio (divide)
The blobbogram!
Blobbogram
Line of no differencebetween treatments
less more
Blobbogram - Difference (taking away)
Line of no differencebetween treatments
less more0
Blobbogram - ratio (dividing)
Line of no differencebetween treatments
less more1
A randomised placebo-controlled trial
Well conducted RCT – no bias
• Five people with backache received Potters
• Five people received placebo
• 4 out of 5 with Potters got better
• 2 out of 5 with placebo got better
<0.00010.010.030.090.29p-value
0.27 to 0.520.10 to 0.640.04 to 0.67-0.04 to 0.71-0.23 to 0.7995% CI
0.40.40.40.40.4Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0.80.80.80.80.8Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
<0.00010.010.030.090.29p-value
0.27 to 0.520.10 to 0.640.04 to 0.67-0.04 to 0.71-0.23 to 0.7995% CI
0.40.40.40.40.4Proportion responding in control arm
408642Responders in control arm
1002015105Number in control arm
0.80.80.80.80.8Proportion responding in treatment arm
80161284Responders in treatment arm
1002015105Number in treatment arm
No backache at 3 months(Results of our Potters tablet
versus placebo trial) Potters PlaceboPotters Placebo
Favours placebo Favours PottersFavours placebo Favours Potters
No backache at 3 months(Results of our Potters tablet
versus placebo trial) Potters PlaceboPotters Placebo
Favours placebo Favours PottersFavours placebo Favours Potters
No backache at 3 months(Results of our Potters tablet
versus placebo trial) Potters PlaceboPotters Placebo
Favours placebo Favours PottersFavours placebo Favours Potters
No backache at 3 monthsDo you think this study proves Potters works?
Potters PlaceboPotters Placebo
Favours placebo Favours PottersFavours placebo Favours Potters
It could be due to chance!
• What if there had 1000 people in each arm and 800 got better with Potters and only 200 got better on placebo?
• Would you believe Potters works now?
• So how many people would you want in each arm to believe the trial?
P-value in a nutshell
The Null Hypothesis
0 1
Impossible Absolutely certain
• p = 0.5
quite likely - evens chance - 50:50 - 1 in 2
• p = 0.001
very unlikely - 1 in 1000• p = 0.01
unlikely - 1 in 100• p = 0.05
fairly unlikely - 1 in 20 - 5 times in 100
Odds ratio (12b)
0
10
20
30
40
50
60
70
80
90
100
Pre and Post Workshop Scores
Perc
enta
ge 5
4
3
2
1
MAAG (9b)
0
10
20
30
40
50
60
70
80
90
100
Pre and Post Workshop Scores
Perc
enta
ge 5
4
3
2
1
Moral:
Any observed difference between two groups, no matter how small, can be made to be “statistically
significant” - at any level of significance - by taking a sufficiently large sample.
• Question: How can we express uncertainty due to chance?
• Answer: the p-value
• But is there a better answer?
Introduction to confidence intervals
• CIs are a way of showing the uncertainty surrounding our point estimate.
No backache at 3 months(Results of our Potters tablet
versus placebo trial) Potters PlaceboPotters Placebo
Favours placebo Favours PottersFavours placebo Favours Potters
No backache at 3 months(Results of our Potters tablet
versus placebo trial) Potters PlaceboPotters Placebo
Favours placebo Favours PottersFavours placebo Favours Potters
No backache at 3 months(Results of our Potters tablet
versus placebo trial) Potters PlaceboPotters Placebo
Favours placebo Favours PottersFavours placebo Favours Potters
No backache at 3 months(Results of our Potters tablet
versus placebo trial) Potters PlaceboPotters Placebo
Favours placebo Favours PottersFavours placebo Favours Potters
Hypothermia vs. control In severe head injury
Mortality or incapacity (n=158)
RR 0.63 (0.46, 0.87)Marion 1997
.1 .2 1 5 10
Total (95%CI)
Clifton 1992
Hirayama 1994
Clifton 1993
RR
Hypothermia vs. control In severe head injury
Mortality or incapacity (n=158)
RR 0.63 (0.46, 0.87)Marion 1997
.1 .2 1 5 10
Total (95%CI)
Clifton 1992
Hirayama 1994
Clifton 1993
RR
Hypothermia vs. control In severe head injury
Mortality or incapacity (n=158)
RR 0.63 (0.46, 0.87)Marion 1997
.1 .2 1 5 10
Total (95%CI)
Clifton 1992
Hirayama 1994
Clifton 1993
Favours intervention RR Favours control
Hypothermia vs. control In severe head injury
Mortality or incapacity (n=158)
RR 0.63 (0.46, 0.87)
Marion 1997
.1 .2 1 5 10
Total (95%CI)
Clifton 1992
Hirayama 1994
Clifton 1993
Favours intervention RR Favours control