Cono Ariti: matched control studies
-
Upload
nuffield-trust -
Category
Health & Medicine
-
view
432 -
download
0
Transcript of Cono Ariti: matched control studies
© Nuffield Trust 22 June 2015
Matched Control Studies:
Methods and case studies
Cono Ariti
© Nuffield Trust
Predictive risk
modelling
Resource
allocation
Descriptive
studies Evaluations
Integrated
care
pilots
nuffield trust
Nuffield Trust Research team – data linkage projects
Risk
sharing
for CCGs
nuffield trust Combined
predictive
model
nuffield trust
Person
based
resource
allocation
nuffield trust
Social
care at
end of life
nuffield trust
Cancer
and social
care
nuffield trust
Predicting
social
care
costs
nuffield trust
Virtual
Wards
nuffield trust
WSD
nuffield trust
British Red
Cross
nuffield trust
© Nuffield Trust
Need for evaluation
Need to know what works
• In a practical setting – “real world evaluation”
• Clarify the debate
• Likely impacts – unbiased results
• Link to qualitative work
Refine programs
• Obtain feedback and learnings – the pain of implementation
• Explore sub-groups – where did it work? Where could it work?
© Nuffield Trust
Issues with evaluations
Randomised control trials
• “Gold standard”
• May not be feasible or ethical
• Inclusion and exclusion rules can limit generalisation
• Are still subject to poor implementation – can induce bias
• Potentially expensive!
Observational studies
• Typically no “natural” experiment exists
• Often no comparable control group to provide a fair assessment
© Nuffield Trust © Nuffield Trust
Matched Control Studies -
Methods
© Nuffield Trust
Matched Control Studies
The basic idea
• Match controls to those treated based on measured characteristics in existing datasets
• The control group and treated group should look similar “on balance”
• Mimics the idea of an RCT
• Based on propensity score theory (Rubin & Rosenbaum, 1983) and earlier work on matching (Cochran, 1965)
© Nuffield Trust
Matched Control Studies
Matching
• Prognostic risk score
• Demographics – age, gender, deprivation, ethnicity
• Prior acute care service use – admissions, OP and A&E attendances
• Prior diagnoses, targeted chronic conditions
Balance
• In this case all matching variables
• Additional variables such as length of stay, additional diagnoses and longer service use history
• Assures comparability between the groups
© Nuffield Trust
Matching Algorithm
Algorithm
• Exact match not possible
• Computer intensive “genetic algorithm”
• Uses a weighted Mahalanobis “distance” to determine closest match
• Automatically assesses balance and moves to an improved solution
Assessing Balance
• On overall group similarity
• Compares means and distribution of variables in the two groups
© Nuffield Trust
Analysis of matched control studies
• Standard statistical methods to estimate the difference in the two groups
• Regression models, difference in difference analysis
• By including matching variables in the statistical adjustment remaining imbalances can be reduced – “doubly robust”
• Methods exist for sensitivity analysis – impact of unobserved variables
• Some controversy around accommodating the matching in analysis
© Nuffield Trust © Nuffield Trust
Case Study 1: Telehealth
Programme
© Nuffield Trust
Case Study 1: Telehealth program
Intervention:
• Remote monitoring for patients with long term conditions
Nuffield commissioned to evaluate impact:
• Primary: Reduction in emergency hospital admissions?
• Secondary: Reductions in Emergency attendances, outpatient attendances, mortality
Methods:
• Retrospective matched control study – use of already existing administrative data
© Nuffield Trust
Description: Telehealth program
© Nuffield Trust
Matched control studies – broad aim
>30,000 individuals – resident in local area June
2010 to March 2013, did not receive telehealth
and were eligible for matching
(local controls)
Aim to find 716 individuals who match
almost exactly on a broad range of
characteristics
Use this group as study control group
716 individuals – enrolled June 2010 to March 2013
& received Telehealth intervention & eligible for
matching
© Nuffield Trust
Datasets available
Telehealth Nuffield trust
N = 716
• person details
• dates of service
• type of service
Identifiers:
Names, DOB,
Addresses, etc
• dates & place
of death for all
people in
England,
• associated
hospital (HES)
records
Identifiers:
Nuffield Trust
specific HESID
Administrative data ONS deaths Hospital inpatient, outpatient, AE
© Nuffield Trust
1
2
3
4
5
6
7
8
9
. .
1
2
3
4
5
6
7
8
9
. .
1
2
3
4
5
6
7
8
9
. .
Telehealth Data Linkage Service Nuffield Trust
New Identifier New Identifier New Identifier
(NHS no) (NHS no)
Names Names
Address Address
DOB DOB
HESID HESID
Telehealth person identifiers (File A)
© Nuffield Trust
Final datasets available for analysis
Nuffield trust
Identifiers:
HESID on all
ONS deaths Hospital inpatient, outpatient, AE Telehealth data - desensitised
Use all this
info to carry
out matched
control
analysis
© Nuffield Trust
Control group – how well matched?
© Nuffield Trust
Control group – how well matched?
Telehealth Controls
© Nuffield Trust
Control group – how well matched?
Telehealth Controls
© Nuffield Trust
Key Result 1: Risk of admissions or death
© Nuffield Trust
Key Result 2: Changes in admissions or attendances
(six months pre and post intervention)
© Nuffield Trust
Results
• Telehealth patients tended to be admitted for an emergency admission earlier than control patients
• There was no difference in mortality between the telehealth and control groups
• There were no statistically significant reductions in hospital admissions when comparing the period six months before and after the telehealth intervention
• In summary the Telehealth program did not have a significant impact on acute care outcomes
• Sensitivity analysis showed little evidence of an important unobserved variable
© Nuffield Trust © Nuffield Trust
Matched Control Studies:
Summary
© Nuffield Trust
Matched Controls: Summary
Benefits
• Makes full use existing data, with relative ease
• Techniques applicable to many different types of services and datasets
• Decisions on what seems to work (and what may not) based on more robust analyses leading to better informed decisions
Caveats
• If important unobserved variables exist results may be biased
• The routine data sources must contain the relevant data
© Nuffield Trust
Implementing locally – key enablers
Do you have … Can you …
• Access to data that contains
the outcomes relevant to your
evaluation?
• Access to data containing
relevant matching
characteristics?
• Do you have consent to
access/link the data?
• Analysis tools to apply
statistical methods to the data?
• Skilled analysts to analyse the
data?
• Link multiple sources of data?
• Handle large amounts of data
(millions of observations)?
• Identify recipients of the
intervention?
• Transform and augment that data
with bespoke variables?
• Apply sophisticated matching
algorithms routinely to this data?
• Analyse the data with a variety of
statistical methods and interpret
the results appropriately?
© Nuffield Trust
www.nuffieldtrust.org.uk
Sign-up for our newsletter www.nuffieldtrust.org.uk/newsletter
Follow us on Twitter: Twitter.com/NuffieldTrust
© Nuffield Trust
Thank you