Challenges in using mathematical modelling for public health decision- making John Edmunds, Jess...
-
Upload
heriberto-laughton -
Category
Documents
-
view
217 -
download
0
Transcript of Challenges in using mathematical modelling for public health decision- making John Edmunds, Jess...
Challenges in using mathematical modelling for public health decision-making
John Edmunds, Jess Metcalf, Justin Lessler
Why use models?• Observational studies of impact are retrospective,
and difficult to interpret• Intervention studies (trials) relatively small & key
endpoints rarely observed, therefore models used for extrapolation – Other endpoints (e.g. deaths)– Other population groups (perhaps excluded from study)– Over time– To others in the population
• Synthesise the results of different studies in a unified framework– Improve our understanding of key drivers
• Identify and quantify knowledge gaps
Why use models?• Observational studies of impact are retrospective,
and difficult to interpret• Intervention studies (trials) relatively small & key
endpoints rarely observed, therefore models used for extrapolation – Other endpoints (e.g. deaths)– Other population groups (perhaps excluded from study)– Over time– To others in the population
• Synthesise the results of different studies in a unified framework– Improve our understanding of key drivers
• Identify and quantify knowledge gaps
Synthesis & prediction
Understanding, research and surveillance
Main role of mathematical modelling in UK infectious disease policy
Epidemics• Contingency planning• Risk assessment• Epidemic forecasting (& “now-casting”)• Impact of interventions
“Endemic” diseases• Vaccine policy• Other control policy (e.g. Screening and treatment)• Burden of disease assessment (priority setting)
Epidemics• Contingency planning• Risk assessment• Epidemic forecasting (& “now-casting”)• Impact of interventions
“Endemic” diseases• Vaccine policy• Other control policy (e.g. Screening and treatment)• Burden of disease assessment (priority setting)
Impact of alternative policies (scenarios)
Transmission models
Transmission models
Main role of mathematical modelling in UK infectious disease policy
Challenges
• Nature of the problem:– “It is difficult to make predictions, particularly of
the future” (Niels Bohr, Yogi Berra, Somebody..)• Multiple agencies:
– Communication, co-ordination, “translation”• Responding to the needs of policy-makers
Nature of the problem: dynamics & uncertainty
Years after introduction
0 20 40 60 80 100
Inci
denc
e pe
r 10
0.00
0
0
200
400
600
800
1000
1200
1400
Vaccinating only elderlyChildhood vaccination - 1 doseChildhood vaccination - 2 doseChildhood vaccination - 2 dose + vaccinating elderly
Years after introduction
0 20 40 60 80 100In
cide
nce
per
100.
000
0
100
200
300
400
500
600
700
Vaccinating only elderlyChildhood vaccination - 1 doseChildhood vaccination - 2 doseChildhood vaccination - 2 dose + vaccinating elderly
Varicella incidence
Zoster incidence
Cost-effectiveness of a combined scheduleover time
Van Hoek et al. Vaccine (2011)
Nature of the problem: unexpected events• Pathogen emergence very difficult to
predict– Drivers (mechanisms) poorly understood
• Low probability high impact events– E.g. Characteristics of next flu pandemic– What is the probability of an epidemic
worse than 1918?– Other infections?
• The next HIV or vCJD?• Remain vigilant but surveillance is
retrospective
1 4 71
01
31
61
92
22
52
83
13
43
74
04
34
64
95
2
00.10.20.30.40.50.60.70.80.9
1917191819191920
weekD
ea
ths
pe
r 1
,00
0
Epidemic Deaths1918 200,000
1957 18,000
1968 46,000
2009 600
Nature of the problem: unexpected events• Models rapidly employed during an
outbreak– Characterise novel pathogen
• Reproduction numbers, case fatality, etc– Inference: can “make sense” of limited data– Though early analyses must be treated with
caution• Delays and censoring of data• Bias• Chance
• Can be employed (later) to forecast the epidemic progress and evaluate interventions
Fraser et al. Science (2009)
Baguelin et al. Vaccine 2010
September predictions• Peak height second wave estimated
to be similar to first• Peak early Nov., end Jan.• Vaccination has limited impact on
casesOctober• Further data reduced uncertainty
but did not alter central estimate
Nature of the problem: unexpected events
Multi-agency: communication & co-ordination
• Modelling is a component of the advice
• Advisory body (e.g. JCVI, NICE) made up of individuals from diverse backgrounds– Lack of practical modelling
experience
• Advice is distilled• Serves different purposes
– E.g. different outcomes may be critical for different “audiences”
– Presented in different ways
DH advises Ministers
JCVI
PHE AcademiaMHRA NIBSC
Communicating the limits of modelling• Public health officials might want clear quantitative statements about the
impact of an intervention– “Single version of the truth”
• Precise quantitative predictions highly likely to be misleading• Uncertainty
– Parametric– Structural– “Rules”
• Representing uncertainty properly, presenting it to decision-makers.– Can clear policy advice still be made?
• Sensitivity analysis– What happens if it can’t?
• Clear guidance of further research• Sensitivity analysis• Value of information
• Assessing the quality of other research
• Predictive power comes from appropriate model structure (mechanism)– As well as parameter
values• If key drivers are omitted
then results will be unreliable
• Review and assess other work – Second opinion
initially infected
R0 Cases at 30 days
Cases at 365 days
10 1.5 31 224,000
10 3 64 774 billion
1,000 1.5 3,094 22 million
1,000 3 6,387 77 trillion
Communicating the limits of modelling: structural uncertainty
Meltzer et al. 2001 EID 7(6)
Mathematical models
Limits of modelling:predictions vs scenarios
Ong et al. PLoS One 2010
Scenarios“things that could happen”
Predictions“things we think will happen”
Khazeni et al. Annal Intern Med 2009
Limits of public health:Optimal vaccine allocation
• Vaccine given before outbreak
• Optimised coverage by age class
• For different outcomes• Compare with CDC
recommended strategy
Medlock & Galvani Science (2009)
Limits of public health:Optimal vaccine allocation
• Vaccine given before outbreak• Optimised coverage by age
class• For different outcomes• Compare with CDC
recommended strategy
• Vaccine coverage cannot be optimised– Not under the control of central
decision-maker
• Many other examples of modelling interventions that are not likely to be considered
Medlock & Galvani Science (2009)
Responding to the needs of policy-makers
• Often want information on whether an intervention is worth implementing when compared to alternatives– Economic analysis
• Explicitly weigh costs and benefits• A number of countries have recommended
methodologies that should be followed– Also journals and societies (e.g. ISPOR/SMDM)– However, many of these “rules of the game” reflect ethical
stances that are not universally shared• E.g. Discounting of health benefits
– May be very influential
Responding to the needs of policy-makers: Vaccines in UK
• Epidemic models and associated economic models used heavily by JCVI in formulating their advice and recommendations
• NHS Constitution– Minister (England) bound to accept recommendation of JCVI if
the programme is cost-effective– CEA is therefore critical
• Transmission model is not required (but usually done)– Critical to be done correctly
• Large scale programmes• Difficult to terminate
Responding to the needs of policy-makers:• Many layers of approval
– Require different outputs, or results presented in different forms
– Can be time consuming• “Mission creep”
• Multiple revisions– Comments from committees,
referees etc.
• Short deadlines• Academics rarely incentivised
to participate in these processes
Date Milestone Activity
Sept ‘06
JCVI subgroup
Review existing models
Refine questions (e.g age at vaccination and delivery route)
Feb ‘07
JCVI subgroup
Review model assumptions
Preliminary results
July ‘07
Papers submitted to JCVI secretariat
Sent for peer review
Sept ‘07
Peer review comments received
Revised analysis
Oct ‘07
JCVI recommendation
Responding to the needs of policy-makers:• Many layers of approval
– Require different outputs, or results presented in different forms
– Can be time consuming• “Mission creep”
• Multiple revisions– Comments from committees,
referees etc.
• Short deadlines• Academics rarely incentivised
to participate in these processes
Choi et al. Vaccine (2010)
Responding to the needs of policy-makers:• Many layers of approval
– Require different outputs, or results presented in different forms
– Can be time consuming• “Mission creep”
• Multiple revisions– Comments from committees,
referees etc.
• Short deadlines• Academics rarely incentivised
to participate in these processes
Jit et al. BMJ (2008)
Tackling challenges: UK vaccines and flu • Relatively simple decision-making framework
– Centralised, clear, limited number of actors
• Relatively healthy (multiple) modelling groups– Mechanisms in place for these to feed into decisions
• E.g. Modelling representative on JCVI• Peer review and scrutiny
• Strong modelling team within the public health agency– Ensure right questions are addressed– Outputs are appropriate– Standing resource (core funding leads to flexibility and rapid response)– Optimises use of all relevant data – adding value– Inputs into design of surveillance system
• Long-term relationships & working patterns built-up• Sophisticated “interpreters” and users of modelling within decision-making
framework• Clear role for economic analyses and clear (?) rules for conducting these• Limited role of industry and lobby groups
Tackling challenges: UK vaccines and flu • Relatively simple decision-making framework
– Top down
• Relatively healthy (multiple) modelling groups– Wasteful
• Strong modelling team within the public health agency– Research best done by academics– No role for un-publishable quick and dirty analyses
• Long-term relationships & working patterns built-up– Club
• Sophisticated “interpreters” and users of modelling within decision-making framework– Results “digested” into summary statements where qualifiers & context lost
• Clear role for economic analyses and clear (?) rules for conducting these– Economists know the price of everything and the value of nothing
• Limited role of industry and lobby groups– Have a legitimate role to play in a healthy democracy
Integrating modelling with public health decision-making• There are technical challenges that will always
remain– Highly non-linear systems– Imperfectly understood natural history– Uncertainty in parameter values– Imperfectly observed process, bias in data sets– unethical to measure, too expensive to design the
experiment, etc
• Other challenges that will always remain due to:– Complex decision-making environment– May be subject to press or political scrutiny– Compromises & time pressures that will inevitably
occur– Inadequate resources
• Enormously rewarding to play a part in improving the health of the nation
Integrating modelling with public health decision-making• There are technical challenges that will always
remain– Highly non-linear systems– Imperfectly understood natural history– Uncertainty in parameter values– Imperfectly observed process, bias in data sets– unethical to measure, too expensive to design the
experiment, etc
• Other challenges that will always remain due to:– Complex decision-making environment– May be subject to press or political scrutiny– Compromises & time pressures that will inevitably
occur– Inadequate resources
• Enormously rewarding to play a part in improving the health of the nation
Integrating modelling with public health decision-making• There are technical challenges that will always
remain– Highly non-linear systems– Imperfectly understood natural history– Uncertainty in parameter values– Imperfectly observed process, bias in data sets– unethical to measure, too expensive to design the
experiment, etc
• Other challenges that will always remain due to:– Complex decision-making environment– May be subject to press or political scrutiny– Compromises & time pressures that will inevitably
occur– Inadequate resources
• Enormously rewarding to play a part in improving the health of the nation
Integrating modelling with public health decision-making• There are technical challenges that will always
remain– Highly non-linear systems– Imperfectly understood natural history– Uncertainty in parameter values– Imperfectly observed process, bias in data sets– unethical to measure, too expensive to design the
experiment, etc
• Other challenges that will always remain due to:– Complex decision-making environment– May be subject to press or political scrutiny– Compromises & time pressures that will inevitably
occur– Inadequate resources
• Enormously rewarding to play a part in improving the health of the nation
IPD incidence E&W, HPASerotypes in PCV7, <2 yrs