Microbial Risk Assessment: lessons learned and future

directions

Greg PaoliDecisionalysis Risk Consultants, Inc.

Ottawa Canada

Or, A “Risk Assessment” of

Microbial Risk Assessment

One View of the Processing Stages in Risk Assessment

• Problem Selection• Model DM’s Values• Outcome Selection• Scope Selection• Tool Selection• Evidence Acquisition• Evid. Characterization• Model Development

• Diagnosis and Experimentation

• Validation-Seeking• Auditing• Documentation• Peer Review• Communication with

Risk Managers• Dissemination

Hazard Identification

• Managerial Hazards

• Scope Hazards

• Evidence Hazards

• Computational Hazards

• Characterization Hazards

• Communication Hazards

Managerial Hazards

• Silence regarding Values

• Valuation of Knowledge Gains

• Linear Processes

• Limited Tool Development

Scope Hazards

• “Keep it simple!”• “This is preposterously simple. You must include

the complexity or it can have no credibility!”• Scope decisions are qualitative risk assessments• Under-valuation of multi-use mitigations• Model resolution balance• Risk-risk tradeoff considerations

Evidence Hazards

• Numerical Hazards (Mean Log Example)

• Evidence regarding Process Deviations

• Regional and Temporal Issues

• Human Factors

• Cross-contamination

• Model Uncertainty

Mean Log Example

2

2

2

2

5

1

1

1

22

Mean Log Example

100

100

100

100

100000

10

10

10

400100030

Risks Differ by a Multiple of 250

Mean Log Example

4

3

4

3

5

1

1

1

3.52

Would this be considered an increase in risk?

Mean Log Example

10000

1000

10000

1000

100000

10

10

10

22000100030

Risks Still Differ by a Multiple of 5

Characterizing the Extremes

• Illnesses result from combinations of rare events• We can expect to characterize most everyday

processes reasonably well.• The devil is in the tails

– no data on the magnitude and probability of deviations.

• Need to prove that this as a common and dominant phenomenon

Cross-Contamination: The Final Frontier

• Is it ‘unmodellable’ for the population?

• How big and black can a black box be?

• How do we incorporate and work with model elements for which our state of knowledge can be best described as ignorance?

Uncertainty Characterization

• If we express our uncertainty for most, but not all, variables or model assumptions, can we still say we have captured uncertainty.

• How much uncertainty is enough?

• Simple Answer: all of it:

• What’s the Real-Life Answer?

Computational Hazards

• Risk Estimate Stability (Rare Events)

• Auditability and Error-Proneness

• Transparency (Strict vs. Real)

• Time consumption (e.g. 2-D Models)

• Inability to Provide Real-Time Decision Support or Managerial Learning

• Lack of Diversity in Approach

Characterization Hazards

• Choice of Measures– Risk to Susceptibles– Population Risk– Risk per Serving/Preparation/Kilogram/Batch

• How Much Uncertainty has to be Included to Pass the “Uncertainty Test”

• Oversimplified Sensitivity Analysis

Communication Hazards

• Dissemination of Models for Review

• Expressing the Magnitude of Uncertainty

• How many audiences can we serve?

• Can stakeholders be meaningfully engaged in complex risk assessments?

Solution Sets

• Methodological Research– Tool Diversification– Modular System Characterization

• Linkages with Epidemiology

• Food Safety Objectives

Tool Diversification

• When all you have is a hammer …• Process Risk Models• Analytical Models• Bayesian Network Models• Expert Systems• Model Learning• Causal Models• Qualitative Risk Assessment• Module Librairies

Tool Diversification

• Within PRM Approaches …– Diversify software, and/or– Perform Needs Assessment– Performance Comparisons– Good Modelling Practice

Qualitative Risk Assessment

• Prone to Inferential Sloppiness– A literature search with conclusions?

• It is possible to impose structure, but its not always welcome

• Absence of an Inferential Trail

• We should be cautious about conferring the label “Risk Assessment” to anything that uses the right terminology.

Risk Modules not Risk Assessments

• Need to build microbial risk assessment ‘infrastructure’– Development away from the ‘bright lights’

– Carefully documented and computationally sound

– Examples of appropriate implementations

– Documented with limitations and caveats

– Shared library and shared experiences

• With this infrastructure, risk assessment can only get better, easier, more reliable

Linkages with Epidemiology

• Attributable Risk Problem

• Integration of Case-Control findings for sporadic cases

• Approaches for model validation

• Epidemiology re-thinking causality criteria

• Epi needs biological plausibility

• QRA needs epidemiological evidence

FSOs, Risk Measures

• Surrogate Variables

• Performance Indices

• Concrete Linkages to Objectives

• Qualitative Measures of Risk

• Farm-Level Measures

Process Characterization

• Distribution of Concentration across Units

• Prevalence of Contamination

• Indicator Levels• Competitive Flora• Homogeneity• Distribution of Strains

Present

• Information Provided with Food to Affect Later Handling

• Downstream Processing• Extent of Pooling• Growth Inhibitors• Packaging and Insulation• Information to Affect

Traceback and Recall• Target Consumer

A Tiered Approach to Methodology Development

• Tier 1:

• systems modelling– Multi-pathogen, indicators, sampling results,

responses to deviations

• Modular abstraction to simpler forms

• Modular integration tools

Tiered Approach (Tier 2)

• risk assessment– farm to fork– plant to fork– farm to plant– Supplier to purchaser

Tiered Approach … (Tier 3)

• risk-based expertise capture– A Risk-Based Process Inspector

• Formal, structured qualitative analysis– Advice replicates quantitative findings through

querying the problem in qualitative terms as well as some quantitative data.

In the Defense of Risk Assessment

• Bending over backwards to meet the demands for:– the best and up-to-date data– the best model and modelling technique– the best software implementation– good documentation of model– high-quality report– technical appendices for peer review– non-technical summaries– peer-reviewed publications– all at once, without a safety net

In the Defense of Risk Assessment: Are we Shooting the Messenger?

• Systems are too complex for human reasoning• Currently, Microbial Risk Assessors are

Methodological Researchers• Carefully formalizing some of the reasons why we

have not been successful in the past– We don’t acknowledge the complexity– We’ve never really understand the systems well enough

to control them reliably.

• Value of Information analysis sorely needed.

Conclusions

• Methodological Research is Key to the Future of Microbiological Risk Assessment– There is no time to think!

• Diversify the portfolio of approaches– Promote competition

• Manage expectations• Assess knowledge gains• Fairly evaluate alternative approaches