Karen Setty,1 Bob McConnell,2 Bob Raucher,3 Jeanne Luh,1,4 Jamie Bartram1

1UNC Chapel Hill, Department of Environmental Sciences & Engineering2Tampa Bay Water (Sponsor)3Corona Environmental Consulting4ICF

Summary: To comparatively evaluate source water risk management frameworks and

tools appropriate for pilot testing in the US, the research team undertook a literature

review coupled with external utility interviews and participating utility surveys. Data

was compared and integrated within a multi-indicator evaluation matrix to synthesize

findings. Results of the external utility interviews were included as a single reference

(personal communication) in the matrix. The Consolidated Framework for

Implementation Research (CFIR) and participating utility survey results helped to

identify and refine the draft evaluation criteria for comparing different program options,

integrating researcher and participant input with existing implementation theory.

Preliminary results were then shared at an in-person workshop. A participatory decision-

making exercise facilitated validation of the evaluation criteria and selection of one or

more robust and appropriate source water risk management frameworks.

WRF Project 4748: Comparative Evaluation of Risk Management Frameworks

and Tools for Managing Source Water Risks in the United States

Abstract• The US Safe Drinking Water Act required states to develop source water assessments to

identify existing and potential contamination sources, but proactive risk management

approaches for these water supplies have seen limited application.

• This study assessed which permutation(s) of risk management frameworks and tools might

benefit utilities with surface water sources by coupling a literature review with external

utility interviews and participating utility surveys.

• Qualitative data provided a basis for categorical assignments of goodness-of-fit with each of

24 evaluation criterion across five categories.

• Weighted integration based on stakeholder input provided a relative ranking of applicability,

later validated at a decision-making workshop.

• Hybridization of the American Water Works Association source water protection standard

(ANSI/AWWA G300) and World Health Organization Water Safety Plan (WSP) guidance

was recommended to develop comprehensive risk management guidance.

• Components of the Australian Drinking Water Guidelines and Techneau guidance were

recommended to incorporate financial considerations into risk ranking and mitigation

decisions.

Methods

Results

Conclusions

Acknowledgements• WRF Project Coordinator: Katie Henderson

• WRF Project Advisory Committee: Jennifer Heymann (American Water), Rich Gullick(RCAP Solutions, Inc.), Reid Campbell (Halifax Water)

• Workshop Participants: Tim Bartrand (Corona Environmental Consulting), Shawn Jones(Tampa Bay Water), Mishelle Nobel (Upper Occoquan Service Authority), Shawn Wiley(Suez North America), Laura Dale (Suez Delaware), Richard Stuck (Greater Cincinnati WaterWorks), Alan Roberson (Association of State Drinking Water Administrators)

Results

What are the risks?

How do you control the risks?

How do you know the risks

are under control?

BackgroundWhat is an intervention?

An effort to change something to protect public health

What is risk management?

Risk management processes focus on prevention and

continuous reevaluation of potential risks before they

might occur.

What is the level?

One drinking water utility/community

What is the geographical setting?

US surface water sources in mixed-use catchments

What is the context?

Infrastructure-heavy sector, dense regulatory

environment, standardized priority contaminants,

budget/expertise/time limitations

Figure 5. A facilitated workshop held May 3, 2018 in Chapel Hill brought

together researchers, participating utilities, funders, and external advisors

to discuss preliminary results and engage in decision-making exercises.

Supplementary techniques are recommended to incorporate cost-benefit considerations into risk

ranking and mitigation decisions, along the lines of examples provided by the ADWG and

Techneau guidance. The WSP guidance was carefully developed with end users in mind and has

spread through the World Health Organization’s active capacity building efforts. The AWWA G300

Source Water Protection Standards Committee is currently updating the guidance for US utilities,

and as a result of this project, will make an effort to incorporate aspects of widely used risk

management programs such as WSPs.

Highlights• This review showed the variety of approaches and tools available to meet individual utilities’

needs and desired degree of sophistication.

• Program hybridization may help address compatibility issues when scaling up or adapting

drinking water safety programs to other high-income settings, such as Canada.

• Renewed attention to risk management may help prevent unexpected contamination and

service interruption events, which ultimately have sizeable effects

on the economy and public health.

Table 2. Summary of risk management framework applicability for surface waters in the US, summarizing

included framework guidance documents and external utility case studies (programs compared to 24 criteria; yes =

2, maybe = 1, no/not applicable = 0; weights = 1–1.5)

Participating Utilities

Tampa Bay Water (Sponsor)

Greater Cincinnati Water Works

Fairfax Water

Suez Delaware

Candidate Risk Management

Frameworks

• American National Standards

Institute/American Water Works

Association (ANSI/AWWA) G300-14:

Standard for Source Water Protection

• ANSI/AWWA J-100-10: Risk and

Resilience Management of Water and

Wastewater Systems

• World Health Organization (WHO): Water

Safety Plan (WSP)

• Hazard Analysis and Critical Control Point

(HACCP)

• International Standards Organization: Food

Safety Management Systems (ISO 22000)

• International Standards Organization: Risk

Management (ISO 31000:2018)

• Australian Standard/New Zealand Standard

(AS/NZS 4360:2004 and ISO 31000:2009)

• Australian Drinking Water Guidelines

(ADWG)

• European Commission: Techneau

Framework and Methods for Integrated

Risk Management in Water Safety Plans

Participant

Survey

Evaluation

Matrix

Ranking of

Frameworks

List of Tools

Literature

Review

External

Utility

Interviews

Framework

Guidance

Systematic

Search

Results

Team

Submissions

Topic-

Specific

Insights

CFIR

Constructs

Evaluation

Criteria Workshop

Validation

Evaluation Criteria (revisions to final criteria in italics; *weighted as higher priority)

1. Implementation Feasibility and Cost for Utility

a. Relies on readily available, and/or readily obtained data

b. *References a strategy for coping with data gaps or uncertainties

c. Relies on user-friendly and readily available tools or methods

d. Relies on modest staff time and available in-house (or external) expertise

e. *Flexible and adaptable to low-to-modest budget or utility resources

f. Applicable to broad range of source water or watershed risks

g. Applicable to many different types of utilities and geographic settings

h. Sustainable over the long term and trialable in the short term

2. Risk Identification

a. Provides examples or list of common hazards

b. Readily reveals potential hazards to source water

c. Provides relatively comprehensive coverage and identification of potential hazards

d. Integrates local or cultural knowledge

3. Risk Characterization

a. *Helps quantify or rank identified risks to source water (e.g., to define priorities based on likelihood

and consequences)

b. Uses sound science in quantifying and characterizing type and relative level of risk

c. *Considers multiple facets of risk (e.g., economic or financial, regulatory compliance, public health,

customer relations and trust or utility reputation)

4. Risk Management

a. Helps identify possible risk-mitigating options and strategies

b. Helps evaluate and prioritize risk mitigation strategies or options

c. *Incorporates monitoring and evaluation strategies (e.g., in real time for critical control points)

d. Offers suggestions for program implementation (e.g., identifies best practices or common pitfalls)

e. Offers advice for managing risks outside immediate control of utility

f. Incorporates regular feedback loops or quality improvement cycles

5. Clarity and Ease of Communication

a. Recommends metrics for measuring progress or demonstrating benefits

b. Beneficial outcomes have been previously demonstrated (e.g., employee satisfaction, water quality,

public health)

c. Supports clarity in conveying risk-based information: (1) within the utility; (2) governing boards,

public officials, and regulators; and (3) watershed stakeholders and the general public

WSPortal(n=17+)

Google

Scholar(n=50)

Web of

Science(n=50)

Scopus(n=58)

Articles+(n=40)

EPA(n=20)

Research

Team (n=29)

Abstract

Screened

Full-Text

Screened

Excluded(n=197)

Excluded(n=14)Included

(n=68)

Framework

Guidance (n=14)

Inclusion criteria Exclusion criteria

Dated 1986 or later (based

on date of first SDWA

amendment)

Relevance to source water

Relevance to high income

nations

Inclusion of risk

management measures

(not just risk

identification)

Written in English

Single-contaminant or

single-site studies

Only describes forecasted

risks (as opposed to

current risks)

Focus on groundwater

rather than surface water

Focus on small community

water systems

Goal:

Select the most feasible intervention

for this setting and context, using a

structured approach

What do we know about implementation?

Since intervention outcomes are strongly influenced by context, it can help to consider

theory, evidence, and input from a variety of sources

Figure 1.

Contextual

considerations

recommended by

the Consolidated

Framework for

Implementation

Research (CFIR)

helped to refine the

evaluation criteria

for intervention

selection.

Figure 4. Literature review search terms (A), inclusion and exclusion criteria (B), and article sources (C)

A

B

C

Figure 6. Summary of participant rankings of risk management framework

applicability for surface waters in the US, upon reviewing research results

(highest ranking = 1; not applicable = N/A)

1

2

3

4

5

6

7

8

9

N/A

Framework1

# Criteria

Met

Weighted

Score Relative Strengths Relative Limitations

Recommended Rank

Based on Context2

WSP 24 53

Recommended for worldwide

application; Extent and nature of

implementation largely flexible;

Many supporting documents

(including free guidance on audits)

Primary focus is on protecting

human health; Less focus on

integrating financial

considerations; Few US-specific

case examples

2

Recommended as

a comprehensive

risk management

framework (source

to tap)

Techneau 17 46

Many supporting documents:

"structure and toolbox"; Spectrum of

quantitative and qualitative tool

options; Intended to improve on

WSPs (e.g., by explicitly including

risk acceptance/tolerability

assessment)

Guidance provision no longer

active (website insecure); Still

somewhat reactive (focused on

early warning/response) 3

Recommended for

those who would

like to explore

more advanced,

technical options

AWWA

G30017 43

US-centric; Respected professional

authority; Excellent example of

watershed outreach in US context

Focused on documentation;

Replication may be difficult;

Few examples provided;

Requires purchase

1

Recommended as

an entry point for

watershed risk

management

ADWG 14 42.5

Comprehensive; User-friendly;

applies elements of HACCP, ISO

9001 and AS/NZS 4360:2004 to

drinking water supply

More than 1K pages (may be

high barrier to entry); Developed

for Australian context;

Somewhat focused on water

quality

3

Recommended as

example for

integrating

financial

considerations

ISO 22000 15 39.5

Internationally recognized; Includes

third-party certification

Tailored to food safety and some

analogies (e.g., pest control) may

not apply to drinking water

facilities; Likely to be some

instinctive rejection of food

chain connotation (Deere and

Davison, 2009)

4

Recommended

add-on only if

international

recognition is

important to utility

ISO 31000 13 37.5

Suggests it can be customized to any

organization, sector, or context;

Simple; Gives advice for

organizational culture

Requires purchase; Cannot be

used for certification purposes;

Brief; Not specific to water-- Not recommended

AS/NZ

Standard

ISO 31000

11 36

Certification pertinent to

organizations of any kind in

Australia and New Zealand;

Contains many definitions and

principles

Requires purchase; Layout

similar to ISO (not user

friendly); Not particular to water

(or food) safety; Diagrams and

guidance are somewhat

theoretical

-- Not recommended

HACCP 11 32

Well-known; Widespread use; Can

use descriptive historical data to plan

for future risks

Specific to food production/not

tailored to drinking water or

catchments; Outdated

-- Not recommended

AWWA

J10010 31

US-centric; Developed by experts

after 9/11

Limited adoption; Seems

burdensome and fairly

prescriptive; Focused on

terrorism (reinforces a piecemeal

approach)

-- Not recommended

SDWA 9 30

Existing guidance from US

regulations; Comprehensive six-step

program

Comprehensive plan not carried

out widely in practice; Level of

expected public involvement

may be unrealistic

--

Currently in use

(included for

comparison)

Framework Utility Country

ISO 22000 Aigües de Barcelona Spain

Australian Drinking Water

Guidelines/

ISO 22000

SEQWater Australia

Australian Drinking Water

Guidelines/WSPs

Water Corporation (Western

Australia)

Australia

Techneau/

ISO 22000

Waternet Netherlands

HACCP East Bay Municipal Utility

District

USA

AWWA G-300 American Water USA

WSP United Utilities UK

32 Candidates

(6 groups of

frameworks)

7 Case

Examples

Figure 3. External utility interviews captured practical experiences with a variety of risk

management frameworks, to supplement literature sources.

Criterion

Framework 1.d. Relies on modest staff time and

available in-house (or external) expertise

AWWA G300-14:

Standard for

Source Water

Protection

Yes; suggests external stakeholders within

the community are critical (and likely

accessible), and suggests slow rate of

progress (allocation of internal staff time)

is acceptable; no reference to requiring

external scientific expertise

AWWA J-100-10

Risk and

Resilience

Management of

Water and

Wastewater

Systems

Yes; "in order to encourage widespread

application... [the methodology is]

appropriate for self-assessment primarily

by on-site staff in a relatively short period

of time (typically less than a week of work

by a team of 3–6 people, after assembly of

the necessary documents)"

Table 1. Excerpt from matrix comparing each

information source to evaluation criteria, addressing

the question “Does framework or guidance meet this

criterion?” (possible answers = yes, no, maybe, or N/A)

49. How important are the following criteria to your utility when selecting a source

water protection and management program?

50. Which other criteria, if any, are important to your utility when selecting a source

water protection and management program?

Figure 2. Questions from a participating utility baseline survey were used to refine evaluation

criteria and assign weights based on shared values and priorities.

Considering the needs of utility personnel

and context for this project, this evaluation

pointed to WSPs and the ANSI/AWWA

G300 as valuable guidance for managing

risks to surface water sources in the US.