“To Ignore or Not to Ignore?”

Follow-up to Statistically Significant Signals"

Biosurveillance Information Exchange Working Group

Reflections from San Diego County

Jeffrey Johnson, MPHSan Diego County Health & Human Services Agency

2/23/06

SAN DIEGO COUNTY

• Nearly 3 million population

• International border

• Large military presence

• Biotechnology Hub

• 21 Emergency Departments

Early Event Detection in San Diego

• Evolving effort since pre - 9/11

• Data sources: ER Visits, Paramedic transports, 911 calls, school surveillance, OTC sales

• Systems: Local SAS/Minitab system, ESSENCE, and BioSense

• Statistical Methods: Descriptive, time series, CUSUM, EWMA, process

control methods (P&U Charts)

• Multiple syndromes • Visualization and alerting• Incident Characterization• Follow-up to signals County of San Diego

Health & Human Services Agency

If We Ignore A Signal……

• We take no action or follow-up

• Save staff resources

• Avoid bothering hospital staff yet again

• Another data source may signal

• “The Feds may pick it up”

• Might lose an earlier start to a response

• We might be dead wrong to ignore

If We Do Not Ignore a Signal……

• Will it be another “false alarm”

• May detect an event earlier

• Earlier response

• Continued interaction with the medical community

• Gain experience with follow-up

• Increased situational awareness

Characterization of Detections

• Detection Method • Syndrome group• % Admitted• Deaths?• Geographic cluster?• Prior day’s level?• Recent level?• Age groups?• Severe syndrome?• Detections in other data sources?• Other epidemiological intelligence?• Other diagnostic information

Follow-up?

Action or

No Actionor

Watch

Detection Follow-up with Medical Community

What is the final diagnosis of Patients A, B, C?

Is there a common pattern among admitted patients?

Did any have lab test results that might suggest a larger event?

Among patients with a common zip code, was there a shared living setting or common exposure?

Can we send someone out to review medical charts?

What is your facility’s assessment of the situation?

County of San DiegoHealth & Human Services Agency

County of San DiegoHealth & Human Services Agency

Routine Surveillance Activities

Rule out system error

YES

NO

Preliminary evaluation

Describe initial results

True Positive

Aberrationdetected

Potential false positive

YES NO“False Positive”

Inform key divisional staff

Intensivemonitoring & surveillance

Evaluate otherdata sources

Cluster check

VERIFY

NOTIFYInform key departmental staff

IDENTIFY

Ignore?

Ignore?

Ignore?

Ignore?

0

10

20

30

40

50

60

70

80

90

10/31/2004 11/30/2004 12/31/2004 1/31/2005 2/28/2005 3/31/2005 4/30/2005 5/31/2005 6/30/2005 7/31/2005

Gastrointestinal 7 Day Moving Average

0

5

10

15

20

25

30

10/31/2004 11/30/2004 12/31/2004 1/31/2005 2/28/2005 3/31/2005 4/30/2005 5/31/2005 6/30/2005 7/31/2005

Gastrointestinal 7 Day Moving Average

0

5

10

15

20

25

10/31/2004 11/30/2004 12/31/2004 1/31/2005 2/28/2005 3/31/2005 4/30/2005 5/31/2005 6/30/2005 7/31/2005

Gastrointestinal/Genitourinary 7 Day Moving Average

GI Syndrome Over Time (10/31/04 – 8/24/05)

ED

911

Par

amed

ic

Ru

ns

The Significant Aspects of Syndromic Surveillance

• Statistical Significance

• Public Health Significance

• Significant Event

• Significant Public Awareness

• Significant Biological Agent Detection

HAZMAT FLAG – 12/04/2004

County of San DiegoHealth & Human Services Agency

Statistical significance vs. public health significance

County of San DiegoHealth & Human Services Agency

Statistical significance vs. public health significance

County of San DiegoHealth & Human Services Agency

Syndromic Surveillance for Natural Disasters

San Diego Wild Fires, 2003

Significant event with statistically significance outcomes

San Diego County

Syndromic surveillance for natural disasters

Significant event with statistically significance outcomes

San Diego County: Normalized Prehospital Transports and ED Visits with "Chest Pain" as Chief Complaint (12/31/03 - 10/08/04)

-3

-2

-1

0

1

2

3

4

5

12/31/03 1/31/04 2/29/04 3/31/04 4/30/04 5/31/04 6/30/04 7/31/04 8/31/04 9/30/04

No

rmal

ized

Co

un

t

ED Visits Ambulance Runs

San Diego County: Prehospital Transports and ED Visits with "Chest Pain" as Chief Complaint (12/31/03 - 10/08/04)

0

10

20

30

40

50

60

12/31/03 1/31/04 2/29/04 3/31/04 4/30/04 5/31/04 6/30/04 7/31/04 8/31/04 9/30/04

Co

un

t

ED Visits Ambulance Runs

“The Clinton Effect”

September 4, 2004

While spikes in both datasets are apparent, normalized counts show a relatively larger increase in ED visits on Sept. 6, 2004.

Significant Public Awareness

7/7/05 London Bombings

San Diego CountyParamedic Transports

for “Chest Pain”

Significant Public Awareness

Application of Syndromic Surveillance Agent:

Syndrome categoriesSpecific word search in CC or DX fields

Sensor site:Zip codes, population (schools)

Date:Temporal based surveillanceNew pre-detection baselines

Biowatch

BioWatch Detection• Tells us agent, sensor site and date• Plume plot may help us narrow surveillance on a geographic area

Significant BT Agent Detection

Anatomy of a Detection(a case example)

Daily Email Report

Attached Table

Feb 5, 2006 911 Call Data

911 Call Center - GI Syndrome Signal

Line listing for review

Non-specific call

complaints

21 Signals since 07/01/03

Various statistical signals

The count for the signals include a consistent range

911 Call Center - GI Syndrome Signal

What did we do?

• Magnitude of cases

• Which method(s) signaled?

• Check the other call centers

• Check the other data sources

(ED data, EMS transports)

• Review the line listing

• Our conclusion…..

…... 14 vs mean of 7.8

…... CUSUM (2), P-Chart, U-Chart

…… No signals

…… No Signals

……. No apparent pattern

>>>>> • Super Bowl Sunday • Fewer trauma calls • Smaller denominator (P-Chart)• Traditional increase in GI on

this day• Watch next day’s results

Case Example #2

Hospital 9 ED DataRespiratory Syndrome

Hospital 9 - Daily Results Table

0

5

10

15

20

25

30

35

40

1/1/2004

2/1/2004

3/1/2004

4/1/2004

5/1/2004

6/1/2004

7/1/2004

8/1/2004

9/1/2004

10/1/ 2004

11/1/2004

12/1/ 2004

1/1/2005

2/1/2005

3/1/2005

4/1/2005

5/1/2005

6/1/2005

7/1/2005

8/1/2005

9/1/2005

10/1/ 2005

11/1/2005

12/1/ 2005

1/1/2006

2/1/2006

Count Signal

Hospital 9 Respiratory Syndrome

01/01/04 - 02/03/06

Many signals….

So what’s the context?

Do we ever ignore the signals?

Hospital 9 Respiratory Syndrome

• 24 signals over a 37 day period• Count range: 11 – 34• Over time an increasing mean

Hospital 9 Influenza-like-illness (ILI) Syndrome

• “ILI syndrome” has greater syndrome specificity than “Respiratory” syndrome”

• 16 signals over a 37 day period

Greater Syndrome Specificity……

• S$gn&ls Happen!

• Make sure you see flames before yelling “Fire”

• CUSUM 2 & 3 STD may be too sensitive

• We lose precision with non-specific syndromes

• Everyone wants to know what’s going on all the time

• Increasing focus on situational awareness

• Further evaluation and testing required

What We Have Learned

Event or

Technology Trigger

Peak of

Inflated Expectations

Trough of

Disillusionment

Slope of

Enlightenment

Plateau of

Productivity

Hype Cycle of Emerging “Syndromic Surveillance” Technologies

Adapted from the Gartner Hype Cycle

Too many signals?, IT Costs, poor syndrome specificity, evaluation

results

Prioritized data sets, protocols in place,

The “magic bullet”

9/11, Anthrax attacks

Dual use, situational awareness, appropriate

signals

• More work in all areas of syndromic surveillance is needed

• Knowledge requires responsibility

• The enemy is studying our efforts

• Current/future funding levels require reliability, efficiency and sustainability of systems and approaches

• The Future:Neural networks and Artificial Intelligence (AI)?

• Are we ready?

Considerations

Contact InformationContact Information

Jeffrey Johnson619-531-4945

jeffrey.johnson@sdcounty.ca.gov

Thank You