Validation of Results Leveraging Navy Medicine’s Analytic Resources.

Validation of Results

Leveraging Navy Medicine’s Analytic Resources

Objectives

Compare analyst’s information to real world knowledge to assess reasonableness.

Review an annotated query panel to confirm analyst understanding and implementation.

Perform or review decompositions for reasonableness in appropriate dimensions.

Selectively exploit the M2 Data Dictionary and data caveat “blasters” for impact on validity.

22

Introduction

Critical Thinking – required of all producers and recipients of information.

Analysts’ powerful tools don’t substitute for managers’ assessment of validity.

In this block:Define validityMethods and illustrations of assessing validityStewardship of a scarce resource: analysts!

33

VALIDITY

Face Validity – the measuring method seems OK.

Construct Validity – either measuring the right thing, or something highly correlated with it.

External Validity – findings are likely to apply to “the bigger world”.

Reliability – Repeated measurements get similar answers.

44

Healthcare Analytics Newsletter

1. REAL WORLD KNOWLEDGE

55

FACTS

VALIDITY

ANALYSTINFORMATION

COMMONSENSE

ILLUSTRATION

HOW MANY ADMISSIONS LAST YEAR AT NAVY MTFs WERE RELATED TO OBSTETRICS?

6

ANALYST’S ANSWER: 286,727

ILLUSTRATION

7


MHS-WIDE, HOW MANY INPATIENTS IN A YEAR?

ILLUSTRATION

8


HAZARD A ROUGH GUESS FOR NAVY OB?

THE ANALYST ANSWER SEEMS ABOUT 7 TIMES TOO BIG!

2. REVIEWING ANNOTATED QUERY PANEL

99

• The analyst tells the computer what to extract via a query panel

• M2 then converts the display into a program which retrieves the data from the M2 database.

• Management can ask the analyst to provide both the query panel, and an explanation for it.

• BUT must also check for filters applied AFTER the data were extracted!

ILLUSTRATION

HOW MANY ACTIVE DUTY NAVY LIVE AROUND NAVAL HOSPITAL, JACKSONVILLE?

10


Used December, although January was available, as most recent month is usually low.Included both Navy, and Navy Afloat

Used Relationship Summary since all AD in it and didn’t need details

Included only the active duty

Residence zip in Jacksonville catchment area

OR enrolled to Jacksonville

Used December, although January was available, as most recent month is usually low.Included both Navy, and Navy Afloat

Used Relationship Summary since all AD in it and didn’t need details

FILTERING THE EXTRACTED DATA?

12

The analyst chose to

omit anyone

enrolled to a non-Navy,

non-contractor site that was not

near Jacksonville

FILTERING THE EXTRACTED DATA?

13

Check slice-and-dice also, since “ranking” can hide data!

RELIABILITY?

The analyst chose December because the most recent month is often immature and unreliable.

14

THE INFORMATION COST TRADE-OFF

15

ANALYST EFFORT

QC AND DOCUMENTINGPRECISION

3. DECOMPOSING RESULTS

1616

• Big sums can hide obvious errors!

• Manager or analyst can decompose, if stratifiers were or are retrieved with the data.

• Just “TLAR”, but at a more granular level.

ILLUSTRATION

HOW DO THE THREE SERVICES COMPARE ON AVERAGE LOS FOR SIMPLE PNEUMONIA?

17

ANALYST’S ANSWER:

Army: 2.36 days

Navy: 2.70 days

AF: 4.41 days

ILLUSTRATION

ANY SURPRISES IF DECOMPOSED BY GENDER?

18

F M

A 2.32 2.37

F 2.18 4.41

N 2.39 2.83

What about Air Force males would cause unusually long LOS?

F

1

F

2

F

3

F

4

M

1

M

2

M

3

M

4

A 2.17 3.00 2.24 2.58 2.14 2.23 2.30 2.47

F 1.85 1.80 2.19 2.64 2.00 1.95 2.66 10.48

N 1.83 1.67 2.32 4.40 4.00 2.17 2.44 3.11

ILLUSTRATION

ANY SURPRISES IF DECOMPOSED BY BENCAT?

19

What about Air Force active duty males would cause unusually long LOS?

Tmt DMIS ID Tmt DMIS ID Name Average LOS

0006 673RD MED GRP-JB ELMENDORF-RICHARDSON 2.00

0014 60TH MED GRP-TRAVIS 3.00

0073 81ST MED GRP-KEESLER 6.00

0079 99TH MED GRP-O'CALLAGHAN HOSP 1.33

0117 59TH MED WING-JB SAN ANTONIO LAF-RAF-FSH 15.59

0120 633RD MED GRP-JB LANGLEY-EUSTIS 2.00

0638 51ST MED GRP-OSAN AB 3.00

0639 35TH MED GRP-MISAWA 1.33

0808 31ST MED GRP-AVIANO 2.00

ILLUSTRATION

SURPRISES IF DECOMPOSED BY MTF (AF only)?

20

Why would Wilford Hall (59th Med Wing) have so much longer stays than any other MTF?

Tmt DMIS ID Tmt DMIS ID Name Record ID Admission Date Discharge Date Average LOS

0117 59TH MED WING-JB SAN ANTONIO LAF-RAF-FSH 6250055 10/09/2010 10/10/2010 1



























ILLUSTRATIONDECOMPOSED BY PATIENT (WH ONLY)?

21

4. EXPLOITING THE DOCUMENTATION

M2 DATA DICTIONARY WORLDWIDE BLASTERS

2222

EXPLOITING DOCUMENTATION

23

• AKIN TO HUNTING WHERE THE LIGHT IS, BECAUSE NOTHING CAN BE FOUND IN THE DARK.

• NEITHER THE DICTIONARY NOR THE BLASTERS ARE NEAR PERFECT. . .

• BUT THEY ARE THE BEST WE HAVE.

• THERE IS NO LIST ANYWHERE OF ALL THE PROBLEMS IN M2 DATA

ILLUSTRATION

24

Both are M2, both report on the same events, why do they differ by a million encounters?

ILLUSTRATION

25

• The M2 Data Dictionary shows that:o CAPERS have more procedure fields and so

will get different estimates of costs.o SADRs are not updated on the same

frequency as CAPERs, so the data are not equally fresh.

• Worldwide blasters were sent:• Cautioning on using the CAPERS, especially

costs.• But subsequent blasters have said that is

fixed.

ILLUSTRATION

26

• Neither Dictionary nor Blasters explain that:o Edits on CAPERS prevent many events from

being reported that are found in the SADRs.o SADRs use an imperfect key so that there are

duplicate records, where these duplicates are not in CAPERs.

Probably good to caveat any M2 answer as no one knows the extent and effect of validity problems in the data.

“According to the data in M2. . .”

Questions?

Rich Holmes

[email protected]

27

Validation of Results Leveraging Navy Medicine’s Analytic Resources.

Documents

Transcript of Validation of Results Leveraging Navy Medicine’s Analytic Resources.