1 University of Nebraska Medical Center Methods Used in Comparative Effectiveness Research September...

1

University of Nebraska Medical CenterUniversity of Nebraska Medical Center

Methods Used inMethods Used in

Comparative Effectiveness ResearchComparative Effectiveness ResearchSeptember 18, 2012September 18, 2012

byby

Susan D. Horn, Ph.DSusan D. Horn, Ph.DInstitute for Clinical Outcomes ResearchInstitute for Clinical Outcomes Research

699 East South Temple, Suite 300 699 East South Temple, Suite 300 Salt Lake City, Utah 84102Salt Lake City, Utah 84102

801-466-5595 (V) 801-466-6685 (F)801-466-5595 (V) 801-466-6685 (F)[email protected] [email protected] www.isisicor.com

• How best to treat the patient in your office right now?

• “Scientific studies” provide imperfect guidance

• Clinical medicine is untidy; innumerable variables describe patients, providers, and practices

• Must consider clinical variability of patient populations, intervention combinations, and outcomes

The Problem

2

1. Describe, compare, and contrast models to conduct comparative effectiveness research, including randomized controlled trials, analysis of claims databases, electronic medical record databases, condition or treatment registries, and practice-based evidence (PBE) study methodology.

2. Demonstrate how evidence-based practice and comparative effectiveness of treatments and treatment strategies can be derived using PBE study designs and health information systems, including clinical examples.

3. Present examples of PBE clinical findings related to quality, policy, and safety improvements.

Objectives of Presentation

3

• Comparative effectiveness research is designed to inform health-care decisions by providing evidence on the effectiveness, benefits, and harms of different treatment options.

• Evidence is generated from research studies that compare drugs, medical devices, tests, surgeries, or ways to deliver health care.

Comparative Effectiveness Research –What is it?

4

There are two ways that evidence is found:1.Systematic reviews of existing evidence. Researchers look at all available evidence about benefits and harms of each treatment choice for different groups of people from existing clinical trials, clinical studies, and other research.2.New studies. Researchers conduct studies that generate new evidence of effectiveness or comparative effectiveness of a test, treatment, procedure, or health-care service.

Comparative Effectiveness Research –What types of evidence are used?

5

We have Efficacy trials that determine whether an intervention produces a specified result(s) under well controlled conditions in a selected population – includes randomized controlled trials (RCTs)

We need Effectiveness trials that measure outcomes of an intervention under “real world” conditions in an unselected clinical population. Hypotheses and study designs of an effectiveness trial are formulated based on conditions of routine clinical practice and on outcomes essential for clinical decisions.

What We Have and What We Need

6

More DefinitionsMore Definitions

• EfficacyEfficacy – benefit under the best possible – benefit under the best possible circumstances (Can it work?)circumstances (Can it work?)» Generation - RCT, explanatoryGeneration - RCT, explanatory» Synthesis – systematic review of RCTs (e.g. Cochrane)Synthesis – systematic review of RCTs (e.g. Cochrane)

• EffectivenessEffectiveness – benefit under ordinary practice (Does – benefit under ordinary practice (Does it work?)it work?)» Generation – effectiveness trial, pragmatic trial, Generation – effectiveness trial, pragmatic trial,

observational studiesobservational studies» Synthesis – systematic review of RCTs and observational Synthesis – systematic review of RCTs and observational

studiesstudies

8

Effectiveness Trials and RCTsEffectiveness Trials and RCTs

RCTRCT

EffectivenessEffectiveness

Progenitor of Progenitor of RCTsRCTs

Practice effects Practice effects of RCT resultsof RCT results

9

Databases for Effectiveness Databases for Effectiveness TrialsTrials

• RCT databasesRCT databases

• Large claims databases, Large claims databases, e.g., Medicare, Medicaid, CDCe.g., Medicare, Medicaid, CDC

• HMO or VA databases from claims and electronic HMO or VA databases from claims and electronic medical recordsmedical records

• Specific condition registries such as arthritis Specific condition registries such as arthritis registryregistry

• Practice-based evidence study registriesPractice-based evidence study registries PBE studies overcome limitations of RCTs (that limit patient types and PBE studies overcome limitations of RCTs (that limit patient types and

treatments)treatments)

More detailed patient, process, and outcome evaluation than is More detailed patient, process, and outcome evaluation than is possible with traditional registries or large claims datasetspossible with traditional registries or large claims datasets

10

RCTRCT Databases Databases for Comparative Effectiveness Researchfor Comparative Effectiveness Research

• RCT databases RCT databases – – gold standard for efficacy and gold standard for efficacy and value of interventionsvalue of interventions

• Advantages includeAdvantages include::

High internal validityHigh internal validity

Causal inferences can be made since patients with Causal inferences can be made since patients with known confounders are excluded and randomization known confounders are excluded and randomization eliminates unknown confounderseliminates unknown confounders

11

RCTRCT Databases Databases for Comparative Effectiveness Research for Comparative Effectiveness Research

• RCT databases RCT databases – – gold standard for efficacy and value of gold standard for efficacy and value of interventionsinterventions

• Limitations includeLimitations include::

Small sample sizes – too small to detect uncommon risksSmall sample sizes – too small to detect uncommon risks

Follow-up periods too short to assess long-term benefits/risksFollow-up periods too short to assess long-term benefits/risks

Higher-risk patients typically excluded; limited external validityHigher-risk patients typically excluded; limited external validity

Level of monitoring more rigorousLevel of monitoring more rigorous than done in routine practice than done in routine practice

High rates of treatment discontinuationHigh rates of treatment discontinuation

12

Electronic Electronic DatabasesDatabases for Comparative Effectiveness Research for Comparative Effectiveness Research

• Three primary sources of electronic databasesThree primary sources of electronic databases::

Large health systems with electronic medical record data, Large health systems with electronic medical record data, e.g., HMO, HMO Research Network (HMORN), and VA e.g., HMO, HMO Research Network (HMORN), and VA databasesdatabases

Insurance claims databases, e.g., Medicare, Medicaid, Insurance claims databases, e.g., Medicare, Medicaid, HCUP dataHCUP data

Disease or procedure-specific registriesDisease or procedure-specific registries

13

Electronic Electronic Databases Databases for Comparative Effectiveness Research for Comparative Effectiveness Research


Identify and track patient populations over timeIdentify and track patient populations over time

Measure treatment exposures over timeMeasure treatment exposures over time

Assess some health status, health behaviors, and other Assess some health status, health behaviors, and other potential confounderspotential confounders

Assess both positive and negative outcomes over timeAssess both positive and negative outcomes over time

Better suited to evaluate safety as opposed to effectivenessBetter suited to evaluate safety as opposed to effectiveness

14

Insurance Claims Electronic Insurance Claims Electronic Databases Databases for Comparative Effectiveness Research for Comparative Effectiveness Research

• Many States and health care and pharmacy insurance Many States and health care and pharmacy insurance providers make administrative claims data available, providers make administrative claims data available, including Medicare, Medicaid, Healthcare Cost and including Medicare, Medicaid, Healthcare Cost and Utilization Project (HCUP), Blue Cross Blue Shield, Utilization Project (HCUP), Blue Cross Blue Shield, and United Health.and United Health.

• Databases contain information on all health care Databases contain information on all health care encounters in which billable services were deliveredencounters in which billable services were delivered

15



Cover millions of peopleCover millions of people

Ability to provide treatment exposures and adverse Ability to provide treatment exposures and adverse events, including hospitalizations and mortality, over events, including hospitalizations and mortality, over extended periods of timeextended periods of time

Provide population and subpopulation-based estimates Provide population and subpopulation-based estimates for various outcomesfor various outcomes

16



Functional and cognitive status, severity of illness, Functional and cognitive status, severity of illness, and health behaviors cannot be obtained and can be and health behaviors cannot be obtained and can be important unmeasured confoundersimportant unmeasured confounders

Possibility of exposure misclassification if patient Possibility of exposure misclassification if patient does not apply for insurance coverage for some does not apply for insurance coverage for some treatmenttreatment

17

Health System Electronic Health System Electronic Databases Databases for Comparative Effectiveness Research for Comparative Effectiveness Research

•Advantages includeAdvantages include::

Data are of high quality usuallyData are of high quality usually

Cover millions of persons including minority and Cover millions of persons including minority and elderly populationselderly populations

Exist in electronic form so Exist in electronic form so somesome data elements may be data elements may be exported for use in comparative effectiveness researchexported for use in comparative effectiveness research

18

Health System Electronic Health System Electronic Databases Databases for Comparative Effectiveness Research for Comparative Effectiveness Research


Restricted ability to capture patientsRestricted ability to capture patients’’ severity of illness, severity of illness, functional and cognitive status, health behaviors other functional and cognitive status, health behaviors other than smoking or pain.than smoking or pain.

Restricted access for CER unless researcher is part of Restricted access for CER unless researcher is part of organization that owns the dataorganization that owns the data

Often required variables are in text so are not Often required variables are in text so are not exportableexportable

19

Clinical Registry Electronic Clinical Registry Electronic Databases Databases for Comparative Effectiveness Research for Comparative Effectiveness Research

• Systematically collected and stored health-related information Systematically collected and stored health-related information on specific patient populations, most often defined by a on specific patient populations, most often defined by a particular illness or procedure.particular illness or procedure.


Designed to collect detailed information related to a particular illness or Designed to collect detailed information related to a particular illness or procedure, e.g., arthritis registry, CABG registry, cancer registriesprocedure, e.g., arthritis registry, CABG registry, cancer registries

• Limitations includeLimitations include: :

Typically created as an add-on or separate database from those used for Typically created as an add-on or separate database from those used for clinical care or paymentclinical care or payment

Limited information outside of particular illness or proceduLimited information outside of particular illness or procedure re

• Minimizing Bias in Existing Data

• Capturing variation in intervention combinations

• Capturing variation in outcomes

The Problems

20

The Balancing ActThe Balancing Act

Experimental

Non-experimental

Different than routine care

Outcomes clearly defined

Balanced groups

Large sample size

Real world settings

Confounded by lifeStrong internal validityStrong external validityLonger follow-up

Defined patient population

21

Causal Inference ComparisonCausal Inference Comparison

Randomized Experiment Quasi-Experiment

Cause precedes effect

Yes Yes

Cause covaries with effect

Yes Yes

Alternate explanations implausible

Yes ?

22

Basic Problems in Basic Problems in Non-Randomized StudiesNon-Randomized Studies

• Non-independence of ObservationsNon-independence of Observations

« Many statistical analyses are based on the assumption Many statistical analyses are based on the assumption that the observations are independent.that the observations are independent.

« Studying patients treated in a single hospital or unit or Studying patients treated in a single hospital or unit or provider breaks this assumptionprovider breaks this assumption

« Intra-correlation among observationsIntra-correlation among observations23

Modeling to Account for Modeling to Account for Non-IndependenceNon-Independence

• Hierarchical designs that account for the Hierarchical designs that account for the intra-correlation intra-correlation

• Generalized estimating equation (GEE) Generalized estimating equation (GEE) regression models for clustered dataregression models for clustered data

• Robust standard errorsRobust standard errors

24

Basic Problems in Basic Problems in Non-Randomized Studies Non-Randomized Studies

• Confounding by IndicationConfounding by Indication» Therapies administered in non-random fashionTherapies administered in non-random fashion

» Prognostic characteristics influence therapyPrognostic characteristics influence therapy

» Recipients of therapy at high risk for outcomesRecipients of therapy at high risk for outcomes

» Users differ from comparators in key respectsUsers differ from comparators in key respects

25

Propensity Score TheoryPropensity Score Theory

• Multivariable scoring method that collapses Multivariable scoring method that collapses multiple multiple observed observed predictors of treatment into a predictors of treatment into a single value (a score)single value (a score)» Probability that a subject with given Probability that a subject with given

characteristics receives specified treatmentcharacteristics receives specified treatment» Removes confounding by components of the Removes confounding by components of the

scorescore» Used to: match, stratify, or modelUsed to: match, stratify, or model

26

Instrumental Variable Analysis-Instrumental Variable Analysis-Theory Theory

• Identify instrument variables that are randomly Identify instrument variables that are randomly associated with individual case, correlated with associated with individual case, correlated with treatment, but uncorrelated with outcomestreatment, but uncorrelated with outcomes

• Therefore, the instrument can effectively Therefore, the instrument can effectively randomize subjects across treatment arms to randomize subjects across treatment arms to achieve equal distribution achieve equal distribution

• Controls for underlying differences in groups that Controls for underlying differences in groups that are are unobservableunobservable (endogeneity bias due to (endogeneity bias due to unobserved heterogeneity)unobserved heterogeneity)

• Supposedly permits estimation of causal effects Supposedly permits estimation of causal effects even when important confounders are unmeasuredeven when important confounders are unmeasured

27

28

Instrumental Variables (IV) to address Instrumental Variables (IV) to address confounding by indication/selection biasconfounding by indication/selection bias

• Used to estimate effect of missing predictor variable or when Used to estimate effect of missing predictor variable or when there is measurement error in predictor variablethere is measurement error in predictor variable

• Instrument itself does not belong in prediction equation but is Instrument itself does not belong in prediction equation but is correlated with missing predictor variable. For example,correlated with missing predictor variable. For example,

• Severity of illness measure is missing; instead use distance Severity of illness measure is missing; instead use distance from patient home to treatment center as IVfrom patient home to treatment center as IV

• No measure of smoking in a community; instead use No measure of smoking in a community; instead use amount of tobacco taxes collected as IV to assess smoking amount of tobacco taxes collected as IV to assess smoking effect on health outcome.effect on health outcome.

• Disadvantage: Assumptions about IV not testable Disadvantage: Assumptions about IV not testable

29

Overcoming Selection Bias/ Overcoming Selection Bias/ Confounding by IndicationConfounding by Indication

• Statistical adjustments:Statistical adjustments:–MatchingMatching–Propensity scoring/instrumental variablesPropensity scoring/instrumental variables–Covariate adjustments (Severity of Illness)Covariate adjustments (Severity of Illness)

• Ongoing debate about the adequacy of Ongoing debate about the adequacy of adjustmentsadjustments

30

PBE Presentation OverviewPBE Presentation Overview

• What is practice-based evidence (PBE) study methodology?What is practice-based evidence (PBE) study methodology?

• How does it incorporateHow does it incorporate patient heterogeneitypatient heterogeneity treatment heterogeneitytreatment heterogeneity outcome heterogeneityoutcome heterogeneity

• Discuss examples of PBE comparative effectiveness Discuss examples of PBE comparative effectiveness findingsfindings

• Discuss implications for health information technologyDiscuss implications for health information technology

31

What makes this approach What makes this approach different?different?

PBE Methodology

31

• Both patients and providers report data

• Data come from existing EMR with standardized data elements about patient characteristics, treatments, processes, patient-reported data, and multiple outcomes

• Data are part of routine documentation, so not an ‘add-on’

• Rapid patient accrual since documentation is standard of care

• Longitudinal and ongoing

Comparative Effectiveness Issues AddressedUsing PBE

32

• Patient comparability is addressed with Comprehensive Severity Index (CSI): disease-specific, physiologic-based, >2,200 criteria, >5,500 disease-specific criteria sets

• CSI addresses confounding by indication/selection bias

• Database includes all treatments with date/dose/intensity/route

• Can assess drug and non-drug combination therapies

• Findings of PBE-CER more readily translated into practice

Comparative Effectiveness Issues Addressed Using PBE (cont)

33

34

Practice Based Evidence (PBE) DesignPractice Based Evidence (PBE) Design

PBE is observational research that: PBE is observational research that:

» captures differential outcomes captures differential outcomes

» associated with naturally occurring variations in associated with naturally occurring variations in

treatmenttreatment

» while adjusting for severity of illness and injury, while adjusting for severity of illness and injury,

co-morbid conditions, and co-occurring treatmentsco-morbid conditions, and co-occurring treatments

34

35

Practice-Based Evidence Study DesignPractice-Based Evidence Study Design

Process FactorsProcess Factors• Management StrategiesManagement Strategies• InterventionsInterventions• MedicationsMedications

Patient FactorsPatient Factors• Psychosocial/demographic FactorsPsychosocial/demographic Factors• Disease(s)Disease(s)• Severity of Disease(s)Severity of Disease(s)

› physiologic signs and symptomsphysiologic signs and symptoms• Genetic informationGenetic information• Measured at Multiple Points in TimeMeasured at Multiple Points in Time

Outcomes of interestOutcomes of interest• Clinical Clinical • Health StatusHealth Status• FunctionalFunctional• Cost/LOS/EncountersCost/LOS/Encounters• ProductivityProductivity

Standardize documentation of :Standardize documentation of :

Control for:Control for:

Measure:Measure:

36

7 Signature Features of 7 Signature Features of PBE StudiesPBE Studies

1.1. Hypotheses can be focused or broadHypotheses can be focused or broad

2.2. All interventions are considered to determine relative All interventions are considered to determine relative contribution of each contribution of each

3.3. Broad patient selection criteria maximize generalizability Broad patient selection criteria maximize generalizability and external validityand external validity

4.4. Detailed characterization of the patient by robust Detailed characterization of the patient by robust measures of patient severity, genetic information, and measures of patient severity, genetic information, and functional statusfunctional status

37

7 Signature Features of 7 Signature Features of PBE StudiesPBE Studies

5.5. Patient differences controlled statistically rather than Patient differences controlled statistically rather than through randomization through randomization

6.6. Facility and clinical/patient buy-in through use of trans-Facility and clinical/patient buy-in through use of trans-disciplinary Clinical Practice Teamdisciplinary Clinical Practice Team

7.7. Strength of evidence built through the research processStrength of evidence built through the research process

PBE findings are more PBE findings are more generalizablegeneralizable and and transportabletransportable than than RCT findingsRCT findings

38

PBE PBE Signature Feature: Interventions Signature Feature: Interventions

2.2. Consider all interventions Consider all interventions to determine relative to determine relative contribution of each. contribution of each.

Uses a detailed characterization of the care Uses a detailed characterization of the care process through a well-designed point-of-care process through a well-designed point-of-care (POC) documentation system(POC) documentation system

– User-defined and user friendlyUser-defined and user friendly– Time sensitive characterization of all Time sensitive characterization of all

interventionsinterventions

39

PBE PBE Signature Feature: Patients Signature Feature: Patients

4.4. Detailed characterization of the patient Detailed characterization of the patient by robust by robust measures of individual severity and functional statusmeasures of individual severity and functional status

Includes Comprehensive Severity Index (CSIIncludes Comprehensive Severity Index (CSI®®))

– Over 2,200 condition-specific signs, symptoms, Over 2,200 condition-specific signs, symptoms, and physical findingsand physical findings

– Continuous score: 0 Continuous score: 0 ∞ ∞– Admission, discharge, maximum during stay, visitAdmission, discharge, maximum during stay, visit

Genetic informationGenetic information Includes Functional Independence Measure (FIM) Includes Functional Independence Measure (FIM)

and/or other measures of functional statusand/or other measures of functional status

40

PBE PBE Signature Feature: Signature Feature: Clinical Practice Team Clinical Practice Team

6.6. Facility and clinical buy-in Facility and clinical buy-in through use of through use of transdisciplinary Clinical Practice Team thattransdisciplinary Clinical Practice Team that:: Develops and frames the questionsDevelops and frames the questions Defines variablesDefines variables Gathers dataGathers data Interprets dataInterprets data Implements findingsImplements findings Fosters clinical and individual buy-in (bottom-up)Fosters clinical and individual buy-in (bottom-up) Facilitates knowledge translationFacilitates knowledge translation

41

PBE PBE Signature Feature: Signature Feature: Strength of Evidence Strength of Evidence

7.7. Strength of evidence Strength of evidence built through the research built through the research processprocess Added confounders preserve the significant Added confounders preserve the significant

associationassociation■ A change in outcomes follows a change in treatment A change in outcomes follows a change in treatment

as predicted by the PBE modelas predicted by the PBE model■ Repeated studies on the same topic yield similar Repeated studies on the same topic yield similar

findingsfindings

42

PBE HallmarksPBE Hallmarks

• Decisions are made by Decisions are made by front-line clinicians front-line clinicians vs. vs. researchersresearchers

• ““Bottom-upBottom-up”” vs. vs. ““Top-downTop-down”” approach approach

• Guidance from researchers (scientific Guidance from researchers (scientific advisory board) and patient experienceadvisory board) and patient experience

42

4343

PBE HallmarksPBE Hallmarks

–Non-experimentalNon-experimental: Follows outcomes of : Follows outcomes of treatments actually prescribedtreatments actually prescribed

–InclusiveInclusive: Uses patient populations undergoing : Uses patient populations undergoing routine clinical careroutine clinical care

–PragmaticPragmatic: Uses actual clinical outcomes: Uses actual clinical outcomes

–LowerLower CostCost than RCTsthan RCTs

–FasterFaster than RCTsthan RCTs

44


• High external validityHigh external validity

- Includes essentially all patients with specific Includes essentially all patients with specific conditioncondition

- Captures confounders that could affect relevant Captures confounders that could affect relevant treatment responses. treatment responses.

- Reduces accidental associations between Reduces accidental associations between treatments and outcomestreatments and outcomes..

45


Process FactorsProcess Factors• Management StrategiesManagement Strategies• InterventionsInterventions• MedicationsMedications

Patient FactorsPatient Factors• Psychosocial/demographic/history factorsPsychosocial/demographic/history factors• Disease(s)Disease(s)• Severity of Disease(s)Severity of Disease(s)

› physiologic signs and symptomsphysiologic signs and symptoms• Genetic informationGenetic information• Multiple Points in TimeMultiple Points in Time

Outcomes of interestOutcomes of interest• Clinical Clinical • Health StatusHealth Status• FunctionalFunctional• Cost/LOS/EncountersCost/LOS/Encounters• ProductivityProductivity

Standardize documentation of :Standardize documentation of :

Control for:Control for:

Measure:Measure:

46

Examples of Severity Systems to address Examples of Severity Systems to address Selection Bias/Confounding by IndicationSelection Bias/Confounding by Indication

Diagnostic/Procedure Based SystemsDiagnostic/Procedure Based Systems

• Clinical definition of severityClinical definition of severity

• Body Systems Count Body Systems Count

•Charlson Comorbidity Index (Charlson Comorbidity Index (1-yr death)1-yr death)

• 3 Disease Staging (hosp death)3 Disease Staging (hosp death)

• Patient Management Categories (hosp Patient Management Categories (hosp death)death)

• Resource-based definition of severityResource-based definition of severity

•Acuity Index Method (LOS)Acuity Index Method (LOS)

•APR DRGs (hosp $)APR DRGs (hosp $)

•Patient Management Categories (hosp $)Patient Management Categories (hosp $)

•Refined DRGs (hosp LOS, $)Refined DRGs (hosp LOS, $)

Physiologic/Clinically Based Physiologic/Clinically Based SystemsSystems

• 2 Apache II & III (ICU death)2 Apache II & III (ICU death)

• 2 Medisgroups (Atlas) (hosp death)2 Medisgroups (Atlas) (hosp death)

CSICSI®®

14

Comprehensive Severity Index (CSIComprehensive Severity Index (CSI®®)) used to avoid selection bias or confounding by indicationused to avoid selection bias or confounding by indication

• Severity defined as Severity defined as ““physiologic complexity physiologic complexity presented to medical presented to medical personnel due to the extent and interactions of a patientpersonnel due to the extent and interactions of a patient’’s diseasess diseases””

• Disease-specificDisease-specific: : 5,500 disease-specific groups; over 2,200 distinct criteria. 5,500 disease-specific groups; over 2,200 distinct criteria. ICD-9 codes trigger disease-specific patient signs, symptoms, and physical ICD-9 codes trigger disease-specific patient signs, symptoms, and physical findings used to score disease-specific and overall severity levelsfindings used to score disease-specific and overall severity levels

• No treatments usedNo treatments used as criteria as criteria

• ComprehensiveComprehensive (all diseases) (all diseases)• Clinically credibleClinically credible: : computes disease-specific and overall severity levelscomputes disease-specific and overall severity levels

• Can measure severity at Can measure severity at multiple time points multiple time points

• Allows statistical comparison Allows statistical comparison of interventions without confounding by of interventions without confounding by

severity of illnessseverity of illness47

48

Characteristics of the CSI SystemCharacteristics of the CSI System

Developed as an addition to ICD-9-CMDeveloped as an addition to ICD-9-CM

Explicit severity criteria established for each ICD-9-CM Explicit severity criteria established for each ICD-9-CM codecode

Severity criteria result in a continuous severity score for Severity criteria result in a continuous severity score for each diagnosiseach diagnosis

Failure to meet enough criteria results in a severity rating of Failure to meet enough criteria results in a severity rating of zerozero

Overall patient severity computed as continuous score, Overall patient severity computed as continuous score, taking all diagnoses into accounttaking all diagnoses into account

49

CSI Severity IndicatorsCSI Severity Indicators

Physiological signs and symptoms of a Physiological signs and symptoms of a diseasedisease- Vital signs- Vital signs

- Laboratory values- Laboratory values

- Radiology findings- Radiology findings

- Other physical findings- Other physical findings

Severity indicators are specific to each Severity indicators are specific to each disease based on ICD-9-CM codingdisease based on ICD-9-CM coding

50

Pneumonia Criteria SetPneumonia Criteria Set480.0-486; 506.3; 507.0-507.1; 516.8; 517.1; 518.3; 518.5; 668.00-668.04; 997.3; 112.4; 136.3; 055.1480.0-486; 506.3; 507.0-507.1; 516.8; 517.1; 518.3; 518.5; 668.00-668.04; 997.3; 112.4; 136.3; 055.1

CATEGORY 1 2 3 4 Cardiovascular

pulse rate 51-100; ST segment changes-EKG; systolic BP 90mmHg

pulse rate 100-129; 41-50; PACs, PAT, PVCs-EKG; systolic BP 80-89mmHg

pulse rate 130; 31-40; systolic BP 61-79mmHg

pulse rate 30; asystole, VT, VF, V flutter; systolic BP 60 mmHg

Fever 96.8-100.4 and/or chills 100.5-102.0 oral; 94.0-96.7

102.1-103.9; 90.1-93.9 and/or rigors

104.0 90.0

Labs ABGs Hematology

pH 7.35-7.45

pO2 61mmHg

WBC 4.5-11.0K/cu mm; bands <10%;

pH >7.46 7.25-7.34

WBC 11.1-20.0K/cu mm; 2.4-4.4K/cu mm; bands 10-20%

pH 7.10-7.24

pO2 51-60mmHg WBC 20.1-30.0K/cu mm; 1.0-2.3K/cu mm; bands 21-40%

pH 7.09;

pO2 50mmHg

WBC 30.1K/cu mm; 1.0K/cu mm; bands 40%

Neuro Status

Lowest Glasgow coma score

12

chronic confusion

9-11

acute confusion

6-8

unresponsive

5

Radiology Chest X-Ray or CT Scan

infiltrate and/or consolidation in 1 lobe; pleural effusion

infiltrate and/or consolidation in >1 but 3 lobes;

infiltrate and/or consolidation in >3 lobes; cavitation or lung necrosis

Respiratory

white, thin, mucoid sputum

dyspnea on exertion; stridor; rales 50%/3 lobes; decreased breath sounds 50%/3 lobes; positive for fremitus; stridor hemoptysis NOS; blood tinged or purulent or frothy sputum

cyanosis present dyspnea at rest; rales >50%/ 3 lobes; decreased breath sounds >50%/ 3 lobes frank hemoptysis

apnea absent breath sounds >50%/ 3 lobes

Copyright 2006. Susan D. Horn. All rights reserved. Do not quote, copy or cite without permission.

51

Uses of PBE FindingsUses of PBE Findings

Create protocols depending on patient Create protocols depending on patient characteristics that result in better outcomescharacteristics that result in better outcomes

Evaluate treatments or programsEvaluate treatments or programs

PBE findings can be used to

52

Examples of PBE Study FindingsExamples of PBE Study Findings

• Children Hospitalized with RSVChildren Hospitalized with RSV. . Birth at 33-35 weeks Birth at 33-35 weeks GA is significantly associated with higher intubation GA is significantly associated with higher intubation rates, longer ICU stays, and longer hospital LOS rates, longer ICU stays, and longer hospital LOS (prompted guideline change for prophylaxis).(prompted guideline change for prophylaxis).

• Low-Level StrokeLow-Level Stroke. . More time spent in high-level More time spent in high-level rehabilitation activities, such as gait, upper extremity rehabilitation activities, such as gait, upper extremity control, and problem solving in the first three hours of control, and problem solving in the first three hours of therapy is significantly associated with higher total, therapy is significantly associated with higher total, motor, and cognitive FIM at discharge.motor, and cognitive FIM at discharge.

53

Pediatric Bronchiolitis StudyPediatric Bronchiolitis StudyLength of StayLength of Stay

4.34

54.3 3.5

7.1

4.8

6.2

4.5

3.6

0

1

2

3

4

5

6

7

8

Site 6

Site 5

Site 7

Site 9

Site 4

Site 1

Site 3

Site 8

Site 2

Site 10

DayDayss

54

Pediatric Bronchiolitis StudyPediatric Bronchiolitis StudyCostCost

8,839

4,908

10,041

4,522

12,373

8,9349,342

6,097

4,122

$0

$2,000

$4,000

$6,000

$8,000

$10,000

$12,000

$14,000

Site 6

Site 5

Site 7

Site 9

Site 4

Site 1

Site 3

Site 8

Site 2

Site 10

55

Pediatric Bronchiolitis StudyPediatric Bronchiolitis StudyOutcome = Length of Stay n=804 ROutcome = Length of Stay n=804 R2=2=.62.62

- Age in months (.0006)- Age in months (.0006)

+ MCSIC (.0001)+ MCSIC (.0001)

+ O+ O22 used (.01) used (.01)

+ Steroids (.0001)+ Steroids (.0001)

+ Antibiotics (.0001)+ Antibiotics (.0001)

+ Intubation (.0001)+ Intubation (.0001)

+ Lasix (.0001)+ Lasix (.0001)

+ Interaction of chest + Interaction of chest physiotherapy and atelectasis physiotherapy and atelectasis (.0001)(.0001)

AssessmentAssessment ProceduresProcedures

56

Pediatric Bronchiolitis StudyPediatric Bronchiolitis StudyOutcome = Cost n=722 ROutcome = Cost n=722 R2=2=.73.73

- Age in months (.0001)- Age in months (.0001)

+ MCSIC (.0001)+ MCSIC (.0001)

+ Admitted to PICU (.0001)+ Admitted to PICU (.0001)+ Arterial line (.04)+ Arterial line (.04)+ Central line (.003)+ Central line (.003)+ Continuous nebulization (.0002)+ Continuous nebulization (.0002)+ Interaction: chest pt & atelectasis (.005)+ Interaction: chest pt & atelectasis (.005)+ Intubation (.0001)+ Intubation (.0001)+ Ipratropium bromide (.005)+ Ipratropium bromide (.005)+ Lasix (.0001)+ Lasix (.0001)+ Ribavirin (.0001)+ Ribavirin (.0001)+ Steroids (.0003)+ Steroids (.0003)

AssessmentAssessment ProceduresProcedures

Willson, et al. PEDIATRICS 2001;108(4):851-855.

57

Prematurity and RSV Prematurity and RSV Hospital OutcomesHospital Outcomes

< < 32 wks32 wks 33-35 wks33-35 wks 36 wks36 wks > > 37 wks37 wks p-valuep-value

IntubationIntubation 21.4%21.4% 38.7%38.7% 20%20% 12.1%12.1% 0.0020.002

ICU LOSICU LOS 5.8 days5.8 days 7.7 days7.7 days 4.2 days4.2 days 3.8 days3.8 days 0.0210.021

Hospital LOSHospital LOS 6.8 days6.8 days 8.4 days8.4 days 4.9 days4.9 days 4.1 days4.1 days <0.0001<0.0001

Admitted to ICUAdmitted to ICU 39.3%39.3% 48.4%48.4% 30.0%30.0% 27.9%27.9% 0.1010.101

HX of Hosp. for HX of Hosp. for RSV /BronchiolitisRSV /Bronchiolitis 14.3%14.3% 16.1%16.1% 6.7%6.7% 6.1%6.1% 0.1370.137

Significant Differences by Gestational Age GroupsSignificant Differences by Gestational Age Groups33-35 week GA infants had highest hospital resource use33-35 week GA infants had highest hospital resource use

58

RSV Hospital Outcomes and Policy ChangesRSV Hospital Outcomes and Policy Changes

ConclusionsConclusions• 33-35 week GA infants had highest hospital resource use33-35 week GA infants had highest hospital resource use

• 36 week infants have risk similar to full term infants36 week infants have risk similar to full term infants

• Changed guidelines for immunoprophylaxis Changed guidelines for immunoprophylaxis for 33-35 week for 33-35 week infantsinfants

• Changed guidelines for intubation Changed guidelines for intubation – try – try ‘‘stimulatingstimulating’’ first first

59

Post-Stroke Rehabilitation StudyPost-Stroke Rehabilitation Study

2001 – 2003; 1,161 patients2001 – 2003; 1,161 patients

PBE study designed to discover what combinations of PBE study designed to discover what combinations of

medical devices, therapies, medications, feeding medical devices, therapies, medications, feeding

approaches, and their interactions worked best for approaches, and their interactions worked best for

specific types of stroke patients treated in real-specific types of stroke patients treated in real-

world practices.world practices.

Study Objectives

60

• PhysiciansPhysicians

• NursesNurses

• Social WorkersSocial Workers

• PsychologistsPsychologists

• Physical TherapistsPhysical Therapists

• Occupational TherapistsOccupational Therapists

• Recreation TherapistsRecreation Therapists

• Speech/Language Speech/Language PathologistsPathologists

Post-Stroke Rehabilitation StudyPost-Stroke Rehabilitation Study

Trans-Disciplinary Project Clinical TeamTrans-Disciplinary Project Clinical Team

61

Outcome: Discharge Motor FIMOutcome: Discharge Motor FIMSevere Stroke – Full Stay

GeneralGeneral AssessmentAssessment

GeneralGeneralInterventionsInterventions

MedicationsMedications

PT PT InterventionsInterventions

– Age

– Black race

+ Mild motor impairment

+ Admission Motor FIM

+ Admission Cognitive FIM

– Days onset to rehab

+ Enteral feeding

– – Anti-ParkinsonsAnti-Parkinsons

– – ModafinilModafinil

–– Old SSRIsOld SSRIs

++ Atypical antipsychotics Atypical antipsychotics

– Formal assessment

– Bed mobility

+ Gait

+ Advanced gait

OT OT InterventionsInterventions

+ Home management

SLP SLP InterventionsInterventions

– Swallowing

– Orientation

+ Reading comprehension

62

Outcome: Discharge Motor FIMOutcome: Discharge Motor FIMSevere Stroke–1st 3 hour Therapy block only

GeneralGeneral AssessmentAssessment

GeneralGeneralInterventionsInterventions

MedicationsMedications

PT PT InterventionsInterventions

– Age – Severe motor impairment+ Admission Motor FIM+ Admission Cog. FIM+ No Dysphagia+ Neurotropic Impairments treated with meds

– Days onset to rehab

+ LOS

+ Enteral feeding

– – Other AntidepressantOther Antidepressant

–– Old SSRIsOld SSRIs

++ Atypical antipsychotics Atypical antipsychotics

– Bed mobility time in 1time in 1stst 3 hrs 3 hrs

+ Gait time in 1in 1stst 3 3 hrshrs

+ Advanced gait time in 1time in 1stst 3 hrs 3 hrs

OT OT InterventionsInterventions

+ Home management

SLP SLP InterventionsInterventions

Horn et al., Arch Phys Med Rehabil 2005;86(12 Supplement 2):S101-S114

Policy Changes from Stroke PBE Study

Early Rehabilitation Admission – get patient into

rehabilitation as soon as possible after stroke onset;

possibly start in Neuro ICU

Early gait in PT – start gait as soon as possible after rehab admission; put patient in harness on treadmill for safety

Early Feeding – continue or start enteral nutrition at rehab admission if patient is not able to eat full meals

Use Opioids for Pain – continue or start opioids at rehab admission if patient misses therapy due to pain 63

RCTs, Observational Cohort Studies,RCTs, Observational Cohort Studies, & PBE Compared & PBE Compared

Feature RCT Observational Cohort PBE

Researcher control of variables

Experimental Observational Observational

Time dimension Prospective Prospective or retrospective Prospective

Intervention(s)

1 or 2 discrete interventions; standardized protocols

1 or many; often a more global intervention, e.g., a program of interventions

All interventions deemed relevant; documented separately and in great detail so that one can examine interactions among interventions

Hypotheses Well-specified Specified, general, or none Focused or broad

Data source Primary dataPrimary data and/or large administrative data sets

Primary data supplemented by abstraction from clinical records

Protocol- vs. data- intensity

Protocol-intensiveNot inherently protocol- or data-intensive

Data intensive

Archives Physical Medicine & Rehabilitation 2012.


Feature RCT Observational Cohort PBE

Exclusion criteria

Extensive exclusions to minimize variation

Minimal exclusions (permits studying specific subsets of patients)

Minimal exclusions (permits studying specific detailed subsets of patients)

Sample sizeTypically small (e.g., <200) because of narrow hypothesis

Usually large Large (e.g., >1,500)

Control for participant differences

Through exclusions and randomization

Through propensity scoring,instrumental variable (IV) analysis, and statistical control

Thorough detailed character-ization of comprehensive patient severity of illness and other relevant characteristics, and statistical control

BlindingSingle, double, or triple blinding

No No

Outcomes Few Few or manyMany- appropriate to study population

Effect size Often small Small or largeUsually small when samples are large



Feature RCTObservational

Cohort PBE

Attitude to Confounders

Not interesting; exclude them

Affect outcomes but may not be measured so hard to control

Affect outcomes, measure as many as suggested by clinical team, and are interesting

ValidityInternal—highExternal—low

Internal—lowExternal—moderate

Internal—moderate to highExternal—high

Causality Assigned AssumedAssumed; with drill-down analyses, implementation, and repeat studies can move close to ‘assigned’

Ability to examine treatment effects on subgroups

Limited, unless pre-designed and powered for subgroup analyses

More likely because of large data sets, but subgroup analyses may be limited by patient control strategies such as propensity scoring

Very likely since measure details about patients that are used to create specific subsets

Research culture (1) Top-down Varies Highly collaborative; bottom-up



Feature RCTObservational

Cohort PBE

Research culture (2)Not depend on local knowledge

VariesLocal knowledge highly valued and sought

Knowledge translation Less buy-in VariesHigh level of buy-in; findings more “transporatable”

Science of …. Confirmation ConfirmationConfirmation, discovery, and innovation

Science of …. Efficacy AssociationsAssociations and effectiveness

Cost High Varies Moderate


1 University of Nebraska Medical Center Methods Used in Comparative Effectiveness Research September...

Documents

Transcript of 1 University of Nebraska Medical Center Methods Used in Comparative Effectiveness Research September...