1 04/21/23

Psychometric Evaluation Psychometric Evaluation and Calibration Planand Calibration Plan

Ron D. Hays, Ph.D. Ron D. Hays, Ph.D.

October 14, 2006 October 14, 2006

ISOQOL (Opala)ISOQOL (Opala)

2:00-3:30 pm2:00-3:30 pm

2 04/21/23

PROMIS Domains

Physical functioning (Hays/Bjorner)

Pain (Revicki/Cook)

Fatigue (Lai)

Emotional distress (Choi/Reise)

Social/role participation (Bode/Hahn)

3 04/21/23

Datasets

Cancer Item Banks (Northwestern)

Digitalis Investigation Group Study--randomized double-blind placebo-controlled trial evaluating effect of digoxin on mortality in 581 patients with heart failure and sinus rhythm.

IMMPACT--internet-based survey of individuals with chronic pain from the American Chronic Pain Association website

Medical Outcomes Study--observational study of persons with hypertension, diabetes, heart disease, and/or depression in Boston, Chicago, and Los Angeles

WHOQOL-100 data (n = 442 from U.S. field center)

4 04/21/23

Types of AnalysesTypes of Analyses

• Classical Test Theory StatisticsClassical Test Theory Statistics

• IRT Model AssumptionsIRT Model Assumptions

• Model FitModel Fit

• Differential Item FunctioningDifferential Item Functioning

• Item CalibrationItem Calibration

5 04/21/23

Classical Test Theory StatisticsClassical Test Theory Statistics

• Out of rangeOut of range

• Item frequencies and distributionsItem frequencies and distributions

• Inter-item correlationsInter-item correlations

• Item-scale correlationsItem-scale correlations

• Internal consistency reliabilityInternal consistency reliability

6 04/21/23

IRT Model AssumptionsIRT Model Assumptions

• (Uni)dimensionality(Uni)dimensionality

• Local independenceLocal independence

• MonotonicityMonotonicity

7 04/21/23

Sufficient UnidimensionalitySufficient Unidimensionality

• Confirmatory factor modelsConfirmatory factor models

One factorOne factor

Bifactor (general and group factors)Bifactor (general and group factors)

8 04/21/23

Local IndependenceLocal Independence

• After controlling for dominant factor(s), item After controlling for dominant factor(s), item pairs should not be associated.pairs should not be associated.

Look at residual correlations (> 0.20)Look at residual correlations (> 0.20)

9 04/21/23

MonotonicityMonotonicity

• Probability of selecting a response category Probability of selecting a response category indicative of better health should increase as indicative of better health should increase as underlying health increases.underlying health increases.

• Item response function graphs withItem response function graphs with

y-axis: proportion positive for item stepy-axis: proportion positive for item step

x-axis: raw scale score minus item scorex-axis: raw scale score minus item score

10 04/21/23

11 04/21/23

Category Response Curves for Category Response Curves for Samejima’s Graded Response ModelSamejima’s Graded Response Model

0.0

0.2

0.4

0.6

0.8

1.0

-3 -2 -1 0 1 2 3

Theta (q)

P(X

=k|q

)

P (X = 3|q)P (X = 1|q)

P (X = 2|q)

P (X = 4|q)

12 04/21/23

Model FitModel Fit

• Compare observed and expected response Compare observed and expected response frequencies by item and response categoryfrequencies by item and response category

• Items that do not fit and less discriminating Items that do not fit and less discriminating items identified and reviewed by content items identified and reviewed by content expertsexperts

13 04/21/23

Differential Item FunctioningDifferential Item Functioning

• Uniform DIF Uniform DIF

Threshold parameterThreshold parameter

• Non-uniform DIF Non-uniform DIF

Discrimination parameter Discrimination parameter

• Gender, race/ethnicity, age, diseaseGender, race/ethnicity, age, disease

14 04/21/23

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

-4 -3.5 -3 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3 3.5 4

Trait level

Pro

bab

ilit

y o

f "Y

es"

Res

po

nse

Location DIF Slope DIF

Dichotomous Items Showing DIF(2-Parameter Model)

White

Hispanic

Hispanic

White

15 04/21/23

Item CalibrationItem Calibration

• Item parameters (threshold, discrimination)Item parameters (threshold, discrimination)

• Mean differences for studied disease groupsMean differences for studied disease groups

16 04/21/23

Example of Lessons Learned in Secondary Analyses

Emotional distress

Cannot be adequately modeled as a unidimensional construct.

Limited representation of positive end of construct

Several items having some response options that provide little information.

17 04/21/23

Documentation

Public website: http://www.nihpromis.org/

Peer-reviewed manuscripts, e.g.:

Hays, R. D. et al. (in press). Item response theory analyses of physical functioning items in the Medical Outcomes Study. Medical Care.

Reeve, B. B. (in press). Psychometric evaluation and calibration of health-related quality of life items banks: Plans for the Patient-Reported Outcome Measurement Information System (PROMIS). Medical Care.

18 04/21/23

19 04/21/23

Datasets Subjected to Psychometric AnalysisDatasets Subjected to Psychometric Analysis

Cancer Fatigue:Cancer Fatigue: Cancer Item Banking Project at NWUCancer Item Banking Project at NWU

Cancer Pain:Cancer Pain: Cancer Item Banking Project at NWUCancer Item Banking Project at NWU

Cancer Social:Cancer Social: Cancer Item Banking Project at NWUCancer Item Banking Project at NWU

CSSCD:CSSCD: Cooperative Study of Sickle Cell Disease (pediatric)Cooperative Study of Sickle Cell Disease (pediatric)

CHC:CHC: Chronic Hepatitis C StudyChronic Hepatitis C Study

CHS:CHS: Cardiovascular Health StudyCardiovascular Health Study

DIG:DIG: Digitalis Investigation Group Quality of Life Sub-studyDigitalis Investigation Group Quality of Life Sub-study

IMMPACT:IMMPACT: Multiple Pain ProjectsMultiple Pain Projects

MOS:MOS: Medical Outcomes StudyMedical Outcomes Study

NGHS:NGHS: National Growth and Health Study (peds.)National Growth and Health Study (peds.)

Q-Score:Q-Score: Cancer Quality of Life Project at NWUCancer Quality of Life Project at NWU

WHOQOL:WHOQOL: World Health Organization Quality of Life ProjectWorld Health Organization Quality of Life Project

20 04/21/23

Datasets Subjected to Psychometric AnalysisDatasets Subjected to Psychometric Analysis

PROMIS Domains

Emotional Distress Fatigue Pain Physical FunctionSocial Role

Participation

CHS(NWU/Cook)

DIG(UCLA/Hays)

Q-Score(NWU/Bode)

CHC(Medtap/Revicki

& Chen)

Cancer Pain(NWU/Lai)

IMMPACT(Medtap/Revicki

& Chen)

CHS(NWU/Cook)

DIG(UCLA/Hays)

CHS(NWU/Cook)

Cancer Social(NWU/Bode)

CHS(NWU/Cook)

Cancer Fatigue(NWU/Lai)

WHOQOL(UCLA/Hays)

WHOQOL(UCLA/Hays)

MOS(UCLA/Hays& Spritzer)

Q-Score(NWU/Lai)

WHOQOL(UCLA/Hays)

WHOQOL(UCLA/Hays)

WHOQOL(UCLA/Hays)

CSSCD(NWU)

NGHS(NWU)

NGHS(NWU)