Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD ([email protected]) February 6,...
-
Upload
dylan-floyd -
Category
Documents
-
view
215 -
download
0
Transcript of Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD ([email protected]) February 6,...
![Page 1: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/1.jpg)
Evaluating Self-Report Data
Using Psychometric Methods Ron D. Hays, PhD ([email protected])
February 6, 2008 (3:30-6:30pm)
HS 249F
![Page 2: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/2.jpg)
H E A L T H
Individual Change• Interest in
– Knowing how many patients benefit from group intervention, or
– Tracking progress on individual patients
•Sample– 54 patients – Average age = 56; 84% white; 58%
female•Method
– Self-administered SF-36 version 2 at baseline and at end of therapy (about 6 weeks later).
![Page 3: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/3.jpg)
H E A L T H
Physical Functioning and Emotional Well-Being at Baseline Physical Functioning and Emotional Well-Being at Baseline
for 54 Patients at UCLA-Center for East West Medicinefor 54 Patients at UCLA-Center for East West Medicine
0 10 20 30 40 50 60
Asymptomatic
Symptomatic
AIDS
General Pop
Epilepsy
GERD
Prostate disease
Depression
Diabetes
ESRD
MS
East-WestEWB
Physical
Hays et al. (2000), American Journal of Medicine
![Page 4: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/4.jpg)
H E A L T H
Change in SF-36 Scores Over Time
0
10
20
30
40
50
PF10 Role-P Pain Gen H Energy Social Role-E EWB PCS MCS
Baseline
Followup
0.13 0.35 0.35 0.21 0.53 0.36 0.11 0.41 0.24 0.30
Effect Size
![Page 5: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/5.jpg)
H E A L T H
t-test for within group change
•XXDD/(SD/(SDdd/n /n ½½))
XXDD = is mean difference, SD = is mean difference, SDd d = standard deviation of difference= standard deviation of difference
![Page 6: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/6.jpg)
H E A L T H
Significance of Group Change (T-scores)Change t-test prob.
PF-10 1.7 2.38 .0208
RP-4 4.1 3.81 .0004
BP-2 3.6 2.59 .0125
GH-5 2.4 2.86 .0061
EN-4 5.1 4.33 .0001
SF-2 4.7 3.51 .0009
RE-3 1.5 0.96 .3400 <-
EWB-5 4.3 3.20 .0023
PCS 2.8 3.23 .0021
MCS 3.9 2.82 .0067
![Page 7: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/7.jpg)
H E A L T H
Reliable Change Index
(X2 – X1)/ (SEM * SQRT [2])
SEM = SDb * (1- reliability)1/2
![Page 8: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/8.jpg)
H E A L T H
Amount of Change in Observed Score Needed for Significant Individual Change
RCI Effect size
PF-10 8.4 0.67
RP-4 8.4 0.72
BP-2 10.4 1.01
GH-5 13.0 1.13
EN-4 12.8 1.33
SF-2 13.8 1.07
RE-3 9.7 0.71
EWB-5 13.4 1.26
PCS 7.1 0.62
MCS 9.7 0.73
![Page 9: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/9.jpg)
H E A L T H
Significant Change for 54 Cases % Improving
% Declining
Difference
PF-10 13% 2% + 11%
RP-4 31% 2% + 29%
BP-2 22% 7% + 15%
GH-5 7% 0% + 7%
EN-4 9% 2% + 7%
SF-2 17% 4% + 13%
RE-3 15% 15% 0%
EWB-5 19% 4% + 15%
PCS 24% 7% + 17%
MCS 22% 11% + 11%
![Page 10: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/10.jpg)
H E A L T H
Multiple Steps in Developing Good Survey•Review literature
•Expert input (patients and clinicians)•Define constructs you are interested in•Draft items (item generation)•Pretest
– Cognitive interviews– Field and pilot testing
•Revise and test again•Translate/harmonize across languages
![Page 11: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/11.jpg)
H E A L T H
What’s a Good Measure?What’s a Good Measure?
• Same person gets same score (reliability)
• Different people get different scores (validity)
• People get scores you expect (validity)
• It is practical to use (feasibility)
![Page 12: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/12.jpg)
H E A L T H
Scales of Measurement and Their Properties
NominalOrdinal +Interval + +Ratio + + +
Type ofScale Rank Order
Equal Interval Absolute 0
Property of Numbers
![Page 13: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/13.jpg)
H E A L T H
Measurement Range for Health Outcome Measures
Nominal Ordinal Interval Ratio
![Page 14: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/14.jpg)
H E A L T H
Indicators of Acceptability
• Unit non-response
• Item non-response
• Administration time
![Page 15: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/15.jpg)
H E A L T H
Variability
• All scale levels are represented
• Distribution approximates bell-shaped "normal"
![Page 16: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/16.jpg)
H E A L T H
Measurement Error
observed = true score
+ systematic error
+ random error
(bias)
![Page 17: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/17.jpg)
H E A L T H
•Coverage ErrorDoes each person in population have an equal chance of selection?
•Sampling ErrorAre only some members of the population sampled?
•Nonresponse ErrorDo people in the sample who respond differ from those who do not?
•Measurement ErrorAre inaccurate answers given to survey questions?
Four Types of Data Collection Errors
![Page 18: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/18.jpg)
H E A L T H
Flavors of Reliability
•Test-retest (administrations)
• Intra-rater (raters) • Internal consistency (items)
![Page 19: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/19.jpg)
H E A L T H
Test-retest Reliability of MMPI 317-362r = 0.75
MMPI 317
True False
169 15
21 95
True
False
MMPI 362
184
116
190 110
I am more sensitive than most other people.
![Page 20: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/20.jpg)
H E A L T H
Kappa Coefficient of Agreement(Corrects for Chance)
(observed - chance) kappa =
(1 - chance)
![Page 21: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/21.jpg)
H E A L T H
Example of Computing KAPPA
1
2
3
4
5
1
1
1
2
2
2 2
2
2
2
2
2
2
2
2
103Column Sum
Rater B
1 2 3 4 5RowSum
Rater A
![Page 22: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/22.jpg)
H E A L T H
Example of Computing KAPPA(Continued)
P =(1 x 2) + (3 x 2) + (2 x 2) + (2 x 2) + (2 x 2)
(10 x 10) = 0.20c
P =9
10 = 0.90obs.
Kappa =0.90 - 0.20
1 - 0.20 = 0.87
![Page 23: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/23.jpg)
H E A L T H
Conclusion Kappa Conclusion Kappa Poor < 0.0
Slight .00 - .20
Poor < .40 Fair .21 - .40
Fair .40 - .59 Moderate .41 - .60
Good .60 - .74 Substantial .61 - .80
Excellent > .74 Almost perfect.81 - 1.00
Fleiss (1981) Landis and Koch (1977)
Guidelines for Interpreting Kappa
![Page 24: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/24.jpg)
H E A L T H
Intraclass Correlation and Reliability
Model Reliability Intraclass Correlation
One-Way MS - MS MS - MS
MS MS + (K-1)MS
Two-Way MS - MS MS - MS
Fixed MS MS + (K-1)MS
Two-Way N (MS - MS ) MS - MS
Random NMS +MS - MS MS + (K-1)MS + K (MS - MS )/NBMS JMS
EMS
BMS WMS
BMS
BMS
EMS
BMS WMS
BMS
BMS
EMS
BMS EMS
EMS EMS
BMS
BMS EMS JMS EMS
WMS
BMS EMS
![Page 25: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/25.jpg)
H E A L T H
Summary of Reliability of Plant Ratings Baseline Follow-up
RTT RII RTT RII
One-Way Anova 0.97 0.95 0.97 0.94Two-Way Random Effects 0.97 0.95 0.97 0.94Two-Way Fixed Effects 0.98 0.96 0.98 0.97
Source Label Baseline MSPlants BMS 628.667Within WMS 17.700Raters JMS 57.800Raters X Plants EMS 13.244
![Page 26: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/26.jpg)
H E A L T H
Raw Data for Ratings of Height (1/16 inch) of Houseplants (A1, A2, etc.) by Two Raters (R1, R2)
A1 R1 120 121 1 R2 118 120
A2 R1 084 085 2 R2 096 088
B1 R1 107 108 2 R2 105 104
B2 R1 094 100 1 R2 097 104
C1 R1 085 088 2 R2 091 096
PlantBaseline Height
Follow-up Height
Experimental Condition
![Page 27: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/27.jpg)
H E A L T H
Ratings of Height of Houseplants (Cont.)
C2 R1 079 086 1 R2 078 092
D1 R1 070 076 1 R2 072 080
D2 R1 054 056 2 R2 056 060
E1 R1 085 101 1 R2 097 108
E2 R1 090 084 2 R2 092 096
PlantBaseline Height
Follow-up Height
Experimental Condition
![Page 28: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/28.jpg)
H E A L T H
Reliability of Baseline Houseplant Ratings
Source DF SS MS F
Plants 9 5658 628.667 35.52
Within 10 177 17.700
Raters 1 57.8 57.800
Raters x Plants 9 119.2 13.244
Total 19 5835
Baseline Results
Ratings of Height of Plants: 10 plants, 2 raters
![Page 29: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/29.jpg)
H E A L T H
Sources of Variance in Baseline Houseplant Height
Source dfs MS
Plants (N) 9 628.67 (BMS)
Within 10 17.70 (WMS)
Raters (K) 1 57.80 (JMS)
Raters x Plants 9 13.24 (EMS)
Total 19
![Page 30: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/30.jpg)
H E A L T H
Cronbach’s Alpha
Respondents (BMS) 4 11.6 2.9 Items (JMS) 1 0.1 0.1 Resp. x Items (EMS) 4 4.4 1.1
Total 9 16.1
Source
df SS MS
Alpha = 2.9 - 1.1 = 1.8 = 0.622.9 2.9
![Page 31: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/31.jpg)
H E A L T H
Alpha for Different Numbers of Itemsand Homogeneity
2 .000 .333 .572 .750 .889 1.000 4 .000 .500 .727 .857 .941 1.000 6 .000 .600 .800 .900 .960 1.000 8 .000 .666 .842 .924 .970 1.000
Numberof Items (k) .0 .2 .4 .6 .8 1.0
Average Inter-item Correlation ( r )
Alphast= k * r 1 + (k -1) * r
![Page 32: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/32.jpg)
H E A L T H
Spearman-Brown Prophecy Formula
alpha y = N • alpha
x
1 + (N - 1) * alphax
N = how much longer scale y is than scale x
)(
![Page 33: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/33.jpg)
H E A L T H
Example Spearman-Brown Calculations
MHI-18
18/32 (0.98) (1+(18/32 –1)*0.98
= 0.55125/0.57125 = 0.96
![Page 34: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/34.jpg)
H E A L T H
Number of Items and Reliability for Three Versions of the
Mental Health Inventory (MHI)
Measure
Number of
I tems
Completion time (min.)
Reliability
MHI -32
32
5-8
.98
MHI -18
18
3-5
.96
MHI -5
5
1 or less
.90
Data f rom McHorney et al. 1992
![Page 35: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/35.jpg)
H E A L T H
Reliability Minimum Standards
• 0.70 or above (for group comparisons)
• 0.90 or higher (for individual assessment)
SEM = SD (1- reliability)1/2
![Page 36: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/36.jpg)
H E A L T H
Reliability of a Composite Score
Mosier 1( j
2w )( j
2S ) ( j2w )( j
2S )( j )( j
2w )( j
2S ) 2( jw )( Kw )( jS )( KS )( jKr )Mosier 1
( j2w )( j
2S ) ( j2w )( j
2S )( j )( j
2w )( j
2S ) 2( jw )( Kw )( jS )( KS )( jKr )
jw weight given to component J
Kw weight given to component K
jS standard deviation of J
j reliability of J
jKr correlation between J and K
jw weight given to component J
Kw weight given to component K
jS standard deviation of J
j reliability of J
jKr correlation between J and K
![Page 37: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/37.jpg)
H E A L T H
Hypothetical Multitrait/Multi-Item Correlation Matrix
Trait #1 Trait #2 Trait #3 I tem #1 0.80* 0.20 0.20 I tem #2 0.80* 0.20 0.20 I tem #3 0.80* 0.20 0.20 I tem #4 0.20 0.80* 0.20 I tem #5 0.20 0.80* 0.20 I tem #6 0.20 0.80* 0.20 I tem #7 0.20 0.20 0.80* I tem #8 0.20 0.20 0.80* I tem #9 0.20 0.20 0.80* *I tem- scale correlation, corrected for overlap.
![Page 38: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/38.jpg)
H E A L T H
Multitrait/Multi-Item CorrelationMatrix for Patient Satisfaction Ratings
Technical Interpersonal Communication FinancialTechnical
1 0.66* 0.63† 0.67† 0.282 0.55* 0.54† 0.50† 0.253 0.48* 0.41 0.44† 0.264 0.59* 0.53 0.56† 0.265 0.55* 0.60† 0.56† 0.166 0.59* 0.58† 0.57† 0.23
Interpersonal1 0.58 0.68* 0.63† 0.242 0.59† 0.58* 0.61† 0.183 0.62† 0.65* 0.67† 0.194 0.53† 0.57* 0.60† 0.325 0.54 0.62* 0.58† 0.186 0.48† 0.48* 0.46† 0.24
Note – Standard error of correlation is 0.03. Technical = satisfaction with technical quality. Interpersonal = satisfaction with the interpersonal aspects. Communication = satisfaction with communication. Financial = satisfaction with financial arrangements. *Item-scale correlations for hypothesized scales (corrected for item overlap). †Correlation within two standard errors of the correlation of the item with its hypothesized scale.
![Page 39: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/39.jpg)
H E A L T H
Construct Validity
• Does measure relate to other measures in ways consistent with hypotheses?
• Responsiveness to change including minimally important difference
![Page 40: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/40.jpg)
H E A L T H
![Page 41: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/41.jpg)
H E A L T H
Construct Validity for Scales MeasuringPhysical Functioning
91 90 87 2 ---
88 78 74 10 5
95 87 77 20 10
None Mild Severe
F-ratioRelative Validity
Scale #1
Scale #2
Scale #3
Severity of Heart Disease
![Page 42: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/42.jpg)
H E A L T H
Responsiveness to Change and Minimally Important Difference (MID)
• HRQOL measures should be responsive to
interventions that changes HRQOL
• Need external indicators of change (Anchors)
– mean change in HRQOL scores among people who have changed (“minimal” change for MID).
![Page 43: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/43.jpg)
H E A L T H
Self-Report Indicator of Change
•Overall has there been any change in your asthma since the beginning of the study?
Much improved; Moderately improved; Minimally improved
No change Much worse; Moderately worse; Minimally worse
![Page 44: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/44.jpg)
H E A L T H
Clinical Indicator of Change
– “changed” group = seizure free (100%
reduction in seizure frequency)
– “unchanged” group = <50% change in
seizure frequency
![Page 45: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/45.jpg)
H E A L T H
Responsiveness Indices
(1) Effect size (ES) = D/SD
(2) Standardized Response Mean (SRM) = D/SD†
(3) Guyatt responsiveness statistic (RS) = D/SD‡
D = raw score change in “changed” group;SD = baseline SD; SD† = SD of D; SD‡ = SD of D among “unchanged”
![Page 46: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/46.jpg)
H E A L T H
Effect Size Benchmarks
• Small: 0.20->0.49
• Moderate: 0.50->0.79
• Large: 0.80 or above
![Page 47: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/47.jpg)
H E A L T H
Treatment Impact on PCS
012345678910
I mpact on
SF- 36 PCS
Treatment Outcomes
Duodenal UlcerMedication
Shoulder Surgery
Heart ValueReplacement
Total Hip Replacement
![Page 48: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/48.jpg)
H E A L T H
Treatment Impact on MCS
0
2
4
6
8
10
12
I mpact on
SF- 36 MCS
Treatment Outcomes
Stayed the same
Low back paintherapy
Hip replacement
Ulcer maintenance
Recovery fromDepression
![Page 49: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/49.jpg)
H E A L T H
IRT
![Page 50: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/50.jpg)
H E A L T H
Latent Trait and Item Responses
Latent Trait
Item 1 Response
P(X1=1)P(X1=0)
10
Item 2 Response
P(X2=1)P(X2=0)
10
Item 3 Response
P(X3=0)0
P(X3=2)2
P(X3=1) 1
![Page 51: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/51.jpg)
H E A L T H
Item Responses and Trait Levels
Item 1 Item 2 Item 3
Person 1 Person 2 Person 3
TraitContinuum
![Page 52: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/52.jpg)
H E A L T H
0%10%20%30%40%50%60%70%80%90%
100%
Trait Leve l
Pro
ba
bil
ity
of
"Ye
s"
Re
sp
on
se
Item 1 (Difficulty = -1) Item 3 (Difficulty = 1)
Item Characteristic Curves(1-Parameter Model)
Pro
b.
of
“Y
es”
![Page 53: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/53.jpg)
H E A L T H
Item Characteristic Curves(2-Parameter Model)
![Page 54: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/54.jpg)
H E A L T H
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
-4 -3.5 -3 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3 3.5 4
Trait level
Pro
ba
bili
ty o
f "Y
es
" R
es
po
ns
e
DIF – Location(Item 1)
DIF – Slope(Item 2)
Hispanics
Whites
Whites
Hispanics
Dichotomous Items Showing DIF(2-Parameter Model)
![Page 55: Evaluating Self-Report Data Using Psychometric Methods Ron D. Hays, PhD (hays@rand.org) February 6, 2008 (3:30-6:30pm) HS 249F.](https://reader036.fdocuments.in/reader036/viewer/2022062518/5697bf811a28abf838c855eb/html5/thumbnails/55.jpg)
H E A L T H