Post on 22-Jan-2016
description
MEASUREMENT: VALIDITY
Lu Ann Aday, Ph.D.The University of Texas School of Public Health
VALIDITY: Definition Extent of systematic departure
in answers to questions in relationship to the meaning of the concept itself (content), equivalent measures of the same concept (criterion), or hypothesized associations with measures of similar or different concepts (construct).
VALIDITY: Types
Content validityCriterion validityConstruct validity
VALIDITY: Computation Requires analyzing the
association between survey measures and documented definitions of the underlying concept (content); an equivalent “gold standard” of the survey measures (criterion); or theoretical associations with other abstract concepts (construct).
VALIDITY: Content
Definition: extent to which the survey measures adequately represent the concept of interest
VALIDITY: Content
Factors affecting: Failure to thoroughly conduct background research on the definition & measurement of the study concept
Failure to clearly define the study concept
VALIDITY: Content Computation:
1. Identify domain or universe of interest through a thorough review of the literature.
2. Select or construct questions based on the review.
3. Ask subject matter experts to review the items you have selected or constructed.
VALIDITY: Content Validity criteria:
1. Is there at least one item to represent the concept?
2. Are there enough items? 3. Are the items well accepted, based on the literature & expert opinion?
4. Are they reliable?
VALIDITY: Criterion
Definition: extent to which the survey measure predicts (predictive) or agrees with (concurrent) a criterion indicator of the concept of interest
VALIDITY: Criterion Factors affecting:
Underreporting due to problems with recalling non-salient or frequently occurring events
Overreporting due to problems with telescoping salient events
VALIDITY: Criterion- Predictive: Correlation Analysis
Computation: Compute correlation coefficient between answers to survey question and future criterion source value for same concept :Respondent Survey Question MD Observation1 ADLs=3 ADLs=3 2 ADLs=2 ADLs=2 3 ADLs=1 ADLs=1 44 ADLs=0ADLs=0 ADLs=3ADLs=35 ADLs=0 ADLs=0
VALIDITY: Criterion- Concurrent: Correlation Analysis
Computation: Compute correlation coefficient between answers to survey question and current criterion source value for same concept:
Respondent Survey Question Clinic Record1 BP=140/90 BP=140/90 2 BP=150/80 BP=150/80 3 BP=145/95 BP=145/95
44 BP=145/95BP=145/95 BP=120/80BP=120/805 BP=140/90 BP=140/90
VALIDITY: Criterion-Concurrent: Sensitivity-Specificity Analysis
CRITERION /SURVEY
IS (+)Hypertensive
IS NOT (-)Hypertensive
IS (+)Hypertensive
a = true +Sensitivity =a/a + c
b = false +
IS NOT (-)Hypertensive
c = false - d = true –Specificity =d/b + d
VALIDITY: Criterion-Concurrent: Sensitivity-Specificity Analysis
CRITERION /SURVEY
IS (+)Hypertensive
IS NOT (-)Hypertensive
IS (+)Hypertensive
a = 130Sensitivity =130/180= 72.2%
b = 1,000
IS NOT (-)Hypertensive
c = 50 d = 64,000Specificity =64,000/65,000=98.5%
VALIDITY: Construct Definition: extent to which
relationships between measures agree with relationships predicted by theories or hypotheses Convergent – Ho: Related Discriminant – Ho: Not Related
VALIDITY: Construct
Factors affecting: Inadequate conceptualization or measurement of study constructs
Inadequate theory to guide specification of hypothesized relationships
VALIDITY: Construct: Correlation Analysis
Computation: Compute correlation coefficient between survey question or scales measuring construct and related (or unrelated) constructs :Respondent Physical Mental Overall
1 Good FairFair Good2 Good FairFair Good3 Good FairFair Good44 Good FairFair Good5 Good FairFair Good
VALIDITY: Construct- Known Groups Validity Analysis (McHorney, et al., 1993, Table 3, p. 255)
COMPARISON GROUPS/MOS-SF 36 SUMMARY SCALES
MINOR MEDICAL
SERIOUS MEDICAL
PSYCHIATRIC ONLY
PSYCHIATRIC& SERIOUSMEDICAL
PhysicalFunctioning
80.53 57.35 80.62 46.37
MentalHealth
82.49 77.59 52.75 56.90
VALIDITY: CONSTRUCT-Factor Analysis(McHorney, et al., 1993, Table 1, p. 251)
FACTOR LOADINGS (HYPOTHESES)/SUBSCALES
Physical Mental
Physical functioning 0.88 (+) 0.04
Role-physical 0.78 (+) 0.30
Bodily pain 0.77 (+) 0.24
Mental health 0.12 0.90 (+)
Role-emotional 0.19 0.81 (+)
Social Functioning 0.44 (~) 0.71 (+)
Vitality 0.59 (~) 0.57 (~)
General health perceptions
0.68 (~) 0.32 (~)
VALIDITY: Construct—Multi-Trait, Multi-Method
Trait X Method
A B C A B C A B C
A (.90)
B (.85)
C (.80)
A .75 (.80)
B .55 (.75)
C .35 (.80)
A .80 .85 (.95)
B .55 .55 (.90)
C .30 .30 (.65)
VALIDITY: Effect of Reliability
Theoretical Variable can be no more valid than it is reliable.
Empirical Rt = theoretical correlation between study &
criterion variables (unobservable) Rx = reliability of study variable (computed) Ry = reliability of criterion variable (computed) Rt * sqrt [ Rx * Ry] = Max. achievable
correlation 1.00 * sqrt [.60 * .80] = .69
WHEN TO UNDERTAKE VALIDITY ANALYSIS
VALIDITY/DIMENSIONS
CONTENT CRITERION CONSTRUCT
QUESTIONS All questions Factual questions for which “gold standard” is available
Summary scales of attitudes or other abstract concepts
STUDIES All types of surveys
Esp. used in behavioral surveys
Esp. used in attitudinal surveys
STAGES Question development
Pretest or final study
Pretest or final study
REFERENCES DeVellis, Robert F. (2003). Scale Development:
Theory and Applications. Second Edition. Thousand Oaks, CA: Sage.
Gandek, B., & Ware, J.E., Jr., for the IQOLA Project (1998). Methods for validating and norming translations of health status questionnaires: The IQOLA Project Approach. J. Clinical Epidemiology, 51 (11), 953-959.
McHorney, C.A., Ware, J.E., Jr., & Raczek, A.E. (1993). The MOS 36-item short-form health survey (SF-36): II. Psychometric and clinical tests of validity in measuring physical and mental health constructs. Medical Care, 31(3), 247-263. [Note: Tables 1 and 3 adapted and reprinted with the permission of Lippincott, Williams & Wilkins, Publishers.]