Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Chapter 17 Assessing...
Transcript of Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Chapter 17 Assessing...
Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins
Chapter 17
Assessing Measurement Quality in Quantitative Studies
Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins
MeasurementMeasurement
•The assignment of numbers to represent the amount of an attribute present in an object or person, using specific rules
•Advantages:
– Removes guesswork
– Provides precise information
– Less vague than words
Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins
Errors of MeasurementErrors of Measurement
Obtained Score = True score + Error
Obtained score: An actual data value for a participant (e.g., anxiety scale score)
True score: The score that would be obtained with an infallible measure
Error: The error of measurement, caused by factors that distort measurement
Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins
Factors That Contribute to Errors of Measurement
Factors That Contribute to Errors of Measurement
• Situational contaminants
• Transitory personal factors
• Response-set biases
• Administration variations
• Problems with instrument clarity
• Item sampling
• Instrument format
Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins
Key Criteria for Evaluating Quantitative Measures
Key Criteria for Evaluating Quantitative Measures
•Reliability
•Validity
Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins
ReliabilityReliability
•The consistency and accuracy with which an instrument measures the target attribute
•Reliability assessments involve computing a reliability coefficient
– most reliability coefficients are based on correlation coefficients
Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins
Correlation CoefficientsCorrelation Coefficients
• Correlation coefficients indicate direction and magnitude of relationships between variables
• Range
from –1.00 (perfect negative correlation)
through 0.00 (no correlation)
to +1.00 (perfect positive correlation)
Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins
Three Aspects of Reliability Can Be Evaluated
Three Aspects of Reliability Can Be Evaluated
•Stability
•Internal consistency
•Equivalence
Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins
StabilityStability
• The extent to which scores are similar on 2 separate administrations of an instrument
• Evaluated by test–retest reliability
– Requires participants to complete the same instrument on two occasions
– A correlation coefficient between scores on 1st and 2nd administration is computed
– Appropriate for relatively enduring attributes (e.g., self-esteem)
Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins
Internal ConsistencyInternal Consistency
• The extent to which all the instrument’s items are measuring the same attribute
• Evaluated by administering instrument on one occasion
• Appropriate for most multi-item instruments
• Evaluation methods:
– Split-half technique
– Coefficient alpha
Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins
EquivalenceEquivalence
• The degree of similarity between alternative forms of an instrument or between multiple raters/observers using an instrument
• Most relevant for structured observations
• Assessed by comparing observations or ratings of 2 or more observers (interobserver/interrater reliability)
• Numerous formula and assessment methods
Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins
Reliability CoefficientsReliability Coefficients
• Represent the proportion of true variability to obtained variability:
r = VT
Vo
• Should be at least .70; .80 preferable
• Can be improved by making instrument longer (adding items)
• Are lower in homogeneous than in heterogeneous samples
Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins
ValidityValidity
• The degree to which an instrument measures what it is supposed to measure
• Four aspects of validity:– Face validity
– Content validity
– Criterion-related validity
– Construct validity
Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins
Face ValidityFace Validity
•Refers to whether the instrument looks as though it is measuring the appropriate construct
•Based on judgment, no objective criteria for assessment
Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins
Content ValidityContent Validity
•The degree to which an instrument has an appropriate sample of items for the construct being measured
•Evaluated by expert evaluation, via the content validity index (CVI)
Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins
Criterion-Related ValidityCriterion-Related Validity
•The degree to which the instrument correlates with an external criterion
•Validity coefficient is calculated by correlating scores on the instrument and the criterion
Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins
Criterion-Related Validity (cont’d)Criterion-Related Validity (cont’d)
Two types of criterion-related validity:
• Predictive validity: the instrument’s ability to distinguish people whose performance differs on a future criterion
• Concurrent validity: the instrument’s ability to distinguish individuals who differ on a present criterion
Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins
Construct ValidityConstruct Validity
Concerned with the questions:
•What is this instrument really measuring?
•Does it adequately measure the construct of interest?
Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins
Methods of Assessing Construct Validity
Methods of Assessing Construct Validity
•Known-groups technique
•Relationships based on theoretical predictions
•Multitrait-multimethod matrix method (MTMM)
•Factor analysis
Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins
Multitrait-Multimethod Matrix Method
Multitrait-Multimethod Matrix Method
Builds on two types of evidence:
• Convergence
• Discriminability
Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins
ConvergenceConvergence
•Evidence that different methods of measuring a construct yield similar results
•Convergent validity comes from the correlations between two different methods measuring the same trait
Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins
DiscriminabililtyDiscriminabililty
•Evidence that the construct can be differentiated from other similar constructs
•Discriminant validity assesses the degree to which a single method of measuring two constructs yields different results