Validity
description
Transcript of Validity
![Page 1: Validity](https://reader035.fdocuments.in/reader035/viewer/2022062521/5681681b550346895ddda8ef/html5/thumbnails/1.jpg)
Validity
• Does test measure what it says it does?• Is the test useful?
• Can a test be reliable, but not valid?• Can a test be valid, but not reliable?
![Page 2: Validity](https://reader035.fdocuments.in/reader035/viewer/2022062521/5681681b550346895ddda8ef/html5/thumbnails/2.jpg)
Types of validity
• Face validity– Important only so far as it doesn’t interfere with
an examinee’s willingness to cooperate.• Content validity
– How well does the test cover areas of content that it should?
– How adequately does it sample the universe of behavior it was designed to assess?
![Page 3: Validity](https://reader035.fdocuments.in/reader035/viewer/2022062521/5681681b550346895ddda8ef/html5/thumbnails/3.jpg)
Content validity (cont.)
• Panel of “experts”– Is the item/content essential?– Lawshe (1975) >50% of experts see skill as
essential• Important for:
– Achievement/classroom tests– Training program exams– Professional exams
![Page 4: Validity](https://reader035.fdocuments.in/reader035/viewer/2022062521/5681681b550346895ddda8ef/html5/thumbnails/4.jpg)
Criterion-Related Validity
• How well does a test score relate to another score/variable of interest?– Correlate test with criterion
• Standard against which test is evaluated
• Concurrent• Predictive
![Page 5: Validity](https://reader035.fdocuments.in/reader035/viewer/2022062521/5681681b550346895ddda8ef/html5/thumbnails/5.jpg)
Criterion-Related Validity (cont.)
• Criterion should be– Reliable
• Reliability limits validity; can’t be valid if not reliable.
– Relevant– Valid– Uncontaminated
• Criterion measure has been based in part on predictor measure
![Page 6: Validity](https://reader035.fdocuments.in/reader035/viewer/2022062521/5681681b550346895ddda8ef/html5/thumbnails/6.jpg)
Criterion-Related Validity (cont.)
• Concurrent validity– Criterion immediately available– Present standing on a criterion
• Diagnosis, score on another test– Used to predict the performance of new test
takers or for people for whom the criterion isn’t available.
![Page 7: Validity](https://reader035.fdocuments.in/reader035/viewer/2022062521/5681681b550346895ddda8ef/html5/thumbnails/7.jpg)
Criterion-Related Validity (cont.)
• Predictive validity– Test given, criterion measured later– Ex. ACT & College GPA; employment test &
job performance• Incremental validity
![Page 8: Validity](https://reader035.fdocuments.in/reader035/viewer/2022062521/5681681b550346895ddda8ef/html5/thumbnails/8.jpg)
Base Rate & Decision Theory
• Base rate: proportion of population who possess a certain trait, characteristic or attribute– % of EIU undergrads who graduate– % of African Americans with sickle cell anemia
• Base rate affects usefulness of tests
![Page 9: Validity](https://reader035.fdocuments.in/reader035/viewer/2022062521/5681681b550346895ddda8ef/html5/thumbnails/9.jpg)
Decision Theory
• 4 outcomes
False rejections/negatives
Valid Acceptances/Positives
Valid Rejections/negatives
False Acceptances/Positives
![Page 10: Validity](https://reader035.fdocuments.in/reader035/viewer/2022062521/5681681b550346895ddda8ef/html5/thumbnails/10.jpg)
Cut scores & Hit rates
False rejections/negatives Valid Acceptances/Positives
Valid Rejections/negatives
False Acceptances/Positives
![Page 11: Validity](https://reader035.fdocuments.in/reader035/viewer/2022062521/5681681b550346895ddda8ef/html5/thumbnails/11.jpg)
Cut scores & Hit rates (cont.)
• Reciprocal relationship between # of false rejections and # of false acceptances
• Which is more acceptable: to limit the number accepted who shouldn’t be, or to minimize the # rejected who could be successful?
![Page 12: Validity](https://reader035.fdocuments.in/reader035/viewer/2022062521/5681681b550346895ddda8ef/html5/thumbnails/12.jpg)
Construct Validity
• Construct:– Scientific idea hypothesized to explain behavior – Postulated attribute of people, assumed to be reflected
in test score– Ex.: intelligence, self-esteem, motivation
• Construct validity: Does the test measure the construct?– Gives theoretical meaning to scores;– Subsumes all other types of validity
![Page 13: Validity](https://reader035.fdocuments.in/reader035/viewer/2022062521/5681681b550346895ddda8ef/html5/thumbnails/13.jpg)
Construct Validity (cont.)
• Convergent evidence/validity• Divergent/discriminant evidence• Factor analysis
– Data reduction/simplification of complex correlational matrices … to reveal major dimensions that underlie a set of items
– A factor is considered to be the construct that best represents relationships among variables
![Page 14: Validity](https://reader035.fdocuments.in/reader035/viewer/2022062521/5681681b550346895ddda8ef/html5/thumbnails/14.jpg)
Factor Analysis (cont.)
• Methods of factor analysis– Exploratory
1. Correlation matrix2. Factor matrix with loadings3. Label factors• Used to develop or eliminate items or scales from
composite scores
![Page 15: Validity](https://reader035.fdocuments.in/reader035/viewer/2022062521/5681681b550346895ddda8ef/html5/thumbnails/15.jpg)
Factor Analysis (cont.)
• Confirmatory factor analysis– Goodness of fit– After test has been developed
![Page 16: Validity](https://reader035.fdocuments.in/reader035/viewer/2022062521/5681681b550346895ddda8ef/html5/thumbnails/16.jpg)
Validity & Bias
• Bias: a factor inherent within a test that systematically prevents accurate, impartial measurement– Bias implies systematic, not random variation
• Can you make equally valid predictions for different groups?
![Page 17: Validity](https://reader035.fdocuments.in/reader035/viewer/2022062521/5681681b550346895ddda8ef/html5/thumbnails/17.jpg)
Bias in Predictions
• Questions of regression– Slope– Intercept– Error of estimate
![Page 18: Validity](https://reader035.fdocuments.in/reader035/viewer/2022062521/5681681b550346895ddda8ef/html5/thumbnails/18.jpg)
Slope Bias
Bias & the DAS
60
80
100
120
140
75 85 100 115General Conceptual Ability Scores
Word Reading Scores
Whites
Asian Americans
Linear (Whites)
Linear (AsianAmericans)
![Page 19: Validity](https://reader035.fdocuments.in/reader035/viewer/2022062521/5681681b550346895ddda8ef/html5/thumbnails/19.jpg)
Intercept Bias
Bias & the DAS
0
20
40
60
80
100
120
140
1 2 3 4 5 6General Conceptual Ability Scores
Basic Number Skills
Series1Series2
![Page 20: Validity](https://reader035.fdocuments.in/reader035/viewer/2022062521/5681681b550346895ddda8ef/html5/thumbnails/20.jpg)
Rating error
• Leniency Error• Severity Error• Central Tendency Error• Halo Effect
![Page 21: Validity](https://reader035.fdocuments.in/reader035/viewer/2022062521/5681681b550346895ddda8ef/html5/thumbnails/21.jpg)
Test Fairness
• Is the test used in an impartial, just, and equitable manner?
• Good tests Discriminate among individuals– Are group differences due to inadequate tests?– Is the test being used fairly?