EPSY 251 M. Young, Ph.D. (860) 486-0182 [email protected] 130 Gentry Bld.
CRITERION-RELATED VALIDITY – PREDICTIVE LECTURE 10 EPSY 625.
-
date post
22-Dec-2015 -
Category
Documents
-
view
230 -
download
3
Transcript of CRITERION-RELATED VALIDITY – PREDICTIVE LECTURE 10 EPSY 625.
EMPIRICAL METHODS FOR VALIDITY
Predictive validity• logistic regression
• discriminant analysis/cluster analysis
• correlation/structural equation modeling
Concurrent validity• correlation/structural equation modeling
• factor analysis
Construct validity• factor analysis
• multitrait-multimethod analysis
PREDICTIVE VALIDITY- logistic regression
Binary group: (0,1) such as hired vs. not hired, general vs. clinical
Transform binary score into logit:
L(y) = log[p/(1-p)]
Predict L(y) = b1 x, where x is a test score
Can use SPSS LOGISTIC option in REGRESSION analysis
Omnibus Tests of Model Coefficients
83.360 14 .000
83.360 14 .000
83.360 14 .000
Step
Block
Model
Step 1Chi-square df Sig.
Model Summary
292.234a .265 .353Step1
-2 Loglikelihood
Cox & SnellR Square
NagelkerkeR Square
Estimation terminated at iteration number 4 becauseparameter estimates changed by less than .001.
a.
Classification Tablea
101 32 75.9
35 103 74.6
75.3
Observed0
1
clinical
Overall Percentage
Step 10 1
clinical PercentageCorrect
Predicted
The cut value is .500a.
Variables in the Equation
-.069 .024 8.374 1 .004 .933
.051 .021 5.854 1 .016 1.052
.004 .016 .059 1 .809 1.004
.026 .017 2.297 1 .130 1.027
.051 .019 7.404 1 .007 1.052
-.046 .028 2.683 1 .101 .955
-.047 .017 7.897 1 .005 .954
-.004 .019 .041 1 .840 .996
-.022 .021 1.101 1 .294 .978
.044 .023 3.793 1 .051 1.045
-.005 .025 .046 1 .829 .995
-.027 .017 2.542 1 .111 .973
.045 .022 4.083 1 .043 1.046
.047 .024 3.843 1 .050 1.048
-2.668 2.529 1.113 1 .291 .069
t1
t10
t11
t12
t13
t14
t2
t3
t4
t5
t6
t7
t8
t9
Constant
Step1
a
B S.E. Wald df Sig. Exp(B)
Variable(s) entered on step 1: t1, t10, t11, t12, t13, t14, t2, t3, t4, t5, t6, t7, t8, t9.a.
VARIABLE LABELSt1 ANXIETYt2 ATTITUDE TO PARENTSt3 ATTITUDE TO SCHOOLt4 ATTITUDE TO TEACHERt5 ATYPICALITYt6 DEPRESSIONt7 INTERPERSONAL RELATIONSt8 SENSE OF INADEQUACYt9 LOCUS OF CONTROLt10SELF ESTEEMt11SELF RELIANCEt12SENSATION SEEKINGt13SOMATICIZATIONt14 SOCIAL STRESS
Multinomial regression
Extension of logistic regression 3 or more groups contrasted
• Ordered groups- compute “threshhold for classification as a “1” or “2” , “2” or “3” etc
• Unordered groups- can do pairwise logistic regression or a priori contrasts among groups as the organizer for binomial contrasting (eg. groups A and B vs. groups C, D, and E)
PREDICTIVE VALIDITY – DISCRIMINANT ANALYSIS
Test scores Group membership
eg, which MMPI scales differentiate/separate/predict manic depressives from normal functioning adults? This will be useful upon intake or commitment hearings in addition to clinical judgement
DISCRIMINANT ANALYSIS
2 Groups: statistical procedure is identical to multiple regression with group (1 or 2) as dependent variable, k test scores as predictors
3 or more Groups: discriminant analysis separates the groups based on a weighted sum of the predictors in standardized form
2 Group Analysis
Model:y = b1x1 + b2x2 + …bkxk + b0
y = 1 or 2 (or any two discrete numbers)
creates single predicted score Dhat which is the predicted score for each person. Can compare this predicted score with actual diagnoses or condition to determine % hit rate
2 Group hit rate
Example: predict male (1) vs. female (2) differences based on interests x1, x2, … xk
Each person receives a score yhat ; if yhat is below 1.5 the person is predicted to be a male, if over 1.5, a female.
Out of 100 persons (50 M, 50 F), by chance we would classify 50 correctly by chance;
2 Group hit rate
Cohen’s kappa will provide evidence for correct classification beyond chance:
k = Pc - P0/[1 - P0]
Alternatively, R2 for the regression provides evidence for classification beyond chance.
Coefficientsa
1.531 .049 31.415 .000
3.476E-03 .012 .008 .283 .777
-4.29E-03 .016 -.009 -.275 .784
1.492E-02 .015 .033 1.017 .309
(Constant)
Country Western Music
Blues or R & B Music
Jazz Music
Model1
B Std. Error
UnstandardizedCoefficients
Beta
Standardized
Coefficients
t Sig.
Dependent Variable: Respondent's Sexa.
ANOVAb
.291 3 9.714E-02 .395 .757a
344.415 1400 .246
344.707 1403
Regression
Residual
Total
Model1
Sum ofSquares df
MeanSquare F Sig.
Predictors: (Constant), Jazz Music, Country Western Music, Blues or R & B Musica.
Dependent Variable: Respondent's Sexb.
Example: Gender predicted from music preferences
R2 = SSb / SStot = .291/344.7 = .001
Descriptive Statistics
1500 1.57 .49 1 2
1404 1.3967 .4894 1.00 2.00
Respondent's Sex
SEXHAT
N MeanStd.
Deviation Minimum Maximum
Respondent's Sex * SEXHAT Crosstabulation
Count
382 226 608
465 331 796
847 557 1404
Male
Female
Respondent'sSex
Total
1.00 2.00
SEXHAT
Total
Symmetric Measures
.045 .094
.042 .025 1.674 .094
1404
Contingency CoefficientNominal by Nominal
KappaMeasure of Agreement
N of Valid Cases
ValueAsymp.
Std. Errora
Approx. Tb
Approx.Sig.
Not assuming the null hypothesis.a.
Using the asymptotic standard error assuming the null hypothesis.b.
Discriminant Analysis
Eigenvalues
.001a 100.0 100.0 .029Function1
Eigenvalue% of
VarianceCumulative
%CanonicalCorrelation
First 1 canonical discriminant functions were used in theanalysis.
a.
Wilks' Lambda
.999 1.185 3 .757Test of Function(s)1
Wilks'Lambda Chi-square df Sig.
Wilks lambda = 1-R2
Standardized Canonical Discriminant Function Coefficients
.263
1.139
-.306
Country Western Music
Jazz Music
Blues or R & B Music
1
Function
Canonical Discriminant Function Coefficients
.241
1.035
-.298
-2.507
Country Western Music
Jazz Music
Blues or R & B Music
(Constant)
1
Function
Unstandardized coefficients
Functions at Group Centroids
-3.33E-02
2.540E-02
Respondent's SexMale
Female
1
Function
Unstandardized canonical discriminant functions evaluated atgroup means
males females
0.0
w
3 Group discriminant analysis
2 or more discriminant functions possible # functions = min (#predictors, #gps-1) Evaluate greatest function (group
separation) first, each function successively
Examine joint classification for all significant functions
3 Group Analysis1st discriminant function
y2
y1
Group 1
means
Group 2
means
D1=b1y1+b2y2
Group 3
means
Maximize SS between groups
3 Group Analysis2nd discriminant function
y2
y1
Group 1
means
Group 2
means
D1=b1y1+b2y2
Group 3
means
D2=b3y1+b4y2
3 Group Analysis
y2
y1
Group 1
means
Group 2
means
D1=b1y1+b2y2
R12 = SSD1 / SStot
Group 3
means
D2=b3y1+b4y2
R22 = SSD2 / SStot
3 Group Analysis
y2
y1
Group 1
means
Group 2
means
D1=b1y1+b2y2
Group 3
means
D2=b3y1+b4y2
Discriminant
function
coefficients
Example: Ethnic music prefsWilks' Lambda
.921 115.483 6 .000
.993 9.691 2 .008
Test of Function(s)1 through 2
2
Wilks'Lambda Chi-square df Sig.
Standardized Canonical Discriminant Function Coefficients
-.644 .072
.483 -.858
.374 1.142
Country Western Music
Jazz Music
Blues or R & B Music
1 2
Function
Structure Matrix
.736* -.243
-.656* .212
.595 .680*
Jazz Music
Country Western Music
Blues or R & B Music
1 2
Function
Pooled within-groups correlations between discriminatingvariables and standardized canonical discriminant functions Variables ordered by absolute size of correlation within function.
Largest absolute correlation between each variable andany discriminant function
*.
Canonical Discriminant Function Coefficients
-.599 .067
.448 -.796
.369 1.127
-.688 -.906
Country Western Music
Jazz Music
Blues or R & B Music
(Constant)
1 2
Function
Unstandardized coefficients
Functions at Group Centroids
.105 -1.70E-02
-.805 -3.12E-02
-6.54E-02 .371
Racew of Respondentwhite
black
other
1 2
Function
Unstandardized canonical discriminant functions evaluated atgroup means
Territorial MapFunction 2 -3.0 -2.0 -1.0 .0 1.0 2.0 3.0 +---------+---------+---------+---------+---------+---------+ 3.0 + 21 + I 21 I I 21 I I 21 I I 21 I I 21 I 2.0 + 21 + + + + + + I 21 I I 21 I I 21 I I 21 I I 21 I 1.0 + 21 + + + + + + I 21 I I 21 I I 21 I I 21 * I I 21 I .0 + 21 + + * +* + + + I 21 I I 21 I I 21 I I 21 I I 21 I -1.0 + 21 + + + + + + I 21 I I 21 I I 21 I I 21 I I 21 I -2.0 + 21 + + + + + + I 21 I I 21 I I 21 I I 21 I I 21 I -3.0 + 21 + +---------+---------+---------+---------+---------+---------+ -3.0 -2.0 -1.0 .0 1.0 2.0 3.0 Canonical Discriminant Function 1
Symbol Group Label------ ----- --------------------
1 1 white 2 2 black 3 3 other * Indicates a group centroid