ASVAB: E Pluribus Unum?
description
Transcript of ASVAB: E Pluribus Unum?
ASVAB: E Pluribus Unum?
Martin J. Ippel, Ph.D.CogniMetrics Inc,San Antonio,TX
Steven E. Watson, Ph.D.U.S. Navy Selection & Classification (CNO 132)
Washington, DC
1
The ASVAB is the principal instrument for selection and classification in the U.S. Armed Forces.
Assumption: measurement invariance acrossfull range of scores.
Relevance: what is the “population of interest”of the ASVAB?
Recent studies cast doubt on this assumption.
2
Two related phenomena suggest a changing factor structure along the dimension of general intelligence (g):
• The g factor gets smaller in high-g samples
• Cognition tests have smaller loadings on “g” in high-g samples
3
Spearman (1927) noticed already a decrease in the positive manifold of cognition variables at higher g levels.
differentiation of intelligence
Spearman’s explanation:
4
5
The present study adheres to an alternative explanation:
The phenomenon follows from the Pearson-Lawley selection rules.
an underlying selection processchanges the variance-covariancestructure and the mean structure
6
One phenomenon:
Decrease in positive manifold of cognition variables in high-g samples
Two explanations:
differentiation of intelligence
selection effects
Consequences of:
• differentiation:
• selection effects:
structure is changing
underlying structureinvariant
7
8
Critical developments in psychometric theory:
• Meredith (1964) showed that both the covariance structure and mean structure change if samples are selected based on one or more latent variables (e.g., the g factor).
• Meredith (1965) developed procedures to derive the single best fitting (i.e., invariant) factor pattern derived from sets of factors obtained on populations differing on a latent variable.
• Jöreskog (1971) formalized this viewpoint as an extension of the common factor model for a parent population to multiple groups based on one or more latent variables in the model.
(df) Measurement Invariance:
If we compare groups, or individuals of different groups, then the expected value of test scores of a person of a given level of ability should be independent of membership of these groups (Mellenbergh, 1989).
In formule:
f (Y | η, ν) = f (Y | η)
y1ij = τ1i + λ1i ηij + ε1ij
f depends on the measurement model of choice:
9
10
y1ij = τ1i + λ1i ηij + ε1ij
change
invariant
ratings cluster 1
ratings cluster 2
η11 η12
Unequal intercepts
11
ratings cluster 1
ratings cluster 2
η12
η11
Unequal factor loadings
12
13
parentpopulation(N = 48,222)
a-selectsample(n=1,000)
hi-gsample(n=600)
av-gsample(n=600)
lo-gsample(n=600)
StatisticalExperiment:
14
parentpopulation(N = 48,222)
a-selectsample(n=1,000)
hi-gsample(n=600)
av-gsample(n=600)
lo-gsample(n=600)
StatisticalExperiment:
determine factorstructure and thensample
eigenvalue database 1 2 3 4 5 61 3.93 3.971 3.876 2.475 3.888 3.811 4.208
2 1.328 1.324 1.314 1.241 1.337 1.386 1.212
3 1.039 1.008 1.061 1.164 1.077 1.039 0.974
4 0.7 0.701 0.713 0.922 0.702 0.689 0.683
N= 48222 1000 1000 1000 966 954 1015
samples
Eigenvalues from a-select samples drawn from the parent population of Air Force recruits
15
General Science (GS): a 25 items knowledge test of physical and biological
sciences.
Arithmetic Reasoning (AR): a 30 items arithmetic word problem test.
Word Knowledge (WK): 35 items testing knowledge of words and synonyms.
Paragraph Comprehension (PC): 15 items testing the ability to extract meaning from short paragraphs.
Auto and Shop I nformation (AS): a 25 items knowledge test of automobiles, shop practices, tools and tool use.
Mathematical Knowledge (MK): a 25 items test of algebra, geometry, fractions, decimals, and exponents.
Mechanical Comprehension (MC): a 25 items test of mechanical and physical principles and ability to visualize how illustrated objects work.
Electronics I nformation (EI ): a 20 items test measuring knowledge about electronics, radio, and electrical principles.
Assembling Objects (AO): a 16 items spatial visualization test.
ASVAB tests and their measurement claims
16
0
QUANTI-TATIVE
MK
0,
e2
1
1
AR
0,
e1
1
0
VERBAL
PC
0,
e4
WK
0,
e3
1
11
0
TECHNICALKNOWLEDGE
MC
0,
e7
GS
0,
e6
1
11
AS
0,
e8
1
EI
0,
e9
1
0,
GENERALINTELLI-GENCE
0,
e11
1
0,
e10
1
0,
e12
1
AO
0,
e5
1
1
Model 1: A hierarchical model of “g”17
0, v
QUANTI-TATIVE
MK
0,
e2
1
1
AR
0,
v1
1
0, v
VERBAL
PC
0,
e4
WK
0,
e3
1
1
1
0, v
TECHNICALKNOWLEDGE
MC
0,
e7
GS
0,
e6
1
11
AS
0,
e8
1
EI
0,
e9
1
AO
0,
e5
1
0,
GENERALINTELLIGENCE
1
Model 2: A “g as first principal factor” model
18
19
parentpopulation(N = 48,222)
a-selectsample(n=1,000)
hi-gsample(n=600)
av-gsample(n=600)
lo-gsample(n=600)
StatisticalExperiment:
determine factorstructure andsample
20
sample N m s.d.skewnes
skurtosi
s
random 1000 41.03 3.7 0.04 -0.68
g-hi 600 44.76 1.73 0.12 -0.11
g-av 600 41.19 1.73 0.12 -0.11
g-lo 600 37.61 1.73 0.12 -0.11
Distributional properties of samples generated from the parent population based on a latent
variable "g"
45.0
47.0
49.0
51.0
53.0
55.0
57.0
59.0
61.0
63.0
MK AR WK PC GS MC AS EI AO
ASVAB tests
mea
n t
est
sco
res
random
g-hi
g-av
g-lo
ASVAB tests mean scores in samples with different levels of "g"
21
y = -0.4812x + 0.3567
R2 = 0.9388
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
-10.0% 0.0% 10.0% 20.0% 30.0% 40.0% 50.0%
Reduction Variance
Ave
rag
e C
orr
elat
ion
22
Average correlation lower with lower variance
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
MK AR WK PC GS MC AS EI AO
ASVAB tests
per
cen
tag
e va
rian
ce r
edu
ctio
n
g-lo
g-av
g-hi
23
The effects of selection based on the latent variable “g”on the variance of ASVAB tests
df par X2 X2 diff. sign. CMI N/ DFRMSEAPCLOSE NFI CFI
109 53 3106.26 - 28.498 0.124 0 0.959 0.961106 56 709.27 2396.99 p < 0.001 6.691 0.056 0.004 0.991 0.99282 80 254.29 454.97 p < 0.001 3.101 0.034 1 0.997 0.99874 88 209.08 45.22 p < 0.0012.825 0.032 1 0.997 0.998
3. RESVAR free model4. Model 3 & some intercepts free
model
1. Full MI model2. RESVAR group-invariant model
Sequence of MCFA model fits and goodness of fit indices
24
Non-standardized MCFA factor loadings.
parent parentestimate estimate s.e. sign. estimate estimate s.e. sign. estimate estimate s.e. sign. estimate estimate s.e. sign.
WK 0.543 0.501 0.115 0 1 1 0 0PC 0.675 0.484 0.044 0 0.461 0.484 0.044 0
MK 1.039 -0.317 0.162 0.05 1 1 0 0AR 1.338 0.453 0.11 0 0.243 0.39 0.063 0
GS 1.093 0.339 0.145 0.02 0.673 0.839 0.073 0 0.335 0.328 0.05 0MC 1.346 0.941 0.261 0 1 1 0 0AS 0.715 0.568 0.249 0.02 1.497 1.8 0.115 0EI 0.96 0.113 0.175 0.52 0.257 0.42 0.061 0 0.962 1.059 0.087 0
AO 1 1 0 0 -0.207 -0.259 0.092 0
mcfa mcfa mcfa mcfaG Verbal Quantitative Technical Knowledge
25
Standardized MCFA factor loadings
aselect high average low aselect high average low aselect high average low aselect high average low aselect high average low
gverbal 0 0 0 0quant. 0 0 0 0 -0.251 -0.003 -0.103 -0.571
TK 0 0 0 0 0.18 0.158 0.157 0.21 -0.054 -0.43 -0.505 -0.573
WK 0.468 0.306 0.24 0.254 0.798 0.841 0.808 0.817 0.856 0.802 0.71 0.732PC 0.482 0.269 0.199 0.212 0.304 0.37 0.335 0.342 0.325 0.393 0.433 0.435
MK 0.656 -0.158 -0.117 -0.137 0.618 0.934 0.832 0.603 0.772 0.897 0.706 0.382AR 0.766 0.242 0.187 0.224 0.129 0.389 0.361 0.269 0.603 0.21 0.165 0.122
GS 0.633 0.154 0.126 0.133 0.361 0.523 0.526 0.529 0.179 0.185 0.22 0.2 0.586 0.373 0.378 0.382MC 0.673 0.382 0.301 0.325 0.463 0.506 0.577 0.539 0.667 0.402 0.423 0.396AS 0.379 0.197 0.146 0.172 0.733 0.779 0.835 0.85 0.681 0.646 0.718 0.751EI 0.528 0.046 0.035 0.039 0.131 0.232 0.217 0.232 0.49 0.532 0.587 0.569 0.559 0.393 0.433 0.435
AO 0.531 0.431 0.333 0.324 -0.102 -0.154 -0.145 -0.135 0.292 0.21 0.132 0.123
Table 6. Standardized MCFA factor loadings.
G Verbal Quantitative Technical Knowledge Communality
26
27
y1ij = τ1i + λ1i ηij + ε1ij
change
should remain invariant
28
y1ij = τ1i + λ1i ηij + ε1ij
change
not invariant
29
Discussion:
• ASVAB is measurement invariant in a limited sense: only factor loadings are invariant across different levels of “g”. (weak factorial invariance).
• ASVAB seems to be measuring too many factors with too few tests.
• more factors than eigenvalues larger than 1.• many tests have communalities < 0.60.• intercepts could not be constrained to be equal
(indicating: other factors influence test scores).