Clyde Tucker and Brian Meekins U.S. Bureau of Labor Statistics and Paul Biemer
description
Transcript of Clyde Tucker and Brian Meekins U.S. Bureau of Labor Statistics and Paul Biemer
www.bls.gov
Using Substantive Diagnostics to Evaluate the Validity of Micro-level
Latent Class Indicators of Measurement Error
Clyde Tucker and Brian MeekinsU.S. Bureau of Labor Statistics
and Paul BiemerResearch Triangle Institute
Background Developed by Lazarsfeld (1950)—
unobserved or “latent” variable drawn from relationships between two or more “manifest” variables
Lazarsfeld and Henry (1968) and Goodman (1974) extended mathematics of theory
Software for latent class analysis (LCA) developed (MLLSA, lEM, M-PLUS)
LCA used to study measurement or response error (VandePol and deLeeuw 1986; Tucker 1992; Van de Pol and Langeheine 1997; Bassi et al. 2000; Biemer and Bushery 2000; Tucker, et al. 2002, 2003, 2004, 2005, 2006, and 2008)
Creation of Manifest Variables
Try to create at least three Try to avoid direct relationships
with outcome variable (expenditures, in this case)
Use LCA to “triangulate” them to produce a latent variable with more information than any one of them alone
Statistical Logic
In mathematical terms, when manifest variables A and B are not independent, the following relationship will not hold:
where i indexes the classes of A, j indexes the classes of B, πij
AB is the probability an individual is in cell ij, πi
A is the probability an individual is in class i, and πj
B is the probability an individual is in class j.
Bj
Ai
ABij
Statistical Logic For the above expression to be true, A and B must
be independent. The purpose of the latent variable X is to achieve that independence. Thus, the following latent class model is desired:
where t indexes the classes of X, πijtABX is the
probability of being in cell ijt of the unobserved ABX table, πt
X is the probability that an individual is in one of the mutually exclusive and exhaustive classes of X, πit
AX and πjtBX are the conditional
probabilities that an individual is in a particular class of A and B, respectively, given that a person is in a certain class of X. Equation (2) indicates that, within a class of X, A and B are independent.
BXjt
AXit
Xt
ABXijt
Purpose of Paper
Concept of LCA relatively straightforward—create a variable to account for common variance among observed variables
Issues: What is the new variable? What do its classes mean? Does it really tell us anything useful?
Statistical diagnostics don’t help us here. We need substantive ones.
Paper explores some of this type of diagnostics
Data Sources
CED 2 week diaries All expenditures Small items and grocery expenditures Used for CPI cost weights
CEQ 5 quarters (first for bounding) PV All consumer expenditures 2 hours Larger consumer items Used for CPI cost weights
Three Examples
1985 CED Operational Test (micro level) 3 treatments—specific, nonspecific, control 800 households in each Latent response error measure of underreporting of grocery
expenditures using manifest performance indicators CEQ (1996-2001) (micro level)
Only analyzed the 2nd wave 43,000 completed 2nd wave interviews Latent response error measure of underreporting for 7
expenditure categories for purchasers using manifest performance indicators
CEQ (1996-2001) (micro level) Analyzed all four waves 14,877 remained in sample throughout Latent response error measure of underreporting for almost
30 expenditure categories for all households (purchasers and nonpurchasers) using manifest performance indicators and indicators of pattern of wave nonresponse
Critical Assumption
Response errors in CE only come from underreporting of expenditures and not overreporting Tedious Time-consuming Recall problems Lack of knowledge
Methodological Issues
Weighted vs. unweighted Variances for complex sample design
vs. SRS Local vs. global maxima Sparse cells (too many manifest
variables) Restricted vs. unrestricted models Boundary problems (no overreporting)
1985 Diary Test
Manifest variables Difference in first and second week grocery
expenditures Difference in usual and average weekly grocery
expenditures Amount of expenditure information collected by
recall Respondent’s attitudes and behavior with
respect to diarykeeping Latent variable
3 classes (low, moderate, high response error)
CEQ Micro-level Manifest Indicators for First Study
Interview level indicators considered:1. Number of contacts2. Ratio of respondents/household members3. Missing income data4. Type and frequency of records used5. Length of interview6. Ratio of expenditures in last month to
quarter7. Combination of type of record and
interview length
Indicator Coding
#contacts (1=0-2; 2=3-5; 3=6+) Resp/hh size (1= <.5; 2= .5+) Income missing (1=present; 2=missing) Records use (1=never; 2=single type or
sometimes; 3=multiple types and always) Interview length (1= <45; 2=45-90; 3= 90+) Month3 expn/all (1= <.25; 2= .25-.5; 3= +.5) Combined records and length (1= poor; 2=
fair; 3=good)
Latent Variables
Three-class latent variables (poor, fair, good reporting) for Kid’s Clothing Women’s Clothing Men’s Clothing Furniture Electricity Minor Vehicle Repairs Kitchen Accessories
Second CEQ Micro-level Study
Based on results of first CEQ study, analysis of purchasers and nonpurchasers together
Used Interviews 2-5 data. Not limited to within-interview indicators Developed model using all Interview 2
respondents Latent variable is still intended to
represent quality of reporting
New Manifest Indicators
Overall Panel level indicators considered1. Number of completed interviews (1-4)2. Attrition combined with # of complete
interviews3. Average number of commodity categories for
which CU had expenditure4. Number of interviews the ratio of third month
expenditure to quarter was between .25 - .55. Panel averages of interview level indicators
from first CEQ study
Model Selection Ran both ordered (fixed or restricted ordinal
constraints) latent class models and unordered. Order was determined based on theoretical
relationship between values of indicators and level of underreporting.
Ran all combinations of indicators in groups of 3 & 4, using 3 or 4 category LC variable for each commodity category & overall
Multiple iterations to avoid local maxima Best model candidates were selected based on
fit From those candidates, models selected based
on relationship of indicators to latent construct
Application of Model For the final models for each commodity:
Each combination of indicators was assigned to a latent class based on probability of being in that class given the value of the indicators
Ran demographic analysis to identify characteristics of members of each latent class
Expenditure means were found for each latent class
Examined the pattern of mean expenditure and the contribution of the latent variable in predicting these expenditures
Second CEQ Micro-level Study– Expenditure Categories
Cable/satellite TV Men’s apparel Women’s apparel Men’s clothing only Women’s clothing only Men’s accessories Women’s accessories Men’s shoes Women’s shoes Kid’s apparel Kid’s clothing only Kid’s Accessories Kid’s shoes Dental care Drugs and medical supplies Electricity Gas (household) Eye care
Sports equipment Televisions, video, & sound
equip. Vehicle service, major Vehicle service, minor Vehicle service, oil changes only Vehicle expenses, other Pets and pet supplies Sports equipment Trash collection Televisions, video, & sound
equip. Vehicle service, major Vehicle service, minor Vehicle service, oil changes only Vehicle expenses, other Pets and pet supplies Kitchen accessories Other household items
Conclusions
When doing LCA for measuring response error, one cannot rely on statistical diagnostics alone. Substantive diagnostics are needed to judge the meaningfulness of the results.
Sometimes the models work and sometimes they don’t. Unfortunately, this is likely to depend on the characteristic you’re analyzing.
We need better manifest variables to explain more variance. We have been unable to develop meaningful latent variables
with more than three or four categories, and, in some cases, we could only identify two. LCA software really does work best with large sample sizes.
Besides only defining a few latent classes, we certainly will not progress beyond the most rudimentary ordinal rankings any time soon.
LCA problems are likely to be multiplied many times for response error measures for non-factual items such as attitudes or opinions.
Contact Information
www.bls.gov
Clyde TuckerSenior Survey Methodologist
OSMR202-691-7371