Scaling Measuring the Unobservable. Scaling Scaling involves the construction of an instrument that...
-
Upload
olivia-fox -
Category
Documents
-
view
216 -
download
0
Transcript of Scaling Measuring the Unobservable. Scaling Scaling involves the construction of an instrument that...
![Page 1: Scaling Measuring the Unobservable. Scaling Scaling involves the construction of an instrument that associates qualitative constructs with quantitative.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e045503460f94af07b7/html5/thumbnails/1.jpg)
Scaling
Measuring the Unobservable
![Page 2: Scaling Measuring the Unobservable. Scaling Scaling involves the construction of an instrument that associates qualitative constructs with quantitative.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e045503460f94af07b7/html5/thumbnails/2.jpg)
Scaling
Scaling involves the construction of an instrument that associates qualitative constructs with quantitative metric units.
Scaling evolved out of efforts to measure "unmeasurable" constructs like authoritarianism and self esteem.
Note: We are talking about constructed scales involving multiple items, not a response scale for a particular question.
![Page 3: Scaling Measuring the Unobservable. Scaling Scaling involves the construction of an instrument that associates qualitative constructs with quantitative.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e045503460f94af07b7/html5/thumbnails/3.jpg)
Scaling
How do we define or “capture” or measure a nebulous concept?
By “taking stabs” from several directions, we can get a more complete picture of a concept we know exists but cannot see.
![Page 4: Scaling Measuring the Unobservable. Scaling Scaling involves the construction of an instrument that associates qualitative constructs with quantitative.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e045503460f94af07b7/html5/thumbnails/4.jpg)
Scaling
In scaling, we have several items that are intended to “capture” a piece of the underlying concept.
The items are then combined in some form to create the scale.
Quite technically, we will talk about scales and indexes interchangeably. Scales are composed of items caused by an underlying construct, whereas indexes are composed of items that indicate the level of a construct and might be useful together to predict outcomes.
![Page 5: Scaling Measuring the Unobservable. Scaling Scaling involves the construction of an instrument that associates qualitative constructs with quantitative.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e045503460f94af07b7/html5/thumbnails/5.jpg)
Scaling
LatentVariable
Observed Item 1
Observed Item 2
Observed Item 3
Observed Item 4
e1 e2 e3 e4
Graphical depiction of a scale:
![Page 6: Scaling Measuring the Unobservable. Scaling Scaling involves the construction of an instrument that associates qualitative constructs with quantitative.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e045503460f94af07b7/html5/thumbnails/6.jpg)
Scaling
+
+
+
Form an Index
Observed Item 1
Observed Item 2
Observed Item 3
Observed Item 4
Graphical depiction of
an index:
![Page 7: Scaling Measuring the Unobservable. Scaling Scaling involves the construction of an instrument that associates qualitative constructs with quantitative.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e045503460f94af07b7/html5/thumbnails/7.jpg)
Scaling
In most scaling, the objects are text statements, usually statements of attitude or belief.
![Page 8: Scaling Measuring the Unobservable. Scaling Scaling involves the construction of an instrument that associates qualitative constructs with quantitative.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e045503460f94af07b7/html5/thumbnails/8.jpg)
Scaling
A scale can have any number of dimensions in it. Most scales that we develop have only a few dimensions.
What's a dimension? If you think you can measure a person's
self-esteem well with a single ruler that goes from low to high, then you probably have a unidimensional construct.
![Page 9: Scaling Measuring the Unobservable. Scaling Scaling involves the construction of an instrument that associates qualitative constructs with quantitative.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e045503460f94af07b7/html5/thumbnails/9.jpg)
Scaling
![Page 10: Scaling Measuring the Unobservable. Scaling Scaling involves the construction of an instrument that associates qualitative constructs with quantitative.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e045503460f94af07b7/html5/thumbnails/10.jpg)
Scaling Many familiar concepts (height, weight,
temperature) are actually unidimensional. But, if the concept you are studying is in fact
multidimensional in nature, a unidimensional scale or number line won't describe it well. E.g., academic achievement: how do you score someone who is a high math achiever and terrible verbally, or vice versa?
A unidimensional scale can't capture that type of achievement.
![Page 11: Scaling Measuring the Unobservable. Scaling Scaling involves the construction of an instrument that associates qualitative constructs with quantitative.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e045503460f94af07b7/html5/thumbnails/11.jpg)
Scaling
Factor analysis can tell you whether you have a unidimensional or multidimensional scale—helping you discover the number of dimensions or scales that exist among a group of variables.
Factor analysis is typically an exploratory process, but it can be confirmatory.
Exploratory factor analysis helps you reduce data by grouping variables into sets that tap the same phenomena.
![Page 12: Scaling Measuring the Unobservable. Scaling Scaling involves the construction of an instrument that associates qualitative constructs with quantitative.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e045503460f94af07b7/html5/thumbnails/12.jpg)
Scaling
Steps in factor analysis (what the computer does):
1. Assumes one factor and checks the correlation of each item with the proposed factor and compares the proposed inter-item correlations with the actual inter-item correlations.
Item 1
Item 2
FactorSum of 1,2
Proposed Model Actual Data
A = 1’s correlation with factor
B = 2’s correlation with factor
By definition, Item 1 & 2’s correlation is A * B
A
B
Compared
with
Do they
Match?
Item 1
Item 2
Correlation
![Page 13: Scaling Measuring the Unobservable. Scaling Scaling involves the construction of an instrument that associates qualitative constructs with quantitative.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e045503460f94af07b7/html5/thumbnails/13.jpg)
Scaling
Steps in factor analysis (what the computer does):
2. If the single concept is not a good model, the computer rejects one factor and forms a residual correlation matrix (real 1,2 – proposed A*B)
3. Identifies a second concept that may explain some of the remaining correlation and checks the proposed inter-item correlation against the real correlations.
4. And so on until the correlations match.
![Page 14: Scaling Measuring the Unobservable. Scaling Scaling involves the construction of an instrument that associates qualitative constructs with quantitative.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e045503460f94af07b7/html5/thumbnails/14.jpg)
Scaling
In actuality, factor analysis will give K factors for K variables. The last residual correlation matrix will result in zeros.
So, how many factors should you use?• You could use statistical criteria: extract factors until
matrix is not statistically significant from zero.
• Historically, number of factors has been determined by substantial needs, intuition, and theory.
![Page 15: Scaling Measuring the Unobservable. Scaling Scaling involves the construction of an instrument that associates qualitative constructs with quantitative.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e045503460f94af07b7/html5/thumbnails/15.jpg)
Scaling
Guideline for subjective analysis: A group of factors should be able to explain a high proportion of total covariance among a set of items.• Eigenvalue test
• Scree Test
![Page 16: Scaling Measuring the Unobservable. Scaling Scaling involves the construction of an instrument that associates qualitative constructs with quantitative.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e045503460f94af07b7/html5/thumbnails/16.jpg)
Scaling
Eigenvalues• An eigenvalue represents the number of units of
information that a factor explains in a k set of variables with k units of information.
• E.g., when k = 10, an eigenvalue of 3 represents 30% of information is explained by the factor.
• An eigenvalue of 1 corresponds with a variable’s worth of information. Therefore, factors with an eigenvalue of 1 or less do not help to reduce data.
• Get rid of factors with eigenvalues less than 1
![Page 17: Scaling Measuring the Unobservable. Scaling Scaling involves the construction of an instrument that associates qualitative constructs with quantitative.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e045503460f94af07b7/html5/thumbnails/17.jpg)
Scaling
Scree Test• Most researchers are looking for stronger,
fewer factors (they want to reduce data). Therefore, they tend to use the scree plot.• Plot the eigenvalues relative to each other
• Strong factors form a steep slope, weaker factors form a plateau
• Retain those factors that lie above the “elbow” of the plot—like with gangrene, cut off the elbow!
Scree plot for 5 variables2
1
![Page 18: Scaling Measuring the Unobservable. Scaling Scaling involves the construction of an instrument that associates qualitative constructs with quantitative.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e045503460f94af07b7/html5/thumbnails/18.jpg)
Scaling
In addition, factors should be composed of similar, logically linked items. This is an especially helpful rule when the number of factors is not that obvious.
![Page 19: Scaling Measuring the Unobservable. Scaling Scaling involves the construction of an instrument that associates qualitative constructs with quantitative.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e045503460f94af07b7/html5/thumbnails/19.jpg)
Scaling
Factor Rotation• Factor rotation involves using an algorithm
to maximize the correlation of items to a factor—making each item appear most relevant to a single factor.
• The point is to identify variables that most similarly form indicators of the same factor—each factor’s variables being most clearly highlighted.
![Page 20: Scaling Measuring the Unobservable. Scaling Scaling involves the construction of an instrument that associates qualitative constructs with quantitative.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e045503460f94af07b7/html5/thumbnails/20.jpg)
Scaling
Factor Rotation• The best-scenario (never happens) is when
all items load (correlate with) as 1 on a single factor and 0 on all the rest. This is called simple structure.
• Factor rotation mathematically takes the items as close as possible to simple structure.
![Page 21: Scaling Measuring the Unobservable. Scaling Scaling involves the construction of an instrument that associates qualitative constructs with quantitative.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e045503460f94af07b7/html5/thumbnails/21.jpg)
Scaling Factor Rotation
• Orthogonal versus oblique rotation• Orthogonal rotation makes factors completely independent of each
other. • This is preferred for finding the most unique factors. Use if factors
ought not be related.• Any item’s variation explained by one factor can be added to that of
another factor to get the total variation explained by the two.• If you find lots of cross-loading, you should consider “Oblique.”
• Oblique rotation makes factors that are allowed to be correlated with each other to some degree.
• Use if the factors ought to be related.• There is redundancy in the variation of any item explained by one
factor versus another, such that they have overlapping explanatory power.
• You might want to try both and look for simple structure.• Strong loadings on two factors may indicate a single factor, high
correlation of two factors may indicate a single factor.
![Page 22: Scaling Measuring the Unobservable. Scaling Scaling involves the construction of an instrument that associates qualitative constructs with quantitative.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e045503460f94af07b7/html5/thumbnails/22.jpg)
Scaling
Factor Rotation
• Items with a high loading on (high correlation with) a factor form the factor’s variable for research purposes.
• Common elements of the items is likely what the factor represents.
![Page 23: Scaling Measuring the Unobservable. Scaling Scaling involves the construction of an instrument that associates qualitative constructs with quantitative.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e045503460f94af07b7/html5/thumbnails/23.jpg)
Scaling
Type of analysis in extracting factors:
• Principal components analysis produces specified proportion of total variance among items explained.
• Common factor analysis produces specified proportion of shared variance among items explained.
• Bottom line: report which you used.
![Page 24: Scaling Measuring the Unobservable. Scaling Scaling involves the construction of an instrument that associates qualitative constructs with quantitative.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e045503460f94af07b7/html5/thumbnails/24.jpg)
Scaling Exploratory versus Confirmatory Analysis
• Exploratory is that which we have been discussing. If using exploratory, with new samples you “rediscover” a structure in each sample—you have persuasive evidence of the structure.
• Confirmatory typically refers to models generated by Structural Equation Modeling where items are specified to form a factor in advance. The question becomes, “How well do the data fit a specified model using statistical inference?” You have to be careful not to overproduce many meaningless factors.
![Page 25: Scaling Measuring the Unobservable. Scaling Scaling involves the construction of an instrument that associates qualitative constructs with quantitative.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e045503460f94af07b7/html5/thumbnails/25.jpg)
Scaling
Validity and Reliability• Like other measures, scales and indexes must
be valid and reliable to be useful.• Validity: Face, Content, Criterion, Construct
• A particular kind of reliability that is particularly useful for scales and indexes is inter-item reliability (internal consistency or high inter-item correlation)
• To the degree that the items are correlated, the common correlation is attributable to the true score of the latent variable.
![Page 26: Scaling Measuring the Unobservable. Scaling Scaling involves the construction of an instrument that associates qualitative constructs with quantitative.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e045503460f94af07b7/html5/thumbnails/26.jpg)
Scaling
Inter-item Reliability—Alpha • Variation in each item is caused by the latent
variable and error (unique for each)
• Common variation is caused by the latent variable.
• Using the variance/covariance matrix, you can see total variance in the sum of components.
• The diagonal (variance) represents unique variation for each item.
• The off-diagonal represents co-variation of items. This also equals 1 – (Unique/Total)
![Page 27: Scaling Measuring the Unobservable. Scaling Scaling involves the construction of an instrument that associates qualitative constructs with quantitative.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e045503460f94af07b7/html5/thumbnails/27.jpg)
Scaling
Inter-item Reliability—Alpha
• The off-diagonal represents co-variation of items. This also equals 1 – (Unique/Total)
• To correct for the ways variance/covariance matrices change with number of items, the formula above is adjusted by k/k-1, where k = number of items. This constrains alpha to range from 0 to 1.
k Unique variance
= k – 1 Total variance
![Page 28: Scaling Measuring the Unobservable. Scaling Scaling involves the construction of an instrument that associates qualitative constructs with quantitative.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e045503460f94af07b7/html5/thumbnails/28.jpg)
Scaling
Inter-item Reliability—Alpha • Some characteristics of alpha
• Holding correlation constant, alpha goes up with more scale items
• To improve a scale, look for effect on alpha if an item were dropped.
• Reliability is not good unless it is .65 or above. “Best” reliability would be around .9.
• Good scales require a balance between reliability and length.
![Page 29: Scaling Measuring the Unobservable. Scaling Scaling involves the construction of an instrument that associates qualitative constructs with quantitative.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e045503460f94af07b7/html5/thumbnails/29.jpg)
Scaling Creating a scale1. Determine what you want to measure
Clarity Specificity
2. Generate an item pool Scales may be generated from 40 to 100 items Good reason to reuse scales Writing
Positive and negative items Stay on topic Avoid lengthy items Keep wording simple Avoid multiple negatives No double-barreled items
![Page 30: Scaling Measuring the Unobservable. Scaling Scaling involves the construction of an instrument that associates qualitative constructs with quantitative.](https://reader035.fdocuments.in/reader035/viewer/2022062421/56649e045503460f94af07b7/html5/thumbnails/30.jpg)
Scaling Creating a scale3. Determine format for measurement.
Response options Broad versus narrow
4. Review of item pool by experts5. Include scale validation items6. Administer to a development sample7. Evaluate items
Differences between items You need item variance Look for means in the middle of the scale Item-scale correlations