psychometric-instrument-development-reliabilities-and-composite-scores-1206978537846832-2.odp
description
Transcript of psychometric-instrument-development-reliabilities-and-composite-scores-1206978537846832-2.odp
Survey Design II
Lecture 6
Survey Research & Design in Psychology
James Neill, 2013
Psychometric Instrument Development
Image source: http://commons.wikimedia.org/wiki/File:Soft_ruler.jpg, CC-by-SA 3.0
7126/6667 Survey Research & Design in PsychologySemester 1, 2013, University of Canberra, ACT, AustraliaJames T. NeillHome page: http://en.wikiversity.org/wiki/Survey_research_and_design_in_psychologyLecture page: http://en.wikiversity.org/wiki/Psychometric_instrument_development
Image source:http://commons.wikimedia.org/wiki/File:Soft_ruler.jpgImage author: Lite, http://commons.wikimedia.org/wiki/User:LiteImage license: GFDL 1.2+, CC-by-SA 3.0 unported
Description: This lecture recaps the previous lecture on exploratory factor analysis, and introduces psychometrics and (fuzzy) concepts and their measurement, including (operationalisation), reliability (particularly internal consistency of multi-item measures), validity and the creation of composite scores.
This lecture is accompanied by a computer-lab based tutorial.http://en.wikiversity.org/wiki/Survey_research_and_design_in_psychology/Tutorials/Psychometrics
Overview
Recap: Exploratory factor analysis
Concepts & their measurement
Measurement error
Psychometrics
Reliability & validity
Composite scores
Writing up instrument development
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svgLicense: Public domain
Bryman & Cramer (1997). Concepts and their measurement. [Chapter - eReserve]
DeCoster, J. (2000). Scale construction notes. http://www.stat-help.com/scale.pdf
Howitt & Cramer (2005). Reliability and validity: Evaluating the value of tests and measures. [Chapter eReserve]
Howitt & Cramer (2011). Reliability in scales and measurement: Consistency and measurement. [Chapter Textbook]
Wikiversity. Measurement error - http://en.wikiversity.org/wiki/Measurement_error
Wikiversity. Reliability and validity - http://en.wikiversity.org/wiki/Reliability_and_validity
Readings: Psychometrics
Recap:
Exploratory Factor Analysis
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svgLicense: Public domain
What is factor analysis?
Factor analysis is: a family of multivariate correlational data analysis methods
used to identify clusters of covariance (called factors)
Two main types (extraction methods):Exploratory factor analysis (EFA)
Confirmatory factor analysis (CFA)
EFA assumptions
Sample size5+ cases per variables (min.)
20+ cases per variable (ideal)
Another guideline: Or N > 200
Check bivariate outliers & linearity
Factorability: check any of:Correlation matrix: Some over .3?
Anti-image correlation matrix diags > .5
Measures of Sampling AdequacyKMO > ~ .5 to 6; Bartlett's sig?
Normality enhances the solution
Summary of EFA steps / process
Test assumptions
Sample size, Outliers & linearity, Factorability
Select type of analysis
Extraction (PC/PAF),
Rotation (Orthorgonal(Varimax)/Oblique(Oblimin)
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svgLicense: Public domainSee also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorialSee also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorial
Summary of EFA steps / process
Determine no. of factors
Theory
Kaiser's criterion
Eigen Values and Scree plot
% variance explained
Interpretability of weakest factor
Select items
Check factor loadings to identify which items belong in which factor
Drop items 1-by-1 if primary loading low
cross-loadings high
item wording doesn't belong to meaning of factor
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svgLicense: Public domainSee also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorialSee also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorial
Summary of EFA steps / process
Name and define factors
Examine correlations amongst factors
Check factor structure for sub-groups
Analyse internal reliability
Compute composite scores
Covered in this lecture
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svgLicense: Public domainSee also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorialSee also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorial
271 UC students responded to 24 university student motivation
statements in 2008 using an 8-point Likert scale (False to True)
e.g.,
I study at university
to enhance my job prospects.
because other people have told me I should.
EFA PC Oblimin revealed 5 factors
Example EFA:
University student motivation
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svgLicense: Public domainSee also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorialSee also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorial
Example EFA:
Pattern matrix
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svgLicense: Public domainSee also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorialSee also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorial
Example EFA:
University student motivation
Career & Qualifications (6 items; = .92)
Self Development (5 items; = .81)
Social Opportunities (3 items; = .90)
Altruism (5 items; = .90)
Social Pressure (5 items; = .94)
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svgLicense: Public domainSee also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorialSee also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorial
Example EFA:
Factor correlations
MotivationCQSDSOALSP
Career & Qualif..26.25.24.06
Self Develop..33.55-.18
Social Enjoyment.26.33
Altruism.11
Social Pressure
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svgLicense: Public domainSee also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorialSee also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorial
Example EFA:
Error-bar graph
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svgLicense: Public domainSee also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorialSee also http://en.wikiversity.org/wiki/Exploratory_factor_analysis/Data_analysis_tutorial
Exploratory factor analysis: Q & A
Questions?
Concepts &
their measurement
Operationalising
fuzzy concepts
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svgLicense: Public domain
Concepts & their measurement:
Bryman & Cramer (1997)
Conceptsform a linchpin in the process of social research
express common elements in the world (to which we give a name)
Hypotheses express relations between concepts
Bryman & Cramer Concepts & their Measurement
(1997):
"Concepts form a linchpin in the process of social research.
Hypotheses contain concepts which are the products of our
reflections on the world. Concepts express common elements in the
world to which we give a name." (p. 53)
Concepts & their measurement:
Bryman & Cramer (1997)
Once formulated, a concept will need to be operationally defined, in order for systematic research to be conducted in relation to it..."
Bryman & Cramer Concepts & their Measurement (1997):Once formulated, a concept and the concepts with which it is purportedly associated, such as social class and authoritarianism, will need to be operationally defined, in order for systematic research to be conducted in relation to it..."
Concepts & their measurement:
Bryman & Cramer (1997)
...An operational definition specifies the procedures (operations) that will permit differences between individuals in respect of the concept(s) concerned to be precisely specified..."
Bryman & Cramer Concepts & their Measurement (1997)
Concepts & their measurement:
Bryman & Cramer (1997)
...What we are in reality talking about here is measurement, that is, the assignment of numbers to the units of analysis - be they people, organizations, or nations - to which a concept refers."
Bryman & Cramer Concepts & their Measurement (1997)
Image (left) source:http://www.flickr.com/photos/iliahi/408971482/By Maui in Vermont: http://www.flickr.com/photos/iliahiLicense: CC-by-A 2.0
Image (right) source: http://commons.wikimedia.org/wiki/File:3DTomo.jpgBy Tomograph: http://de.wikipedia.org/wiki/Benutzer:TomographLicense: GFDL
Operationalisation
...is the act of making a fuzzy concept measurable.
Social sciences often use multi-item measures to assess related but distinct aspects of a fuzzy concept.
Image source:
http://commons.wikimedia.org/wiki/Image:Die_bone.jpgCreative
Commons Attribution 3.0 Unported
Operationalisation steps
Brainstorm indicators of a concept
Define the concept
Draft measurement items
Pre-test and pilot test
Examine psychometric properties
how precise are the measures?
Redraft/refine and re-test
This is a cyclical or iterative process.
Operationalisating a fuzzy concept: Example (Brainstorming indicators)
Image source: James Neill, Creative Commons Attribution-Share Alike 2.5 Australia, http://creativecommons.org/licenses/by-sa/2.5/au/
Fuzzy concepts - Mindmap
Nurse empowerment
Image source: James Neill, Creative Commons Attribution 3.0; Adapted from original image source: http://www.uwo.ca/fhs/nursing/laschinger/image002.gif (Retreived 20 March, 2007). As of 25 March, 2008, the original image could no longer be found. It was based on research by Laschinger and colleagues on nurse empowerment.
Factor analysis process
Image source: Figure 4.2 Bryman & Cramer (1997)
Image source: Figure 4.2 Bryman & Cramer (1997)
Measurement error
Image source: http://commons.wikimedia.org/wiki/File:Noise_effect.svg
Image source: http://commons.wikimedia.org/wiki/File:Noise_effect.svgImage license: GFDL 1.2+
Measurement precision & noise
The lower the precision, the more subjects you'll need in your study to make up for the "noise" in your measurements. Even with a larger sample, noisy data can be hard to interpret. And if you are an applied scientist in the business of testing and assessing clients, you need special care when interpreting results of noisy tests.
http://www.sportsci.org/resource/stats/precision.html
Measurement error
Measurement error is any deviation from the true value caused by the measurement procedure.Observed score =
true score + measurement errorMeasurement error =
systematic error + random error
Sources of measurement error
Non-sampling
(e.g., unreliable
or invalid
tests)Sampling
(e.g., non-rep. sample)Researcher bias
(e.g., researcher favours a hypothesis)Paradigm
(e.g., Western focus on individualism)
Respondent bias
(e.g., social desirability)
Image source:s Unknown.Paradigm (e.g., assumptions, focus, collection method)
Personal researcher bias (conscious & unconscious)
Sampling (e.g., representativeness of sample)
Non-sampling (e.g, non-response, inaccurate response due to unreliable measurements, misunderstanding, social desirability, faking bad, researcher expectancy, administrative error)
To minimise measurement error
Use well designed measures:Multiple indicators for fuzzy constructs
Sensitive to target constructs
Clear instructions and questions
Reduce demand effects:Train interviewers
Use standard administration survey protocol
To minimise measurement error
To minimise measurement error
Obtain a representative sample:Use probability-sampling if possible
Minimise bias in selection for non-probability sampling
Maximise response rate:Pre-survey contact
Minimise length / time / hassle
Offer rewards / incentives
Coloured paper
Call backs / reminders
Ensure administrative accuracy:Set up efficient coding, with well-labelled variables
Check data (double-check at least a portion of the data)
To minimise measurement error
Psychometrics
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svgLicense: Public domain
Image source: http://www.flickr.com/photos/ajawin/2614485186/By akawin http://www.flickr.com/photos/ajawin/License: Creative Commons by Attribution 2.0 http://creativecommons.org/licenses/by/2.0/deed.en
Psychometrics: Goal
To validly measure differences between individuals and groups in psychosocial qualities such as attitudes and personality.
Psychometrics is the field of study concerned with the theory and technique of educational and psychological measurement, which includes the measurement of knowledge, abilities, attitudes, and personality traits. The field is primarily concerned with the study of differences between individuals and between groups of individuals. http://en.wikipedia.org/wiki/Psychometrics
Psychometrics: As test-taking grows, test-makers grow rarer
"Psychometrics, one of the most obscure, esoteric and cerebral
professions in America, is now also one of the hottest.
- As test-taking grows, test-makers grow rarer, David M.
Herszenhor, May 5, 2006, New York Times
e.g., due to increased testing of educational and psychological capacity and performance
Image source: http://commons.wikimedia.org/wiki/Image:Framework_complexity_of_the_Pater_Noster_lighthouse.jpgGNU Free Documentation license, Version 1.2 or any laterhttp://www.nytimes.com/2006/05/05/education/05testers.htmlIn Australia, probably the largest collection of psychometricians is at the Australian Council for Educational Research:http://www.acer.edu.au/
Psychometric tasks
Develop approaches and procedures (theory and practice) for measurement of psychological phenomena
Design and test psychological measurement instrumentation
e.g., examine and improve reliability and validity
Psychometrics is the field of study concerned with the theory and technique of educational and psychological measurement, which includes the measurement of knowledge, abilities, attitudes, and personality traits. The field is primarily concerned with the study of differences between individuals and between groups of individuals. http://en.wikipedia.org/wiki/Psychometrics
Not everything that counts...
But remember
Image source: Unknown
Psychometric methods
Factor analysisExploratory
Confirmatory
Classical test theoryReliability
Validity
Item response modeling
Reliability & Validity
Image source: http://www.flickr.com/photos/psd/17433783/in/photostream, CC-by-A 2.0
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svgLicense: Public domain
Image source: http://www.flickr.com/photos/psd/17433783/in/photostreamBy psd http://www.flickr.com/photos/psd/License: Creative Commons by Attribution 2.0 http://creativecommons.org/licenses/by/2.0/deed.en
Reliability and validity
(Howitt & Cramer, 2005)
Reliability and validity are the means by which we evaluate the value of psychological tests and measures.Reliability is about the consistency of the items within the measure
the consistency of a measure over time
Validity concerns the evidence that the measure actually measures what it is intended to measure.
Internal consistency (single administration) and Test-reliability (multiple administrations) are based on classical test theory. A more advanced/refined reliability technique is based Item Response Theory, which involves examining the extent to which each item discriminates between individuals with high/low overall scores.For more info see: http://en.wikipedia.org/wiki/Reliability_(psychometric)
Reliability vs. validity
In classical test theory, reliability is generally thought to be necessary for validity, but it does not guarantee validity.
In practice, a test of a relatively changeable psychological construct such as suicide ideation, may be valid (i.e., accurate), but not particularly reliable over time (because suicide ideation is likely to fluctuate).
Image source: http://www.socialresearchmethods.net/kb/relandval.phpReliability is about the consistency of a measure. Validity is about whether a test actually measures what its meant to measure.Validity requires reliability, but reliability alone is not sufficient for validity.See the more complex discussion of this at:http://en.wikiversity.org/wiki/Reliability_and_validity
Reliability and validity
(Howitt & Cramer, 2005)
Reliability and validity are not inherent characteristics of measures. They are affected by the context and purpose of the measurement a measure that is valid for one purpose may not be valid for another purpose.
Internal consistency (single administration) and Test-reliability (multiple administrations) are based on classical test theory. A more advanced/refined reliability technique is based Item Response Theory, which involves examining the extent to which each item discriminates between individuals with high/low overall scores.For more info see: http://en.wikipedia.org/wiki/Reliability_(psychometric)
Reliability
Reproducibility of a measurement
Image source: UnknownReliable; not validReliable; valid
Types of reliability
Internal consistency: Correlation among multiple items in a factorSplit-half reliability
Odd-even reliability
Cronbachs Alpha ()
Test-retest reliability: Correlation between test at one time and anotherProduct-moment correlation (r)
Inter-rater reliability: Correlation between one observer and another:Kappa
Internal consistency (single administration) and Test-reliability (multiple administrations) are based on classical test theory. A more advanced/refined reliability technique is based Item Response Theory, which involves examining the extent to which each item discriminates between individuals with high/low overall scores.For more info see: http://en.wikipedia.org/wiki/Reliability_(psychometric)
Reliability rule of thumb
.95 = may be overly reliable or redundant this is subjective and whether a scale is overly reliable depends also on the nature what is being measured
Image source: http://commons.wikimedia.org/wiki/File:Circle-Thumb.pngImage author: Acadac, http://commons.wikimedia.org/wiki/User:AcadacImage license: Public domain
Reliability rule of thumb
Table 7 Fabrigar et al (1999).
Table 7 Fabrigar et al. (1999)
Rule of thumb - reliability coefficients should be over .70, up to approx. .90
Image source: Table 7 Fabrigar et al (1999).
Internal consistency
(or internal reliability)
Internal consistency refers to:How well multiple items combine as a measure of a single concept
The extent to which responses to multiple items are consistent with one another
Internal consistency can measured by:Split-half reliability
Odd-even reliability
Cronbach's alpha
Internal consistency
(Recoding)
Remember to:Ensure that negatively-worded items are recoded
Types of internal consistency:
Split-half reliability
Sum the first half of the items.
Sum the second half of the items.
Compute a correlation between the sums of the two halves.
ReferenceHowitt & Cramer (2005), p. 221
Types of internal consistency -
Odd-even reliability
Sum items 1, 3, 5, etc.
Sum items 2, 4, 6, etc.
Compute a correlation between the sums of the two halves.
ReferenceHowitt & Cramer (2005), p. 221
Types of internal reliability:
Alpha reliability (Cronbach's )
Averages all possible split-half reliability coefficients.
Akin to a single score which represents the degree of intercorrelation amongst the items.
ReferenceHowitt & Cramer (2005), p. 221
More items greater reliability
(The more items, the more rounded the measure)
Law of diminishing returns
Min. = 2?
Max. = unlimited?
Typically ~ 4 to 12 items per factor
Final decision is subjective and depends on research context
How many items per factor?
Internal reliability example:
Student-rated
quality of maths teaching
10-item scale measuring students assessment of the educational quality of their maths classes
4-point Likert scale ranging from:
strongly disagree to strongly agree
This example is from Francis 6.1
Quality of mathematics teaching
My maths teacher is friendly and cares about me
The work we do in our maths class is well organised.
My maths teacher expects high standards of work from everyone.
My maths teacher helps me to learn.
I enjoy the work I do in maths classes.
+ 5 more
Internal reliability example: Quality of maths teaching
Image source: James Neill, Creative Commons Attribution-Share Alike 2.5 Australia, http://creativecommons.org/licenses/by-sa/2.5/au/SPSS Analyze Scale Reliability Analysis
SPSS:
Corrected Item-total correlation
Image source: James Neill, Creative Commons Attribution-Share Alike 2.5 Australia, http://creativecommons.org/licenses/by-sa/2.5/au/SPSS Analyze Scale Reliability Analysis
SPSS: Cronbachs
Image source: James Neill, Creative Commons Attribution-Share Alike 2.5 Australia, http://creativecommons.org/licenses/by-sa/2.5/au/SPSS Analyze Scale Reliability Analysis
Item-total Statistics
Scale Scale CorrectedMean Variance Item- Alphaif Item if Item Total if ItemDeleted Deleted Correlation Deleted
MATHS1 25.2749 25.5752 .6614 .8629MATHS2 25.0333 26.5322 .6235 .8661MATHS3 25.0192 30.5174 .0996 .9021MATHS4 24.9786 25.8671 .7255 .8589MATHS5 25.4664 25.6455 .6707 .8622MATHS6 25.0813 24.9830 .7114 .8587MATHS7 25.0909 26.4215 .6208 .8662MATHS8 25.8699 25.7345 .6513 .8637MATHS9 25.0340 26.1201 .6762 .8623MATHS10 25.4642 25.7578 .6495 .8638
Reliability Coefficients
N of Cases = 1353.0 N of Items = 10
Alpha = .8790
SPSS: Reliability output
Item-total Statistics
Scale Scale CorrectedMean Variance Item- Alphaif Item if Item Total if ItemDeleted Deleted Correlation Deleted
MATHS1 22.2694 24.0699 .6821 .8907MATHS2 22.0280 25.2710 .6078 .8961MATHS4 21.9727 24.4372 .7365 .8871MATHS5 22.4605 24.2235 .6801 .8909MATHS6 22.0753 23.5423 .7255 .8873MATHS7 22.0849 25.0777 .6166 .8955MATHS8 22.8642 24.3449 .6562 .8927MATHS9 22.0280 24.5812 .7015 .8895MATHS10 22.4590 24.3859 .6524 .8930
Reliability Coefficients
N of Cases = 1355.0 N of Items = 9
Alpha = .9024
SPSS: Reliability output
Reliabilities: Example table
Image source: James Neill, Creative Commons Attribution-Share Alike 2.5 Australia, http://creativecommons.org/licenses/by-sa/2.5/au/Life Effectiveness Questionnaire - http://wilderdom.com/leq.html
Validity
Validity is the extent to which an instrument actually measures what it purports to measure.
Validity = does the test measure what its meant to measure?
Image source: UnknownReliable; validhttp://en.wikipedia.org/wiki/Construct_Validity
Validity
Validity is multifaceted and includes:Correlations with similar measures
How the measure performs in relation to other variables
How well the measure helps to predict the future
From http://www.socialresearchmethods.net/kb/measval.htm
Types of validity
Face validity
Content validity
Criterion validityConcurrent
Predictive
Construct validityConvergent
Discriminant
From http://www.socialresearchmethods.net/kb/measval.htm
Face validity
(low-level of importance overall)
Asks:
"Do the questions appear to measure what the test purports to
measure?"
Important for:
Respondent buy-in
How assessed:
Read the test items
Adapted from: http://www.psyasia.com/supportsuite/index.php?_m=knowledgebase&_a=viewarticle&kbarticleid=85&nav=0Prima facie extent to which an item is judged to reflect target construct.e.g., a test of ability to add two-digit numbers should cover the full range of combinations of digits. A test with only one-digit numbers, or only even numbers, would (on the face of it) not have good coverage of the content domain.
Content validity
(next level of importance)
Asks:
"Are questions measuring the complete construct?"
Important for:
Ensuring holistic assessment
How assessed:
Diverse means of item generation (lit. review, theory, interviews,
expert review)
Adapted from: http://www.psyasia.com/supportsuite/index.php?_m=knowledgebase&_a=viewarticle&kbarticleid=85&nav=0Extent to which test content covers a representative sample of the domain to be measured. Compare test with: existing literature
expert panels
qualitative interviews / focus groups with target sample
Criterion validity
(high importance)
Asks:
"Can a test score predict real world outcomes?"
Important for:
Test relevance and usefulness
How assessed:
Concurrent validity: Correlate test scores with recognised external
criteria such as performance appraisal scores
Predictive validity: Correlate test scores with future outcome e.g., offender risk rating with recidivism
Adapted from: http://www.psyasia.com/supportsuite/index.php?_m=knowledgebase&_a=viewarticle&kbarticleid=85&nav=0Involves the correlation between the test and a criterion variable (or variables) taken as representative of the construct.
Concurrent validityCorrelation between the measure and other recognised measures of the target construct
Predictive validityExtent to which a measure predicts something that it theoretically should be able to predict.
Criterion refers to the true score. Criterion validity refer to measures of relationship between the criterion variable and the measured variable.http://en.wikipedia.org/wiki/Concurrent_validityhttp://www.socialresearchmethods.net/kb/measval.htm
Construct validity
(high importance)
Asks:
Does the test assess the construct it purports to? ("the truth, the
whole truth and nothing but the truth.")
Important for:
Making inferences from operationalisations to theoretical
constructs
How assessed:
- Theoretical (is the theory about the construct valid?)- StatisticalConvergent correlation with similar measures;Divergent not correlated with other constructs
Adapted from: http://www.psyasia.com/supportsuite/index.php?_m=knowledgebase&_a=viewarticle&kbarticleid=85&nav=0Totality of evidence about whether a particular operationalisation of a construct adequately represents what is intended.
Not distinct from support for the substantive theory of the construct.
Involves empirical and theoretical support.
Includes:Statistical analyses of the internal structure of the test including the relationships between responses to different test items (internal consistency).
Relationships between the test and measures of other constructs.
Convergent validityExtent to which a measure correlates with measures with which it theoretically should be associated.
Discriminant validityExtent to which a measure does not correlate with measures with which it theoretically should not be associated.
Construct validity
(high importance)
Image source:http://www.socialresearchmethods.net/kb/considea.phpImage license: 2006, William M.K. Trochim, All Rights Reserved
Composite Scores
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svgLicense: Public domain
Image source: http://www.flickr.com/photos/generated/2585844966/By generated http://www.flickr.com/photos/generated/License: Creative Commons by Attribution 2.0 http://creativecommons.org/licenses/by/2.0/deed.en
Composite scores
(Factor scores)
Combine item-scores into overall scores which represent individual differences in the target constructs.These new 'continuous' variables can then be used for:Descriptive statistics and histograms
Correlations
As IVs and/or DVs in inferential analyses such as MLR and ANOVA
Composite scores
(Factor scores)
There are two ways of creating composite scores:Unit weighting
Regression weighting
Unit weighting
Average (or total) of item scores within a factor.
(each variable is equally weighted)
X = mean(y1yp)
Unit Weighting
.25.25.25.25
Creating composite scores: Dealing with missing data
It can be helpful to maximise sample size by allowing for some missing data.
Composite scores:
Missing data
SPSS syntax:Compute X = mean (v1, v2, v3, v4, v5, v6)Compute X = mean.4 (v1, v2, v3, v4, v5, v6)Specify a min. # of items. If the min. isn't available, the composite score will be missingHow many items can be missed? Depends on overall reliability. A rule of thumb:Allow 1 missing per 4 to 5 items
Allow 2 missing per 6 to 8 items
Allow 3+ missing per 9+ items
A researcher may decide to be more or less conservative depending on the factors reliability, sample size, and the nature of the study.
compute X = mean (v1, v2, v3, v4, v5.v6)
(note: a missing value will be returned for a case if 1 or more of
its items are missing data)compute X = mean.4 (v1, v2, v3, v4,
v5,v6)
(note: in this case a missing value will be returned for a case if
3 or more of its items are missing data the .4 means calculate a
mean using data from 4 or more items, otherwise make X
missing)Image source:
http://commons.wikimedia.org/wiki/File:Circle-Thumb.pngImage
author: Acadac, http://commons.wikimedia.org/wiki/User:AcadacImage
license: Public domain
Regression weighting
Factor score regression weighting
The contribution of each
item to the composite score
is weighted to reflect some
items more than other
items.X = 20*a + .19*b + .27*c + .34*d
X
.20.19.27.34a
b
c
d
This is arguably more valid, but the advantage may be marginal, and it makes factor scores difficult to compare.
X = r1y1 +. + rpypRegression weighting will also include small weights for non-target items
Regression weighting
Two calculation methods:Manual (use Compute)
Automatic (use Factor Analysis Factor Scores)
Image source: James Neill, Creative Commons Attribution-Share Alike 2.5 Australia, http://creativecommons.org/licenses/by-sa/2.5/au/
Regression weighting SPSS output
Data view
Variable viewImage sources: James Neill, Creative Commons Attribution-Share Alike 2.5 Australia, http://creativecommons.org/licenses/by-sa/2.5/au/
Writing up
instrument development
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svgLicense: Public domain
Image source: http://www.flickr.com/photos/mezone/21970578/By http://www.flickr.com/photos/mezone/License: Creative Commons by Attribution 2.0 http://creativecommons.org/licenses/by/2.0/deed.en
Writing up instrument development
IntroductionLiterature review about underlying factors theory and research
MethodMaterials/Instrumentation summarise how the measures were
developed and the expected factor structure
e.g., present a table of the expected factors and their operational
definitions.
Writing up instrument development
ResultsFactor analysisAssumption testing
Extraction method & rotation
# of factors & items removed
Names & definitions of factors
Item factor loadings & communalities
Factor correlations
Reliability
Composite scores
Factor correlations
Writing up instrument development
DiscussionTheoretical underpinning Was it supported by the data? What adaptations should be made to the theory?
Quality / usefulness of measure Provide an objective, critical assessment, reflecting the measures' strengths and weaknesses
Recommendations for further improvement
Writing up a factor analysisSee downloadable example
Summary
Operationally define concepts
Brainstorm measurement items
Draft measure aiming to minimise measurement error
Pre-test & pilot
Use EFA, reliability, and validity
Create composite scores
Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svgLicense: Public domain
Allen, P. & Bennett, K. (2008). Reliability analysis (Ch 15) in SPSS for the health & behavioural sciences (pp. 205-218). South Melbourne, Victoria, Australia: Thomson.
Bryman, A. & Cramer, D. (1997). Concepts and their measurement (Ch. 4). In Quantitative data analysis with SPSS for Windows: A guide for social scientists (pp. 53-68). Routledge.
DeCoster, J. (2000). Scale construction notes. http://www.stat-help.com/scale.pdf (pdf)
Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272-299.
Fowler, F. (2002). Designing questions to be good measures. In Survey research methods (3rd ed.)(pp. 76-103). Thousand Oaks, CA: Sage. Ereserve.
Howitt, D. & Cramer, D. (2005). Reliability and validity: Evaluating the value of tests and measures (Ch. 13). In Introduction to research methods in psychology (pp. 218-231). Harlow, Essex: Pearson. eReserve.
References
Open Office Impress
This presentation was made using Open Office Impress.
Free and open source software.
http://www.openoffice.org/product/impress.html
Click to edit the title text format
Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelEighth Outline LevelNinth Outline Level
Click to edit the title text format
Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelEighth Outline LevelNinth Outline Level
Click to edit the title text format
Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelEighth Outline LevelNinth Outline Level
Click to edit the title text format
Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelEighth Outline LevelNinth Outline Level
Click to edit the title text format