Discussion Overview: Measurement I) Reliability of Measures I) Reliability of Measures II) Construct...
-
Upload
kelley-griffith -
Category
Documents
-
view
217 -
download
0
Transcript of Discussion Overview: Measurement I) Reliability of Measures I) Reliability of Measures II) Construct...
Discussion Overview: Discussion Overview: MeasurementMeasurement
I) Reliability of MeasuresI) Reliability of Measures
II) Construct ValidityII) Construct Validity
III) Measurement scalesIII) Measurement scales
I) Reliability of MeasuresI) Reliability of Measures ReliabilityReliability
– The consistency or stability of a measure The consistency or stability of a measure Assessing a restaurant’s foodAssessing a restaurant’s food
Three important variablesThree important variables– How many testers? (Observers)How many testers? (Observers)
Interrater reliabilityInterrater reliability– How many different entrees? (Observations)How many different entrees? (Observations)
Internal consistencyInternal consistency– How many times? (Occasions)How many times? (Occasions)
Test-retestTest-retest
Interrater ReliabilityInterrater Reliability
The degree to which The degree to which independent raters independent raters agree on an observationagree on an observation
Have two (or more) Have two (or more) judges rate the same judges rate the same peoplepeople
Trained and Trained and independent raters, independent raters, using a coding schemeusing a coding scheme
Observer 1 Observer 2
Complain about injection
-2 3
First negative comment
0 1
Second negative comment
-2 2
Rip up questionnaire -2 3
Interrater ReliabilityInterrater Reliability
Observer 1 Observer 2
Complain about injection
22 22
First negative comment
00 00
Second negative comment
-2-2 -2-2
Rip up questionnaire 22 33
Interrater ReliabilityInterrater Reliability
Internal ConsistencyInternal Consistency
Internal consistencyInternal consistency – the degree to – the degree to which all specific items of a measure which all specific items of a measure behave the same waybehave the same way
Measure the same people with Measure the same people with multiple itemsmultiple items– Different questions in a surveyDifferent questions in a survey– Different behaviors in observationDifferent behaviors in observation
ExtraversionExtraversion
1 2 3 4 5
Not at all
true
Very true
1.I am outgoing. ____
2.I am friendly. ____
3.I am talkative. ____
4.I am gregarious.____
Internal consistencyInternal consistency
Split-half reliabilitySplit-half reliability – correlation of – correlation of scores on one half of the test with scores on one half of the test with scores on the other halfscores on the other half
Cronbach’s alphaCronbach’s alpha – the average of all – the average of all possible correlations between itemspossible correlations between items
‘‘One of these things just One of these things just doesn’t belong’doesn’t belong’
One of these things is not like the others, One of these things is not like the others, One of these things just doesn't belongOne of these things just doesn't belong
Student 1 Student 2 Student 3
Ques 1 Ques 1 (Chpt 12)(Chpt 12)
1010 22 99
Ques 2 Ques 2 (Chpt 12)(Chpt 12)
99 33 88
Ques 3 Ques 3 (Chpt 3)(Chpt 3)
22 66 11
Ques 4 Ques 4 (Chpt 12)(Chpt 12)
1010 22 99
Test-Retest ReliabilityTest-Retest Reliability The degree to which a measure correlates The degree to which a measure correlates
positively with itself over timepositively with itself over time– Consistency of the measure over timeConsistency of the measure over time
Measure the same people at two (or more) Measure the same people at two (or more) points in timepoints in time
Desirable for stable traits, but not for transient Desirable for stable traits, but not for transient statesstates
The “More is Better Rule”The “More is Better Rule”
Reliability is likely to increase as we Reliability is likely to increase as we increase the number of…increase the number of…– Observers (or raters)Observers (or raters)– Observations (or items)Observations (or items)– OccasionsOccasions
Measurement error will average outMeasurement error will average out
II) Construct Validity II) Construct Validity
How well an How well an operational definition operational definition represents the represents the construct of interestconstruct of interest
The degree to which The degree to which the construct can be the construct can be inferred from the inferred from the operational definition operational definition of that constructof that construct
Indicators of Construct Indicators of Construct ValidityValidity
Face validityFace validity Criterion validityCriterion validity
– Predictive validityPredictive validity– Concurrent validityConcurrent validity– Convergent validityConvergent validity– Discriminant validityDiscriminant validity
Face ValidityFace Validity
Face validityFace validity – Does the measure – Does the measure appear to measure the construct of appear to measure the construct of interest?interest?– Does the measure “on the face of it” Does the measure “on the face of it”
look like what it’s supposed to look like what it’s supposed to measure?measure?
Not necessary or sufficient for a Not necessary or sufficient for a good measuregood measure
Predictive ValidityPredictive Validity
Predictive validityPredictive validity – Is the measure – Is the measure associated with variables it should associated with variables it should theoretically predict?theoretically predict?
LSAT – Law school performanceLSAT – Law school performance Self-esteem – DepressionSelf-esteem – Depression Shyness – Social anxiety Shyness – Social anxiety
Concurrent ValidityConcurrent Validity
Concurrent validityConcurrent validity – Does the – Does the measure differ between groups it measure differ between groups it ought to differ between?ought to differ between?– Also called “known groups validity”Also called “known groups validity”
E.g., clinically depressed versus non-E.g., clinically depressed versus non-depressed groupsdepressed groups
Convergent ValidityConvergent Validity
Convergent validityConvergent validity – Is the – Is the measure associated with other measure associated with other established measures of the same established measures of the same construct?construct?
Self-report - ObservationsSelf-report - Observations Physiological measure - Self-reportPhysiological measure - Self-report Self-report 1 – Self-report 2Self-report 1 – Self-report 2
Discriminant ValidityDiscriminant Validity
Discriminant validityDiscriminant validity – Is the – Is the measure NOT associated with measure NOT associated with measures of other constructs?measures of other constructs?
Self-esteem scores not associated Self-esteem scores not associated with locus of control scoreswith locus of control scores
Problem solving knowledge not Problem solving knowledge not associated with factual knowledgeassociated with factual knowledge
Measurement Reliability & Measurement Reliability & ValidityValidity
ReliabilityReliability: Is the measure consistent?: Is the measure consistent? ValidityValidity: Does the measure : Does the measure
adequately reflect the construct of adequately reflect the construct of interest?interest?
Reliable and Valid Reliable, not Valid Not Reliable, not Valid
Relationship between Relationship between Reliability and ValidityReliability and Validity
Can be reliable but not validCan be reliable but not valid To be valid it must be reliable
– But reliability is not sole condition for validity
Both reliability and validity are necessary for accurate measurement in a research study.
Measurement Scales Measurement Scales
Nominal scalesNominal scales Ordinal scalesOrdinal scales Interval scalesInterval scales Ratio scalesRatio scales
Nominal ScalesNominal Scales AKA Categorical scalesAKA Categorical scales No numerical/quantitative properties. No numerical/quantitative properties.
Categories or group simply differ from Categories or group simply differ from one anotherone another
Examples:Examples:– Men or womenMen or women– Right or left handedRight or left handed– Catholic, Protestant, Jewish, Hindu, Catholic, Protestant, Jewish, Hindu,
Buddhist…Buddhist…– Numbers on basketball jerseysNumbers on basketball jerseys– Zip codesZip codes
Ordinal ScalesOrdinal Scales
Allow us to rank order the levels of Allow us to rank order the levels of the variables being studiedthe variables being studied
ExamplesExamples– Social classSocial class
lower class, working class, middle class, lower class, working class, middle class, and upper classand upper class
– College football standingsCollege football standings– Letterman’s Top TenLetterman’s Top Ten
Top Ten Questions to ask Yourself Top Ten Questions to ask Yourself Before Eating Spinach?Before Eating Spinach?
10.10. Was my spinach properly sprayed with Lysol? Was my spinach properly sprayed with Lysol? 9.9. Isn't it still safer than eating a New York City Isn't it still safer than eating a New York City
hot dog?hot dog? 8.8. So all those years my mom made me eat So all those years my mom made me eat
spinach, she was trying to kill me?spinach, she was trying to kill me? 7.7. Is this the right side dish for my Mad Cow Is this the right side dish for my Mad Cow
burger?burger? 6.6. Are my papers in order? Are my papers in order? 5.5. If I get sick, will my wife TiVo Ventriloquist If I get sick, will my wife TiVo Ventriloquist
Week on the Late Show?Week on the Late Show? 4.4. Should I also avoid kale? Should I also avoid kale? 3.3. If I'm going to eat something deadly, shouldn't If I'm going to eat something deadly, shouldn't
it be delicious Pop-Tarts?it be delicious Pop-Tarts? 2.2. What would Popeye do? What would Popeye do? 1.1. Do I really want my obituary to read: "Man Dies Do I really want my obituary to read: "Man Dies
A La Florentine?"A La Florentine?"
Interval ScalesInterval Scales
The difference between the numbers The difference between the numbers on the scale is meaningfulon the scale is meaningful
Scores separated by equal intervalsScores separated by equal intervals ExamplesExamples
– Temperature (Fahrenheit or Celsius)Temperature (Fahrenheit or Celsius)– Scores on personality measureScores on personality measure
Ratio ScalesRatio Scales
Scores separated by Scores separated by equal intervals and equal intervals and there is an absolute there is an absolute zerozero
ExamplesExamples– LengthLength– Weight Weight – TimeTime– Number of responsesNumber of responses
LevelLevel
QualitativeInfo
Has inherent order
‘more to less’
EqualIntervals
Has zero point
Nominal XX
Ordinal XX XX
Interval XX XX XX
Ratio XX XX XX XX
Scales of MeasurementScales of Measurement
Concept Check Concept Check
Which scale of measurement best Which scale of measurement best describes the following:describes the following:– Telephone numbersTelephone numbers– Distances from Budapest to cities in the USDistances from Budapest to cities in the US– Scores on an extraversion personality Scores on an extraversion personality
assessmentassessment– Ranking of basketball teams in the Big TenRanking of basketball teams in the Big Ten