Types of Reliability. Reliability of Consistency of What? l Observers or raters l Tests over time l...
-
Upload
maleah-zane -
Category
Documents
-
view
217 -
download
2
Transcript of Types of Reliability. Reliability of Consistency of What? l Observers or raters l Tests over time l...
Types of ReliabilityTypes of Reliability
Reliability of Consistency of What?Reliability of Consistency of What?
ObserversObservers or raters or raters Tests over Tests over timetime Different Different versionsversions of the same test of the same test A test at A test at one point one point in timein time
Inter-Rater or Inter-Observer ReliabilityInter-Rater or Inter-Observer Reliability
Object orObject orphenomenonphenomenon
Inter-Rater or Inter-Observer ReliabilityInter-Rater or Inter-Observer Reliability
Observer 1Observer 1Observer 1Observer 1
Object orObject orphenomenonphenomenon
Inter-Rater or Inter-Observer ReliabilityInter-Rater or Inter-Observer Reliability
Observer 1Observer 1Observer 1Observer 1 Observer 2Observer 2Observer 2Observer 2
Object orObject orphenomenonphenomenon
Inter-Rater or Inter-Observer ReliabilityInter-Rater or Inter-Observer Reliability
Observer 1Observer 1Observer 1Observer 1 Observer 2Observer 2Observer 2Observer 2
Object orObject orphenomenonphenomenon
==??
Inter-Rater or Inter-Observer ReliabilityInter-Rater or Inter-Observer Reliability
Are different observers Are different observers consistentconsistent?? Can establish this outside of your study Can establish this outside of your study
in a in a pilot studypilot study.. Can look at Can look at percent of agreement percent of agreement
(especially with category ratings).(especially with category ratings). Can use Can use correlation correlation (with continuous (with continuous
ratings).ratings).
Test-Retest ReliabilityTest-Retest Reliability
Time 1Time 1 Time 2Time 2
Test-Retest ReliabilityTest-Retest Reliability
TestTest TestTest
Time 1Time 1 Time 2Time 2
==
Test-Retest ReliabilityTest-Retest Reliability
TestTest TestTest
Time 1Time 1 Time 2Time 2
==
Stability over timeStability over time
Test-Retest ReliabilityTest-Retest Reliability
Measure instrument at Measure instrument at two times two times for for multiple persons.multiple persons.
Compute Compute correlationcorrelation between the two between the two measures.measures.
Assumes there is Assumes there is no change no change in the in the underlying trait between time 1 and time underlying trait between time 1 and time 2.2.
Parallel-Forms ReliabilityParallel-Forms Reliability
Time 1Time 1 Time 2Time 2
Parallel-Forms ReliabilityParallel-Forms Reliability
Form BForm B
Time 1Time 1 Time 2Time 2
Form AForm A
==
Parallel-Forms ReliabilityParallel-Forms Reliability
Form BForm B
Time 1Time 1 Time 2Time 2
Stability across formsStability across formsForm AForm A
==
Parallel-Forms ReliabilityParallel-Forms Reliability
Administer both forms to the Administer both forms to the samesame people.people.
Get Get correlationcorrelation between the two forms. between the two forms. Usually done in Usually done in educational contexts educational contexts
where you need alternative forms where you need alternative forms because of the frequency of retesting because of the frequency of retesting and where you can sample from lots of and where you can sample from lots of equivalent questions.equivalent questions.
Internal Consistency ReliabilityInternal Consistency Reliability
Average inter-item correlationAverage inter-item correlationAverage inter-item correlationAverage inter-item correlation
Internal Consistency ReliabilityInternal Consistency Reliability
TestTest
Average Inter-Item CorrelationAverage Inter-Item CorrelationAverage Inter-Item CorrelationAverage Inter-Item Correlation
Internal Consistency ReliabilityInternal Consistency Reliability
TestTest
Item 1
Item 2
Item 3
Item 4
Item 5
Item 6
Average Inter-Item correlationAverage Inter-Item correlationAverage Inter-Item correlationAverage Inter-Item correlation
Internal Consistency ReliabilityInternal Consistency Reliability
TestTest
Item 1
Item 2
Item 3
Item 4
Item 5
Item 6
1.001.00.89.89 1.00 1.00.91.91 .92.92 1.00 1.00.88.88 .93.93 .95.95 1.00 1.00.84.84 .86.86 .92.92 .85.85 1.00 1.00.88.88 .91.91 .95.95 .87.87 .85.85 1.00 1.00
II11 II22 II33 II44 II55 II66
II11
II22
II33
II44
II55
II66
Average inter-item correlationAverage inter-item correlationAverage inter-item correlationAverage inter-item correlation
Internal Consistency ReliabilityInternal Consistency Reliability
TestTest
Item 1
Item 2
Item 3
Item 4
Item 5
Item 6
1.001.00.89.89 1.00 1.00.91.91 .92.92 1.00 1.00.88.88 .93.93 .95.95 1.00 1.00.84.84 .86.86 .92.92 .85.85 1.00 1.00.88.88 .91.91 .95.95 .87.87 .85.85 1.00 1.00
II11 II22 II33 II44 II55 II66
II11
II22
II33
II44
II55
II66
Average inter-item correlationAverage inter-item correlationAverage inter-item correlationAverage inter-item correlation
.90.90
Average item-Average item-totaltotal correlationcorrelationAverage item-Average item-totaltotal correlationcorrelation
Internal Consistency Internal Consistency ReliabilityReliability
TestTest
Average item-Average item-totaltotal correlationcorrelationAverage item-Average item-totaltotal correlationcorrelation
Internal Consistency Internal Consistency ReliabilityReliability
Average item-Average item-totaltotal correlationcorrelationAverage item-Average item-totaltotal correlationcorrelation
Internal Consistency Internal Consistency ReliabilityReliability
TestTest
Item 1
Item 2
Item 3
Item 4
Item 5
Item 6
1.001.00.89.89 1.00 1.00.91.91 .92.92 1.00 1.00.88.88 .93.93 .95.95 1.00 1.00.84.84 .86.86 .92.92 .85.85 1.00 1.00.88.88 .91.91 .95.95 .87.87 .85.85 1.00 1.00.84.84 .88.88 .86.86 .87.87 .83.83 .82.82 1.001.00
II11 II22 II33 II44 II55 II66
II11
II22
II33
II44
II55
II66
TotalTotal
Average item-Average item-totaltotal correlationcorrelationAverage item-Average item-totaltotal correlationcorrelation
Internal Consistency Internal Consistency ReliabilityReliability
TestTest
Item 1
Item 2
Item 3
Item 4
Item 5
Item 6
1.001.00.89.89 1.00 1.00.91.91 .92.92 1.00 1.00.88.88 .93.93 .95.95 1.00 1.00.84.84 .86.86 .92.92 .85.85 1.00 1.00.88.88 .91.91 .95.95 .87.87 .85.85 1.00 1.00.84.84 .88.88 .86.86 .87.87 .83.83 .82.82 1.001.00
II11 II22 II33 II44 II55 II66
II11
II22
II33
II44
II55
II66
TotalTotal
Average item-Average item-totaltotal correlationcorrelationAverage item-Average item-totaltotal correlationcorrelation
.85.85
Internal Consistency Internal Consistency ReliabilityReliability
TestTest
Item 1
Item 2
Item 3
Item 4
Item 5
Item 6
Split-half correlationsSplit-half correlationsSplit-half correlationsSplit-half correlations
Internal Consistency Internal Consistency ReliabilityReliability
TestTest
Split-half correlationsSplit-half correlationsSplit-half correlationsSplit-half correlations
Internal Consistency Internal Consistency ReliabilityReliability
Split-half correlationsSplit-half correlationsSplit-half correlationsSplit-half correlations
Internal Consistency Internal Consistency ReliabilityReliability
TestTest
Item 1
Item 2
Item 3
Item 4
Item 5
Item 6
Split-half correlationsSplit-half correlationsSplit-half correlationsSplit-half correlations
Item 1Item 1 Item 3Item 3 Item 4Item 4
Internal Consistency Internal Consistency ReliabilityReliability
TestTest
Item 1
Item 2
Item 3
Item 4
Item 5
Item 6
Split-half correlationsSplit-half correlationsSplit-half correlationsSplit-half correlations
Item 2Item 2 Item 5Item 5 Item 6Item 6
Internal Consistency Internal Consistency ReliabilityReliability
Item 1Item 1 Item 3Item 3 Item 4Item 4
TestTest
Item 1
Item 2
Item 3
Item 4
Item 5
Item 6
Split-half correlationsSplit-half correlationsSplit-half correlationsSplit-half correlations
.87.87
Internal Consistency Internal Consistency ReliabilityReliability
Item 2Item 2 Item 5Item 5 Item 6Item 6
Item 1Item 1 Item 3Item 3 Item 4Item 4
TestTest
Item 1
Item 2
Item 3
Item 4
Item 5
Item 6
Cronbach’s alpha (Cronbach’s alpha ())Cronbach’s alpha (Cronbach’s alpha ())
Internal Consistency Internal Consistency ReliabilityReliability
TestTest
Cronbach’s alpha (Cronbach’s alpha ())Cronbach’s alpha (Cronbach’s alpha ())
Internal Consistency Internal Consistency ReliabilityReliability
Cronbach’s alpha (Cronbach’s alpha ())Cronbach’s alpha (Cronbach’s alpha ())
Internal Consistency Internal Consistency ReliabilityReliability
TestTest
Item 1
Item 2
Item 3
Item 4
Item 5
Item 6
Cronbach’s alpha (Cronbach’s alpha ())Cronbach’s alpha (Cronbach’s alpha ())
.87.87
item 1item 1 item 3item 3 item 4item 4
item 2item 2 item 5item 5 item 6item 6
.87.87
item 1item 1 item 3item 3 item 4item 4
item 2item 2 item 5item 5 item 6item 6
.87.87
item 1item 1 item 3item 3 item 4item 4
item 2item 2 item 5item 5 item 6item 6
Internal Consistency Internal Consistency ReliabilityReliability
TestTest
Item 1
Item 2
Item 3
Item 4
Item 5
Item 6
SHSH11 .87.87
SHSH22 .85.85
SHSH33 .91.91
SHSH44 .83.83
SHSH55 .86.86
......SHSHnn .85.85
Internal Consistency Internal Consistency ReliabilityReliability
Cronbach’s alpha (Cronbach’s alpha ())Cronbach’s alpha (Cronbach’s alpha ())
.87.87
item 1item 1 item 3item 3 item 4item 4
item 2item 2 item 5item 5 item 6item 6
.87.87
item 1item 1 item 3item 3 item 4item 4
item 2item 2 item 5item 5 item 6item 6
.87.87
item 1item 1 item 3item 3 item 4item 4
item 2item 2 item 5item 5 item 6item 6
TestTest
Item 1
Item 2
Item 3
Item 4
Item 5
Item 6
SHSH11 .87.87
SHSH22 .85.85
SHSH33 .91.91
SHSH44 .83.83
SHSH55 .86.86
......SHSHnn .85.85
= .85= .85 = .85= .85
Internal Consistency Internal Consistency ReliabilityReliability
Cronbach’s alpha (Cronbach’s alpha ())Cronbach’s alpha (Cronbach’s alpha ())
.87.87
item 1item 1 item 3item 3 item 4item 4
item 2item 2 item 5item 5 item 6item 6
.87.87
item 1item 1 item 3item 3 item 4item 4
item 2item 2 item 5item 5 item 6item 6
.87.87
item 1item 1 item 3item 3 item 4item 4
item 2item 2 item 5item 5 item 6item 6
TestTest
Item 1
Item 2
Item 3
Item 4
Item 5
Item 6
SHSH11 .87.87
SHSH22 .85.85
SHSH33 .91.91
SHSH44 .83.83
SHSH55 .86.86
......SHSHnn .85.85
= .85= .85 = .85= .85
Like the average Like the average of all possible of all possible
split-half split-half correlationscorrelations
Like the average Like the average of all possible of all possible
split-half split-half correlationscorrelations
Internal Consistency Internal Consistency ReliabilityReliability
Cronbach’s alpha (Cronbach’s alpha ())Cronbach’s alpha (Cronbach’s alpha ())
.87.87
item 1item 1 item 3item 3 item 4item 4
item 2item 2 item 5item 5 item 6item 6
.87.87
item 1item 1 item 3item 3 item 4item 4
item 2item 2 item 5item 5 item 6item 6
.87.87
item 1item 1 item 3item 3 item 4item 4
item 2item 2 item 5item 5 item 6item 6
TestTest
Item 1
Item 2
Item 3
Item 4
Item 5
Item 6
Internal Consistency Reliability - SummaryInternal Consistency Reliability - Summary
Average inter-item correlationAverage inter-item correlation Average item-total correlationAverage item-total correlation Split-half reliabilitySplit-half reliability Cronbach’s alpha (Cronbach’s alpha ())