Types of ReliabilityTypes of Reliability

Reliability of Consistency of What?Reliability of Consistency of What?

ObserversObservers or raters or raters Tests over Tests over timetime Different Different versionsversions of the same test of the same test A test at A test at one point one point in timein time

Inter-Rater or Inter-Observer ReliabilityInter-Rater or Inter-Observer Reliability

Object orObject orphenomenonphenomenon

Inter-Rater or Inter-Observer ReliabilityInter-Rater or Inter-Observer Reliability

Observer 1Observer 1Observer 1Observer 1

Object orObject orphenomenonphenomenon

Inter-Rater or Inter-Observer ReliabilityInter-Rater or Inter-Observer Reliability

Observer 1Observer 1Observer 1Observer 1 Observer 2Observer 2Observer 2Observer 2

Object orObject orphenomenonphenomenon

Inter-Rater or Inter-Observer ReliabilityInter-Rater or Inter-Observer Reliability

Observer 1Observer 1Observer 1Observer 1 Observer 2Observer 2Observer 2Observer 2

Object orObject orphenomenonphenomenon

==??

Inter-Rater or Inter-Observer ReliabilityInter-Rater or Inter-Observer Reliability

Are different observers Are different observers consistentconsistent?? Can establish this outside of your study Can establish this outside of your study

in a in a pilot studypilot study.. Can look at Can look at percent of agreement percent of agreement

(especially with category ratings).(especially with category ratings). Can use Can use correlation correlation (with continuous (with continuous

ratings).ratings).

Test-Retest ReliabilityTest-Retest Reliability

Time 1Time 1 Time 2Time 2

Test-Retest ReliabilityTest-Retest Reliability

TestTest TestTest

Time 1Time 1 Time 2Time 2

==

Test-Retest ReliabilityTest-Retest Reliability

TestTest TestTest

Time 1Time 1 Time 2Time 2

==

Stability over timeStability over time

Test-Retest ReliabilityTest-Retest Reliability

Measure instrument at Measure instrument at two times two times for for multiple persons.multiple persons.

Compute Compute correlationcorrelation between the two between the two measures.measures.

Assumes there is Assumes there is no change no change in the in the underlying trait between time 1 and time underlying trait between time 1 and time 2.2.

Parallel-Forms ReliabilityParallel-Forms Reliability

Time 1Time 1 Time 2Time 2

Parallel-Forms ReliabilityParallel-Forms Reliability

Form BForm B

Time 1Time 1 Time 2Time 2

Form AForm A

==

Parallel-Forms ReliabilityParallel-Forms Reliability

Form BForm B

Time 1Time 1 Time 2Time 2

Stability across formsStability across formsForm AForm A

==

Parallel-Forms ReliabilityParallel-Forms Reliability

Administer both forms to the Administer both forms to the samesame people.people.

Get Get correlationcorrelation between the two forms. between the two forms. Usually done in Usually done in educational contexts educational contexts

where you need alternative forms where you need alternative forms because of the frequency of retesting because of the frequency of retesting and where you can sample from lots of and where you can sample from lots of equivalent questions.equivalent questions.

Internal Consistency ReliabilityInternal Consistency Reliability

Average inter-item correlationAverage inter-item correlationAverage inter-item correlationAverage inter-item correlation

Internal Consistency ReliabilityInternal Consistency Reliability

TestTest

Average Inter-Item CorrelationAverage Inter-Item CorrelationAverage Inter-Item CorrelationAverage Inter-Item Correlation

Internal Consistency ReliabilityInternal Consistency Reliability

TestTest

Item 1

Item 2

Item 3

Item 4

Item 5

Item 6

Average Inter-Item correlationAverage Inter-Item correlationAverage Inter-Item correlationAverage Inter-Item correlation

Internal Consistency ReliabilityInternal Consistency Reliability

TestTest

Item 1

Item 2

Item 3

Item 4

Item 5

Item 6

1.001.00.89.89 1.00 1.00.91.91 .92.92 1.00 1.00.88.88 .93.93 .95.95 1.00 1.00.84.84 .86.86 .92.92 .85.85 1.00 1.00.88.88 .91.91 .95.95 .87.87 .85.85 1.00 1.00

II11 II22 II33 II44 II55 II66

II11

II22

II33

II44

II55

II66

Average inter-item correlationAverage inter-item correlationAverage inter-item correlationAverage inter-item correlation

Internal Consistency ReliabilityInternal Consistency Reliability

TestTest

Item 1

Item 2

Item 3

Item 4

Item 5

Item 6

1.001.00.89.89 1.00 1.00.91.91 .92.92 1.00 1.00.88.88 .93.93 .95.95 1.00 1.00.84.84 .86.86 .92.92 .85.85 1.00 1.00.88.88 .91.91 .95.95 .87.87 .85.85 1.00 1.00

II11 II22 II33 II44 II55 II66

II11

II22

II33

II44

II55

II66

Average inter-item correlationAverage inter-item correlationAverage inter-item correlationAverage inter-item correlation

.90.90

Average item-Average item-totaltotal correlationcorrelationAverage item-Average item-totaltotal correlationcorrelation

Internal Consistency Internal Consistency ReliabilityReliability

TestTest

Average item-Average item-totaltotal correlationcorrelationAverage item-Average item-totaltotal correlationcorrelation

Internal Consistency Internal Consistency ReliabilityReliability

Average item-Average item-totaltotal correlationcorrelationAverage item-Average item-totaltotal correlationcorrelation

Internal Consistency Internal Consistency ReliabilityReliability

TestTest

Item 1

Item 2

Item 3

Item 4

Item 5

Item 6

1.001.00.89.89 1.00 1.00.91.91 .92.92 1.00 1.00.88.88 .93.93 .95.95 1.00 1.00.84.84 .86.86 .92.92 .85.85 1.00 1.00.88.88 .91.91 .95.95 .87.87 .85.85 1.00 1.00.84.84 .88.88 .86.86 .87.87 .83.83 .82.82 1.001.00

II11 II22 II33 II44 II55 II66

II11

II22

II33

II44

II55

II66

TotalTotal

Average item-Average item-totaltotal correlationcorrelationAverage item-Average item-totaltotal correlationcorrelation

Internal Consistency Internal Consistency ReliabilityReliability

TestTest

Item 1

Item 2

Item 3

Item 4

Item 5

Item 6

1.001.00.89.89 1.00 1.00.91.91 .92.92 1.00 1.00.88.88 .93.93 .95.95 1.00 1.00.84.84 .86.86 .92.92 .85.85 1.00 1.00.88.88 .91.91 .95.95 .87.87 .85.85 1.00 1.00.84.84 .88.88 .86.86 .87.87 .83.83 .82.82 1.001.00

II11 II22 II33 II44 II55 II66

II11

II22

II33

II44

II55

II66

TotalTotal

Average item-Average item-totaltotal correlationcorrelationAverage item-Average item-totaltotal correlationcorrelation

.85.85

Internal Consistency Internal Consistency ReliabilityReliability

TestTest

Item 1

Item 2

Item 3

Item 4

Item 5

Item 6

Split-half correlationsSplit-half correlationsSplit-half correlationsSplit-half correlations

Internal Consistency Internal Consistency ReliabilityReliability

TestTest

Split-half correlationsSplit-half correlationsSplit-half correlationsSplit-half correlations

Internal Consistency Internal Consistency ReliabilityReliability

Split-half correlationsSplit-half correlationsSplit-half correlationsSplit-half correlations

Internal Consistency Internal Consistency ReliabilityReliability

TestTest

Item 1

Item 2

Item 3

Item 4

Item 5

Item 6

Split-half correlationsSplit-half correlationsSplit-half correlationsSplit-half correlations

Item 1Item 1 Item 3Item 3 Item 4Item 4

Internal Consistency Internal Consistency ReliabilityReliability

TestTest

Item 1

Item 2

Item 3

Item 4

Item 5

Item 6

Split-half correlationsSplit-half correlationsSplit-half correlationsSplit-half correlations

Item 2Item 2 Item 5Item 5 Item 6Item 6

Internal Consistency Internal Consistency ReliabilityReliability

Item 1Item 1 Item 3Item 3 Item 4Item 4

TestTest

Item 1

Item 2

Item 3

Item 4

Item 5

Item 6

Split-half correlationsSplit-half correlationsSplit-half correlationsSplit-half correlations

.87.87

Internal Consistency Internal Consistency ReliabilityReliability

Item 2Item 2 Item 5Item 5 Item 6Item 6

Item 1Item 1 Item 3Item 3 Item 4Item 4

TestTest

Item 1

Item 2

Item 3

Item 4

Item 5

Item 6

Cronbach’s alpha (Cronbach’s alpha ())Cronbach’s alpha (Cronbach’s alpha ())

Internal Consistency Internal Consistency ReliabilityReliability

TestTest

Cronbach’s alpha (Cronbach’s alpha ())Cronbach’s alpha (Cronbach’s alpha ())

Internal Consistency Internal Consistency ReliabilityReliability

Cronbach’s alpha (Cronbach’s alpha ())Cronbach’s alpha (Cronbach’s alpha ())

Internal Consistency Internal Consistency ReliabilityReliability

TestTest

Item 1

Item 2

Item 3

Item 4

Item 5

Item 6

Cronbach’s alpha (Cronbach’s alpha ())Cronbach’s alpha (Cronbach’s alpha ())

.87.87

item 1item 1 item 3item 3 item 4item 4

item 2item 2 item 5item 5 item 6item 6

.87.87

item 1item 1 item 3item 3 item 4item 4

item 2item 2 item 5item 5 item 6item 6

.87.87

item 1item 1 item 3item 3 item 4item 4

item 2item 2 item 5item 5 item 6item 6

Internal Consistency Internal Consistency ReliabilityReliability

TestTest

Item 1

Item 2

Item 3

Item 4

Item 5

Item 6

SHSH11 .87.87

SHSH22 .85.85

SHSH33 .91.91

SHSH44 .83.83

SHSH55 .86.86

......SHSHnn .85.85

Internal Consistency Internal Consistency ReliabilityReliability

Cronbach’s alpha (Cronbach’s alpha ())Cronbach’s alpha (Cronbach’s alpha ())

.87.87

item 1item 1 item 3item 3 item 4item 4

item 2item 2 item 5item 5 item 6item 6

.87.87

item 1item 1 item 3item 3 item 4item 4

item 2item 2 item 5item 5 item 6item 6

.87.87

item 1item 1 item 3item 3 item 4item 4

item 2item 2 item 5item 5 item 6item 6

TestTest

Item 1

Item 2

Item 3

Item 4

Item 5

Item 6

SHSH11 .87.87

SHSH22 .85.85

SHSH33 .91.91

SHSH44 .83.83

SHSH55 .86.86

......SHSHnn .85.85

= .85= .85 = .85= .85

Internal Consistency Internal Consistency ReliabilityReliability

Cronbach’s alpha (Cronbach’s alpha ())Cronbach’s alpha (Cronbach’s alpha ())

.87.87

item 1item 1 item 3item 3 item 4item 4

item 2item 2 item 5item 5 item 6item 6

.87.87

item 1item 1 item 3item 3 item 4item 4

item 2item 2 item 5item 5 item 6item 6

.87.87

item 1item 1 item 3item 3 item 4item 4

item 2item 2 item 5item 5 item 6item 6

TestTest

Item 1

Item 2

Item 3

Item 4

Item 5

Item 6

SHSH11 .87.87

SHSH22 .85.85

SHSH33 .91.91

SHSH44 .83.83

SHSH55 .86.86

......SHSHnn .85.85

= .85= .85 = .85= .85

Like the average Like the average of all possible of all possible

split-half split-half correlationscorrelations

Like the average Like the average of all possible of all possible

split-half split-half correlationscorrelations

Internal Consistency Internal Consistency ReliabilityReliability

Cronbach’s alpha (Cronbach’s alpha ())Cronbach’s alpha (Cronbach’s alpha ())

.87.87

item 1item 1 item 3item 3 item 4item 4

item 2item 2 item 5item 5 item 6item 6

.87.87

item 1item 1 item 3item 3 item 4item 4

item 2item 2 item 5item 5 item 6item 6

.87.87

item 1item 1 item 3item 3 item 4item 4

item 2item 2 item 5item 5 item 6item 6

TestTest

Item 1

Item 2

Item 3

Item 4

Item 5

Item 6

Internal Consistency Reliability - SummaryInternal Consistency Reliability - Summary

Average inter-item correlationAverage inter-item correlation Average item-total correlationAverage item-total correlation Split-half reliabilitySplit-half reliability Cronbach’s alpha (Cronbach’s alpha ())