Validity/Reliability Matters Really? Beverly Mitchell, Kennesaw State University.

Validity/Reliability MattersReally?

Beverly Mitchell, Kennesaw State University

Can a test be valid and not be reliable?


Can a test be reliable and not be valid?


JustifiableRelevant

True to its purpose(consistently)

Validity


Validity

Design Issues

Application Issues


Validity

Design IssuesApplication Issues


Design: Creating the Instrument

1-Inference 2-Complexity


Inference

Low High


HighInference

To draw a conclusion

To guess, surmise

To suggest, hint


LowInferenceStraightforward

Language = precise & targeted

Clear – no competing interpretations of words

No doubt as to what point is being made


Inference

Low High


Complexity

Low High


HighComplexity

Complicated

Comprised of interrelated parts or sections

Developed with great care or with much detail


LowComplexity

Simplistic

Plain

Unsophisticated


Complexity

Low High


Low HighInference

Complexity

Low

High

How They Are Related


Low HighInference

Complexity

Low

High

Designing the Instrument


Low HighInference

Complexity

Low

High

Due “Yesterday”!


Low HighInference

Complexity

Low

High

“Overachieving”


Low HighInference

Complexity

Low

High

How Much Error Are You Willing to Risk?

Error

Error


Low HighInference

Complexity

Low

High

Compromise


Does the OBSERVED Behavior = TrueBehavior?

Observed SCORE ≠ TRUE SCORE

E R R O R


Design: Creating the Instrument

1-Inference

General Rubric - high

Qualitative analytic rubric – low

2-Complexity

Easy to develop – question worthiness, guidance, single interpretation - low

Time to develop – labor intensive, onerous, long - high


Validity

Design Issues

Application Issues


Application Issues

Designated Use

Limitations/Conditions


Application Issues

Designated UseDon’t borrow from neighbor!


Application Issues

Limitations/ConditionsOne size does not fit all or apply to all circumstances


Ways to Increase Probability for Accuracy

Compare language: standards & concepts

The concepts/expectations in the standards are apparent in the assessments – same depth and breadth

Good example of Content Validity

Behavior (performance) expected in the standard matches the performance expected in the assessment – i.e., knowledge of…demonstrating skill…

Identify Key/Critical items/concepts to evaluate

Give it away for analysis (many eyes)

Invite external “expert” review

Be receptive to feedback

Surveys from P-12 partners, candidates

Regular evaluation and analysis: revise, revise, revise

Awareness of design and application issues


Ways to Increase Reliability

Begin with a valid instrument Two reliability issues:

Reliability of the instrument: repeated use of instrument by same evaluators

If problematic: revise, re-think, abandon Reliability of the scoring: performance rated same by

different evaluators, i.e., objectivity If problematic: ensure qualifications of evaluators, check

rubric, check language, minimize generalized concepts applied to all subject areas

Train evaluators frequently


AN APPLICATION: A KSU Workshop (Handouts Available)

Thirty experienced teachers participated in a daylong workshop to help us evaluate three student teaching observation rating forms.


Three Instruments

Traditional Candidate Performance Instrument (CPI) Observation of Student Teaching. Observer is asked to indicate strengths and weaknesses and areas for improvement in three broad outcomes (Subject matter, Facilitation of Learning, and Collaborative Professional).

Modified CPI Observation of Student Teaching (Observer is asked to explicitly rate each proficiency within each outcome and then provide narrative indicating any strengths, weaknesses, suggestions for improvement.

Formative Analysis Class Keys: Observer is asked to rate 26 elements from Georgia Department of Education’s Class Keys. No required narrative.


Generally we were interested in two areas……………….

Validity/Accuracy – Which instrument provides us the best inference about the present of positive behaviors (proficiencies) we deem important? AND

Reliability/Consistency – Which instrument demonstrates the best inter-rater reliability?


Study Design Instrument Group 1 Group 2 Group 3

Period 1:Traditional CPI-Narrative

Video A Video B Video C

Period 2: Modified CPI Rating and Narrative

Video B Video C Video A

Period 3: Class Key Formative Analysis

Video C Video A Video B


Reliability

Strongest inter-rater agreement between Modified CPI with performance level rating followed by Class Keys Formative Assessment Instrument with a performance level rating.

Very little agreement between behaviors noted in Traditional CPI narratives and no performance level ratings were available. Probably not a reliable instrument for rating student teaching behaviors.


Validity

Both the traditional CPI and Modified CPI are explicitly aligned with institutional (and other) standards but the Traditional CPI is a global assessment and the Modified CPI requires a rating and narrative for each proficiency.

However, the traditional CPI has not demonstrated reliability….so

Participants were also asked to provide information about the language, clarity, ease of use for all instruments.


Validity/Reliability Matters Really? Beverly Mitchell, Kennesaw State University.

Documents

Transcript of Validity/Reliability Matters Really? Beverly Mitchell, Kennesaw State University.