Using Mixed Methods to Develop a Questionnaire

on Accountability for Teachers and Principals

Nora Gannon, Tysza Gandha & Katherine Ryan 2009 CREATE Conference

October 9, 2009

Accountability and Evaluation

Using Questionnaires in Evaluation

Using Mixed Methods in Questionnaire Development

Our Mixed Methods Design and

Implementation

Summary

Overview

No Child Left Behind (NCLB)

Credible Evidence

Improving Evaluation Methods

Accountability and Evaluation

Key Questions◦ What are the intended and unintended

consequences of the accountability assessments utilized in this state?

◦ What are the perspectives of teachers and principals on how these consequences influence: Instructional practices? Local assessment practices? Use of test data? School policies and practices? The Teaching profession?

Instate Assessment Consequences Evaluation (IACE)

◦Illuminate large-scale patterns in schools reform as a result of accountability systems in place

◦Provide broad-based views of stakeholders’ perspectives (e.g., teachers and principals)

◦Time and cost efficient

Use of Questionnaires in Large-scale Educational Evaluations

Reference: Desimone & le Floch, 2004

Advantages to Using Questionnaires

Challenge for Researchers

• Administer to many people to learn perspectives of diverse stakeholders

How do we know if a respondent’s understand questionnaire items as we intended?

• Large scale survey study can examine a wide range of contexts (e.g., school settings)

How can we strengthen our inferences across contexts?

Reduce response bias

Reduce response burden

Improve item clarity

GOAL: Understand thought process and improve data

interpretability

Reference: Czaja & Blair, 2005

◦Purposeful

◦Systematic

◦Comprehensive

Mixed Methods for Instrument Development

Questionnaire Development: Combination

Sequential/Concurrent Mixed Methods Design

qual Data analysis

qual Data collection

QUAN Data analysis

QUAN Data collection QUAL

Data analysis

QUAL Data collection

QUAN & qual

Phase 2

QUAL

Phase 1

Integration of Entire Analysis

PHASE 1: QUALITATIVE

qual Data analysis

qual Data collection

QUAN Data analysis

QUAN Data collection QUAL

Data analysis

QUAL Data collection

QUAN & qual

Phase 2

QUAL

Phase 1

Integration of Entire Analysis

PHASE 1: QUALITATIVE

Methods• Item Development• Cognitive Labs• Expert Review

Literature Review◦ 5 dimensions: Instructional practices, Local assessment

practices, Use of test data, School policies and practices, Teaching profession

◦ Dimension Task Lists

Item development◦ Item Bank◦ Reverse engineering

Content Review

Item Development

PHASE 1: QUALITATIVE

Item DevelopmentExample of Partial Task List for Changes in Instructional Practices

Test Preparation Kinds of test preparation strategies and frequency of use o Test taking tips (marking answers correctly,

eliminating wrong answers etc) o Using tests of similar format to state test o Using items of similar format of items on

state test o Using old test items from state test o Using entire tests from past as practice o Commercial test prep o Focus on content that is likely to be tested

Teachers’ perspectives in test preparation (i.e. ethics) o Easy to understand and pull information from o Teachers receive training in reading reports

PHASE 1: QUALITATIVE

Literature Review◦ 5 dimensions: Instructional practices, Local assessment

practices, Use of test data, School policies and practices, Teaching profession

◦ Dimension Task Lists

Item development◦ Item Bank◦ Reverse engineering

Content Review

Item Development

PHASE 1: QUALITATIVE

What is the purpose of a cognitive lab?◦ Gain insight into participant understandings.

How do you sample for cognitive labs?◦ Choose participants that represent your

population.

Implementation: the think-aloud process◦ Be careful not to influence responses!

Cognitive Labs

PHASE 1: QUALITATIVE

Single vs. Dual Rating Scales

Cognitive Lab Findings

PHASE 1: QUALITATIVE

How have you changed your instructional strategies as a result of NCLB testing?

Overall, my use of this strategy has… In

crea

sed

a

lot

Incr

ease

d

som

ewh

at

Not

ch

ange

d

Dec

reas

ed

som

ewh

at

Dec

reas

ed a

lot

I do

n’t

use

this

st

rate

gy

a. Project-based assignments

5 4 3 2 1

Single vs. Dual Rating Scales

Cognitive Lab Findings

PHASE 1: QUALITATIVE

Since NCLB testing began in grades 3-8, how have you changed your instructional strategies? To what extent were the changes a result of NCLB testing? [Please rate the item on both scales]

Overall, my use of this strategy has… To what extent was it a result

of NCLB testing?

Incr

ease

d

a lo

t

Incr

ease

d

som

ewh

at

Not

ch

ange

d

Dec

reas

ed

som

ewh

at

Dec

reas

ed a

lot

I do

n’t

use

this

st

rate

gy

To

a gr

eat

exte

nt

To

a m

oder

ate

exte

nt

To

a sm

all

exte

nt

Not

at

all

a. Project-based assignments

5 4 3 2 1 4 3 2 1

Global Questions

Cognitive Lab Findings

To what extent, and how, have NCLB testing changed your instruction since it began in all grades 3-8?

Changed it a lot,

for the better

Changed it somewhat,

for the better

Did not change

it

Changed it somewhat,

for the worse

Changed it a lot, for the worse

5 4 3 2 1

PHASE 1: QUALITATIVE

Cognitive Lab Findings

1. To what extent has NCLB testing had positive effects on your instruction since it began in grades 3-8?

To a great

positive extent

To a moderate positive extent

To a small positive extent

Not at all

4 3 2 1

2. To what extent has

NCLB testing had negative effects on instruction in your school since it began in grades 3-8?

To a great

negative extent

To a moderate negative

extent

To a small negative

extent

Not at all

4 3 2 1

PHASE 1: QUALITATIVE

Language

What does “test preparation materials” mean?

Cognitive Lab Findings

PHASE 1: QUALITATIVE

Since NCLB testing began in grades 3-8, how has the use of the following instructional strategies changed in your school? To what extent were the changes a result of NCLB testing?

[Please rate each item on both scales]

My school’s use of this strategy has… To what extent

was it a result of NCLB testing?

Inc

reas

ed

a

lo

t

Inc

reas

ed

s

om

ew

ha

t

No

t c

ha

ng

ed

De

cre

ase

d

so

me

wh

at

De

cre

ase

d

a l

ot

We

do

n’t

u

se

th

is

str

ate

gy

A l

ot

So

me

Ve

ry l

ittl

e

No

t a

t a

ll

a. Using test preparation materials to help students prepare for the NCLB test

5 4 3 2 1 4 3 2 1

Language

◦ Strategies for Addressing Language Issues

Examples

Alternate words

Multiple questions

Cognitive Lab Findings

PHASE 1: QUALITATIVE

Why use Expert Reviewers?

Expert Reviewers

PHASE 1: QUALITATIVE

When to engage expert reviewers?

Early• Can solicit broad, big picture feedback

• Have time to make major changes with feedback

Later• Can solicit highly specific (item-level) feedback

• Useful for final review of instruments in almost-final draft

Reference: Ramirez, 2002

PHASE 2: QUANTITATIVE and qualitative (concurrent)

qual Data analysis

qual Data collection

QUAN Data analysis

QUAN Data collection QUAL

Data analysis

QUAL Data collection

QUAN & qual

Phase 2

QUAL

Phase 1

Integration of Entire Analysis

Methods• Pilot Questionnaire• Cognitive Lab

Focus Group

PHASE 2: QUANTITATIVE AND qualitative

Teacher and Principal Questionnaires◦ Approximately 200 items; 60% shared between

teacher and principal versions

Sample- 383 principals - 2415 teachers

(1940 online, 475 mail)

Proportionate Probability Sample (N=400 schools)

◦ Stratified by geographic region and school type (i.e. elementary schools and middle schools)

◦ Sampling Criteria (AYP Subgroups)

Pilot QuestionnaireAdministration

PHASE 2: QUANTITATIVE AND qualitative

Teachers: 37% (902)

◦ Online: 44%◦ Paper-and-pencil: 9%

Principals: 44% (175)

Pilot QuestionnaireResponse Rates

PHASE 2: QUANTITATIVE AND qualitative

Population proportions

Response rate

proportions

Elementary 60% 57%

Middle School

40% 43%

Pilot QuestionnaireFindings

Refine content of Questionnaire based on◦ Low response endorsement◦ Low response rates◦ Low variance on scaled items

Pilot Questionnaire Findings

PHASE 2: QUANTITATIVE AND qualitative

IMPLEMENTATION: Cognitive lab in small group setting (4-6 people)

PURPOSE: Group narrative

SAMPLE: Depends on what you want to know

Cognitive Focus Group

Strengths Limitations• Group setting allows for

conversations that elicit normal narrative

• Cost-efficient way to learn about potentially different interpretations

• Have to use smaller number of items than individual cognitive lab

• Group relations need to be considered (e.g., include teachers and principal?)

PHASE 2: QUANTITATIVE AND qualitative

To what extent is the change in local assessment due to the state accountability assessment?

Mr. Smith: I think assessment is just a component of good teaching.

Ms. Brown: I think you are naïve…it’s all about preparing for the state test.

Ms. Rogers: Well, it depends on how you look at it, I guess… if you are cynical everything is because of accountability assessment!

Cognitive Focus Group

PHASE 2: QUANTITATIVE AND qualitative

Goal Method• Refine scope of

questionnaire• Comprehensive

literature review • Expert reviewers• Pilot administration

• Improving the interpretability of the questionnaire

• Cognitive labs• Cognitive focus groups

• Strengthen capacity to warrant inferences within and across contexts

• Cognitive labs• Purposeful sampling for

cognitive labs• Deliberate sample frame

in pilot

Summary: the value in using mixed methods

Questions?Thank you!