http://www.soe.umich.edu/lmt/

Measuring Effectiveness in Measuring Effectiveness in Mathematics Education for Mathematics Education for

TeachersTeachers

Heather HillUniversity of Michigan School

of EducationLearning Mathematics for

Teaching2007 MSRI

June 1, 2007

http://sitemaker.umich.edu/lmt/

Avoiding Arbitrariness! Avoiding Arbitrariness!

• 16 is my favorite number

http://sitemaker.umich.edu/lmt/

Avoid Arbitrariness!Avoid Arbitrariness!

QuickTimeª and aMotion JPEG OpenDML decompressor

are needed to see this picture.

http://sitemaker.umich.edu/lmt/

ChallengeChallenge

• Knowing you’ve added (relevant) knowledge to prospective or in-service teachers– Not going to discuss student achievement as

outcome

• Issues to consider as you pursue understanding impact:– Getting clear on your question– Research design– Instrument selection– Comparability to other projects

http://sitemaker.umich.edu/lmt/

Getting clear on your questionGetting clear on your question

• Do you want to know the effect of:– A set of materials?– A course?– Course & instructor?– Sequence of courses/instructors?

• Different questions imply different designs– Simplest design: What is effect of

course and instructor?

http://sitemaker.umich.edu/lmt/

Getting clear on your questionGetting clear on your question

• Do you want to know the effect of:– A set of materials?– A course?– Course & instructor?– Sequence of courses/instructors?

• Different questions imply different designs– Simplest design: What is effect of

course and instructor?

http://sitemaker.umich.edu/lmt/

Research DesignResearch Design

• Question: What would these people have known and been able to do in the absence of our program?– Estimate difference between actual and

“counterfactual”

• Problem: Cannot estimate with program and without program at the same time– e.g., Marcia in December WITH and WITHOUT TE401– Random assignment provides best estimate of

counterfactual– Quasi-experimental designs more possible

http://sitemaker.umich.edu/lmt/

Stop. Design. Stop. Design.

• 1 minute: Think about how you would evaluate your work with teachers– What is your question?– How can you gather evidence about

your question?

• 3 minutes: Share & critique with neighbors

http://sitemaker.umich.edu/lmt/

Best Solution: Best Solution: Random AssignmentRandom Assignment

• Problem– Rules out easiest research question: you +

your materials– Treatment/random assignment of students

occurs in classes – Statistical tests must be performed at the level

of treatment (e.g., compare this class to that)• Using students = cheating by boosting your power

– Need large N of classrooms or programs for statistical power

• Even mathematicians aren’t this prolific

• Another: Technically complex

http://sitemaker.umich.edu/lmt/

Quasi-Experimental DesignsQuasi-Experimental Designs

• Definition: No randomization to treatment

• Problems:– Not causal -- always threat to inferences

• Selection, pre-test controls, “natural” learning

– “Assignment” is still class level for some questions

– But easier to implement

http://sitemaker.umich.edu/lmt/

Quasi-Experimental DesignsQuasi-Experimental Designs

• Worst:

– Threats: selection, no comparison, no pre-test control

• Second-Worst

– Threats: Selection into T and C, no pre-test control

Tpost

Tpost

Cpos

t

http://sitemaker.umich.edu/lmt/

Quasi-Experimental DesignsQuasi-Experimental Designs

• Slightly less bad, but still not good:

– Threats: “Natural” learning over time; learning from instrument; selection

• Good:

– Threats: Selection

Tpre Tpost

Tpost

Cpos

t

Tpre

Cpre

http://sitemaker.umich.edu/lmt/

Quasi-Experimental DesignsQuasi-Experimental Designs

• Best:

– Threats: Selection– Advantage: Allows for growth modeling

T3

C3

T2T1

C2C1

http://sitemaker.umich.edu/lmt/

Quasi-Experimental Design: Quasi-Experimental Design: Unit of Analysis Problem Does Unit of Analysis Problem Does

Not Go AwayNot Go Away

• To understand YOUR effect with YOUR materials, unit of analysis can be student– E.g., comparing 32 pre/post tests

• To separate materials effect from instructor effect, need multiple classrooms

http://sitemaker.umich.edu/lmt/

Example: Quasi-Experimental Example: Quasi-Experimental DesignDesign

• Hill/Ball study of MPDI (2002-2003 data):– Pre/post for “treatment” group (1000 teachers

in about 25 sites)– Pre/post for “comparison” group (300

teachers who signed up for MPDIs but did not attend)

• Can compare change in treatment to change in comparison– MKT instrument

• Compare among 25 programs

http://sitemaker.umich.edu/lmt/

InstrumentationInstrumentation

• Criteria:– Aligned to your program’s content– Technically checked and validated– Linked to student achievement

• Types of instruments:– Teacher knowledge– Teacher “practice” – Mathematical quality of teaching

http://sitemaker.umich.edu/lmt/

Teacher Knowledge: Multiple Teacher Knowledge: Multiple ChoiceChoice

• LMT: K-5, 6-8 measures in number/operations, algebra, geometry (soon: rational number, proportional reasoning)

• www.sitemaker.umich.edu/lmt

• KAT: Algebra • www.msu.edu/~kat/

• DTAMS: K-5, 6-8 measures in Whole Number Computation, Rational Number Computation, Geometry/Measurement, Probability/Stats/Algebra

• http://louisville.edu/edu/crmstd/diag_math_assess_elem_teachers.html

http://sitemaker.umich.edu/lmt/

Knowledge: Other MethodsKnowledge: Other Methods

• Kersting (LessonLab): Teacher analysis of video segments

• Discourse analysis, clinical interviews (e.g., TELT -- see Ball’s personal website), videos of clinical teaching experiences

• Home-grown tests

http://sitemaker.umich.edu/lmt/

Possible Instruments: Possible Instruments: ObservationalObservational

• Of “practice”:– Reformed Teaching Observation

Protocol– Horizon’s Inside the Classroom

• Of “mathematical quality” of instruction– LMT Mathematical Quality of Instruction– TIMSS instruments

http://sitemaker.umich.edu/lmt/

Plea from Meta-Analysts: Plea from Meta-Analysts: ComparabilityComparability

• Use common measures across teacher education efforts. Why?– Knowledge is built by comparing effects

of different programs• Knowing that program A has a .5 effect is

good• But knowing that Program A =.5 and

Program B = .3 is better; can ask what aspects of program A “worked”

• Must do with large “N” of programs

http://sitemaker.umich.edu/lmt/

Comparison ExampleComparison Example

• Example: Carnegie (Matt Ellinger)– Formative assessment (feedback to programs

involved)– Four programs with math/math ed

collaboration • Seven sections

– Place value is content focus– LMT instrument focused on place value is

pre/post– No comparison/control; internal variation

http://sitemaker.umich.edu/lmt/

Comparison ExampleComparison Example

• Mathematical Education of Elementary Teachers (Raven McCrory)– 37 sections, 27 instructors, 13 institutions– 588 total matched-pair student responses– Can compare outcomes by program

characteristics• Instructor surveys of topics taught• Textbook used, chapters covered• Cognitive demand measure (based on Adding

It Up)• Instructor characteristics

http://sitemaker.umich.edu/lmt/

Randomized Example: Hill (fall Randomized Example: Hill (fall 2007)2007)

Videopre

Videopre

Videopre

Videopre

Lesson StudyMath ContentCoaching

Records of Practice

Videopost

Videopost

Videopost

Videopost

http://sitemaker.umich.edu/lmt/

ConclusionConclusion

• Don’t be arbitrary• Link to many instruments described

here– www.sitemaker.umich.edu/lmt

• Good design advice:– Institute for Social Research: Robin

Jacob (rjacob@umich.edu)– Local university-based evaluators