Assessment as the

“Hidden Variable” in Achievement

Michael ZeilikDept. of Physics & AstronomyThe University of New Mexico

(NSF DUE 99-85511)

What Works in Higher Ed? What Works in Higher Ed? 20th Century Gain Results20th Century Gain Results

Active Learning, Mastery Learning (SD ≥ 0.5SD ≥ 0.5, PSI, discussion, debates, games, role playing)

Cooperative Learning (SDSD≥ 0.5≥ 0.5, cognitive and affective; century of research, all disciplines)

One-on-One Tutoring (SD = 2 = 2,, with trained tutors)

Lectures:Reinforce memorization; promote illusion of competence; educational seduction (Dr. Fox)!

Still not better than lectures: Computer-based, instructional

television, Web-based

What Does What Does NotNot Work? Work? Standard Models (Standard Models (SDSD < 0.3< 0.3))

Everything Else!Everything Else!

Evidence-Based ReformEvidence-Based ReformActive learning

+ Classroom assessment=

Achievement: Deep conceptual understanding

(“sticky learning”)

What is What is ActiveActive Learning?Learning?

Instructional activities that engage students’ minds (the “sticky” factor)

Interactive lectures, discussion, cooperative learning teams, role playing, peer teaching, minute papers, concept mapping, games, cooperative tests, controversies

Require alternative assessments

What Works in Context: What Works in Context: Physics & Astronomy Physics & Astronomy (PAER)(PAER)

Disciplinary education research, “tools of the trade” for experiments

Empirical robust knowledge base about learning in physics & astronomy

Probe initial state (prior knowledge), final state (learning outcomes)

Measure change: Assessment! Field-tested Learning Assessment Guide (FLAG: www.flaguide.org)

FLAG CATsFLAG CATs Attitudinal Surveys: E.

Seymour, E. Lewis Concept Tests:

A. Ellis Concept Maps:

M. Zeilik Conceptual Diagnostic

Tests: M. Zeilik

Interviews: M. Smith, S. A. Southerland

Performance Assessments: T. Slater

Portfolios: T. Slater

Scoring Rubrics: D. Ebert-May

Student Assessment of Learning Gains: E. Seymour

Weekly Reports: E. Etkina

Mathematical Thinking: M. Swan,

J. Rideway Multiple-Choice Tests: J. Parkes Minute Papers:

M. Zeilik

Conceptual Diagnostic TestsConceptual Diagnostic Tests

Assess prior knowledge of students“Misconceptions” research,

“think aloud” interviews; “natural language”

Measure pre/post conceptual gains (normalized gain, <g>)

Force Concept Inventory (FCI); Astronomy Diagnostic Test: “Astro 101”, nationally normed, reliable & valid

Pre/post: ADT National Project/UNMPre/post: ADT National Project/UNM<g> = (post% - pre%)/(100 - pre)<g> = (post% - pre%)/(100 - pre)

UNM

UNM

Standard errors plotted

From Quarks to CosmosFrom Quarks to Cosmos(Committee on the Physics (Committee on the Physics

of the Universe; NRC)of the Universe; NRC)

11 interface areas (21st century!) UNM Physics 102 (Conceptual Physics) poll, top 5:

1. What is dark matter?2. What is the nature of

dark energy?3. How did the universe begin?4. Are there additional spacetime dimensions?5. What are the masses of neutrinos?

Student Assessment of Student Assessment of Learning Gains (SALG)Learning Gains (SALG)

Probes learning gains students perceive for specific aspects

Avoids performance critiques (“Dr. Fox”) of instructor (not correlated with conceptual gain)

Easily customized (15 min); access through FLAG (hosted by University of Colorado/Boulder)

UNM P102 SALG Spring 2003(Mean = 3.38)

3.68

3.52

3.64

3.71

4.15

3.75

3.99

1.00 2.00 3.00 4.00 5.00

Instructor talk

Discussions

Team work

Activities

Coop quizzes

CT/CRS

Videos

Rating (scale 1 - 5)

P102 UNM Spring 2003 Gender Matters—Sometimes!

1.00 2.00 3.00 4.00 5.00

Math Self-Aptitude

Science Self-Aptitude

Last math

Pretestconfidence

PosttestConfidence

Score (1 = very poor, 5 = very good)

Female Male All

NS!

NS!

p < 0.002

p < 0.0001

p < 0.002

Cooperative Quiz GainsCooperative Quiz Gainsby Genderby Gender

P102 UNM Spring 2003 Quizzes

0.00

0.20

0.40

0.60

0.80

1.00

Practice Quiz #1 Quiz #2 Quiz #3 Mean

<g> Female Effect size (F) <g> Male Effect size (M)

Does it stick? Does it stick? (Coop quizzes/Test)(Coop quizzes/Test)

Grade ExpectationsGrade Expectations

Assessment “C” LessonsAssessment “C” Lessons(Gender (Gender SometimesSometimes Matters!) Matters!)

Content matters! Concepts matter!

Connections matter! Context matters!

Confidence? Competence?

Assessment probes all & increases achievement by 0.5 SD!