Assessing multiple research-based transformations in second semester physics.

Steven J. Pollock University of Colorado at Boulder (per.colorado.edu)

We examine the impact of several proven reforms1,2 in a large course, including Peer Instruction3 and Tutorials4

Goals:Study impacts of multiple layered coordinated activities Investigate necessary features of individual reforms and the dynamics among them. Improve/examine student learning & expectations/beliefs. Increased engagement with physics and teaching of physics

Guiding Research Questions:Which elements most closely associate with learning gains?What traditionally not-studied factors shape students’

conceptual development, attitudes, beliefs, and success?What are the impacts of/on undergrad Learning Assistants?

Phys 1120500+ students3 lectures/wk

(No lab)

LON-CAPA5

onlinepersonalized homework

Text6

HRW

Tutorials50 min/wk, 30 students, 1 grad TA

+ 1 undergrad Learning Assistant (LA)Weekly prep sessions + LA seminar

Interactive LecturesPeer Instruction with

H-ITT personal response system

Introduction

TOPIC: PRE POST (CLICK)

POST (EXAM)

Flux through closed surface (0 charge) 15% 75% 90%

Capacitance: dependence of voltage difference on plate separation

35 NA 80

Circuits: compare bulb brightness

(current model sufficient)

15 70 80 *

Circuits: compare bulb brightness

(current and voltage models needed)

5

BEMA: 25

60 60

BEMA: 65

RC circuits: short time behaviour 40, BEMA=45 90 85, BEMA=85

Lenz’ law (qualitative) post lect: 55 pre tut: 80 85

Data are for selected concepts from tutorials. (Results as %, rounded to nearest 5%) "Post Click" are multiple choice questions answered in class via clickers Clicker q's encourage peer discussion, but exams are individual.

“LA” = Undergrad Learning Assistant. “TA” = grad teaching assistant Data from other institutions shown for comparison at bottom 8

44% average gain. Note: BEMA is hard - normalizing to physics grad TA pre-score would yield a normalized class gain of 60%.)

LEARNING GAINS: Pre/post (BEMA) 8

LEARNING GAINS: Tutorials

End NotesConclusions

1. McDermott, L., Redish, E., American Journal of Physics, 67(9), 755-767, 1999, Redish, E., Teaching Physics with the Physics Suite, Wiley 2003

2. Hake, R., Am. J. Phys 66, 64-74, (‘98) Hake R., Cons Ecology 5(2):28, (‘02)

3. Mazur, E., Peer Instruction, Prentice Hall 1997

4. McDermott, Shaffer, Tutorials in Introductory Physics, Prentice Hall 02.

5. CAPA homework system: see www.lon-capa.org

6. Halliday, Resnick, Walker, Fundamentals of Physics Wiley 2004

7. http://cosmos.colorado.edu/phet/survey/fall2003.html Adams et al., "Design and Validation of the Colorado Learning About Science Survey", Perkins et al., "Correlating student attitudes with student learning using the CLASS", both in review for Proceedings of the PERC 2004, Sacramento, CA.

8. BEMA: Private communication, R. Chabay and B. Sherwood, see also http://www.ncsu.edu/per/TestInfo.html

9. McDermott, L.C. and P.S. Shaffer (1992). Am. J. Phys. 60, 994 (1992), and part II, p. 1003

10. Elby, A. Am. J. Phys 69(7), (2001)

11. Redish, Saul, Steinberg, Am. J. Phys. 66 212-224 (1998)

12. Pollock, S. “No Single Cause…”, PERC 2004.

13. Finkelstein, N., Pollock, S., to appear in PRST (2005)

Learning gains (via multiple measures) are high => this is a successful transformed course.

(Average posttest BEMA exceeds current CU phys majors at end of their Jr. level E&M.)

Attitudes/beliefs/expectations 10,11 show modest declines (towards “novice”), but less so for higher learning-gain students. (Student evaluations of course are high.) No single reform element alone appears to account for the measured successes, but the

combination of course elements has a large impact 12,13. LA program is successful (pre/post content gains, LA satisfaction, and interest in teaching) In 3 semesters, 24 STEM majors have been LA’s. Of those, 8 were LA’s twice or more

(continuing with more School of Ed courses) and (at least) 3 are headed to K-12 teaching.

Course Structure

LEARNING GAINS: GENDER

Female pretest, and gain, on BEMA marginally lower. Final course grades are statistically identical: females outperform on hw

and particip, compensating for slightly lower exam scores.

• Bins are course grades (which correlate strongly with BEMA gain). • CLASS7 is a Likert-scale survey of attitudes and beliefs about learning. Note correlation of final grade with CLASS beliefs at the start More negative shift in beliefs for lower performing students.

ATTITUDES AND BELIEFS

EXAMINING LOW OUTCOMES (on BEMA post)

. Can we characterize students who do NOT end up with high BEMA scores?

Bottom quartile on BEMA post

The rest of the class

Ave online participation grade * 87% 87% *

Pretest BEMA ** 22% 28%

Grade in mechanics ** 76% 84%

Pretest CLASS (all categories) ** 63% 72%

Normalized learning gain on BEMA 24% 52%

Self-reported outside study time/week 6.8 hrs 5.6 hrs

Females 31% 22%

Undeclared Majors 11% 5%

Course Grade *** Solid C Solid B

* Participation is not significantly different: these are not "slackers"**Weak at-risk markers. Can not easily/uniquely pre identify this population.***low gainers not largely D/F students (for whom we rarely get BEMA score) Statistical uncertainty 1-2% for entries in "tail" column, typically half that for rest. Bottom quartile => roughly BEMA (post)<50% CLASS shows “percent favorable” in all categories combined.

Special thanks to PhysTEC (APS/AIP/NSF), NSF STEM-TP, NSF CCLI (DUE #0410744), the CU Physics Dep’t, the entire Physics Education Research at Colorado group, and the Physics Education Group at the Univ. of Washington.

Acknowledgements

Data Collected

Shown here is all new data from Fall ‘04 “Phys 1120” (Physics II)We have collected data for 2 semesters of reformed Phys I (reported elsewhere)

• Pre and post conceptual exams: BEMA8 (Brief E&M Assessment) (Fa ‘04): N=331 matched of 462 students

• Pre/post CLASS (Colorado Learning about Science Survey7)• Weekly online Tutorial pretest4

• Posttests given in class (collaborative, with clickers)• Posttests on course exams: 75% mult. choice, 25% long answer.• Attendance (lecture and Tutorial)

Collected but not shown here:

• Weekly online attitude, and course surveys/long response• Homework: online and long answer

INDIVIDUAL REFORM ELEMENTS

Correlating normalized BEMA gain with: Correlation coefficient

Pretest BEMA 0.01

Average course exam score 0.62 (p<<.05)

Overall course grade 0.56 (p<<.05)

Grade in Phys 1 0.51 (p<<.05)

Tutorial homework 0.32 (p<<.05)

CLASS attitudes/beliefs (all categories, pre) 0.29 (p<<.05)

Long answer homeworks 0.18 (p<<.05)

In class clicker score 0.15 (p<.05)

Conventional homework 0.11 (p<.05)

Lecture attendance * 0.09

Tutorial Attendance * 0.04

• Tutorial attendance is close to 90%, lecture attendance is close to 80%

INFLUENCING FACTORS.

BEMA gain

BEMA PRE

Course grade

Tut. HW

Lec. Att.

CAPAHW.

Mech. grade

BEMA PRE .01

Course grade .56 .26

Tutorial HW .32 .07 .79

Lecture Att. .09 .04 .35 .42

CAPA homework .11 .01 .63 .66 .36

Mechanics grade .51 .21 .77 .58 .23 .40

CLASS .29 .08 .25 .08 0.0 0.0 .31

Matrix of correlation coefficients Correlation coefficients in red are not statistically significant (p>.05) CLASS => % favorable on all scored categories on pretest Normalized BEMA gain does not correlate with pretest (c.f. Hake2) Course grade based heavily (60%) on exams (25% long answer, highly conceptual) covering

far more material than BEMA. No single course reform correlates strongly with BEMA gain

References

Examining Impacts of Reforms Population EffectsEvidence for Conceptual Learning

BEMA (matched) (CU scoring) Fa04g(ave)=.44, N=331

0

5

10

15

20

25

30

35

40

45

50

0 6 12 18 24 30 36 42 48 54 60 66 72 78 84 90 96

Score (%) (CU scoring)

# of students

Pre (ave=26)

Post (ave=59)

LA pre

LA post

TA post

TA pre

CU upp div post

1120 post

NCSU tradpost

NCSU honor& CMU tradpost

CMU M&Ipost

CMU uppdiv post

Favorably compares to primary implementation of Tutorials

* U. Wash. Tutorials on Circuits:9 (bulb brightness, current model) => 50% post instruction without Tutorials, 75% with Tutorials.

ROLE OF TUTORIALS: Relation to Course Goals

A significant fraction of exams cover material beyond Tutorial coverage.

High correlation of grade with Tutorial hw (graded by TA's) => tight coupling with other hw/exam questions/course goals

1120 SP04: Course grade (excluding Tut HW) vs Tut HW (N=457, r=.76)

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0 10 20 30 40 50 60 70 80 90 100

Tut HW (%)

Course score (w/o tut hw)

IMPACT ON LEARNING ASSISTANTS.

23.4 26.4

55.560.3

77.3 76.9

0102030405060708090

100

Females (23%) Males (77%)

BEMA pre

BEMA post

Course Grade

Normalized BEMA learning gain for LA’s 45%. Average final score (75%) is statistically indistinguishable from Physics Grad students pre-score.

Some quotes from first time “initially tentative” Learning Assistants Fa ‘04:

LA #1: ...in the course of this semester I have advanced my conception of teaching far beyond where it was... My model of how classes should be run has also been radically changed... I hardly realized that there were so many options and methods for running a class... Now that I have a base knowledge of techniques and philosophies I believe that through self-evaluation and awareness my teaching can grow immensely.

#2: I gain more confidence and ideas in my teaching styles... The Tutorials have helped me determine how to build the correct rapport with my students... I have enjoyed the experience in physics... More importantly, it has showed me teacher preparation is the foundation for student success in learning.

#3: I was expecting to learn more about physics as an LA, but had no idea it would be so much... Personally, being an LA has been one of the most rewarding experiences of my college career.... I have found it to be more challenging and involving than I expected, which makes me like it much more... Where before I had considered teaching as a possibility for my future, I have now determined I definitely want it to be part of my life in some fashion or another

FA04: CLASS all categories

50

55

60

65

70

75

80

D/F's (N=21) C's (N=108) B's (N=125) A's (N=80)

% favorable

PRE CLASS

POST CLASS