Calibrated Peer ReviewCalibrated Peer Review (CPR) (CPR)TMTM

at Texas A&M Universityat Texas A&M University

by Dr. Wendy Keeney-Kennicuttby Dr. Wendy Keeney-Kennicutt

Associate Director, First Year Chemistry ProgramAssociate Director, First Year Chemistry Program

CPR Master Administrator, TAMUCPR Master Administrator, TAMU

k-keeney@tamu.eduk-keeney@tamu.edu

HISTORY AT TAMUHISTORY AT TAMUcpr.tamu.educpr.tamu.edu

20022002 - - CPRCPRTMTM was introduced to First Year was introduced to First Year

Chemistry in an NSF-sponsored Multi-Chemistry in an NSF-sponsored Multi- Initiative Initiative Dissemination spring workshop and a Dissemination spring workshop and a CPR CPR summer workshop. We started in the fall.summer workshop. We started in the fall.

20032003 - - To avoid FERPA* issues & because of a To avoid FERPA* issues & because of a joint joint NSF CPR grant with UCLA and our Center NSF CPR grant with UCLA and our Center for for Teaching Excellence, CPR was housed on a Teaching Excellence, CPR was housed on a secure TAMU server.secure TAMU server.

20032003 - - I volunteered to be the CPRI volunteered to be the CPRTMTM mastermaster

administrator because of experience.administrator because of experience.*Family Educational Rights and Privacy Act (1974) – protects the privacy of *Family Educational Rights and Privacy Act (1974) – protects the privacy of student education recordsstudent education records

USE OF CPRUSE OF CPRTMTM AT TAMU AT TAMU

In the last 3 years, the use of CPRIn the last 3 years, the use of CPRTMTM at at TAMU been used by approximately TAMU been used by approximately 10,00010,000 undergraduate and graduate undergraduate and graduate

students doing students doing 320320 new assignments in new assignments in 200200 courses spread over courses spread over 3030 majors in majors in 77 colleges.  colleges. 

MAJORS USING CPRMAJORS USING CPRTMTM

AccountingAccounting Ag. EconomicsAg. Economics Animal ScienceAnimal Science ArcheologyArcheology BiochemistryBiochemistry BiologyBiology BotanyBotany Bus. Admin.Bus. Admin. ChemistryChemistry EnglishEnglish

Ed. PsychologyEd. Psychology FilmFilm FrenchFrench KinesiologyKinesiology Learn. Comm.Learn. Comm. MathMath MicrobiologyMicrobiology NutritionNutrition PhysicsPhysics Poultry SciencePoultry Science

PsychologyPsychology Secondary Ed.Secondary Ed. Vet Integr. Bio. Sci.Vet Integr. Bio. Sci. Wildlife & Fish. Sci.Wildlife & Fish. Sci. ZoologyZoology

MISSING:MISSING:EngineeringEngineering

•Starting F06Starting F06

GeosciencesGeosciences

How Did CPRHow Did CPRTMTM Become Become So Popular?So Popular?

(1) Creation of “W” Courses(1) Creation of “W” Courses Our Faculty Senate decided that for graduation, there would be a Our Faculty Senate decided that for graduation, there would be a

writing-intensive (W) discipline-specific course graduation writing-intensive (W) discipline-specific course graduation requirement. requirement.

The Core Curriculum Review Committee report entitled The Core Curriculum Review Committee report entitled "Educational Leadership at the Beginning of the 21st Century""Educational Leadership at the Beginning of the 21st Century" (March 28, 2000, as amended and approved by the Faculty (March 28, 2000, as amended and approved by the Faculty Senate, May 8, 2000), established this requirement. The first Senate, May 8, 2000), established this requirement. The first writing-intensive course graduation requirement went into effect in writing-intensive course graduation requirement went into effect in the Fall 2004 catalog. The requirement for a second course is the Fall 2004 catalog. The requirement for a second course is

scheduled to be in place in the Fall 2006 catalog.scheduled to be in place in the Fall 2006 catalog.

(2) CPR was recognized as good pedagogy campus-wide(2) CPR was recognized as good pedagogy campus-wide I co-ran a CPR workshop for TAMU and local community college I co-ran a CPR workshop for TAMU and local community college

faculty, supported by TAMU College of Science mini-grant in faculty, supported by TAMU College of Science mini-grant in Summer 2003 and invited Summer 2003 and invited everyoneeveryone..

The Director of the TAMU Writing Center participated, The Director of the TAMU Writing Center participated, recognized its effective pedagogy and gave CPR her blessing for recognized its effective pedagogy and gave CPR her blessing for use in “W” courses.use in “W” courses.

By Fall 2003, we had CPRBy Fall 2003, we had CPRTMTM on our server, TAMU had quit using on our server, TAMU had quit using SSN as SIDs and we were set to go.SSN as SIDs and we were set to go.

How Did CPRHow Did CPRTMTM Become Become So Popular?So Popular?

(3) Continued Support for our TAMU users:(3) Continued Support for our TAMU users: A good website: cpr.tamu.edu with extended FAQsA good website: cpr.tamu.edu with extended FAQs Regular workshops supported by WALS*, Center for Teaching Regular workshops supported by WALS*, Center for Teaching

Excellence (CTE), Information Technology Services, The Writing Excellence (CTE), Information Technology Services, The Writing Center, and the Office of the Vice-President for Information Center, and the Office of the Vice-President for Information TechnologyTechnology

A local users group which has started to meet regularlyA local users group which has started to meet regularly 2 months summer salary for the TAMU CPR master administrator; 2 months summer salary for the TAMU CPR master administrator;

two other administrators are paid through their position at CTEtwo other administrators are paid through their position at CTE Access to the actual program to solve problemsAccess to the actual program to solve problems

*WALS = Writing for Assessment and Learning in the Natural and Mathematical Sciences (the UCLA/TAMU NSF project)

How Did CPRHow Did CPRTMTM Become Become So Popular?So Popular?

What are the issues?What are the issues?

TAMU has its own CPR webpage, so students occasionally log on to TAMU has its own CPR webpage, so students occasionally log on to the wrong one.the wrong one.

TAMU is disconnected from the UCLA server, so we don’t have TAMU is disconnected from the UCLA server, so we don’t have access to the library assignments. This will be changing under the access to the library assignments. This will be changing under the upgraded system.upgraded system.

Note: We have found that there is more student buy-in when Note: We have found that there is more student buy-in when instructors write their own assignments:instructors write their own assignments:

The assignments reflect the instructor’s wording and toneThe assignments reflect the instructor’s wording and tone CPR fits into the course flow betterCPR fits into the course flow better The instructor is much more familiar with the program after they write The instructor is much more familiar with the program after they write

assignments.assignments.

So what happened to CPR in So what happened to CPR in Chemistry?Chemistry?

In Fall 2002 and Spring 2003, we incorporated CPR in all the In Fall 2002 and Spring 2003, we incorporated CPR in all the Chemistry 101/102 classesChemistry 101/102 classes

~ 2600 students per semester~ 2600 students per semester Individual instructors (besides myself) had minimal training in CPR Individual instructors (besides myself) had minimal training in CPR

and were not involvedand were not involved Assignments were taken directly from the libraryAssignments were taken directly from the library

4 assignments, dropping 1 to allow for absences4 assignments, dropping 1 to allow for absences I and a staff member managed the problems list, all the grades, I and a staff member managed the problems list, all the grades,

and student questionsand student questions So what do you think was the outcome? I kept track using SALG.So what do you think was the outcome? I kept track using SALG.

Student Assessment of Learning GainsStudent Assessment of Learning Gainswww.wcer.wisc.edu/salgains/instructor/www.wcer.wisc.edu/salgains/instructor/

First Year’s Results First Year’s Results from My Classfrom My Class

SALG* Results on How Students View CPR (*Student Assessment of Learning Gains)

0

10

20

30

40

50

CPR isEnjoyable

Helps usLearn

Chemistry

HelpsImproveWriting

HelpsImprove

Critiquing

OthersShould Do

CPR

Term - Chem Class

Perc

en

t o

f S

tud

en

ts W

ho

A

gre

e

F'02-102

S'03-101

This was This was notnot good…… Everyone but me stopped doing CPR. good…… Everyone but me stopped doing CPR.I knew that scientific writing/critiquing were invaluable skills, so I I knew that scientific writing/critiquing were invaluable skills, so I kept trying new approaches to improve student attitude.kept trying new approaches to improve student attitude.

Overview of Study Overview of Study (manuscript in preparation)(manuscript in preparation)

I collected SALG data for 7 semesters from my students (1515 total; 94-98% I collected SALG data for 7 semesters from my students (1515 total; 94-98% of classes participated), as I continued to make CPR more palatableof classes participated), as I continued to make CPR more palatable Likert scale (1=strongly disagree…….5=strongly agree)Likert scale (1=strongly disagree…….5=strongly agree)

• I enjoyed doing the CPR assignmentsI enjoyed doing the CPR assignments• The CPR assignments helped me learn some chemistryThe CPR assignments helped me learn some chemistry• The CPR assignments helped me improve my writing skillsThe CPR assignments helped me improve my writing skills• The CPR assignments helped me learn to critique my own writing and that of The CPR assignments helped me learn to critique my own writing and that of

othersothers Yes/No + explanation Yes/No + explanation

• Do you think that future classes should do CPR?Do you think that future classes should do CPR? I had both quantitative and qualitative data for analysisI had both quantitative and qualitative data for analysis

Improvements Made Over TimeImprovements Made Over Time Prepared more thorough instructions and a supportive websitePrepared more thorough instructions and a supportive website Wrote most of my own assignmentsWrote most of my own assignments Became more proactive at listening to students & adjusting grades when appropriateBecame more proactive at listening to students & adjusting grades when appropriate Told my students upfront in the syllabus that the class was a writing-intensive class Told my students upfront in the syllabus that the class was a writing-intensive class

and included my teaching philosophyand included my teaching philosophy Gave a “CPR lab holiday” – since I did 7 labs rather than 10Gave a “CPR lab holiday” – since I did 7 labs rather than 10 Invited students to let me review their essays before submissionInvited students to let me review their essays before submission Increased importance of the text entry from 20% to 30%Increased importance of the text entry from 20% to 30% Increased CPR’s worth from 3-5% to 12% of class gradeIncreased CPR’s worth from 3-5% to 12% of class grade Took classroom time to demonstrate CPRTook classroom time to demonstrate CPR Used Bloom’s Taxonomy to show importance of critiquingUsed Bloom’s Taxonomy to show importance of critiquing

Why is writing Why is writing important?important?

Synthesis

Analysis

Application

Use of information Use of information to solve problems; to solve problems; transfer of abstract transfer of abstract or theoretical ideas or theoretical ideas

to practical to practical situations.situations.

Interpretation Identification of Identification of connections and connections and

relationshipsrelationshipsTranslation

Restatement in your Restatement in your own words; own words;

paraphrase; summaryparaphrase; summary

RecallVerbatim information; Verbatim information; memorization with no memorization with no

evidence of evidence of understandingunderstanding

EvaluationCombination of Combination of

information to form a information to form a uniqueunique

product; requires product; requires creativity and originalitycreativity and originality

Judgment: the ability toJudgment: the ability tomake decisions and make decisions and supportsupportviews; requires views; requires understanding of valuesunderstanding of values

Identification of Identification of component parts; component parts; determination ofdetermination ofarrangement, arrangement, logic, semanticslogic, semantics

Bloom’s Taxonomy – categorizingBloom’s Taxonomy – categorizing level of abstraction of questionslevel of abstraction of questions

Improvements Made Over TimeImprovements Made Over Time Prepared more thorough instructions and a supportive websitePrepared more thorough instructions and a supportive website Wrote most of my own assignmentsWrote most of my own assignments Became more proactive & adjusted grades when appropriateBecame more proactive & adjusted grades when appropriate Told my students upfront in the syllabus that the class was a writing-intensive class and included my Told my students upfront in the syllabus that the class was a writing-intensive class and included my

teaching philosophyteaching philosophy Gave a “CPR lab holiday” – since my class did 7 labs, not 10Gave a “CPR lab holiday” – since my class did 7 labs, not 10 Invited students to let me review their essays before submissionInvited students to let me review their essays before submission Increased importance of the text entry from 20% to 30%Increased importance of the text entry from 20% to 30% Increased CPR’s worth from 5% to 12% of class gradeIncreased CPR’s worth from 5% to 12% of class grade Related that CPR is a grade in its own right & helps poor test-takersRelated that CPR is a grade in its own right & helps poor test-takers Took classroom time to demonstrate CPRTook classroom time to demonstrate CPR Used Bloom’s Taxonomy to show importance of critiquingUsed Bloom’s Taxonomy to show importance of critiquing Emphasized that most students are novice reviewers and that I would gladly look at anyone’s gradeEmphasized that most students are novice reviewers and that I would gladly look at anyone’s grade

SALG* Results on How Students View CPR (*Student Assessment of Learning Gains)

0

10

20

30

40

50

60

70

80

CPR is Enjoyable Helps us LearnChemistry

Helps Improve Writing Helps Improve Critiquing Others Should Do CPR

Survey Questions

Per

cent

of S

tude

nts

Who

A

gree

F'02-102

S'03-101

S'04-101

F'04-102

S'05-101

F'05-102

S'06-101

What I saw was a significant increase in student acceptanceWhat I saw was a significant increase in student acceptance and understanding of CPR over time.and understanding of CPR over time.

QuantitativeQuantitative ResultsResults

Group 1: Students with negative CPR experience, wanted future classes to do CPRGroup 1: Students with negative CPR experience, wanted future classes to do CPRGroup 2: Students with negative CPR experience, did not want others to do CPRGroup 2: Students with negative CPR experience, did not want others to do CPRGroup 3: Students with positive CPR experience, wanted others to do CPRGroup 3: Students with positive CPR experience, wanted others to do CPRGroup 4: Students with positive CPR experience, did not want others to do CPR Group 4: Students with positive CPR experience, did not want others to do CPR

Relationship Over Time between Student Experience with CPR and Their Promotion of CPR

for Future Classes

0.010.020.030.040.050.060.0

Fall2002

Spring2003

Spring2004

Fall2004

Spring2005

Fall2005

Spring2006

Semester

Perc

en

t o

f S

tud

en

ts Group 1 (%)

Group 2 (%)

Group 3 (%)

Group 4 (%)

QualitativeQualitative ResultsResults““Do you think that future classes should do CPR? Explain.”Do you think that future classes should do CPR? Explain.”

Over 7 semesters, there wereOver 7 semesters, there were 550 totally positive responses550 totally positive responses 515 totally negative responses515 totally negative responses 174 mixed responses174 mixed responses 25 neutral responses25 neutral responses Total: 1264 responsesTotal: 1264 responses

The qualitative part of this study gave invaluable insight into The qualitative part of this study gave invaluable insight into student attitude about CPR and how it changed as I made student attitude about CPR and how it changed as I made changes in presentation, student support and grade changes in presentation, student support and grade intervention. intervention.

QualitativeQualitative Results – Negative CommentsResults – Negative Comments

On writing in a chemistry class:On writing in a chemistry class: ““I have never viewed chemistry as being a subject where you write things;” I have never viewed chemistry as being a subject where you write things;”

“We could take English to learn how to write correctly;” “I didn’t understand “We could take English to learn how to write correctly;” “I didn’t understand why writing a paper and grading other students papers had anything to do why writing a paper and grading other students papers had anything to do with chemistry.”with chemistry.”

On the peer review process:On the peer review process: ““They ask you to grade the essays, but then your opinion of how that person They ask you to grade the essays, but then your opinion of how that person

did would be wrong. I just don’t see how your opinion could be wrong.”did would be wrong. I just don’t see how your opinion could be wrong.”

Other:Other: Too time consuming, waste of time, not related to the subject; it harmed their Too time consuming, waste of time, not related to the subject; it harmed their

grade; was worse than lab; their peers lacked motivation; added to stressgrade; was worse than lab; their peers lacked motivation; added to stress

QualitativeQualitative Results – Positive CommentsResults – Positive Comments

On writing in a chemistry class:On writing in a chemistry class: ““Calibrated Peer Reviews forces the student to look into the topic way more Calibrated Peer Reviews forces the student to look into the topic way more

than what he or she would do out of a textbook. I know the CPR has than what he or she would do out of a textbook. I know the CPR has tremendously helped me understand each topic better although I didn’t tremendously helped me understand each topic better although I didn’t exactly enjoy it.”exactly enjoy it.”

““The CPR really helped me understand the topics. It reinforced the material The CPR really helped me understand the topics. It reinforced the material by forcing me to teach myself and explain it to others through writing. It was by forcing me to teach myself and explain it to others through writing. It was very helpful.”very helpful.”

““I think the first one is bad because you don’t really know what you’re doing I think the first one is bad because you don’t really know what you’re doing and how to approach the whole thing, but after doing it you realize that you and how to approach the whole thing, but after doing it you realize that you are learning the subject because you had to write a paragraph on it. It was a are learning the subject because you had to write a paragraph on it. It was a big help whether people will admit it or not.”big help whether people will admit it or not.”

QualitativeQualitative Results – More Positive CommentsResults – More Positive Comments

Overall:Overall: ““Although CPR was one of my least favorite things to do in this class, I think Although CPR was one of my least favorite things to do in this class, I think

the good in it outweighs the bad. I think that especially in the science fields, the good in it outweighs the bad. I think that especially in the science fields, students don’t have to do a lot of writing and so they don’t develop students don’t have to do a lot of writing and so they don’t develop communication skills that they will need later on in life. I think communication communication skills that they will need later on in life. I think communication is very important and it is something that you just have to work on. I think is very important and it is something that you just have to work on. I think students will look back and wish they would have done more stuff like CPR.”students will look back and wish they would have done more stuff like CPR.”

““It seems like a pain at the time, but I can already see how much I learned It seems like a pain at the time, but I can already see how much I learned from it. Please continue to do it, it helps more than people realize.”from it. Please continue to do it, it helps more than people realize.”

Other positives:Other positives: Helped link chemistry to real life and their professional future, developed time Helped link chemistry to real life and their professional future, developed time

management and research skillsmanagement and research skills

QualitativeQualitative Results – OverviewResults – Overview

Percentage of Positive Comments per Semester

0.010.020.030.040.050.060.070.0

Fall 2002 Spring2003

Spring2004

Fall 2004 Spring2005

Fall 2005 Spring2006

Semester

Perc

en

tag

e (%

)

Final CommentsFinal Comments

CPR is an example of the move from teacher-centered CPR is an example of the move from teacher-centered learning to student-centered learninglearning to student-centered learning

If you add writing, peer review and technology to the mix, If you add writing, peer review and technology to the mix, should I have been surprised at the level of student angst?should I have been surprised at the level of student angst?

Students have a more positive experience when the instructor Students have a more positive experience when the instructor actively promotes & demonstrates CPR in the classroom, actively promotes & demonstrates CPR in the classroom, makes the assignments a significant part of grade,makes the assignments a significant part of grade, is involved in writing assignments,is involved in writing assignments, gives personalized supportgives personalized support regrades when warrantedregrades when warranted

Thanks to my co-authors:Thanks to my co-authors:

Dr. Nancy Simpson,Dr. Nancy Simpson,Director, Director, Center for Teaching ExcellenceCenter for Teaching Excellence

Ms. A. Baris Gunersel,Ms. A. Baris Gunersel,Ph.D. Student, Educational PsychologyPh.D. Student, Educational PsychologyGraduate Research AssistantGraduate Research AssistantCenter for Teaching ExcellenceCenter for Teaching Excellence