Peer evaluation in TBL: A Peer evaluation in TBL: A student-developed student-developed approachapproach

Derek R. Lane, Ph.D.Associate DeanCollege of Communications and Information Studies H. Lester Reynolds Engineering Endowed Professor

June 1, 2007 9:45 - 10:45 Room C150

TBL Conference 2007TBL Conference 2007REFLECTIONSREFLECTIONS

People have problems working together TBL is distinctive and importantDee Fink

“Students don’t like doing peer evaluations.”Interesting examples of TBL Self/Peer Assessment

Cooperative Learning Skills Self-directed Learning Interpersonal Skills Rudolph Navari

TBL Conference 2007TBL Conference 2007REFLECTIONSREFLECTIONS

iPeer 2.0 can help facilitate TBL peer evaluationsSophie Spiridonoff / Jim Sibley (UBC)

Medical students have mixed opinions about satisfaction with peer evaluation Dean Parmlee, Dan DeStephen, Nicole Borges (Wright

State U.)

What Makes a Team Work?

“Individual commitment to a group effort: this is what makes a team work, a company work, a society work, and a civilization work.”

--Vince Lombardi,Former Green Bay Packers Coach

Driving Questions

RQ1What is the purpose of peer evaluation?

RQ2 How do we improve our methods of helping students assess professionalism?

Students support that which they help to create.

Session Overview

I. Purpose of Peer Evaluation: A TBL Retrospective

II. Characteristics of High Performing Teams

III. Importance of Helpful Feedback

IV. Strategies for Establishing Meaningful Formative (PROCESS) and Summative (OUTCOME) Assessment

I. Purpose of Peer Evaluation in TBL

A TBL Retrospective

TBL Retrospective

Michaelsen, L.K., & Black, R. H. (1994). Building Learning Teams: The Key To Harnessing the Power of Small Groups in Higher Education. Growth Partners.

Performance Evaluation

Three Essential Components of Grading System Individual Performance

Individual Accountability for Reading/Homework

Assess Higher-Level Thinking Skills Group Performance

Provides Incentives to Support Cohesiveness

Justification for Putting Effort into Group Work

Peer Evaluation

Peer Evaluation

Solves Important Motivational Problems

Provides Incentives for Equitable Group Participation

Removes students’ fear that they will have to choose between receiving a low grade on the group assignments and having to “carry” group work (when other group members fail to do their fair share)

Performance Evaluation

Three Essential Components of Grading System Individual Performance Group Performance Peer Evaluation

Give sufficient weight to each of the components

Must be clear to students that the instructor thinks each is important (minimum 10%)

Must be responsive to student concerns for fairness

and equity (reduce social loafing)

Necessary Components of Necessary Components of TBL TasksTBL Tasks

Heterogeneous composition of diverse interdependent work teamsClear, specific, and widely shared team goals which encourage group cohesionSufficiently difficult and meaningful team activities that do not allow one person to accomplish the task alone.Regular, descriptive, specific, relevant, timely and usable internal peer feedbackExternal comparisons which are emphasized through immediate and ongoing feedback about organizational performances relative to other teams.

II. Characteristics of High Performing Teams

Team Performance and Effectiveness

In a well-run team, the overall performance is superior to that of individual efforts “The whole is greater than a sum of its

parts”

In a poorly run team, the overall performance is worse than what would be obtained by linearly combining the contributions of many individuals

The Wisdom of Teams: Creating the High Performance Organization Katzenbach & Smith ©1999

Six team basics define the discipline required for team performance:

1. Small Number (5-7; generally less than 12) 2. Complementary Skills3. Common Purpose4. Common Set of Specific Performance Goals5. Commonly Agreed Upon Working Approach

(structure, norms)6. Mutual Accountability

Team Effectiveness

PerformanceImpact

WorkingGroup

RealTeam

High-Performance Team

Pseudo-Team

PotentialTeam

The Team Performance Curve

From: Katzenbach & Smith, The Wisdom of Teams:Creating the High-Performance Organization.Used by permission.

Building Team Performance

Establish urgency, demand performance standards and directionMembers should be selected for skill and skill potential--not personalityPay particular attention to first meetings and actionsSet clear rules of behaviorSet immediate performance-oriented tasks and goalsChallenge the group with fresh facts and informationSpend lots of time togetherExploit the power of positive feedback, recognition, and reward

Katzenbach & Smith, 1993

Eight Characteristics of High Performance Teams

A Clear, Challenging Goal A Results-Driven Structure (clear roles,

effective communication) Competent Team Members Unified Commitment Collaborative Climate Standards of Excellence External Support and Recognition Effective Principled Leadership

LaFasto, F., & Larson, C. (2001).  When teams work best: 6,000 team members and leaders tell what it takes to succeed.  Thousand Oaks, CA:  Sage.

III. Importance of Helpful Feedback

Formative vs. Summative Peer Evaluation Formative (process) peer evaluation

procedures can potentially provide a results-driven structure and establish the criteria or standards of excellence to ensure mutual accountability.

Summative (outcome) peer evaluations provide an overall standardized assessment of the extent to which individuals contributed to team success. (Helping Behavior)

IV. Strategies for Meaningful Assessment

TEAM FORMATIO

NDear God, please don’t make me work with THEM!

Assign a number from 1 - 10

Preliminary Individual Peer Evaluation Criteria

Reflect on past group experiencesConsider + - experiences What are the four or five most important issues that contributed to the group success or failure?Develop a list of preliminary criteria you believe are important for group success / peer evaluation

Preliminary List of Evaluative Criteria

Instructions: Develop a list of evaluative criteria that you believe is important for students to use when working in TBL instructional teams in order to be successful.

Final TBL Evaluative Criteria

Instructions: 1) Make introductions and share

contact information. 2) After introductions have been made

and contact information shared, decide on a name for your team. The final decision must be unanimous and the name should reflect positively on the qualifications of the individuals in your group. Provide the decision on the line below.

Group # ___ Team Name: __________________________________

Final TBL Evaluative Criteria

Share individual preliminary criteria with all group members. Consolidate the preliminary list of criteria and develop a final group list of the criteria that the group is comfortable using for peer evaluations.

Use the directions on the next page to develop procedures that incorporate the group criteria that will allow efficient feedback.

Submit the final peer evaluation procedures and criteria as well as the Group # and Firm Name along with all of the individual criteria sheets at the beginning of the next class.

Team Peer Evaluation Procedures & Criteria

Instructions: Before proceeding, read the four-page article written by Michaelsen and Schultheiss’ (1988) Making Feedback Helpful.

Develop a system for providing performance feedback to the members of your team. The performance feedback system should include:

1. Provide a statement of the team goals and objectives that you intend to achieve. These goals should reflect an integration of individual team members’ goals for the course.

2. A description of how your team intends to collect the data on which the feedback will be based. Please include a copy of a specific peer grading form that clearly indicates how data will be collected.

3. A description of the feedback process you intend to use. Please specify:a. When the feedback will be given? Be specific.b. Who will give it? Be specific.

4. Assess the difficulties that you are likely to encounter in implementing your performance feedback system. How will the team overcome these difficulties?

5. A statement of how the system provides input into the summative helping behavior grade at the end of the class.

Characteristics of Helpful Feedback

descriptive, not evaluative, and is owned by the sender (not anonymous)specific, not general (focus on specific issues and behaviors)honest and sincereexpressed in terms relevant to the self-perceived needs of the receivertimely and in contextdesired by the receiverusable; concerned with behavior over which the receiver has control

Michaelsen, L.K., & Schultheiss, E. E. (1988). Making feedback helpful. Organizational Behavior Teaching Review 13(3), 109-113.

Team Peer Evaluation Procedures & CriteriaYour performance feedback system will be evaluated using the

following criteria:

1. Is the team collecting data they will need to support the achievement of course goals?

2. Will the procedures they intend to use support the achievement of their objective(s)?

3. Are the procedures they intend to use consistent with the seven characteristics of helpful feedback?

4. Are the procedures practical (i.e., can they be implemented effectively given the specific situation in which they will be used)?

5. Have they accurately anticipated the problems they are likely to encounter in implementing their performance feedback system and have they developed strategies for overcoming the difficulties?

Key Elements Critical to Cooperation

Positive Interdependence individuals succeed only when group succeeds

Individual Accountability learn together to perform better as an individual

Promotive Interaction help, assist, support, encourage, praise efforts

Social Skills (Professionalism) communication, leadership, decision-making,

trust-building, and conflict management skills

Group Processing

Code of Cooperation

EVERY member is responsible for the team's progress and success.Attend all sessions and be on time.Listen to and show respect for the contributions of other members; be an active listener.Criticize ideas, not persons.Resolve conflicts constructively.Pay attention - avoid disruptive behavior.Avoid disruptive side conversations.Only one person speaks at a time.Everyone participates -- no one dominates.Be succinct, avoid long anecdotes and examples.No rank in the room.Attend to your personal comfort needs at any time but minimize team disruption.HAVE FUN! Adapted from the Boeing

Commercial Airplane Group

IU School of Medicine - South BendTBL Self/Peer Assessment Criteria

Cooperative Learning Skills Attendance Listening Interaction / Participation Information Sharing Defer to Needs of the Group

Self-Directed Learning Preparation Depth of Knowledge Identifies Limits of Knowledge Problem Solving Confidence

Interpersonal Skills Gives instructive feedback Accepts instructive feedback Shows sensitivity and concerns for others Clarity in rationale/reasoning process

Jackson, Hamlett, & Navari

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Robert Herring, III

Winston-Salem State U.

TBL and the Business Strategy Game

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

Levine, R. E., Kelly, A., Karakoc, T., & Haidet, P. (2007). Peer evaluation in a clinical clerkship: Student attitudes, experiences, and correlations with traditional assessments. Academic Psychiatry, 31(1), 19-24.

Who seeks formative feedback?

360 Year 3 Undergraduate Medical Students at University of Aberdeen, UK

Less than half the students (46%) collected their formative feedback.

Females (p=.004) and higher achievers (p=.020) were more likely to seek formative feedback.

Sinclair & Cleland (June, 2007)Journal of Medical Education

Civil Engineering Design Team Criteria

Attendance Punctuality Attitude Respect Preparedness Completed Readings Completed Homework Ready for Meetings Preparation Goal Oriented Flexibility Quality Work Completed the First Time Accountability Trustworthy and Reliable Gives Proper Notice if Can’t Attend a Meeting Individual Initiative Team commitment Leadership Willingness to accept alternative viewpoints

Little and Cardenas (2001)

Quality of Technical Work

Ability to Communicate

Ability to Provide Leadership

Commitment to Team and Project

Demonstrated Effectiveness

Helping Behavior

Summative (outcome) peer evaluations provide an overall standardized assessment of the extent to which individuals contributed to team success. (Helping Behavior)

Standardized base 10 n-1 X 10 (6-1)*10 = 50 pts to assign Mean of group scores

Percent multiplier 100 point distribution (peer*GRAT)

Successful Teams

employ self examination and assessmentchange basic operating assumptions when necessarymonitor their own interaction patterns and progressare willing to make changesare more attentive to group processesbegin their discussion by attempting to analyze the problem before trying to search for viable solutions

Session Summary

I. Purpose of Peer Evaluation: A TBL Retrospective

II. Characteristics of High Performing Teams

III. Importance of Helpful Feedback

IV. Strategies for Establishing Meaningful Formative (PROCESS) and Summative (OUTCOME) Assessment

Student-Developed Peer Evaluation Procedures & Criteria

Feedback—both formative (process) feedback and summative (helping behavior) feedback should be direct, clear, and ensure that group members are accountable to the rest of the team. 

The value of the process feedback is that all of the members of the team have the opportunity to discuss the feedback and make performance improvements over the course of the semester.