The use of Web-PA as a formative peer assessment instrument for team working

Tom Joyce, Nuala Davis* and Clare Hopkins

School of Mechanical & Systems Engineering

*Information Systems and Services

Newcastle University

24th June 2013

Outline

Context for the use of Web-PA

Use of online peer assessment tool (Web-PA) as a means of identifying and addressing team difficulties

Technical outline of the use of Web-PA

School of Mechanical and Systems Engineering, Newcastle University

Offer BEng and suite of MEng degrees

‘Typical’ engineering student – male, post ‘A’ level, UK national

Increasing number of international students (24% of Stage 1 2012/13)

Like many engineering Schools, progression is an issue

School of Mechanical and Systems Engineering, Newcastle University

Year of student

registration

Number registered

Year 1

Number progressing

to Year 2

Percentage progressing

to Year 2

2006/7 75

2007/8 90

2008/9 114

School of Mechanical and Systems Engineering, Newcastle University

Year of Year of student student

registrationregistration

Number Number registered registered

Year 1Year 1

Number Number progressing progressing

to Year 2to Year 2

Percentage Percentage progressing progressing

to Year 2to Year 2

2006/72006/7 7575 6262 83%83%

2007/82007/8 9090 7272 80%80%

2008/92008/9 114114 9595 83%83%

A proposed solution – “Engineering Teams”

Introduced in 2009/10 academic year Pre-selected group of 5 first-year students ‘Balanced’ in terms of academic ability Overlap with tutorial system Work together on Team projects in two

modules (“Design & Manufacturing” and “Professional Skills” – total 45 credits) during first year

Why Engineering Teams?

Increase in student integration – students get to know other students more quickly

Informal peer instruction and learning, sharing skills and knowledge

Team-working on projects gives shared goals

School of Mechanical and Systems Engineering, Newcastle University

Year of student

registration

Number registered

Year 1

Number progressing

to Year 2

Percentage progressing

to Year 2

2006/7 75 62 83%

2007/8 90 72 80%

2008/9 114 95 83%

School of Mechanical and Systems Engineering, Newcastle University

Year of student

registration

Number registered

Year 1

Number progressing

to Year 2

Percentage progressing

to Year 2

2006/7 75 62 83%

2007/8 90 72 80%

2008/9 114 95 83%

2009/10* 107

2010/11* 127

2011/12* 126

* intervention year* intervention year

School of Mechanical and Systems Engineering, Newcastle University

Year of student Year of student

registrationregistration

Number Number registered registered

Year 1Year 1

Number Number progressing progressing

to Year 2to Year 2

Percentage Percentage progressing progressing

to Year 2to Year 2

2006/72006/7 7575 6262 83%83%

2007/82007/8 9090 7272 80%80%

2008/92008/9 114114 9595 83%83%

2009/10*2009/10* 107107 100100 93%93%

2010/11*2010/11* 127127 114114 90%90%

2011/12*2011/12* 126126 114114 91%91%

* intervention year* intervention year

82%

91%

Students’ views on team working

In 2012/13 academic year 91% of students said that they enjoyed/enjoyed very much working in an Engineering Team

86% said that their Engineering Team had formed a good working relationship

71% said that working in an Engineering Team had helped them to feel part of the School

But when there are problems

Problems occur for a small number of Engineering Teams when members fail to contribute in a variety of ways

“Some people in the team don't do anything at all. They don’t turn up to any meetings or labs and I’m surprised they even know we're making a wind turbine. They should get no marks at all if they haven’t contributed to the team.”

Peer–moderated marking

Used in Design and Manufacturing module Teams decide division of marks at end of 1st

and 2nd terms Anecdotally, we hear of some heated

discussions Some groups want to avoid face-to-face

conflict

Measures taken to further address team-working (2012/13)

Enhanced team building exercises on second day of induction period

An Engineering Team handbook with information and advice about team working, including ways of addressing non-contribution

Use of Web-PA peer assessment to facilitate feedback between team members

WebPA

What contribution did each member make to the group effort?

Average = Teacher awarded mark

Tutor I

Tutor creates a form with assessment criteria

Defines the start and end times for the assessment

Reminds students to fill it out

Students

Students fill the form out online Confidential

Tutor II

Enters group scores in WebPA

Generates a “Mark Sheet” based on their chosen weighting for peer marked element

Examines and moderates results

Using Web-PA as formative peer assessment

Took place eight weeks into first term 111 students took part 15 of whom were

women and 22 international students self and peer assessment using freetext Conducted in computer lab where teams

could sit separately from one another Feedback given to teams in anonymised

form under the guidance of tutors

Key learning points from the use of Web-PA

Of the 537 pieces of feedback given, the majority (92%) were constructively written

Large degree of agreement between students’ self assessment and the feedback provided by their peers

Only 43 comments (relating to 20 students) 8% of the responses were negative

Common reservations about using peer assessment

Students may be unprofessional and give highly critical feedback, leading to acrimony and breakdown within teams

Students may assess themselves more positively than they assess their peers as a way of gaining advantage

Students may use anonymous feedback as a way of expressing personal resentments

Reservation 1 – students may give unprofessional/critical feedback

What we found: Of the 537 pieces of feedback 92% was

balanced and constructive Four students who had contributed to a

lesser extent for a reason that their peers believed to be valid received supportive, uncritical feedback

Reservation 2 – students may be over-positive about themselves

What we found: A very high level of agreement between

team members self assessment and how they were rated by their peers (only 2 cases where this was not the case)

In 15 cases self assessment was deprecatory. In only 7 of these was this negative assessment confirmed by peers

Reservation 3 – criticism on the basis of personal feelings

What we found: Only 43 comments were negative (related to

20 people) – 8% of responses Criticisms were related to team performance

– absence; personal disorganisation; failure to contribute to the team; poor quality of work.

Fear 3 – criticism on the basis of personal feelings (cont)

Even when the majority of comments were negative, in more than half there was also an acknowledgement of some positive aspect “attended all meetings” “however he didn’t lack effort” “but he does try”

In a very small percentage (5%) there was a disparity between single negative comments and other feedback

Issues to be addressed before next use of Web-PA formative assessment

The reluctance/avoidance of a small number of students

Addressed by – provision of information about use of Web-PA during induction period/EngineeringTeam handbook

The tendency of some international students to be tentative/self-deprecatory

Addressed by – targeted measures to engage them in the process/help them overcome reluctance

Students’ perceptions of Web-PA

91% of those who completed it rated the system as technically easy to use

49% said that they were able to be more honest in their ratings because they did not have to share them directly with their team mates

“It is good to be able to rate your team mate, and share your opinion of them without having the embarrassment of a face to face confrontation”

Students’ perceptions of Web-PA

It was seen as a good way of addressing problems within a team

“It allowed me to focus on the aspects of the team that were good but more importantly, it helped to highlight the shortcomings of the team so that they could be addressed and remedied”.

Students’ perceptions of Web-PA

As a useful way of bring up issues without embarrassment

“Allows people to bring up any issues anonymously. Without worry.”

Thank you

Any questions?