The impact of case studies formatively and summatively assessed on students’ examination...
-
date post
19-Dec-2015 -
Category
Documents
-
view
214 -
download
0
Transcript of The impact of case studies formatively and summatively assessed on students’ examination...
The impact of case studies formatively and summatively assessed on students’ examination performance.
Geeta Hitch (Senior Lecturer, Dept of Pharmacy)
Janet Webber (Senior Lecturer, Dept of Physiotherapy/Principal Lecturer LTI)
Background
• PMM module /2nd year MPharm degree • Previously taught as lectures only• Concern- ‘decontextualised’ as well as there
was a divide between experience of learning and that as a practitioner
Aim
• To investigate the the impact of case studies formatively and summatively assessed on students’ examination performance
Seven Principles of Good Practice in ‘Higher’ Education (Chickering & Gamson, 1987).
• Encourages contact between students and lecturers • Develops reciprocity and cooperation among students • Encourages active learning • Gives prompt feedback • Emphasises time on task • Communicates high expectations • Respects diverse talents and ways of learning
Method
• Use of ‘hybrid’ PBL in PMM module– Bridge the divide between theory and practice; real-life
situation problems. • Incorporation of constructive alignment (Biggs, 2000)
in the use of problem solving exercise– exam question format is case study style – Inclusion summative assessment of presentation &
poster– PMM module 50% exam (25% on case study style
question); 50% CW (of which 5% was allocated to this assignment)
Method: encompassed Seven Principles of Good Practice in ‘Higher’ Education (Chickering & Gamson, 1987).
• Students placed in groups of 6; communicated with peers online via Studynet discussion site
• ‘Hybrid’ PBL- case study with structured questions
• Tutor acted as facilitator
• Students given a timeline• Constant feedback was provided online• Had an oral presentation /poster (10 mins)
– Followed by Q&A– Assessed by panel of 3 lecturers– CW mark of 5% in summatively
assessed cohort
– Encourages contact between students and lecturers
– Develops reciprocity and cooperation among students
– Encourages active learning – Respects diverse talents and ways of
learning • Emphasises time on task • Gives prompt feedback
– Encourages contact between students and
lecturers – Develops reciprocity and cooperation
among students – Encourages active learning – Communicates high expectations
• Respects diverse talents and ways of learning
Exam question:25% of the marks- had a choice of doing one of the 2 case study style based questions
Method• Cohorts in which study was carried out:
Academic year Lectures only (no hybrid PBLs so used as Control Group)
Lectures plus Hybrid PBLs (formatively assessed)
Lectures plus Hybrid PBLs (summatively assessed)
2007-08 (n=95) ×
2008-09(n=123)
×
2009-10(n=132)
×
Feedback Questionnaire
× ×
Data analysis
• Overall examination performance between all 3 cohorts was compared- Statistical Package for Social Science (SPSS for MS windows version 16). – The Students t test was used to compare the probability level set
at 5% i.e. P < 0.05. If the calculated p- value was below the threshold chosen for statistical significance of 0.05, then the null hypothesis was rejected in favour of the alternative hypothesis.
– Therefore any P values less than 0.05 were regarded as significant. • Null hypothesis: assessed case studies should make NO
impact on students’ final exam performance• Alternative hypothesis: assessed case studies should make
an impact on students’ final exam performance
Data analysis
• Questionnaires– A qualitative analysis of the student feedback
questionnaire was also carried out from the academic cohorts of 2008-09 (‘formative cohort’) and 2009-10 (‘summative cohort’)
– Group dynamics and the impact of inclusion of summative and formative assessment was examined as to whether it was a driving force in participation of group work.
Results: Percentage of student cohorts in 2007-08 (Control cohort), 2008-09 (formative cohort) and 2009-10 (summative cohort) showing marks range scored (%) in the final PMM Examination question.
0 to
und
er 1
0
10 to
und
er 2
0
20 to
und
er 3
0
30 to
und
er 4
0
40to
und
er 5
0
50 to
und
er 6
0
60 to
und
er 7
0
70 to
und
er 8
0
80 to
und
er 9
0
90 to
100
0
5
10
15
20
25
30% of students 2007-08 cohort (Lectures only; No case studies)
% of students 2008 -09 cohort lec-tures and (Formative assessment of case studies)
% of students 2009-10 cohort lectures and (Summative assess-ment of case studies)
% s
tude
nts
Range of marks (%)
Mean marks:2007-08: 38.48 %2008-09: 38.41 %2009-10: 46.04 %
Results
40 to
und
er 5
0
50 to
und
er 6
0
60 to
und
er 7
0
70 to
und
er 8
0
80 to
und
er 9
0
90 to
100
tota
l % o
f stu
dent
s sc
orin
g fr
o...
0
10
20
30
40
50
60
70
80
% of students 2007-08 cohort (Lectures only; No case studies)
% of students 2008 -09 cohort lectures and (Formative assessment of case stud-ies)
% of students 2009-10 cohort lectures and (Summative assessment of case studies)
% s
tude
nts
Percentage of student cohorts in 2007-08, 2008-09 and 2009-10 scoring more than 40% in final PMM Examination question
More than 65% of students have scored greater than 40% in their final PMM exam question in 2009-10 cohort (summative vs lectures only) p≤0.0001;
Results
• No significant difference in exam performance for students when formatively assessed case studies were used (2008-09 cohort) and when lectures only were used (2007-08 cohort) to deliver the curriculum (student T test; p≤0.97).
• Significantly better overall performance in the relevant exam question when the case studies were summatively assessed in comparison to formative assessment (student T test; p≤0.0005);
2007-08 (control cohort)
0 to under
10
11 to under 20
20 to under 30
30 to under 40
40 to under 50
50 to under 60
60 to under 70
70 to under 80
80 to under 90
90 to 1000
5
10
15
20
25
30
35
40
45
2007-08 MPP3 exam
2007-08 PMM exam
Mean mark: PMM: 38.48 %MPP3: 55.38 %
% s
tude
nts
Marks range %
Comparison of final exam performance in the same cohort between 2 different pharmacy modules (PMM- pharmaceutical Microbiology & Manufacture and MPP3- Medicines & Professional Practice Level 3)
2008-09 (formative cohort)
0 to under
10
11 to under
20
20 to under
30
30 to under
40
40 to under
50
50 to under
60
60 to under
70
70 to under
80
80 to under
90
90 to 100
0
5
10
15
20
25
30
35
40 2008-09 MPP2 exam
2008-09 PMM exam
Mean mark:PMM: 38.41 %MPP2: 51.5 %
% s
tude
nts
Marks range %
Comparison of final exam performance in the same cohort between 2 different pharmacy modules (PMM- pharmaceutical Microbiology & Manufacture and MPP2- Medicines & Professional Practice Level 2)
2009-10 (summative cohort)
0 to under
10
11 to under
20
20 to under
30
30 to under
40
40 to under
50
50 to under
60
60 to under
70
70 to under
80
80 to under
90
90 to 100
0
5
10
15
20
25
30
35
40
452009-10 MPP1 exam
2009-10 PMM exam
Mean Mark:PMM: 46.05 %MPP1: 51.2 %
% s
tud
ents
Marks range %
Comparison of final exam performance in the same cohort between 2 different pharmacy modules (PMM- pharmaceutical Microbiology & Manufacture and MPP1- Medicines & Professional Practice Level 1)
Cohorts general performance in other examinations
• All 3 cohorts appear to be performing at the same level in 2 sets of exams results compared.
• This bears a significance in terms of inclusion of case studies because:– The overall performance appears to be similar in
2007-08 and 2008-09 cohorts. – The overall exam performance in case study style
based question is performed significantly better in the 2009-10 summative assessed cohort.
Questionnaires• Response rate:
– 65% from 2008-09 (n=123) (Formative cohort)– 73% from 2009-10 (n=132) (Summative cohort)
• Enjoyed taking part in group work:– 75% of formatively assessed cohort - yes– 90% of summative assessed cohort- yes• 58% of students participated in equal input into group work when case
studies were formatively assessed in comparison to 83% when summatively assessed.
• When students were asked if they participated more when summative assessment took place,
– 76% replied yes and 66% of these stated that they were driven by a genuine desire to study by participating in the case study.
– The other 34% were just performing a task as it was summative and exam marks mattered more to them.
Questionnaires
– 56% of the formatively assessed cohort felt that they learnt better in group work than working alone
– 64% of the summative assessed cohort felt that they learnt more in group work than by working alone.
Questionnaires
• Formative assessments were not viewed as important to the students in achieving their learning outcomes for a variety of reasons. In the case of this cohort (2008-09), several reasons were cited by students in the feedback questionnaire:– “Too many summative assignments to hand in and therefore
formative assignments are my last priority”– “Many students tend to rely on others to do the group work and
then when they are told to get on with it, the work they hand in is not reliable”
– “Some students were clearly not willing to prepare or participate in the presentations as there was no marks awarded for this work and so the rest of the group ended up doing their part of the work”
Discussion
• Gibbs, (1999) - “Assessment is perceived by students as the curriculum and as such the power of assessment needs to be used strategically to help students learn”
• Biggs (2002) identifies this- states that “students will only learn what they think will be assessed on as opposed to what is required of them to learn in the curriculum”.
Discussion
• General observations: inclusion of case studies does improve overall performance- students engage more effectively when learning is contextualised and is of relevance to practice
• Students also perform better when marks are allocated to assignments adding to overall module mark
• If marks are not allocated towards assignments and towards module marks in general, students can be reluctant to participate in CW elements of the curriculum.
Finally we conclude:
“That by inclusion of the 7 principles of good practice in undergraduate education, we have shown in this small study that how we practice the teaching is as much as important as to what the students learn.”
Seven Principles of Good Practice in ‘Higher’ Education (Chickering & Gamson, 1987).
• Encourages contact between students and lecturers • Develops reciprocity and cooperation among students • Encourages active learning • Gives prompt feedback • Emphasises time on task • Communicates high expectations • Respects diverse talents and ways of learning
References
1. Chickering, A. W. & Gamson, Z. F. (1987). ‘Seven principles for good practice in undergraduate education.’ American Association for Higher Education Bulletin, 39(7), 3–7.
2. Gibbs, G (1999) Using assessment strategically to change the way students learn, in: S. Brown & A. Glasner (Eds) Assessment Matters in Higher Education (Buckingham, SRHE & Open University Press).
3. Biggs, J. (2002) Aligning the curriculum to promote good learning, paper presented at the Constructive Alignment in Action: Imaginative Curriculum Symposium, LTSN Generic Centre, November 2002. Available online at: www.ltsn.ac.uk
Thank you
Any questions?