Why Assessment Format Matters

Philip Hedges

Department of Economics & Quantitative Methods Westminster Business School

University of Westminster

UK

Aims

• To present an analysis of coursework performance and the influence of two independent variables:

1) the learner’s mode of study 2) the learner’s nationality status

• Casual empiricism suggests that part-time and home/EU students have an advantage over full-time and overseas students – based partly on this the decision was made to alter the coursework format for the 2014/15 academic year

• To quantify the influence of the student’s study mode and nationality on performance, and examine the stability of their influence when an assessment format changes

2

Format of Original Coursework Assessment

• Context: Core PG module for MA HRM students hosted by Economics dept. – original assessment pattern: 50% in-module coursework, 50% end-of-module exam

• Original coursework assessment format: 2,500 words STEEPLE analysis report of a certain market or self-selected markets for students with HR experience

• The original coursework format was designed to be student-centred, useful, engaging and differentiated according to student characteristics

• The brief was set at the beginning of the module and reports were due to be submitted in week 7 of 12 teaching weeks 3

• Original coursework format was satisfactory but a number of issues became apparent:

• a) Marking & moderation of reports on this large PG module slowed down the feedback process (3-4 weeks)

• b) Coverage of the module’s content – not all students used Economics concepts & principles in the report

• c) Differences in differentiated learners’ access to info. etc. seemed to have led to different levels of achievement between Part-Time/Home/EU and Full-time/Overseas students

4

Problems with the Original Format

Distribution of Learner Characteristics: 2010/11 - 2014/15, N = 610

5

Coursework Mean Marks (%)by Mode of Study 2010-11 to 2013-14

6

Coursework Mean Marks (%) by Nationality 2010-11 to 2013-14

7

Opportunity for Change

• The module had to be reviewed internally & externally during Spring 2014 as part of a university course revalidation & accrediting body review by the Chartered Institute of Personnel and Development

• This presented an opportunity to reflect on the assessment strategy with a focus on altering the coursework format

• The new format would hopefully reduce problems a), b) and c) associated with the original format

8

Solution: New Coursework Format

• The STEEPLE report was replaced by an in-class test based on the first 5 weeks’ Economics material

• The first iteration of the test (for 2014-15) was exclusively multiple-choice based and students were given 1 hour 15 minutes to complete 15 questions each worth between 1 and 6 marks

• Students were issued with 2 mock in-class tests & answers between weeks 1 & 5 , given a reading week to revise in week 6 and took the real test in week 7

9

Results: 2 out of 3 Problems Solved• The in-class test was double-marked and given back to

students in week 9 (2 weeks faster turnaround!)

• For the first time in the coursework for this module most students displayed a good level of familiarity with Economics principles and concepts

• Marks were higher than with the original report format with no increase or decrease in the coursework mean differences across the different characteristics

• Therefore, for problems a) and b) the change in assessment feedback was successful – the difference in means across certain characteristics may be insurmountable?

10

Coursework Means (%) by Mode of Study:

2010-11 to 2013-14 and 2014-15

11

Coursework Means (%) by Nationality: 2010-11 to 2013-14 and 2014-15

12

7 percentage points

difference

Unintended Outcomes: Student Feedback• Compared with the 2013-14 cohort the 2014-15

students facing the new assessment format gave much higher ratings on the module evaluation survey:

• The response rate to the survey increased from 29% to 56% - plus unsolicited positive remarks concerning the utility of the Economics discipline for HRM PGs

13

Area of FeedbackAverage % of 4 and 5 scores

(2013-14 average % in parentheses)

Teaching & Academic Support 92 (74)

Resources & Learning Environment 76 (75)

Assessment 92 (79)

Coursework Feedback 89 (76)

Overall 91 (70)

Quantifying the Significance and Influence of Study Mode & Nationality

• The data permits the estimation of the sign, strength and significance of the two independent characteristics on learner performance

• For each cohort coursework marks were regressed on study mode and nationality – then repeated for all cohorts using time dummies

• “Before” and “after” regressions allow us to see if study mode and/or nationality effects are independent of assessment format 14

Coursework Marks Regression Results

15

Independent Variable (1) C1 (2) C2 (3) C3 (4) C4 (5) C5 (6) C1 to C4 (7) C1 to C5

Constant 51.35*** 54.07*** 59.30*** 59.19*** 84.84*** 58.25*** 85.01***

(2.165) (1.614) (1.214) (1.229) (2.496) (-1.039) (1.181)

Study Mode (=1 if Part-Time Student) 1.06 1.62 -1.57 1.11 1.25 0.57 0.67

(3.301) (2.385) (1.781) (1.629) (3.634) (1.106) (1.090)

Nationality (=1 if Home/EU Student) 10.58*** 4.22* 9.53*** 4.85*** 6.21 6.60*** 6.55***

(3.542) (3.542) (1.749) (1.770) (3.771) (1.153) (1.135)

C1 (=1 for Cohort 1 Students) -3.87*** -30.66***

(1.481) (1.638)

C2 (=1 for Cohort 2 Students) -5.15*** -31.93***

(1.062) (1.216)

C3 (=1 for Cohort 3 Students) 1.48 -25.31***

(1.138) (1.285)

C4 (= 1 for Cohort 4 Students) -26.79***

(1.246)

Adj R-squared 0.242 0.035 0.222 0.090 0.042 0.169 0.596

n 55 167 129 146 113 497 610

Baseline for regressions (1) to (5) = Full-Time International students for each cohort from 2010-11 to 2014-15Baseline for regression (6) on 4 cohorts uses the same baseline as (1) to (5) but for 2013-14 studentsBaseline for regression (7) on 5 cohorts uses the same baseline as (1) to (5) but for 2014-15 students

Standard errors in parentheses; * p < 0.1, ** p < 0.05, *** p < 0.01

Interpretation of Regression Results• “Before” regressions (1) to (4) for each annual cohort (writing reports)

suggest that mode of study is not significant in explaining coursework marks; nationality is significant and Home/EU learners outperform their Overseas peers – according to the pooled “before” regression (6) this is worth an additional 6.6 percentage points (which rises to 7 points when study mode is omitted)

• The “after” regression (5) for the 2014-15 cohort suggests that study mode and nationality are no longer significant (though nationality is only marginally insignificant at the 10% level) – the format change has had some effect in terms of problem c)

• The pooled “before & after” regression (7) suggests that nationality is a significant variable in explaining coursework performance though this sample is limited to 19% coming from after the format change (i.e. 81% of observations are before the format change)

16

Future Directions• The first year’s experience of the new format suggests that

the statistical influence of nationality has weakened despite the persistent difference in mean mark between Home/EU and Overseas students

• Control variables for gender, age and UG degree classification may weaken the significance of nationality

• The influence of mode of study and nationality on exam performance is yet to be modelled

• The new coursework format increased total module marks so much that the module exceeded our KPI mean mark ceiling – possible change in the test to include open questions and reducing the time allowed to complete the test 17