TRANSFORMING LIVES

INSPIRING CHANGE

ACC3015 Accounting & Finance for Managers

Gail Capper | Pearson

KEY

FINDINGS

Engagement, motivation and activity patterns are generally seen to be linked to the mode of course delivery. Campus and Distance Learners use MyAccountingLab (MAL) with differing levels of participation and achievement.

The findings in this report relate specifically to the campus and distance learning cohorts of Northampton Business School (NBS) ACC3015. This is an exploratory study within which cohorts vary from 8 to 144 students and whose findings assist us in understanding how MAL helps in supporting learners at NBS

Course: Accounting & Finance - 7 different cohorts both on campus and distance learners (see below for detail)

Resource: MyAccountingLab for Atrill/McLaney Accounting & Finance for Non-Specialists 8/e

Data Collection: MAL Gradebook on NILE (Northampton VLE)

Data Type & Period: Quantitative, 2013-2014 Summary of Results

Students engage differently across courses. Distance Learning and Campus learning seems to be intrinsically

different.

The exception to this was the general DL course which seemed to perform differently to all the other distance

learning courses. Motivation, activity and achievement all differed for this group. Their achievement was far

more diverse, and their engagement in the resource far higher on self-study than any other Distance Learning

cohort and their dropout rate was the highest at 18%.

Participation in the homework and homework scores are higher on campus than in any distance learning

course. Motivation appears to drop after the first two homework assignments have been completed for

distance learners.

More specific findings include:

Distance & Campus students show clear differences in the level and continuity of engagement with MyAccountingLab (MAL)

Deadlines and credit- bearing assessment influence activity online

Cohorts who completed more homework, gained higher average scores in their formative homework assignments

Cohorts showing most activity on the resource, were not those who achieved the highest grades in summative assessment

Cohorts who were actively engaged with the resource show sustained motivation to access the resource throughout the course

Pag

e2

Summary & Study Purpose

This study was completed in order to better understand how campus based and distance learners using

MyAccountingLab (MAL) were interacting with it. The University wanted to understand more about how

students were making use of the resource and whether it was effective in helping to engage them.

Northampton Business School (NBS) will move their campus location and begin a flipped classroom delivery

from September 2015 for their Year 1 intake. Understanding how students are using and engaging with

resources will help to inform the way in which online resources can be delivered most effectively.

Of particular interest to NBS was a report covering retention, achievement, activity patterns or peaks

(especially around deadlines) and overall student engagement on different courses. There was also a question

around how well the resource engaged campus versus distance learners.

NBS has invested significant amounts of time and money in MAL and so this study should inform a business

case considering whether the investment is well spent or how to improve the return on that investment by

improving the effectiveness of MAL in NBS.

General study question: ‘How is the resource helping and is the resource effective?’

Especially:

‘Are students engaging with MAL across all courses?

‘What differences are there in usage of and achievement in MAL between the courses? ‘

‘How can these findings inform NBS plans to deliver courses differently in e.g. flipped classrooms?’

NBS are unable to share final course marks for the purpose of this study and so there is no cross reference

between achievement on MAL tests and grades at the end of the course available.

MyAccountingLab – Overview of Resource

MyAccountingLab (MAL) is an online learning resource designed to work alongside specific textbooks as an

online homework, tutorial and assessment program. It provides a variety of different question types which can

be studied independently or assigned by the lecturer. Assigned work is created and set with a deadline either

as homework (for practice) or a test (for assessment). Settings can be adjusted to allow for single or multiple

attempts. In anything other than a test, students will have access to a range of features to provide support and

feedback such as ‘Help Me Solve This’ which offers a similar exercise, broken down into steps.

Independent study is completed in a study plan which contains practice and test questions. These tests are

called sample tests.

Achievement on the course can be measured in a number of ways and mastery is one way. Students gain

mastery points as they complete the study plan, usually to a maximum of 54 points. They must score 100% on

the relevant questions to gain 1 mastery point (the % required for a point can be adjusted manually to any

score the lecturer thinks appropriate). Mastery is represented as both points and translated as a percentage

for the overall study plan score. This allows us to compare it to assessed scores and homework scores.

Pag

e3

Integration of MAL in Northampton Business School

Overview:

NBS first started using MyLabs in 2010 and Pearson labs are integrated into the Northampton Integrated

Learning Environment (NILE – Blackboard Learn) for 6 disciplines in the school, one of which is Accounting. The

university integrated MyLabs into NILE following a couple of years of difficulty in self-registration using access

codes. Self-registration was time-consuming and ultimately unmanageable for the staff. VLE integration works

well, is cost efficient, gives reliable access for students and is time-saving for staff who are generally happy with

the setup.

The eLearning Development & IT Support Manager in NBS said that Northampton are committed to providing

students with a u-Learning (ubiquitous learning) experience in a timely, cost effective and cost transparent

way… a compelling eLearning experience.

On ACC3015 specifically:

This study reports on 7 courses run by NBS. They are all the same course delivered to different cohorts in

different locations: ACC3015 Accounting & Finance. All these courses used Atrill & McLaney’s 8/e textbook and

the accompanying MAL. Two of the courses are campus based, face-to-face courses while five courses are

Distance Learning (DL) courses delivered in Asia. The distance courses are classed as ‘supported distance

learning’ meaning they meet the lecturer and attend locally run workshops for the course with the exception of

the DL course who have no face-to-face contact with teaching staff. That said, students could be located in any

geography to attend the distance courses.

This VLE introduction to MAL on the Northampton VLE (NILE) sets out the intended use for all students.

Course & Assessment Structure

All 7 courses use MAL to set assessed tests for each of the cohorts. Test 1 represents 20% of the final mark to

assess ‘Financial Accounting’, Test 2 represents 20% of the final mark assessing ‘Management Accounting’ and

the last part of the course is assessed by an essay worth 60% which covers ‘Financial Management’.

In addition, students were assigned a mixture of practice tests, homework and given open access to additional

self-study contained in the study plan section of MAL. The study plan is open to all students and contains

exercises which are not used in homework or assessed tests.

One difference in implementation for DL and campus students is that DL students were set 9 homeworks for

the duration of their courses, October campus students were set 2 homeworks and January campus students

were set 2 homeworks with 4 additional homework assessments. None were credit bearing.

Campus courses: (short reference used throughout the study is in bold) • STD Oct - 144 students. Accounting & Finance for Managers • STD Jan - 24 students. Accounting & Finance for Managers

Pag

e4

Distance learning courses: • DL (General) - 40 students, Accounting & Finance for Managers

• DL 1 (H) - 35 students, Accounting & Finance for Managers

• DL 2 (S) – 16 students, Accounting & Finance for Managers

• DL 3 (XHC) – 8 students, Accounting & Finance for Managers

• DL 4 (HCX) – 10 students, Accounting & Finance for Managers

Methodology & Metrics

The metrics used in this report are drawn from each course Gradebook in MAL. As these cohorts have moved on there is no interview or survey data, but the students do complete Module Evaluation forms. Over time, the study can be added to as further years complete the course to allow for a more longitudinal study to be completed. All data provided is anonymised as stipulated by the university. The list below explains which data which has been accessed from the MAL Gradebook for each course.

• Mastery points: These points represent student achievement and ‘mastery’ of the content in MMLG. Mastery points are earned by answering questions correctly (100% correct) in the study plan (self-study). Assigned homework, tests and quizzes selected from chapter material may also contribute to mastery points.

• Average grades: available from sample tests (study plan tests), study plan, homework, tests and quizzes.

• Sample tests are used as a measure of self-study or independent study. They are accessed by individuals at any time in the course but are not assigned.

• Time spent on task: available in the study plan and assigned homework or tests. Total time represents logged in time to any part of the resource. It may not always indicate learner activity.

Pag

e5

Results

Student Access & Activity

Charts 1.1 and 1.2 below show that there is more activity on MAL when credit bearing assessments are due,

especially for distance learners.

Four out of the seven courses have been used as example activity patterns, the results show:

Distance Learners access materials in a need-to-do manner. There are peaks of activity around credit-

bearing assessment deadlines.

Campus-based learners also demonstrate peaks of activity around credit-bearing assessment deadlines.

However, there are more peaks and more sustained activity for campus based learners overall. Peaks

may be linked to lecture dates and/or being campus-based with more access to a learning

environment.

Campus-based learners have more peaks of activity at times when assessments are not due than

distance learners (over the duration of the course)

Announcements on the university VLE (NILE) show no impact on activity patterns on any courses.

How can we explain the other peaks of activity? For example, on the STD Oct course student activity shows peaks on the 3rd November when 23 students accessed the homeworks and on 24th November when 37 students accessed homework. Did the cohort attend lectures on that day, or the previous day where a lecturer reminded them to access the resource? Students were grouped in order to lecture on the course and had different days for their lectures. However, they appeared to access their work most often on a Sunday or Monday in preparation for the week – whatever day the lectures were. Did the availability of assigned homework affect activity? Homework can be completed at any time, and varies

by course. For example, STD Jan had to complete their Homework 1 & 2 by 7th March 2014. STD Sep had to

complete theirs by 15th November 2013. These dates did not seem to influence access to the homework.

The STD January cohort were given additional Homework Assessment 1-4 compared to STD September.

According to Chart 1.1, the STD January cohort had the highest percentage of students opening their

homework. The two campus cohorts also scored the highest average homework scores (see chart 1.4)

Chart 1.1 below tracks when homework was accessed, by date, for campus and distance students.

Pag

e6

Chart 1.1

Pag

e7

DL 2 (S) in particular has low levels of activity outside the deadlines for assessment. In addition, DL 2(S) also had

the lowest levels of access to testing within the study plan as shown in Chart 1.7. Combining this information

helps to draw a clearer picture of student engagement with MAL

Engagement with Homework & Assessed Tests (40% credit)

Accessing the assigned homework can be used as a measure of how engaged or motivated students are as it is

not credit-bearing, but is recommended for practice. The number of students failing to complete Test 1 AND

Test 2 shows a failure to retain the students on the course.

If we compare the average number of students who logged in and accessed assigned homework and also those

who failed to complete credit-bearing tests in ACC3015 it shows that for distance learning DL 1 (H) and DL

(General) have the highest non-completion rate and lowest number of homework assignments accessed.

STD Oct has comparatively high homework completion but is the highest non-completion number for the

campus courses.

Overall, retention is good on all courses except on the general DL (General) course.

Chart 1.2

N.B. Distance Learning courses, the students had more homework to complete.

What impact did this have on their sustained motivation to complete the homework and on what

scores they achieved?

Pag

e8

Table 1.1 shows the percentage of students who completed all homework set for their course against the total

number of students on the course. The yellow highlights show comparatively high or low numbers:

Course % Completing all

HW % Not Completing

Course

STD Oct 81% 4%

STD Jan 94% 0%

DL (General) 48% 18%

DL 1 (H) 40% 2%

DL 2 (S) 75% 0%

DL 3 (XHC) 75% 0%

DL 4 (HCX) 70% 0% Table 1.1

Table 1.2 shows the average number of students completing homework in more detail, showing how many

students completed each homework and giving an overall average for homework completion.

Cohort HW 1 HW 2 HW 3 HW 4 HW 5 HW 6 HW 7 HW 8 HW 9 Median Median % (rounded)

DL 1 (H) 21 14 12 11 13 10 15 15 17 14 40%

DL 2 (S) 11 11 11 15 12 11 14 14 12 12 75%

DL 3 (XHC) 8 7 6 6 6 6 8 7 5 6 75%

DL 4 (HCX) 7 7 7 6 6 5 7 7 7 7 70%

DL General 30 19 15 11 14 14 24 21 20 19 48%

HW 1 HW 2 Median Median % (rounded)

STD Oct 117 115 116 81%

STD Jan 23 22 22.5 94%

Table 1.2

Chart 1.3 shows that more students completed the first two homework assignments than the later homework

assignments. DL 2 (S) is the small, but notable exception. Is this linked to a decrease in motivation and

engagement in distance learning resources over the duration of a course?

Chart 1.3

Pag

e9

If we check the scores for those first two homework assignments against the scores for all homework on chart

1.4, it shows that students gain higher marks on their first homework assignments.

Chart 1.4

Engagement with the Study Plan.

Motivation to study without direction is a useful measure of how engaged students might be, in addition to

activity pattern for homework access which is completed at any time. It requires interpretation and those

students who choose to self—study might be struggling, but motivated or they may be highly motivated.

The number of sample tests opened in the study plan (which are not assigned by lecturers) can be used to

measure engagement and participation in the course over time. The charts below show how many sample

tests students from each course accessed.

Legend:

Pag

e10

Pag

e11

The DL (General) and STD Oct cohorts score well in self-study access as a positive sign of motivation:

DL (General) course cohort has the fewest number of students accessing zero sample tests.

DL (General) and STD Oct have the highest number of students opening more than 26 tests

On ALL other distance learning courses the majority of students do not open any sample tests for self-directed study.

Time Spent

As the study plan represents self-motivated, self-directed work it is interesting to look below the surface for

further trends in time spent and in how much of the resource was used.

Course Average Total Time on MAL Average Number of Questions (zero score students removed) Worked in Study Plan (rounded) STD Oct 779 minutes 61 STD Jan 1748 minutes 120 DL 1 (H) 818 minutes 1 DL 2 (S) 1761 minutes 6 DL 3 (XHC) 1712 minutes 5 DL 4 (HCX) 2244 minutes 2 DL (General) 1877 minutes 70 Table 1.3

The average time remains high for all courses. However, the number of questions students worked in their study plan is dramatically different for campus and DL courses.

DL 2 (S) is the least active / engaged and scored among the lowest in chart 1.4 for homework scores. However, in the achievement section below we can see that they performed well in their assessed work.

This raises the question ‘where is the time being spent and is it active engagement? ‘

In DL 2 (S) to continue the example, the time spent online is disproportionately affected by one or two students spending a large amount of time with their homework ‘open’ but not necessarily completing it - from 20 minutes per homework to 24 hours per homework Student A spent 86.5 hours in total time Student B spent 54 hours in total time

Chart 1.5

Pag

e12

The number of questions worked in the study plan is a better indicator of engagement or motivation than time spent on it. Chart 1.6 below demonstrates that STD Oct, DL (General) and STD Jan worked the greatest number of questions. However, the achievement in assessed work is higher in all the other supported distance learning courses.

Chart 1.6

Pag

e13

Motivation & Achievement Engagement and motivation are not shown to correlate with high scores in assessed tests in ACC3015. So how does participation in MAL compare to average course scores in the MAL resource?

If we use self-study as a measure of motivation on MAL (sample tests accessed, see page 4) and compare it to overall scores from all parts the MAL resource, Chart 1.7 shows no discernable link between the two. However, it does show how few distance learners were motivated to access the study plan sample tests.

Chart 1.7

If we look at the average number of ‘questions worked’ in the study plan which includes all exercises accessed

by students (see Table 1.3 & Table 1.4, Chart 1.6) it clearly shows supported distance learners* undertook

comparatively little self study. The reasons for this are not clear.

Course Number of students who worked Average number of questions

questions in study plan/total students worked in Study Plan (rounded)

STD Oct 69 / 144 (47.9%) 61 STD Jan 20 / 24 (83.3%) 120 DL (General) 21 / 40 (52.5%) 70 DL 1 (H)* 2/13 (15.4%) 1 DL 2 (S)* 1/16 (6.3%) 6 DL 3 (XHC)* 4/8 (50%) 5 DL 4 (HCX)* 2/10 (20%) 2 Table 1.4

This shows that half the DL (General) and half the DL 3 (XHC) accessed the study plan. Those who used it in DL (General) cohort simply did far more than DL 3 (XHC). Once again STD Jan have high numbers as 83.3% of the cohort worked questions and completed the highest average number of questions each. Why do distance learners access less of the digital learning resource and show less engagement in MAL despite the online delivery of the course?

Pag

e14

Achievement - Homework

Identifying where students perform well and where they perform less well informs any strategies for improving

engagement or achievement.

Chart 1.8 below shows the difference between homework and assessment scores for campus cohorts versus

Distance Learning cohorts. Campus students invariably gain higher average scores in homework than

assessments. However, Distance Learning courses gain higher average scores in assessed work rather than their

homework.

Chart 1.8

It is worth remembering that the Distance Learners have more homework than Campus students and that their

motivation to complete it appears to decrease across the duration of the course

The mastery scores are similar across courses – this is an average score for the study plan section of MAL (self-

study, non-assigned). If we were able to compare exam results we could compare achievement on the course.

Applying average examination results to this chart would be valuable in order to show how the cohort

performed on the course, not just on MAL. If they were to reflect the mastery score on the study plan,

for example, lecturers would be able to assign more activities on which students can gain mastery

points.

Pag

e15

0102030405060708090

100

0 10 20 30 40 50 60 70 80 90 100

Test

1 S

core

(%

)

Test 2 Score (%)

DL 2 (S) student achievement in assessed work

Pag

e16

Charts (all) 1.9

Achievement - Test 1 & 2 (20% credit each)

If we look at the spread of results over the assessed component of the course it shows the range of abilities in

the various cohorts over Test 1 and Test 2.

The trends which are most evident in Chart 1.9 below are:

Campus students have a wide range of average scores for both Test 1 and Test 2 in the October and January intake

Campus students tended to do better in Test 2 than in Test 1 Distance Learning students’ results were interesting.

DL 2 (S) and DL 3 (XHC) and DL 4 (HCX) achieved high scores in both Test 1 and Test 2

DL and DL 1 (H) had the greatest range of results in distance courses though they generally scored better in Test 2.

DL (General) had least students in the top, right quadrant (over 50% in both tests)

The questions this raises:

Why is it harder to engage the seemingly most able, best performing students in MAL?

Is it harder to engage these students in distance learning online generally?

Why is there a difference with sustaining participation for different groups of learners on MAL?

The high scores in the courses listed below

appear unusual considering the lower

homework scores in chart 1.8 and high total

time commitment in chart 1.10. It may

warrant some further investigation for NBS:

- DL 3 (XHC)

- DL 4 (HCX)

0102030405060708090

100

0 10 20 30 40 50 60 70 80 90 100

Test

1 S

core

(%

)

Test 2 Score (%)

DL (General) student achievement in assessed work

Pag

e17

Time Spent on MAL

Average total time spent on all aspects of MAL is shown below. It is important to remember that time spent

simply indicates that a particular activity was open on a student’s device. It is not an indication of activity and

so can be misleading e.g. 24 hours to do a homework is unlikely, even though one student spent this length of

time with the activity open.

STD Oct had the lowest average time in campus students

DL 1 (H) had the lowest average time in DL

DL 4 (HCX) had the highest average time online

Chart 1.10 How might time and achievement relate? The median averages have been used to represent time as it can be a misleading number. The lengths of time which students spend online varies to a large degree within each cohort e.g. from 131 hours at the highest down to just 11 minutes in STD Oct.

Course

Assessment Score %(based on students completing all assigned assessments)

Average Score % (all forms of testing, quizzes, assessed, homework, sample testing)

Mean Total Time (zero score students removed) (mins)

Median Total Time (zero score students removed) (mins)

STD Oct 70.9 72.6 799 474

STD Jan 73.5 56.2 1748 2007

DL (General) 67.7 44.6 1877 1075

DL 1 (H) 82.5 79.8 818 239

DL 2 (S) 86.9 62.7 1761 1775

DL 3 (XHC) 96.3 92.7 1712 1612

DL 4 (HCX) 86 54.1 2244 1054 Table 1.5

Pag

e18

Charts 1.11 and 1.12 below are show an overall trend upwards for total time spent on MAL in relation to the credit-bearing Test 1 & 2 scores i.e. the more time spent, the better the score (generally). However, the opposite picture is shown for total time and average scores on all forms of activity on MAL in chart 1.12. That is to say, more time in total on MAL, the worse the score.

Chart 1.11

Chart 1.12

Time as an average seems to be an unreliable marker of achievement or engagement at a course level – though perhaps if we used it for individual students it might be more revealing e.g. did the student who spent 131 hours online in STD Oct achieve high scores?

Pag

e19

Conclusion & Recommendations

1. Are students engaging with MAL across all courses? 2. What differences are there in usage of and achievement in MAL between the courses? Overall: How can these findings inform NBS plans to deliver courses differently in e.g. flipped classrooms?

1. Students engage differently across courses. Distance Learning and Campus learning seems different.

The exception here is the DL General course which seemed to perform differently to all the other distance

learning courses. Is this because the DL General cohort is a more diverse group of students or is not a

‘supported’ distance learning course? Motivation, activity and achievement all differed for this group. Their

achievement was far more diverse (Chart 1.9), their engagement in the resource far higher on self-study (Chart

1.6) than any other Distance Learning cohort and their dropout rate was the highest at 18%.

RECOMMENDATION. Consider how distance learning delivery needs to differ for different cohorts to

maximize participation.

RECOMMENDATION. Consider the general DL course for changes to ensure these students can achieve

their potential and be supported throughout the course.

RECOMMENDATION. Consider how best to motivate other distance learners to engage with MAL.

Consider setting fewer homework assignments, some ‘homework assessments’ and more testing

deadlines such as a practice test 1 and 2.

2. Participation in the homework and homework scores are higher on campus than in any distance learning

course. Why is this? Motivation drops after the first two homework assignments have been completed for

distance learners (Chart 1.4).

RECOMMENDATION. As NBS move to a flipped classroom delivery and have less students in proximity

to lecturers and learning resources consider how lecturers interact with distance students in order to

encourage participation in the way lectures might. Is a ‘live’ video of or group chat with a lecturer able

to recreate this motivation to participate in online activities for example?

RECOMMENDATION. Consider how to make announcements to students on distance learning courses

more personal and motivational.

RECOMMENDATION. Homework ‘assessment’ worked well in engaging the STD Jan cohort and may

have contributed to their high level of engagement with the work.

The questions this raises might be:

Why is it harder to engage the most able, best performing students in MAL?

Is it harder to engage these students in distance learning online generally?

Why is there a difference when sustaining participation for different cohorts on MAL?

Overall.

Recommendations in this report may offer insight into how flipped classrooms will work best with campus

students. There may also be insight into ways to ensure a consistent student experience with MAL on the same

course (by specification) delivered in different ways and at different times by different people. Alterations to

the course may be desirable for the different distance learning cohorts based on their activity, engagement

and achievement levels.

Pag

e20

Background Information

Northampton is proud of its achievements in university rankings as well as its social and entrepreneurial

achievements. Two of the highlights on the website are below along with recent highlights for context.

2014 Student satisfaction increases to 89% (National Student Survey 2014)

Maintains top 50 UK university placing in the Guardian University Guide

Midlands Business Awards Most Enterprising University of the Year (winner 2012, 2013 and 2014) 2013

Times Higher Education University of the Year (shortlisted) Business School of the Year (shortlisted)

Recognised as the highest climbing university in the Complete University Guide 2014 and the Times/Sunday Times University Guide 2014

Awarded Changemaker Campus status by AshokaU and recognised as number one for social enterprise.

2012

Times Higher Education Awards Entrepreneurial University of the Year (shortlisted) Recognised by HESA as number one for employability amongst full-status universities in England The Business School prides itself on the following statements which can be found on their website: ‘Northampton Business School (NBS) has over 3500 students across the globe, offering a wide choice of flexible courses at undergraduate, postgraduate and professional levels. The school is renowned for high quality teaching and research supported by strong links with industry and commerce’*

Flexible study – part-time & online learning

Professional development

Eligibility for professional body recognition on some courses

Teaching excellence

Improving employability through work, industry connections and social enterprise

*http://www.northampton.ac.uk/about-us/academic-schools/northampton-business-school 21st April 2015 16:17