Conscientiousness as an indicator of Degree Success in Non-Traditional Students

Catherine Marshall Centre Director

Presented by Dr Mary Dodd

Learning and Teaching LeaderFoundation Centre Durham University

∂

Perhaps the greatest challenge yet to face the educational system of most nations is how best to address issues of equity and excellence within a context of diversity

(Cushner, 1994)

∂

Foundation Programmes• Foundation Programmes were recently highlighted in

the Milburn report as an initiative for Higher Education Institutions to engage in for improving social mobility

• Widening participation activities involve risk for both the student and the institution

• Identifying factors that can predict student success could be useful in recruitment and retention

∂

Many Different UK Foundation Programme Models

1. Home students - remediation programmes for students who have studied to A Level, but not achieved a high enough level – deepening participation

2. Home students – widening participation programmes – offering an alternative route for students who have not studied at A level

3. International students – programmes which offer a bridging year for those whose national education system does not reach A level standard

4. International students – programmes which focus solely on Language Provision

∂

Durham Foundation Programme

Combination of widening participation for mature, non-traditional home students and international students who need a bridging year

Durham also has a Pre-sessional programme run by the English Language Centre which offers year-long English Provision

∂

Home Students•Predominantly local

•Over the age of 21

•Do not have the requirements for immediate entry to their chosen degree

•Low socio-economic background

International Students •From many different countries

•Between 17 and 20

•Have school leaver’s certificate

•High socioeconomic background

∂

Foundation Centre teachingFocus is on holistic approach

International education – synergy between home and international students invaluable

– International students want to learn alongside home students

– Local home students benefit from sharing experiences with a diverse student group

∂

Widening Participation The Foundation Centre contributes to the University’s

current widening participation target for home students

• Mature students: FC is a major contributor to the 19.3% of mature students in the University

• State School Entrants: all home students are from state schools

• LPN and social class: 60% of Foundation students in the last 2 years had household income below £25K

∂

Numbers of home and international students

Year of study

∂

Percentage Mature Students

Year of Study

∂

Percentage and Number of Local Students

Year of study

∂

Percentage Students with non-A level qualifications

∂

Ratio of female to male

Year of Study

∂

Assessment

• Follows University regulations

• One-time-only submission and no negotiation of marks

• Internal examination with some coursework

• One resit opportunity

∂

Outcomes

• There are direct progression routes to every department at Durham University.

• Students must pass the whole programme with 50% in every module

∂

Success rate• Typically the successful completion rate for all

students is between 75% and 80%

• The progression rate to Durham university departments for students is between 65% and 70%

• Of those who do not progress to Durham some withdraw from the course, and some transfer to other universities

∂

Percentage of Foundation Students passing, failing and withdrawing from the first year of their degree

∂

Measuring Potential• For traditional undergraduates there is a heavy reliance on A

level grades, but there is no consistent academic measure for non-traditional students

• Ensuring that students who progress will succeed is important • for the student, particularly with fees of £9,000

• the institution, given the higher rate of non-completion by non-traditional students

• and subsequent students, the pathway needs to be shown to be successful if it is to be maintained

∂

Design of the Study

• This particular study was based on 234 cases of students who studied on the Foundation Programme between 2004 and 2007 and progressed to degrees at Durham University

• Data was collated on the students’ age, sex, origin, qualifications prior to entry, programme of study, average score on the Foundation Programme, their attendance percentage and timely submission of assignments and year 1, year 2 and degree outcomes.

∂

Calculating a ‘Conscientiousness Index’ • Analysis of attendance showed that students who

attended fewer than 80% of classes were much less likely to pass the programme.

• The data on attendance percentage and percentage of timely submission of assignments were combined to create a Conscientiousness Index (CI).

• This follows work by the Medical Education Research Group at Durham University who have correlated a measure on Conscientiousness to perceptions of professionalism (McLachlan et al., 2009) and performance in both skills and knowledge exams (Kelly et al., 2012).

∂

CI in FoundationAttendance at all teaching sessions is compulsory•18 hours formal teaching time per week

– 6 compulsory sessions

lectures, workshops, practical sessions– over 21 weeks• CI component calculated as % attendance of all possible

sessions

Assignment submission•On average 12 summative assignments during the year•CI component calculated as % assignments submitted on time

∂

Qualitative Comparative Analysis• Proposed by Charles Ragin• Boolean Algebra• Formalises the logic of qualitative analysis• Possible to bring the logic and empirical intensity of

qualitative approaches to studies that embrace more than a handful of cases -- research situations that normally call for the use of variable-oriented, quantitative methods.

• Consider various inputs and analyse their effects on the output

∂

Truth Tables• The following fictitious example is used to help describe how the process

of QCA works. The fictitious data considered the output of whether students were successful in an English Language module, based on the inputs of GCSE standard ability, good attendance and reading set texts.

 

• QCA uses binary data i.e. 1 and 0 only, where 1 indicates the presence of a factor and 0 indicates its absence. In order to recode data into this system, a specific cut off point needs to be identified for continuous data and categorical data with more than one category can be shown with a range of binary variables. In the case of GCSE standard, the membership of the groups is likely to be either yes or no and simply dichotomised. For good attendance, however, there would need to be a decision made about what percentage of attendance was considered “good”.

∂

Truth Tables

• This data is then presented in a truth table. A truth table is a way of representing the raw data into all the possible combinations of values of the independent variables. Each row of the truth table represents one logical combination of values and is then assigned a 1 or a 0 depending on whether a particular output is present. The truth table also gives the number of cases with that particular set of conditions and in some instances there may be rows with no cases.

∂

Inputs Output Successs in

English Module

Number of Cases

A B C S  

1 1 1 1 56

1 1 0 1 23

1 0 1 1 7

1 0 0 0 14

0 1 1 0 12

0 1 0 0 25

0 0 1 0 3

0 0 0 0 5

A Ability to read and write to GCSE standardB Good attendanceC Reading set texts

∂

Necessity and Sufficiency

• Traditional statistical methods are usually based in significance based on variables being both necessary and sufficient – focussing on one pathway to the outcome

• QCA allows analysis of necessity and sufficiency separately – can identify several pathways to the outcome

∂

Necessity and Sufficiency

• Any condition which must be present to produce a particular outcome is considered necessary i.e. if within all the cases with a particular condition there is a subset with the outcome, that condition is considered necessary.

• Any condition that will always produce the outcome is considered sufficient i.e. if within all the cases with an outcome there is a subset with a particular condition, that condition is considered sufficient.

∂

Necessity and Sufficiency

All the cases with the condition

Cases with the outcome

Necessity Sufficiency

Cases with the condition

All the cases with the outcome

∂

Necessary but not sufficient

• In the example given, condition A – the ability to read and write to GCSE standard is a necessary condition, because all combinations of the conditions which have the outcome include the condition A

• It is not sufficient on its own, as there needs to be at least one other condition present, either class attendance or reading the set texts for a successful outcome

∂

Inputs Output Successs in

English Module

Number of Cases

A B C S  

1 1 1 1 56

1 1 0 1 23

1 0 1 1 7

1 0 0 0 14

0 1 1 0 12

0 1 0 0 25

0 0 1 0 3

0 0 0 0 5

A Ability to read and write to GCSE standardB Good attendanceC Reading set texts

∂

Quasi-Sufficiency, Quasi-necessity• QCA originally designed for use with small datasets, e.g. Ragin’s

hypothetical example (1987 pp 90) has a data set of 22 countries.

• In this study, the data set is larger and is made up of individual people rather than countries. In this case relationships of perfect sufficiency or necessity are unlikely and the real world situation is much more likely to involve quasi-sufficiency or quasi-necessity (Cooper, 2005).

 

• In the example below the first case shows complete sufficiency, with all members of the subset demonstrating the outcome, whereas in the second case 87.5% of the subset demonstrate the outcome, but that means that 12.5% of them do not, but it is a high enough percentage to be considered quasi-sufficient.

∂

Quasi-sufficiency

Cases showing the outcome

Subset of cases showing the outcome

Cases showing the outcome

Subset of cases

12.5% no outcome

87.5% outcome

Sufficiency Quasi-sufficiency

∂

Creating a solution to the truth table

• Using quasi-sufficiency means that there needs to be an element of judgement in deciding which of the configurations of cases will be included in the solution to the truth table which identifies which pathways will produce the outcome

• Any configurations with complete consistency with sufficiency i.e. a consistency score of 1.000000 are included

• For quasi-sufficient configurations the normal threshold is 0.8 although any large discrepancies in consistencies between adjacent rows can be taken into account as long as the threshold is no lower than 0.7

∂

Results – Honours Degree

The output analysed was achieving an Honours Degree

The inputs considered were •Age

•Sex

•Programme of study

•Qualification

•Students’ origin

•Performance on the Foundation Programme

•Conscientiousness at Foundation level

∂

Row no.

Over21 female science tradqual fa65 ci80 int local No. Hons deg

consistency

1 1 1 0 0 1 1 0 1 7 1 1

2 0 1 1 1 1 1 0 1 6 1 1

3 1 1 1 1 1 1 0 1 10 1 0.9

4 0 1 0 0 0 1 1 0 8 1 0.875

5 0 0 1 0 1 1 1 0 12 1 0.833333

6 1 1 1 0 1 1 0 1 28 1 0.821429

7 0 0 0 0 0 1 1 0 5 1 0.8

8 1 1 1 0 0 1 0 1 9 0 0.777778

9 0 1 0 0 1 1 1 0 13 0 0.769231

10 1 0 1 0 1 1 0 1 8 0 0.75

11 1 1 0 0 0 1 0 1 14 0 0.642857

12 1 0 0 0 0 1 0 1 5 0 0.6

13 0 0 0 0 1 1 1 0 9 0 0.555556

14 1 1 0 0 0 0 0 1 9 0 0.555556

∂

Solution for the Outcome Achieving an Honours Degree

  Raw Coverage

Unique coverage

Consistency

female*science*tradqual*fa65*ci80*local 0.084746 0.084746 0.937500

Over21*female*~tradqual*fa65*ci80*local 0.169492 0.169492 0.857143

~over21*~science*~fa65*ci80*int 0.062147 0.062147 0.846154

~over21*~female*science*fa65*ci80*int 0.056497 0.056497 0.833333

solution coverage: 0.372881 solution consistency: 0.868421

∂

Necessary Conditions

• All four solutions have the element of high conscientiousness as part of their configurations and this is the only element that is found in all four pathways. Consequently a

• An analysis of necessity for high conscientiousness for the outcome of achieving an honours degree was shown to be approaching necessity at 83.0%.

 Analysis of Necessary Conditions for the outcome variable: honsdeg

Conditions tested Consistency Coverage

CI80 0.830508 0.803279

∂

Good Honours Degree

Many universities use good honours degrees as a key performance indicator, so as well as whether students achieved an honours degree, the outcome of achieving either a first class or a 2:1 degree was analysed to elucidate whether any configurations of inputs were sufficient for that outcome

∂

Truth Table for Good Honours DegreeRow No. female tradqual fa65 ci80 number goodhons consistency

1 1 1 1 1 22 1 0.818182

2 0 1 1 1 5 0 0.63 1 0 1 1 53 0 0.547174 0 0 1 1 35 0 0.457143

5 1 0 1 0 5 0 0.46 1 1 0 1 5 0 0.27 0 0 0 1 19 0 0.157895

8 1 0 0 1 41 0 0.146341

9 1 0 0 0 17 0 0.117647

11 0 0 0 0 12 0 0.083333

12 0 0 1 0 4 0 013 1 1 0 0 4 0 014 0 1 0 1 3 0 015 1 1 1 0 2 0 016 0 1 0 0 5 0 0

∂

Solution for Achieving a Good Honours Degree• Only one pathway was found to be sufficient

QCA Output for the Outcome Achieving a Good Honours Degree

  Raw Coverage

Unique coverage

Consistency

female*tradqual*fa65*ci80 0.219512 0.219512 0.818182

∂

Necessary Conditions

Analysis of necessary conditions was conducted for all four inputs•High conscientiousness is approaching necessity at 0.927 •Doing well on the Foundation Programme is also quite high at 0.841•Traditional qualifications are not a necessary condition at all for the outcome of achieving a good honours degree at 0.280•at 0.71, being female is just on the borderline of being considered necessary - it may be that being female is associated with having high conscientiousness

∂

Potential Uses: Selection?

• IT may be possible to use conscientiousness as a selection too e.g. the medical team found strong correlation with the NEO-P-IR inventory and their measure of conscientiousness

• However, issues with whether conscientiousness is remediable

∂

Other Potential uses:

• Formative– Used for feedback to student– To identify students who need greater support for

retention and success• Summative

– Used to determine progression. Can identify outliers and major changes. Can be used to explore ‘failure to remediate’

• Informative– if the data is retained and cohort study conducted

might be used to inform organisation of potential issues which may occur in the future

∂

Practicality

• Objective• Can largely be done by admin staff• Does not require a ‘decision to report’• Runs continuously all year• Requires registers to be taken• Requires centralisation of information previously

gathered but not collated

∂

Extending it to other groups

• In use by medical and dental schools• Being developed by surgeons, anaesthetists, clinical

psychologists and general practitioners• Rolled out to paramedics and physiotherapists• Being explored for teachers in training

For further questions, please contact:

Catherine Marshall c.a.marshall@dur.ac.uk

∂∂

Let the journey begin …