Assessing the academic writing skills of ‘home’ students; the next big thing in EAP?

Jonathan Rees and Dorota Pacek English for International Students Unit

University of Birmingham

Evidence of a deficit.It would be funny if it wasn’t serious…

• This essay will look at whether development can be predicted or unpredicted.

(student essay: register, style, incoherence)

• The menopause is similar to puberty only backwards.

(student report: register, incoherence)

• Mr X became upset and slightly agitated and asked how long he would be at the clinic as he had a dog. The staff nurse, student doctor and associate psychiatrist left the meeting; the social worker decided it was appropriate to discuss the situation with Mr X’s dog.

(student observation report: lack of logical sequencing)

Evidence of a deficit.Not even funny…..

This structure still plays many of important roles in biological proteins

(student essay: grammatical errors, confusing ellipsis)

The most important interaction within proteins hydrophobic interactions do a lot in causing the protein to fold into the correct conformation

(student descriptive assignment: lexical issues, punctuation problems and academic style.

Overview of presentation

I. Investigations to date of deficits

II. Three exacerbating factors

III. University of Birmingham Response

IV. Background to the test development

(The Biosciences Project)

V. Test construction

VI. Findings

VII. Conclusion

Acknowledged but poorly researched problem

• There has been a longstanding recognition of deficits in academic literacy amongst students in HEIs in the UK (Jones, 2004 and Lillis and Scott, 2007)

• Most of the evidence is anecdotal• Only one meaningful study; the MASUS study

at the Open University (Erling and Richardson 2010)

MASUS STUDY(Measuring the Academic Skills of University

Students)

MASUS procedure developed at the University of Sydney to identify students in need of academic writing development. Procedure employs 4 point rating scales for 5 assessment criteria:

1. Use of source materials

2. Structure and development of texts

3. Control of academic writing style

4. Grammatical correctness

5. Qualities of presentation

MASUS STUDY(Measuring the Academic Skills of University

Students)

• The procedure was employed by Erling and Richardson (2010) with sample of 78 students from the Open University.

• Agreed low ratings on each of the five criteria ranged between 17% and 47% of the students in the sample.

According to the study, this indicated that a

considerable number of students in the sample were

at risk of underperformance due to deficits in

academic writing skills.

• Outdated student ‘categorisation’ discourse • Diversity of ‘home’ student intake and

consequences of increasing student mobility.

• Diversity of teaching staff background

Worsening situation?

Home student?

A. The archetypal ‘home’ student: (sheltered UK background, knowledge of only UK education system, very limited travel experience, monolingual)

B. The mixed exposure ‘home’ student:(UK and other nationality parentage, lived and studied in two countries, bilingual, early years education in the UK, returned to the UK to finish secondary education and to enter HE)

C. The ‘home’ student without borders: (mixed non UK parentage, born outside the UK, lived and studied in different countries, multilingual, came to the UK to complete secondary education before university study

Diversity of teaching staff background

• At the UoB, 31% of academic teaching staff are from outside the UK (average across Colleges).

• Certain Schools/Departments have a higher concentration of international teaching staff e.g. Business School - 43% (68/159)

• Many of these staff studied up to 1st degree level in their home country

The University of Birmingham Response

• Piecemeal support from academic tutors

• Recent attempt to draw together all those involved in AW support (Graduate School)

• Development of ‘online’ resources (e.g. Studyskills4campus, Guide for effective learning)

• Centralised Academic Skills Centre

• Experimental projects (College of Arts and Law) for tutorial support

Opportunity knocks; the Biosciences Project

Small survey (2010) of key staff involved with undergraduate teaching in 3 Schools at the University of Birmingham:

• Grammar and sentence structure

• Text organisation – lack of logical order

• Style too colloquial, or inconsistent

• Lack of proofreading: correcting and refining

• Not understanding the purpose of referencing, literature

reviews, introductions

• Not following instructions, guidance and feedback

The Biosciences Project: Aims

• Develop an additional mechanism to identify home students with the most significant problems in academic writing (particularly essay writing)

• Provide targeted support in a cost/time effective way at an early stage of the student’s university career.

Mission Impossible for EISU

• to develop a version of the test within 1 month• to pre-test (particularly to assess face validity)• to deliver a test with the following specifications:

i. no more than 40 minutes in length

ii. assessing skills related to essay writing

iii. easy to score/allowing quick turnaround of results

Content considerations

Five ‘ability’ areas to be assessed:

i. ability to employ ‘academic’ grammar (e.g. verb complementation)

ii. ability to employ academic lexis (e.g. academic collocations)

iii.ability to employ an academic register (e.g avoidance of contractions,

iv. ability to sequence logically and structure discourse (e.g. cohesive devices, discourse markers)v. ability to comprehend and respond to instructions

Speed Development Process

• recycle

• use authentic materials

• go with intuition

• pre-test with small sample of Biosciences postgraduates (face validity issue)

RESULTS/FINDINGS

PRE-TEST QUESTIONNAIRE ITEMS (Likert Scale)YEAR 1 AND YEAR 2 OF PROJECT

MEAN 2011 2012

MEDIAN2011 2012

ITEM 1 PG 2.45UG 2.50 2.52

PG 2.0UG 2.0 2.0

ITEM 2 PG 2.27UG 2.61 2.77

PG 2.0UG 2.0 3.0

ITEM 3 PG 3.64UG 3.53 3.58

PG 4.0UG 4.0 4.0

ITEM4 PG 2.91UG 2.89 3.01

PG 3.0UG 3.0 3.0

MAIN MESSAGES FROM THE QUESTIONNAIRE DATA

• Both cohorts are over-optimistic about their academic writing skills (item 1 and 3)

• Both cohorts are neutral about whether they have developed their AW skills over the last year (item 2)

• Both cohorts are neutral about the need for further AW training (item 4)

TEST SCORES YEAR 1 AND YEAR 2 OF PROJECT

2O11 (n= 238) 2012 (n= 226)

MEAN SCORE 30.87 33.54

LOW 10 13

HIGH 41 45

Score distribution for 2011 cohort(Biosciences undergraduates, n =231)

MAIN MESSAGES FROM THE TEST SCORE DATA

• Apparently consistent measurement of ‘skills’

However, highly significant difference in mean

scores (p < .001)

• Negatively skewed distribution; long tail in performance

(2011 cohort, bottom 20% score < 28)

(2012 cohort, bottom 20% score < 30)

• Mean overall performance for both cohorts is much better than that of a small sample (n = 37) of MBA international students (M = 23.4)

Conclusion

• Further evidence (‘quasi-experimental’) of deficit in AW skills for a substantial number of ‘home’ undergraduates

• Any assessment only useful if part of a change in HEI culture (resources/timetabling)