Introduction - Welcome to AALL | Association for Academic...

PELA Symposium 15 Feb 2017

http://about.curtin.edu.au/who/aboriginal-welcome/

Introduction

The Association for Academic Language and Learning

The Association for Academic Language and Learning (AALL) is a national association that was formed in November 2005. It is the organisational body for specialists who work with university students to enhance their learning and academic English. More broadly, it aims to enhance and extend teaching and learning opportunities through policies, practices and research. Academic Language and Learning specialists work with both local and international students at every level from first year through to postgraduate. They recognise that the linguistic challenges that students face are not remedial, but intrinsic to the learning experience of every student. All students need to understand, participate and communicate effectively in the discourses and cultures of enquiry in their disciplines.

Aims and Context of the SymposiumPost-entry language assessments (PELAs) take many forms, but all share a common goal to identify the language development needs of students. As you are aware, their use is increasing rapidly in Australian Higher Education. It is in this context that university staff, and academic language and learning specialists in particular, are called on to design and implement language diagnostics. Language diagnostics have a specific purpose to identify language strengths and weaknesses so that feedback can be provided and action taken.

The Association of Academic Language and Learning, in collaboration with Curtin University and with support from Murdoch, Edith Cowan University and the University of Western Australia have come together to bring two leading experts on language testing (Dr Ute Knoch and Professor John Read) to Perth share their knowledge and expertise with us. In particular, this Symposium aims to advance our theoretical knowledge and practical skills in academic language testing. In particular, the aim is to build our capacity to evaluate the effectiveness of existing PELAs in our institutions, and to improve our capacity to design and validate PELAs to meet the learning needs of all students.

Rationale

Post-entry language assessments (PELAs) take many forms, adopt differing processes of implementation, and serve many purposes (Knoch & Elder, 2013), yet all share a common goal to identify the language development needs of students (Dunworth, 2009, 2013; Murray, 2010). Their use is increasing rapidly in Australian Higher Education (Harris, 2013) as regulatory bodies focus on post-enrolment language assessment and development (Murray, 2010). In this context, there are expectations on ALL staff to design and implement valid and reliable diagnostic instruments. Given the diversity of the disciplines that inform academic language and literacy practice, it is not surprising that ALL staff are from “diverse and various discipline backgrounds” (Carmichael, Hicks, McGowan and Van der Wal, 1999). And, even though many have expertise in Applied Linguistics, particularly in English language teaching/TESOL, few have specific expertise to design and validate language tests.

Diagnostic AssessmentsDiagnostic language testing is a notoriously difficult and under-researched area of language assessment (Elder & Read, 2015), but it is agreed that diagnostic tests have a specific purpose to identify language strengths and weaknesses so that feedback can be provided and action taken (Alderson, 2005 as cited in Elder & Read, 2015).

The lack of research and attention on diagnostic language testing means that insufficient resources are readily available to assist staff to develop, validate and implement PELA. Coupled with insufficient financial resourcing that would otherwise allow them to purchase commercially available products, ALL staff often bear the burden of responsibility for design and implementation of PELA alone.

PELAsThe current situation has led to the proliferation of in-house PELAs that are rarely formally validated (Knoch & Elder, 2013) and while these PELAs may be well-designed and valid instruments, universities may be currently unable to provide evidence of this. It is in this context that regulatory bodies are likely to call upon institutions to defend the veracity of PELA processes and procedures and ALL staff will be on the front line in articulating and defending the work they do. Within this process, the legitimacy of ALL work will be scrutinised.

This symposium aim is to advance the theoretical knowledge and practical skills of staff involved in post-entry language testing. The symposium will develop the skills and knowledge of attendees and build their capacity to: a) evaluate the effectiveness of existing PELAs in their institutions and, b) to design and validate a range of different types of PELA to suit institutional purposes that will withstand external scrutiny. It is anticipated that this initiative will reinforce the expertise, significance and influence of the Association for Academic Language and Learning in quality assurance in Australian Higher Education.

The speakers

Jill DownieProfessor Downie is an established national and international researcher with numerous grants, publications and conference presentations, including keynote addresses.

In her role as DVC, Academic, Jill is responsible for Curtin Learning and Teaching and the quality assurance processes of learning and teaching across the University including the Academic Language Learning Policy, Library and Learning Centre; Student Support Services including Counselling and Health Services, Sport and Recreation, Housing, Child Care; Student Services from Admissions though the continuum to Graduation; Academic Registrar's Office; Ethics, Equity and Social Justice, and the Centre for Aboriginal Studies (CAS) and Indigenous strategy.

Alex BarthelAlex Barthel is a higher education consultant in academic language and learning. He was the Director of the University of Technology, Sydney Academic Language & Learning Centre.

He was the inaugural President of the Association for Academic Language & Learning and a member of the DEEWR/AUQA Steering Committee that developed the ‘English language standards in higher education’. His main interests include the development of English language policy and procedures in the context of English language standards in higher education.

Alex has received recognition for his achievements through Teaching and Learning awards from the Australian Government, University of Technology Sydney, and from the NSW TAFE Commission. Alex is formally registered as Expert in Academic Language and Learning with the national Tertiary Education Quality Standards Agency (TEQSA).

Professor John ReadJohn Read is a Professor of Applied Linguistics. He is the former Head of the Applied Language Studies and Linguistics at the University of Auckland. His main area of specialisation is language testing and assessment, with a particular focus on vocabulary assessment and the testing of English for academic and professional purposes. He has been extensively involved with the International English Language Testing System (IELTS) as a researcher, consultant and test administrator, and has directed three IELTS-funded research projects on various aspects of the test. He has been a consultant to Aviation Services Ltd and Air New Zealand on the design of English language assessments for pilots and air traffic controllers. At the University of Auckland, he is Academic Coordinator of the Diagnostic English Language Needs Assessment (DELNA), which assesses the academic language needs of students entering the university.John was President of the International Language Testing Association (ILTA) from 2011 to 2012, having previously served as Vice-President (2009-10) and a member of the Executive Board (1992-93, 2000-01). He was co-editor of the international research journal Language Testing from 2002 to 2006. More recently, he completed a four-year term as a member of the Committee of Examiners for the Test of English as a Foreign Language (TOEFL) (2011-15).

In his recent publication Post-Admission Language Assessment of University Students (PALA)John emphasises that follow up and follow through are crucial element in the successful use of PELAs. As John reminds us – post-admission language assessment is not an end in itself. In this volume, John also highlights the practical considerations of balancing resources with outcomes with a reminder that language support is a central responsibility that universities have to their students – an effective diagnostic instrument cannot reach its goals without strong institutional support.

Dr Ute KnochDr Ute Knoch is the Director of the Language Testing Research Centre at the University of Melbourne.

She has published widely with nearly 40 peer-reviewed publications which have been published in journals such as Language Testing, Language Assessment Quarterly, TESOL Quarterly, Applied Linguistics, Assessing Writing, Journal of Second Language Writing and Language for Specific Purposes.

Her research interests are in the area of writing assessment, rating processes, assessing languages for academic and professional purposes, and placement testing.

She is currently the Co-president of the Association for Language Testing and Assessment of Australian and New Zealand (ALTAANZ) and on the Executive Board of the International Language Testing Association (ILTA).

In 2014, Ute was awarded the TOEFL Outstanding Young Scholar Award by the Educational Testing Service (Princeton, US), recognizing her contribution to language assessment. In her recent chapter in Post-admission Language Assessment of University Students co-authored with Cathy Elder and Sally O’Hagan Ute argues that PELA must be ‘embedded in a more enlightened university policy which places a premium on the provision of opportunities for English language development’.

Presentations

Title page

Teaching & Learning Aymposium

Post-entry language assessment

Association for Academic Language & Learningand

Curtin University15 February 2017

Alex BARTHELHE consultant in ALL

0

PELA symposium

• 1. Definitions

• 2. Do you PELA?

• 3. Questions

16© 2017, A. Barthel

0

1. Definitions

• Post-Enrolment/Entry Language Assessment (PELA) includes any form of integrated language assessment used to diagnose the level of ELP of all or specified cohorts of students. Types, designs, content, modes, etc. of currently used PELA in Australia are summarised in the table below:


0

1. Features of PELA in Oz universities (DoP, 2013)


characteristic range of types

organisationdesign

content

mode

target

feedback

follow-up

0




organisation • institution wide / faculty based / unit based

design

content

mode

target

feedback

follow-up

0





design • in-house / co-designed /purchased ‘off the shelf’• single instrument / 2 part instrument: screening & diagnostic component

content • •

mode

target

feedback

follow-up

0






content • skills-based tasks with emphasis on writing / ‘indirect’ tasks e.g. cloze texts / oral presentations• general proficiency / generic academic register / discipline specific register

mode

target

feedback

follow-up

0







mode • paper based / online• timed / untimed• compulsory / optional• supervised / self-administered• secure / freely available• available all year / available at certain times• automated marking / marked by hand

target

feedback

follow-up

0








target • all students / international students / new students / EAL students entering HE below a certain identified level / students enrolled in a particular course/unit

feedback

follow-up

0









feedback • score / band / descriptor / individual written feedback• confidential / distributed to academic coordinator or ALL unit

follow-up

0









feedback • score / band / descriptor / individual written feedback• confidential / distributed to academic coordinator or ALL unit

follow-up • at initiative of student / mandatory enrolment in an ALL program / contact from an ALL educator to discuss needs

0

2. Do you PELA?


responses 2011 n=37 98% 2017 n=33 85%

01 do you PELA YES n=23 62 % YES n=23 70 %

0

2. Do you PELA?


responses 2011 n=37 98% 2017 n=33 85%


02 university wide of the 23: 46 % of the 23: 52 %

0

2. Do you PELA?


responses 2011 n=37 98% 2017 n=33 85%



03 faculties 21 % 61 %

0

2. Do you PELA?


responses 2011 n=37 98% 2017 n=33 85%




04 compulsory 83 % 52 %

• for all students 78 % 78 %

• selected cohorts 13 % 17 %

0

2. Do you PELA?


responses 2011 n=37 98% 2017 n=33 85%




04 compulsory 83 % 52 %

• for all students 78 % 78 %

• selected cohorts 13 % 17 %

05 assessing general English 70 % 78 %

• academic writing 83 % 83 %

• reading 52 % 61 %

• listening 30 % 13 %

• speaking 9 % (n=1) 4 %

0

2. Do you PELA? (continued)


responses 2011 n=37 98% 2017 n=33 85%

06 locally developed of the 23: 83 % of the 23: 83 %

• commercially purchased 17 % 17 %

0



responses 2011 n=37 98% 2017 n=33 85%



07 paper based 71 % 57 %

• online 52 % 57 %

0



responses 2011 n=37 98% 2017 n=33 85%




• online 52 % 57 %

08 marked by A.L.L. staff 78 % 61 %

• by other uni staff 30 % 22 %

• externally/automatically 6 % 30 %

0



responses 2011 n=37 98% 2017 n=33 85%




• online 52 % 57 %




09 results sent to students 100 % 100 %

• to faculties 65 % 61 %

• to others 39 % 22 %

0



responses 2011 n=37 98% 2017 n=33 85%




• online 52 % 57 %






• to others 39 % 22 %

10 URL with info about PELA 40 % 39 %

0



responses 2011 n=37 98% 2017 n=33 85%




• online 52 % 57 %






• to others 39 % 22 %

10 URL with info about PELA 40 % 39 %

11 additional comments 57 % 26 %

0

3. Questions1. how useful / effective is PELA? assessing R? W? L? S?

2. should PELA be generic or discipline specific?

3. optional or compulsory? enforceable?

4. stand-alone or linked to A.L.L. strategies?

5. post-entry (pre-course) or integrated?

6. validation / reliability?

7. how to link PELA to academic performance?

8. ……?37© 2017, A. Barthel

thank you

Alex BarthelHigher Education consultant in academic language & learning

[email protected]

© 2017, A. Barthel 38

mailto:[email protected]

ISSUES IN THE DESIGN OFPOST-ENTRY LANGUAGE ASSESSMENTSAALL SYMPOSIUM, CURTIN UNIVERSITY, 15 FEB 2017

JOHN READUNIVERSITY OF AUCKLAND

OUTLINE OF THE ISSUES

The nature of PELAs, as compared to international proficiency tests

The diagnostic dimension The constructs that may underlie a PELA Design implications of the constructs Postgraduate vs. undergraduate PELAs The interaction between PELAs and provisions for language

development

THE DISTINCT FEATURES OF PELASAS COMPARED TO PROFICIENCY TESTS

Proficiency Tests (IELTS, TOEFL, PTE)

PELAs

Used for high-stakes admissions decisions

Funded by high fees charged to test-takers

Generic and inflexible in nature

Highly professional in design, administration and score reporting

Results evaluated by users primarily on a pass/fail basis

Used for low(er)-stakes advising after admission

Funded by the university or the faculty

Can address local needs Variable in the quality of

the assessment, depending on the expertise available

Results in profile form to inform subsequent language development

THE DIAGNOSTIC DIMENSION

THE DIAGNOSTIC DIMENSION

Several PELAs have the term “diagnostic” in their title. The naming of these PELAs tends to predate the current interest

in language diagnosis among language testers. It is useful to consider post hoc the extent to which the PELAs are

diagnostic in nature.

LANGUAGE DIAGNOSISSOME KEY REFERENCES

Alderson, J.C. (2005). Diagnosing foreign language proficiency. London: Continuum. Alderson, J. C., Brunfaut, T., and Harding, L. (2015). Towards a theory of diagnosis in

second and foreign language assessment: Insights from professional practice across diverse fields. Applied Linguistics, 36(2), 236−260.

Alderson, J. C., Haapakangas, E.-L., Huhta, A., Nieminen, L., & Ullakonoja, R. (2014). The diagnosis of reading in a second or foreign language. New York: Routledge.

Lee, Y-W. (Ed.) (2015). Special issue: Future of diagnostic language testing. Language Testing, 32(3).

SOME FEATURES OF DIAGNOSTIC ASSESSMENT(ALDERSON, 2005)

A focus on areas of difficulty and the reasons for poor performance.

An analytic approach to language ability◦ Language elements: vocabulary, grammar, phonology◦ Subskills in reading and listening; speaking and writing

A basis for informed decisions about subsequent teaching and learning.

For adult learners, a process of raising awareness and encouraging learner autonomy.

THE DELNA PROCESS AT AUCKLAND

46

Computer-based Screening

of all 1st Year Undergraduates

No further assessment

Paper-based Diagnosis

OK

Not OK

General advice on academic language enhancement OK

Consultation with Language

Advisor

Not OK

Individual advice on recommendedor required language development

DELNA AS A DIAGNOSTIC ASSESSMENT

The basic purpose of identifying students whose academic language proficiency may not be adequate.

The use of measures of language knowledge (vocabulary, timed reading) in the Screening phase, including the speeded element.

The analytic rating scale for Writing, with separate ratings for Fluency, Content and Form and further sub-categories.

The role of the Language Advisor in interpreting a student’s Diagnosis results and advising on language enhancement options.

THE UNDERLYING CONSTRUCTSAND THEIR DESIGN OPTIONS

THE CONSTRUCTS UNDERLYING A PELAFOUR OPTIONS

GENERIC (Academic) language knowledge/competence Academic language proficiency/academic literacy

DISCIPLINE-SPECIFIC Academic literacies Professional communication skills

THE CONSTRUCTS UNDERLYING A PELASOME USEFUL REFERENCES

Davies, A. (2008). Assessing academic English. Testing English proficiency 1950-89: The IELTS solution. Cambridge: Cambridge University Press.

Murray, N. (2010). Considerations in the post-enrolment assessment of English language proficiency: From the Australian context. Language Assessment Quarterly, 7(4),343-358.

Murray, N. (2016). Standards of English in higher education: Issues, challenges and strategies. Cambridge: Cambridge University Press. Chapter 3.

Read, J. (2015). Assessing English proficiency for university study. Basingstoke, UK: Palgrave Macmillan. Chapters 6 and 7.

DESIGN OPTIONS(ACADEMIC LANGUAGE KNOWLEDGE

DISCRETE-POINT Vocabulary: multiple-choice, matching, gap-filling, (timed) Yes/No Grammar: multiple-choice, gap-filling, sentence rewriting

INTEGRATIVE Cloze procedure / C-Test Cloze-elide Dictation / Elicited imitation

DESIGN OPTIONSACADEMIC LANGUAGE PROFICIENCY

Assessment tasks with a skills focus, designed to simulate to some extent academic study tasks

Exemplified by the tasks in the major international proficiency tests (IELTS, TOEFL, PTE)

May be independent (focused on a single macro-skill) or integrated (requiring a combination of skills, such as Listening + Reading Writing)

THE DELNA TASKS AND THEIR CONSTRUCTS

SCREENING (Computer-based) Vocabulary Timed Reading (cloze-elide)

DIAGNOSIS (Paper-based) Listening to a mini-lecture Reading academic-type texts Writing an interpretation of a graph

Academic Language Knowledge

Academic Language Proficiency

DESIGN OPTIONSACADEMIC LITERACIES

Key feature: tailored to the study skills, language knowledge and discourse norms required in a particular discipline or faculty.

Normally requires consultation, if not collaboration, with subject lecturers.

Best-known model: the integrated writing task used in Sydney’s MASUS (Measuring Academic Skills of University Students)

A SAMPLE MASUS TASKBONANNO & JONES, 2007

Design Architecture (undergraduate) Input: campus tour, library research session, information on referencing

conventions Prompt:

Identify two or three architectural styles present on the main campus of the University of Sydney. Do you think that having so many different architectural ideals reflected in the buildings and spaces on the campus gives an unharmonious effect?

http://sydney.edu.au/stuserv/documents/learning_centre/MASUS.pdf55

http://sydney.edu.au/stuserv/documents/learning_centre/MASUS.pdf

BUILDINGS AT SYDNEY UNIVERSITY

56

PROFESSIONAL COMMUNICATION SKILLS(MURRAY, 2010, 2016)

Intercultural competence A cultural relativistic orientation Interpersonal skills Conversancy in the discourses and behaviours associated with

particular domains Non-verbal communication skills Group and leadership skills

DESIGN OPTIONSPROFESSIONAL COMMUNICATION SKILLS

Questionable whether such skills can or should be assessed in a PELA.

Better assessed through appropriate course work or an exit/graduating assessment, especially where students need to seek professional registration after graduating.

May influence the design of a PELA based on the academic literacies construct.

OTHER PELA DESIGN ISSUES

PELAS FOR POSTGRADUATE STUDENTS

Few if any (?) PELAs designed specifically for postgraduates. If their undergraduate study was non-English-medium, the same

kind of PELA (as for undergrads) may be appropriate. Despite experience of co-authored articles and conference

presentations in English, diagnosis of academic language knowledge may still be important.

DELNA for doctoral candidates replaces the independent writing task (for undergrads) with a more integrated reading-writing task.

THE INTERACTION OF PELAS AND SUBSEQUENT LANGUAGE DEVELOPMENT

If provision for language development is largely generic in nature, PELAs based on academic language knowledge and proficiency are most suitable.

Discipline-specific language development opens the opportunity for a PELA based on an academic literacies construct, depending on the level of cooperation between language/literacy tutors and subject specialists.

MODELS OF COLLABORATION IN LANGUAGE DEVELOPMENT (JONES, BONANNO & SCOULLER, 2001)

Weak Adjunct ModelGeneric tutorials by learning advisors on academic writing outside of class hours

Strong Adjunct ModelA stronger focus on writing genres that are relevant to the students’ discipline

Integrative ModelLearning advisors give presentations or workshops on discipline-specific aspects of academic literacy during class hours

Embedded ModelA course curriculum focused on literacy in the discipline, designed collaboratively by learning advisors and the subject lecturers who will actually teach the course.

AN INCLUSIVE MODEL(WINGATE, 2015)

Academic literacy instruction should focus on an understanding of the genres associated with the students’ academic subjects, rather than taking the generic EAP approach.

All students should have access to this instruction, regardless of language background. Language support for non-native speakers should be provided in addition to the academic literacy instruction.

The instruction needs to be integrated with the teaching of content subjects so that ideally academic literacy is assessed as part of the subject curriculum.

Academic literacy instruction requires collaboration between writing experts and subject experts to develop the curriculum jointly.

EMBEDDED LANGUAGE DEVELOPMENTSOME KEY REFERENCES

Arkoudis, S., Baik, C., & Richardson, S. (2012). English language standards in higher education. Camberwell, Vic.: ACER Press.

Arkoudis, S., & Kelly, P. (2016). Shifting the narrative: International students and communication skills in higher education. IERN Research Digest, 8. International Education Association of Australia. Available: www.ieaa.org.au/documents/item/664.

Jones, J., Bonanno, H., & Scouller, K. (2001). Staff and student roles in central and faculty-based learning support: Changing partnerships. Paper presented at the 2001 National Language and Academic Skills Conference. Available at: http://learning.uow.edu.au/LAS2001/selected/jones_1.pdf.

Wingate, U. (2015). Academic literacy and student diversity: The case for inclusive practice. Bristol: Multilingual Matters.

http://www.ieaa.org.au/documents/item/664

http://learning.uow.edu.au/LAS2001/selected/jones_1.pdf

OUTLINE OF THE ISSUES

The nature of PELAs, as compared to international proficiency tests

The diagnostic dimension The constructs that may underlie a PELA Design implications of the constructs Postgraduate vs. undergraduate PELAs The interaction between PELAs and provisions

for language development

OKHave I

covered them all?

Thanks for your attention!

EVALUATING POST-ENTRY LANGUAGE ASSESSMENTS (PELAS)Ute KnochLanguage Testing Research CentreUniversity of Melbourne

PELA EVALUATION

Studies set out to evaluate specific aspects of PELAs and policies (e.g. Read, 2008; Elder & von Randow, 2008; Davies and Elder, 2005; Bright & von Randow, 2004; Knoch, 2012)

DEGREES OF PROFICIENCY PROJECT

Presented Evaluation as one aspect to consider when designing/implementing a PELA

Shown as the last tile

PELA EVALUATION

2013 - Knoch & Elder proposed framework for PELA

evaluation (PLTA) Read (2015) applied this framework to the DELNA

used at the University of Auckland

AIMS OF TODAY’S TALK1. Outline the key aspects of the framework

(Knoch & Elder, 2013) show examples of such an evaluation in light of three different PELAs I have

worked on2. Show impact of policy on evaluation outcome

3. Discuss the impact of practicality

4. Timing of the evaluation

PELA EVALUATION ARGUMENT Based on validity theory from educational evaluation Includes a number of building blocks which are evaluated

together to form an overall argument Important: focusses not just narrowly on the assessment but

also evaluates the decisions made based on the assessment and the consequences for the candidates and other stakeholders

Directly examines the policy that supports the assessment implementation

OVERVIEW OF PELA VALIDITY ARGUMENT

Beneficial consequences

Appropriacy of decisions

Relevance of PELA domain

Reliability of results across administrations

Characteristics of test,

scoring and test

administration

SUMMARY OF PELA VA

LIDITY ARGUMENT

Consequences of P

ELA are beneficial

Decisions made based onapprop

PELA are equitable and riate

Relevance of assess

dom

Reliability of

ment to academic ain

results


scoring and test

administration

Warrants Sources of supporting evidence

1. Raters can implement scoring procedures consistently

Statistical analysis of ratings

2.Test administration conditions are clearly articulated and appropriate

Student questionnaires/interviews; review of test administration protocol; observation of test sessions; interviews with test invigilators

3. Instructions and tasks are clear to all test takers

Student questionnaires/interviews

4.Test is pitched at appropriate difficulty level and test tasks/items discriminate consistently between more and less able candidates

Statistical analysis of test properties (i.e. item difficulty, discrimination, internal consistency)

OVERVIEW OF PELA VALIDITY ARGUMENT




Reliability of results


scoring and test

administration

THREE SAMPLE PELAS

Diagnostic English

Language Assessment

(DELA) - UniMelb• Assessment of listing,

reading and writing

• Pen-and-paper

Academic English Screening Test (AEST)

– trial UniMelb

• Online Screening test

Diagnostic Pharmacy Writing Assessment

– University Y• Detailed diagnosis of writing

• Pen-and-paper

DELA (DIAGNOSTIC ENGLISH LANGUAGE ASSESSMENT) – UNIVERSITY OF MELBOURNEPen-and-paper assessment of listening, reading, and

writingListening and reading marked using detailed marking

keysWriting assessed by human raters (single marked due to

costs) using three criteria (organisation, content, form)Various equated versions of the test materials availableStudents receive feedback descriptors for reading andlistening and three writing criteria

DELA (DIAGNOSTIC ENGLISH LANGUAGE ASSESSMENT) – UNIVERSITY OF MELBOURNE

Assessment compulsory for specific student groups (IELTS below 7 or equivalent)

students placed into one of three groups: at risk, borderline, proficient. At risk students required to enrol in compulsory

support Language support following DELA depends on faculty Support not well-linked to needs of students Diagnostic potential of test not mined as basis for support No follow-up of students who do not enrol in

‘compulsory support’

CHARACTERISTICS OF TEST, SCORING AND ADMINISTRATION

Warrants DELAInstructions and tasks are clear to all test takers Test is pitched at appropriate difficulty level Test tasks/items discriminate consistently betweenmore or less able candidates

Scoring is consistent /?Test administration conditions are clearly articulatedand appropriate

Evaluation

RELIABILITY OF RESULTS ACROSS ADMINISTRATIONSWarrants DELA

Different test forms are parallel in design Appropriate equating procedures are used to ensure equivalency across forms

Sufficient tasks are included to provide stable estimatesof test taker ability

/?

Test administration conditions are consistent

Evaluation

RELEVANCE OF PELA TO DOMAINWarrants DELA

Characteristics of test tasks are similar to those required ofstudents in the academic domain

Scoring criteria and rubrics capture relevant aspects Test results are good predictors of language performance inacademic domain

Scores derived from the test provide sufficient information about candidates’ academic language proficiency

/?

Performance on PELA relates to performance on other assessments of academic language proficiency

Evaluation

APPROPRIACY OF DECISIONS BASED ON TEST

Warrants DELAStudents are correctly categorised based on theirtest scores

/?

The test results include feedback on test performance and a recommendation

Recommendation is closely linked to on-campus support

Assessment results are distributed in a timelymanner

/

Test results are available to all relevant stakeholders

Test users understand the meaning and intended uses of scores

Evaluation

CONSEQUENCESWarrants DELA

All targeted test takers sit for the test The test does not result in stigma or disadvantage for students Test takers’ perceptions of the test and its usefulness are positive /?The feedback from the test is useful and directly informs students’ future learning /?Students act on the test recommendation Follow-up language development options provided for students are appropriate and ongoing if needed

Learners taking up support options improve their English overthe course of their studies

/?

Students who fail to act on the recommendations are more likely to struggle in their academic studies

Evaluation

SUMMARY OF DELA EVALUATION ARGUMENT

DELACharacteristics of test, scoring and test administration

Reliability of results Relevance of PELA domain Appropriacy of decisions


THREE SAMPLE PELAS

Diagnostic English

Language Assessment


reading and writing

• Pen-and-paper

Academic English Screening Test (AEST)

– UniMelb trial

• Online Screening test



• Pen-and-paper

ACADEMIC ENGLISH SCREENING TEST (AEST) – TRIAL UNIMELB

Online screening test: two test sections: text completion & speeded reading – 25 minutes Automated scoring & instant test result

Students take assessment at home in their own timeCombined scores on the screening tasks

are used to predict which students are likely to be linguistically at risk

Students grouped into one of three groups: proficient,borderline, at risk

ACADEMIC ENGLISH SCREENING TEST (AEST)

Administered universally to ALL incoming undergraduate and postgraduate students

Students receive recommendation and list ofsuggested courses (if applicable)At risk student encouraged to take uprecommended support


Warrants AESTInstructions and tasks are clear to all test takers Test is pitched at appropriate difficulty level Test tasks/items discriminate consistently between more or less able candidates

Scoring is consistent Test administration conditions are clearly articulated and appropriate /?

Evaluation

RELIABILITY OF RESULTS ACROSS TEST ADMINISTRATIONS

Warrants AESTDifferent test forms are parallel in design Appropriate equating procedures are used to ensure equivalency across forms

Sufficient tasks are included to provide stable estimates of test taker ability Test administration conditions are consistent ?

Evaluation

RELEVANCE OF PELA TO DOMAINWarrants AESTCharacteristics of test tasks are similar to those required of students in the academic domain

Test results are good predictors of languageperformance in academic domain /?Scores derived from the test provide sufficient information about candidates’ academic language proficiency

/?

Performance on PELA relates to performance on other assessments of academic language proficiency

Evaluation /?


Warrants AESTStudents are correctly categorised based on their test scores /?The test results include feedback on test performanceand a recommendation

/

Recommendation is closely linked to on-campus support Assessment results are distributed in a timely manner Test results are available to all relevant stakeholders Test users understand the meaning and intended usesof scores

?Evaluation

/?

CONSEQUENCESWarrants AEST

All targeted test takers sit for the test The test does not result in stigma or disadvantage for students Test takers’ perceptions of the test and its usefulness are positive

/?The feedback from the test is useful and directly informs their future learning /?Students act on the test recommendation Follow-up language development options provided for students are appropriate and ongoing if needed Learners who take up support options improve their English over the course of their studies

?Students who fail to act on the recommendations are more likely to struggle in their academic studies

?

Evaluation

/?

SUMMARY OF AEST (UNIMELB)AEST

Characteristics of test, scoring and test administration

Reliability of results across administrations

Relevance of PELA domain /?Appropriacy of decisions /?Beneficial consequences /?

THREE SAMPLE PELAS

Diagnostic English

Language Assessment


reading and writing

• Pen-and-paper

Academic English Screening Test

(AEST)–UniMelb• Online

Screening test



• Pen-and-paper

PHARMACY – DIAGNOSTIC WRITING ASSESSMENT

Administered to all new incoming undergraduatesPen-and-paper assessment of writingWriting task based on pharmacy/pharmaceutical

sciences-relevant topicsWriting assessed by human raters (single marked due

to costs) using very detailed diagnostic criteria

PHARMACY – DIAGNOSTIC WRITING ASSESSMENT

All students, regardless of background, are asked to sit the assessment during class time

Students receive these detailed feedbackdescriptors three days following administrationStudents are also grouped into one of three groups

(proficient, borderline, at risk)Lowest group receives support – workshops are

designed to align with the key areas of feedback on the sheet


Warrants PharmacyInstructions and tasks are clear to all test takers Test is pitched at appropriate difficulty level Test tasks/items discriminate consistently between more or less able candidates

Scoring is consistent /?Test administration conditions are clearly articulated and appropriate Evaluation /?

RELIABILITY OF RESULTS ACROSS ADMINISTRATIONS

Warrants PharmacyDifferent test forms are parallel in design Appropriate equating procedures are used to ensure equivalency across forms n/aSufficient tasks are included to provide stable estimates oftest taker ability

Test administration conditions are consistent Evaluation /?

RELEVANCE OF PELA TO DOMAINWarrants PharmacyCharacteristics of test tasks are similar to those required of students inthe academic domain

Scoring criteria and rubrics capture relevant aspects Test results are good predictors of language performance in academic domain /?Scores derived from the test provide sufficient information about candidates’ academic language proficiency

/?Performance on PELA relates to performance on other assessments of academic language proficiency Evaluation


Warrants PharmacyStudents are correctly categorised based on their test scores /?The test results include feedback on test performance and a recommendation

Recommendation is closely linked to on-campus support

Assessment results are distributed in a timely manner Test results are available to all relevant stakeholders Test users understand the meaning and intended uses ofscores

Research underway

Evaluation

CONSEQUENCESWarrants Pharmac

yAll targeted test takers sit for the test The test does not result in stigma or disadvantage for students

Test takers’ perceptions of the test and its usefulness are positive The feedback from the test is useful and directly informs their future learning Students act on the test recommendation Follow-up language development options provided for students are appropriate and ongoing if needed Learners who take up support options improve their English over the course of their studies tbcStudents who fail to act on the recommendations are more likely to struggle in their academic studies tbc

Evaluation

SUMMARY OF PHARMACY PELAPharmacy

Characteristics of test, scoring and testadministration

/?

Reliability of results /?Relevance of PELA domain /?Appropriacy of decisions Beneficial consequences

IMPACT OF POLICY ON EVALUATION OUTCOME

Institutional policy plays a key part in determining evaluation outcomesAEST UniMelb trial AEST University XCharacteristics of test, scoring and test administration

Reliability of results Relevance of PELA domain /? /?

Appropriacy of decisions /? /?

Beneficial consequences /?

PRACTICALITYBeneficial consequences



Reliability of resultsCharacteristics of test, scoring and test administration

Practicality

Low-stakes environment with little funding and large test taker populations

PRACTICALITY – SUMMARY OF THREE PELAS

DELA AEST PharmacyCharacteristics of test, scoring and test administration

/?

Reliability ofresults

/?Relevance of PELA domain

/? /?

Appropriacy ofdecisions /?


TIMING AND DIRECTIONALITY OF PELA VALIDITY ARGUMENT




Reliability of results


scoring and test

administration

IN SUMMARYPELA Evaluation involves a number of levelsPolicy in which PELA is embedded has

outcome on evaluation

Practicality

Evaluation should be fore fronted at the time of PELA design and policy formation

Thanks• Thanks to AALL for awarding the grant to bring these two speakers to Perth

• Thanks to the AALL membership, without whom this even would not be possible

• Thanks to the WA-ALL Subcommittee – Steve Johnson - Murdoch, Meriel Griffiths – the Uni of WA, and Jo Ashton Edith Cowan Uni for their assistance in organising this event

• Thanks to my faculty counterparts Trish Dooey, Janine Rutledge and Reva Ramiah – who keep my feet on the ground and my head above water – thank you

• I’d also like to thank Craig Zimitat and Sonia Ferns from Curtin Learning and Teaching, Celia Cornwell, the Dean International Curtin Health Sciences, Jana Brock, the DVCE academic Murdoch University, Siri Barret-Lennard from Study Smarter at the University of WA and Michael Stein from Edith Cowan Learning and Teaching for their generous contributions to this event

• Finally, please join me once again in thanking our speakers who have been so generous in coming here to share their knowledge and expertise with us

Introduction - Welcome to AALL | Association for Academic...

Documents

Transcript of Introduction - Welcome to AALL | Association for Academic...