Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and...
-
Upload
bryan-brady -
Category
Documents
-
view
222 -
download
0
Transcript of Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and...
![Page 1: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/1.jpg)
Language teaching and language assessment in a multilingual Europe. A
case of quality, diversity and transparency
Piet van AvermaetCentre for Language and Migration
KULeuven
![Page 2: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/2.jpg)
Economical, cultural
and social integration
(unification?) in Europe
Language Policy Context in Europe
National interests (19th
century: one nation, one
state, one language)
![Page 3: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/3.jpg)
Multilingual policy in
Europe
Language Policy Context in Europe: paradoxes
Monolingual policy in
different European
countries
![Page 4: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/4.jpg)
Positive attitude in Europe towards use of more than one language
Language Policy Context in Europe: paradoxes
Negative attitude in European countries towards use of non European languages
![Page 5: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/5.jpg)
Society handles with language diversity
in a schizophrenic way
Language Policy Context in Europe
Argue in favour of multilingualism
Demand/require the use of one language
Diversity is important and has an extra value
Unity, nation building, uniformity
![Page 6: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/6.jpg)
Language Policy Context in Europe
Striving for monolingualism is easier to argue for
Surplus value of multlingualism is difficult to make a case for; difficult to bring up arguments
Economical,practicalVague surplus value of cultural and linguistic diversity
![Page 7: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/7.jpg)
Language and dealing with languages is a continuous
balancing between
Language Policy Context in Europe
•Uniformity•clarity•transparency•economical way of thinking•consistency•straightforwardness
•Diversity•identity•be opposed to•react against
![Page 8: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/8.jpg)
There will always be a tendency towards uniformization and there will always be a counter
reaction by people
Language Policy Context in Europe
Process of language evolution and language change. Languages are not static but dynamic. It is
important to have people who stress nowadays
the value of diversity
![Page 9: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/9.jpg)
Now two major tendencies
Language Policy Context in Europe
In a climate of globalisation and economic thinking there is a tendency towards uniformization and uniformity (one language/one standard)
People and media making more and more use of different communication ‘modes’: pictures/graphs/…/ linguistic code mixing
Policy / perception / discourse Daily behaviour
![Page 10: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/10.jpg)
In the context just described, recent developments in Europe like Bologna agreement, Barcelona indicators, the CEF and the Manual have to be situated and interpreted.
Bologna agreement, Barcelona indicators, CEF and Manual
Keywords are diversity and transparency
Often interpreted and translated in many different ways:
diversity often only at level of European official languages
transparency often as uniformity and loss of specificity
![Page 11: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/11.jpg)
One concept people often forget to mention or it is often taken for granted
Bologna agreement, Barcelona indicators, CEF and Manual
quality in language teaching
quality in language assessment
![Page 12: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/12.jpg)
Define for yourself: quality, diversity and transparency
Organise teaching and assesment along these lines: levels and content
Make a clear statement
Consequences of Bologna and CEF for University Language teaching Institutes
![Page 13: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/13.jpg)
Original aims
What?
CEF as POINT OF REFERENCE
Background and role of CEF
![Page 14: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/14.jpg)
Only the overall proficiency scale is known/used
Educational misuse
Assessment misuse
Ethical misuse
Misinterpretations, misuse and impact of CEF
![Page 15: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/15.jpg)
Quality needs, context and functionality
reliability, validity, efficiency, impact/effect
Diversity (linguistic and language) and transparency
needs analysis
domains
description of aims/context/level
relate to CEF
Bologna - CEF and language teaching/assessment
![Page 16: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/16.jpg)
Instruments
CEF itself
Manual*
Population/students
ALTE/COP/checklists/minimum standards*
Bologna - CEF and language teaching/assessment
![Page 17: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/17.jpg)
Evaluation is gathering and judging (assessing) students PERFORMANCES
3 central questions:
Why?
What?
How?
Language assessment: tests and alternative assessment
![Page 18: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/18.jpg)
Shifting on 4 axes. Position on each axis depends on answer on 3 question
TIME
INVOLVEMENT IN SCORING/MARKING
TYPE OF DATA/INFORMATION GATHERED
WAYS OF GATHERING INFORMATION/DATA
Alternative assessment
![Page 19: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/19.jpg)
Alternative assessment is NEW and NOT NEW
NOT: we do it daily
NEW: try to make it more formal/explicit. Give it a higher status in making/taking decisions for reasons of reliability, validity and efficiency
Alternative assessment
![Page 20: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/20.jpg)
Two examples
ALTE Code of Practice
Checklists
Minimum standards
CEF Manual to relate examinations to CEF
Quality, diversity and transparency
![Page 21: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/21.jpg)
The ALTE Code of Practice
Aim - to guarantee fairness for the users of the examinations
Two main groups of user:
Primary users are the candidates who take the exams and whose lives are affected by the results
Secondary users include: sponsors of the candidates, teachers, parents, funding agencies, employers, language teaching institutions, etc.
![Page 22: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/22.jpg)
The Code of Practice identifies the roles of three groups of stakeholder in the testing process:
the examination developers - e.g. members of ALTE
the examination takers - primary users - who take the examinations by choice, direction or necessity
the examination users – secondary users - who require the examination for some decision-making or other purpose
Developing the ALTE Code of Practice
![Page 23: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/23.jpg)
The Code of Practice lays down four broad areas of responsibility:
developing examinations interpreting examination results striving for fairness informing examination takers
Developing the ALTE Code of Practice
![Page 24: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/24.jpg)
Quality Considerations for ALTE Members
Two aspects of Quality
Better understanding of Principles of Good Practice - theoretical concerns and knowledge
The practical application of the principles within each ALTE organisation
developing better systems managing innovation and change monitoring standards
![Page 25: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/25.jpg)
The ALTE Principles of Good Practice
Based on VRIP features Validity Reliability Impact Practicality
Plus additional feature
Quality of service
![Page 26: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/26.jpg)
How to set ALTE Quality Standards?
Quality standards
Quality Standard
In need of improvement
Satisfactory
“Best Practice Models”
Good practicet
![Page 27: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/27.jpg)
QMS approach:
Monitor quality through self-assessment
Seek confirmation that standards are being met
e.g. through peer review within ALTE
ALTE Quality Management System - QMS
![Page 28: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/28.jpg)
Discuss and agree on minimum standards
But establish “best practice” models as long-term targets
Establish desired outcomes and impacts within each member organisation
Move towards good practiceAim at continuous improvement
Applying an ALTE QMS
![Page 29: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/29.jpg)
BUT each ALTE organisation is different !
Organisation
ExamsUnit
External regulation &
fundinge.g. by Govt.
ALTE
Admin.
DecisionMaking
Unit
ALTE member institution(Internal stakeholders)
External collaboratorse.g. Item writers,Examiners, etc
Test-taking contextExternal
stakeholders
USERLearners &test-takers
Schools USERUSERUSERS
External regulation &
fundinge.g. Govt.
Test centres
![Page 30: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/30.jpg)
Adopt the QMS approach and the revised Code of Practice checklists
Apply the checklists to the 4 aspects of the Test Development and Administration Cycle
Test Design and Construction
Administration
Processing - marking, grading, issue of results
Analysis and Review
ALTE QMS
![Page 31: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/31.jpg)
Identify within each organisation:
Current strengths
Areas in need of immediate improvement
Areas for long-term development
Jointly agree on activities to support ALTE Members in raising standards:
e.g. workshops, sub-groups projects etc.
ALTE QMS
![Page 32: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/32.jpg)
Manual
Preliminary Pilot Version
Relating Language Examinations to the CEF
![Page 33: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/33.jpg)
What the brief was…..
Provide guidelines and suggest procedures to
facilitate a common understanding of the CEF
levels.
![Page 34: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/34.jpg)
Project Timeline
July 2002 Helsinki seminar
September Project approval
November Authoring group starts work
Nov-January 2003 Consultation on structure
Jan-June Consultation on content
June Consultants’ meeting (v.9)
September Preliminary Pilot version
January 2004 Standardised videos and benchmarked items
![Page 35: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/35.jpg)
The Authoring Group
Coordinator: Brian North
Neus Figueras
Sauli Takala
Piet van Avermaet
Norman Verhelst
+
Sounding board Consultants
![Page 36: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/36.jpg)
The Manual offers guidance to users to...
describe the examination coverage, administration and analysis procedures,
relate results reported from the examination to the “Common
Reference Levels” presented in Chapter 3 of the CEF,
provide supporting evidence tht reports the procedures followed to do so.
![Page 37: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/37.jpg)
The Manual ….
provides a guide specifically focussed on procedures involved in the validation of a claim that a certain examination or test is linked to the CEF,
but.........
![Page 38: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/38.jpg)
The Manual does not….
provide a general guide how to construct good
language tests or examinations, or
prescribe any single approach to constructing
language tests or examinations.
![Page 39: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/39.jpg)
Structure of the pilot version
Chapter 1: Introduction
Chapter 2: The Common European Framework
Chapter 3: Familiarisation
Chapter 4: Specification
Chapter 5: Standardisation
Chapter 6: Empirical validation
Chapter 7: Guidelines for Reporting
Appendix: Report forms
![Page 40: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/40.jpg)
To be followed by.....
Benchmarked performances
Benchmarked reading items
Reference supplement
![Page 41: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/41.jpg)
Approach adopted
Four inter-related sets of procedures which
contribute to the validation process:
Familiarisation
Specification
Standardisation
Empirical Validation
![Page 42: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/42.jpg)
SPECIFICATION OF EXAMINATION CONTENT
Internal familiarisation with CEF
Internal validity: description and analysis of° General examination content;° Process of test development;° Marking, grading, results;° Test analysis and post-examination review.
External validity: Relate° general examination description to CEF scale;° description of communicative activities tested to CEF scales;° description of aspects of communicative language competence tested to CEF scales.
CLAIM of link to CEF
![Page 43: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/43.jpg)
Graphical presentation of examination(s) related to CEF levels
C2
C1
B2
B1
A2
A1
overall listening reading spoken spoken written written
interaction production interaction production
![Page 44: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/44.jpg)
STANDARDISATION OF JUDGEMENTS
Familiarisation training
Training in assessing performance in relat -ion to CEF levels using standardised samples
Training in judging theDifficulty of test itemsin relation to CEFstandardised items
Benchmarking localperformance samples to CEF levels
Judging the difficultyof local items in relat-ion to CEF levels
Dissemination and implementation
CLAIM (reinforced on basis of specification)
![Page 45: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/45.jpg)
EMPIRICAL VALIDATION THROUGH ANALYSIS OF TEST DATA
Data collection
Internal validation:
Confirming the psychometric quality of the test
External validation:
Confirming the relationship to the CEF through an independent measure
CLAIM (confirmation on basis of specification and standardisation)
![Page 46: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/46.jpg)
VISUAL REPRESENTATION OF PROCEDURES TO RELATE EXAMINATIONS TO THE CEF
"BUILDING AN ARGUMENT"
THEORETICAL RATIONALE
QUALITATIVE VALIDATION PROCEDURE QUALITATIVE VALIDATION PROCEDURE QUANTATIVE VALIDATION PROCEDURE
SPECIFICATION OF EXAMINATION CONTENT
STANDARDISATION THROUGH TRAINING
EMPIRICAL VALIDATION THROUGH DATA ANALYSIS
Internal familiarisation with CEF Familiarisation training Data collection
Training in assessing performance to CEF using standardised samples
Training in judging thedifficulty of test itemsto CEF standardised items
Internal validity: description and analysis of° General examination content;° Process of test development;° Marking, grading, results;° Test analysis and post-examination review.
Internal validation:
To be completed
Benchmarking localperformance samples to CEF
Item difficulty Estimation to CEF
External validity: relate° General examination description to CEFscale;° Description of communicative activities tested to CEF scales;° Description of aspects of communicative language competence tested to CEF scales.
Dissemination and implementation
External validation:
To be completed
CLAIM of link to CEF CLAIM (reinforced CLAIM (confirmation on basis of
on basis of specification) specification and standardisation)
Recommended procedures encouragealignment of examinations to CEF atdiffering degrees of rigour appropriateto different testing contexts and to theextent permitted by resources andexpertise available.
![Page 47: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/47.jpg)
Contributions most welcome!
Feedback, on the reading of the Manual
Piloting, on the basis of using the Manual
A Case Study, writing up the experience in using the Manual
Collecting Performance Samples for Speaking and Writing
![Page 48: Language teaching and language assessment in a multilingual Europe. A case of quality, diversity and transparency Piet van Avermaet Centre for Language.](https://reader034.fdocuments.in/reader034/viewer/2022051412/5514a49c550346ea6e8b5b53/html5/thumbnails/48.jpg)
Language teaching and language assessment in a multilingual Europe. A
case of quality, diversity and transparency
Piet van AvermaetCentre for Language and Migration
KULeuven