Language Proficiency Testing in Aviation - Language Testing€¦ · 2 – it’s a language test! 3...
Transcript of Language Proficiency Testing in Aviation - Language Testing€¦ · 2 – it’s a language test! 3...
Language Proficiency
Testing in Aviation – 10 years of
challenges and solutions
Neil Bullock,
International Civil Aviation English Association
Introduction
Communication
‘a code that is used in a
very restricted context and
register associated with
distinctive probabilities of
discourse functions ’ Alderson 2009
Intro to LPRs
• Testing – initial & recurrent
• Pilots and ATCOs
• Speaking & listening only
• 5 Holistic Descriptors
• 6 skills
• 6 levels
• Level 4 minimum
• HIGH STAKES TESTING
ICAO Doc9835
Communicative competence
Bagarić, (2007)
A specific purpose language test ... Allows us
to make inference about a test taker’s
capacity to use language in the specific
purpose domain (Douglas, 2000)
1 – standardisation
2 – it’s a language test!
3 – ambiguity
4 – face validity
5 – testing skills
6 – test driven system
7 – fairness
Key challenges
A group of pilots in Korea
seriously doubted whether
the proficiency test they
were obliged to take
corresponded to the real
tasks and demands of their
job. Kim (2013).
English proficiency …
is only one of the
many factors that
influence performance
in a professional
setting (Kim, 2018)
… miscommunication
between pilots and air traffic
personnel could not be attributed
solely to
limited English proficiency on
the part of non-native speakers
of English (Kim & Elder 2009)
Solution in Testing 1 : TD in CH
The challenges - LP exams for pilots
• Disconnect – TSP / TTs = zero face
validity.
• Poorly produced non-contextual material.
• Little test security.
• No sample material.
• Lack of testing expertise.
• Limited training.
Solution in Testing 1 : TD in CH
The solutions: 2010 >
• Test development team
• Weir / Fulcher /Alderson
• TD plan & Spec
• SMEs – real world + Disc. Analysis
• Training, research and studies.
• Statistics > Analysis > Validity & Reliability
• RTF > Trialling > Samples
• Feedback > Engaged with stakeholders
• 3-year review
Voice only interaction (role play) = 0.84
Positive co-efficients From TT feedback
Face-to-Face Interview = 0.90
Listening Test = 0.91
Exam Organisation = 0.96
After the Exam = 0.94
Overall = 0.91
• 280/551 TTs = 51% response
Standard Setting towards validity 2017-2018
Construct Context Cognitive Content Face Consequential
0.80 0.80 0.81 0.81 0.83 0.81
VALIDITY
Co
nst
ruct
Co
nte
xt
Co
gnit
ive
Co
nte
nt
Face
Co
nse
qu
enti
al
Scores out of 5 (5 = fully agree / 1= Fully disagree)
1 I found the sample test material easy to download ✔ ✔
2 I found the sample test material useful ✔ ✔ ✔ ✔ ✔ ✔
Speaking Ability Exam – VOI Simulated Flight
3 The role-play reflects real operational situations ✔ ✔ ✔ ✔ ✔ ✔
4 The material content for the role-play is appropriate for
the test ✔ ✔ ✔ ✔ ✔ ✔
5 The tasks focus on non-routine & unexpected
operational events for pilots. ✔ ✔ ✔ ✔ ✔ ✔
6 Preparation time for the role-play is appropriate ✔
7 The duration of the role-play is appropriate ✔ ✔
78 TTs (59%) 31 questions
• 30+ years
• 600 members in 80 countries
• Board – Pres + 4 VPs
• Support with testing and training
• Involvement with ICAO/EASA
• Presentations & Papers
• Social media & Linked in forum
Solutions 2 – ICAEA
Solutions 2 – ICAEA Projects
• Test Design Guidelines project
• https://drive.google.com/open?id=
1VJvzXTAxkt-zxJwxrJOJNQl-
aUyWwu-W
• Help & guidance:
o LPR test design
o Select and audit tests
o Design & improve tests
o Confidence and face validity
o Standardisation
ICAEA Workshops
Roll-out phase:
1. 2019 and 2020 (2.5 days / 50 max):
Regional participation
• Europe (Luxembourg)
• Asia (Thailand)
• Americas (Argentina)
• Middle East (TBC)
• Africa (TBC)
2. TDGs (published online): 2019
3. ICAEA support – ongoing 2019/20
The Future of the LPRs & testing…
• Assessment literacy
• Domain communication
• Interactive competences
• Socio-cultural competences
• L1 awareness
• Research, papers & books chapters
• Using skills and expertise
• Engaging with all stakeholders
• Building relationships EASA/ICAO/TF
References
• Alderson, C. J. (2009). Air safety, language assessment policy, and policy implementation: The case
of aviation English. Annual Review of Applied Linguistics, 29, 168–187.
• Bullock, N. (2015). Wider considerations in teaching speaking of English in the context of
aeronautical communications, IATEFL ESPSIG Journal, 45, 4-11.
• Bullock, N. (2015). Defining meaningful material for the teaching of English for
• aeronautical communications, in A. Borowska & A. Enright, A. (Eds.) Changing perspectives on
Aviation English training. Warsaw : Uniwersytet Warszawski.
• Douglas, D. (2000). Assessing language for specific purposes, Cambridge: CUP.
• International Civil Aviation Organisation. (2010). Doc 9835 Manual on the implementation of ICAO
language proficiency requirements. Montreal: ICAO.
• International Civil Aviation Organisation. (2007). Doc 9432 Manual of radiotelephony. Montreal: ICAO.
• Kim, H & Elder, C. (2015). Interrogating the construct of aviation english: Feedback from test takers in
Korea. Language Testing, 32 (2), 129-150.
• Kim, H. (2013). Exploring the construct of radiotelephony communication: A critique of the ICAO
English testing policy from the perspective of Korean aviation experts. Papers in Language Testing
and Assessment, 2 (2), 103–110.
• Moder, C. & Halleck, G. (2009). Planes, politics and oral proficiency: Testing international air traffic
controllers. Australian Review of Applied Linguistics, 32(3).
• Paltridge, B. & Starfield, S. (2013). The Handbook of English for Specific Purposes, Chichester: John
Wiley and Sons Inc.
• Weir, C.J. (2005) Language Testing and Validation – An Evidence Based Approach. London:
Palgrave Macmillan.