R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre...
-
Upload
may-daniel -
Category
Documents
-
view
216 -
download
0
Transcript of R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre...
![Page 1: R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cc45503460f9498def9/html5/thumbnails/1.jpg)
RATERS TRAINING AS A SAFETY CONCERN
Sergey MELNICHENKOCompLang Aviation Training Centre
(Moscow), Deputy Director
Associate Member
ICAO LPR Implementation WorkshopRussia, St-Petersburg, May 24-26, 2011
![Page 2: R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cc45503460f9498def9/html5/thumbnails/2.jpg)
RATERS TRAINING AS A SAFETY CONCERN
Associate Member
- RATER AS A DECISION MAKER - SELECTION OF PERSONNEL - AB INITIO TRAINING - RATERS CERTIFICATION - INTER-RATER RELIABILITY - INTRA-RATER RELIABILITY - RASCH ANALYSIS - RECURRENT TRAINING - EVALUATION OF RATER’S ADEQUACY
![Page 3: R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cc45503460f9498def9/html5/thumbnails/3.jpg)
Associate Member
Raters Training as a Safety Concern
• RATER AS A DECISION MAKER• SELECTION OF PERSONNEL• AB INITIO TRAINING• RATERS CERTIFICATION• INTER-RATER RELIABILITY• INTRA-RATER RELIABILITY• RASCH ANALYSIS• RECURRENT TRAINING• EVALUATION OF RATER’S ADEQUACY
![Page 4: R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cc45503460f9498def9/html5/thumbnails/4.jpg)
Associate Member
Raters Training as a Safety Concern
Aviation English test results are felt around the globe
![Page 5: R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cc45503460f9498def9/html5/thumbnails/5.jpg)
Raters Training as a Safety Concern
Associate Member
Inadequacy in Level Evaluation
Level 3 instead of Level 4:
Loss of job and/or occupationFinancial losses for AO or ANSPMarket lossesRedistribution of air transportation
servicesLack of confidence for test-takers
![Page 6: R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cc45503460f9498def9/html5/thumbnails/6.jpg)
Associate Member
Inadequacy in Level Evaluation
Level 4 instead of Level 3:
Aviation safety is compromisedDegradation to Level 2Overconfidence for test-takers
Raters Training as a Safety Concern
![Page 7: R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cc45503460f9498def9/html5/thumbnails/7.jpg)
Associate Member
Raters Training as a Safety Concern
• RATER AS A DECISION MAKER
• SELECTION OF PERSONNEL• AB INITIO TRAINING• RATERS CERTIFICATION• INTER-RATER RELIABILITY• INTRA-RATER RELIABILITY• RASCH ANALYSIS• RECURRENT TRAINING• EVALUATION OF RATER’S ADEQUACY
![Page 8: R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cc45503460f9498def9/html5/thumbnails/8.jpg)
Associate Member
Raters Training as a Safety Concern
ICAO recommendations on interlocutors and raters qualifications
![Page 9: R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cc45503460f9498def9/html5/thumbnails/9.jpg)
Raters Training as a Safety Concern
Associate Member
ICAO recommendations on interlocutors and raters qualifications (Doc 9835)
Familiarity with ICAO documentationAnnex 1;Doc 9835;ICAO Rated Speech Samples CD;ICAO Circ 318
![Page 10: R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cc45503460f9498def9/html5/thumbnails/10.jpg)
Raters Training as a Safety Concern
ICAO recommendations on interlocutors and raters qualifications (Doc 9835)
Familiarity with ICAO documentationOperational and/or language specialist
expertiseLevel 5 or Level 6 proficiencyFamiliarity with Aviation EnglishRATERS CODE OF ETHICS
Associate Member
![Page 11: R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cc45503460f9498def9/html5/thumbnails/11.jpg)
Associate Member
Raters Training as a Safety Concern
• RATER AS A DECISION MAKER• SELECTION OF PERSONNEL
• AB INITIO TRAINING• RATERS CERTIFICATION• INTER-RATER RELIABILITY• INTRA-RATER RELIABILITY• RASCH ANALYSIS• RECURRENT TRAINING• EVALUATION OF RATER’S ADEQUACY
![Page 12: R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cc45503460f9498def9/html5/thumbnails/12.jpg)
Associate Member
Raters Training as a Safety Concern
• Theory of language testing • ICAO documents: Annex 1, Annex 10, Doc 9432, Doс 9835, Circ 318• The ICAO Proficiency Rating Scale and holistic descriptors• Correlating speech samples with the ICAO LPR Scale
![Page 13: R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cc45503460f9498def9/html5/thumbnails/13.jpg)
Associate Member
Raters Training as a Safety Concern
Raters training course should be aimed at experienced non-native and native English language teaching professionals and subject-matter experts who are currently teaching Aviation English and who want to be able to rate according to the ICAO Language Proficiency Scale
![Page 14: R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cc45503460f9498def9/html5/thumbnails/14.jpg)
Associate Member
Raters Training as a Safety Concern
• RATER AS A DECISION MAKER• SELECTION OF PERSONNEL• AB INITIO TRAINING
• RATERS CERTIFICATION• INTER-RATER RELIABILITY• INTRA-RATER RELIABILITY• RASCH ANALYSIS• RECURRENT TRAINING• EVALUATION OF RATER’S ADEQUACY
![Page 15: R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cc45503460f9498def9/html5/thumbnails/15.jpg)
Associate Member
Raters Training as a Safety Concern
• Certification by national Civil Aviation Authority or by international aviation organization is mandatory
![Page 16: R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cc45503460f9498def9/html5/thumbnails/16.jpg)
Associate Member
Raters Training as a Safety Concern
• Is geographical principle necessary?
![Page 17: R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cc45503460f9498def9/html5/thumbnails/17.jpg)
Raters Training as a Safety Concern
Associate Member
• Are quotas necessary?
![Page 18: R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cc45503460f9498def9/html5/thumbnails/18.jpg)
Associate Member
Raters Training as a Safety Concern
• RATER AS A DECISION MAKER• SELECTION OF PERSONNEL• AB INITIO TRAINING• RATERS CERTIFICATION
• INTER-RATER RELIABILITY• INTRA-RATER RELIABILITY• RASCH ANALYSIS• RECURRENT TRAINING• EVALUATION OF RATER’S ADEQUACY
![Page 19: R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cc45503460f9498def9/html5/thumbnails/19.jpg)
Associate Member
Raters Training as a Safety Concern
Different raters but same results
![Page 20: R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cc45503460f9498def9/html5/thumbnails/20.jpg)
Associate Member
Raters Training as a Safety Concern
• RATER AS A DECISION MAKER• SELECTION OF PERSONNEL• AB INITIO TRAINING• RATERS CERTIFICATION• INTER-RATER RELIABILITY
• INTRA-RATER RELIABILITY• RASCH ANALYSIS• RECURRENT TRAINING• EVALUATION OF RATER’S ADEQUACY
![Page 21: R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cc45503460f9498def9/html5/thumbnails/21.jpg)
Raters Training as a Safety Concern
Associate Member
Different times but same results
![Page 22: R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cc45503460f9498def9/html5/thumbnails/22.jpg)
Associate Member
Raters Training as a Safety Concern
• RATER AS A DECISION MAKER• SELECTION OF PERSONNEL• AB INITIO TRAINING• RATERS CERTIFICATION• INTER-RATER RELIABILITY• INTRA-RATER RELIABILITY
• RASCH ANALYSIS• RECURRENT TRAINING• EVALUATION OF RATER’S ADEQUACY
![Page 23: R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cc45503460f9498def9/html5/thumbnails/23.jpg)
Associate Member
Raters Training as a Safety Concern
Evidence of the validity of the responses
![Page 24: R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cc45503460f9498def9/html5/thumbnails/24.jpg)
Associate Member
Raters Training as a Safety Concern
Evidence of the validity of the responses
Correctional refresher course
![Page 25: R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cc45503460f9498def9/html5/thumbnails/25.jpg)
Raters Training as a Safety Concern
Associate Member
Evidence of the validity of the responses
Correctional refresher course
Administrative measures to marginal raters
![Page 26: R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cc45503460f9498def9/html5/thumbnails/26.jpg)
Associate Member
Raters Training as a Safety Concern
• RATER AS A DECISION MAKER• SELECTION OF PERSONNEL• AB INITIO TRAINING• RATERS CERTIFICATION• INTER-RATER RELIABILITY• INTRA-RATER RELIABILITY• RASCH ANALYSIS
• RECURRENT TRAINING• EVALUATION OF RATER’S ADEQUACY
![Page 27: R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cc45503460f9498def9/html5/thumbnails/27.jpg)
Associate Member
Raters Training as a Safety Concern
Annual refresher course for raters
![Page 28: R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cc45503460f9498def9/html5/thumbnails/28.jpg)
Associate Member
Raters Training as a Safety Concern
Annual refresher course for raters
New materials
![Page 29: R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cc45503460f9498def9/html5/thumbnails/29.jpg)
Raters Training as a Safety Concern
Associate Member
Annual refresher course for raters
New materials
Practical training
![Page 30: R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cc45503460f9498def9/html5/thumbnails/30.jpg)
Associate Member
Raters Training as a Safety Concern
• RATER AS A DECISION MAKER• SELECTION OF PERSONNEL• AB INITIO TRAINING• RATERS CERTIFICATION• INTER-RATER RELIABILITY• INTRA-RATER RELIABILITY• RASCH ANALYSIS• RECURRENT TRAINING
• EVALUATION OF RATER’S ADEQUACY
![Page 31: R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cc45503460f9498def9/html5/thumbnails/31.jpg)
Associate Member
Raters Training as a Safety Concern
Rater is not a lifetime profession
RATER TRAINING IS A SAFETY CONCERN
![Page 32: R ATERS T RAINING AS A S AFETY C ONCERN Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director Associate Member ICAO LPR Implementation.](https://reader036.fdocuments.in/reader036/viewer/2022062515/56649cc45503460f9498def9/html5/thumbnails/32.jpg)
Associate Member
Raters Training as a Safety Concern
Thank you