ENHANCING TRAINING AND TESTING EFFECTIVENESS THROUGH LESSONS LEARNED
description
Transcript of ENHANCING TRAINING AND TESTING EFFECTIVENESS THROUGH LESSONS LEARNED
ENHANCING TRAINING AND ENHANCING TRAINING AND TESTING EFFECTIVENESS TESTING EFFECTIVENESS
THROUGH LESSONS THROUGH LESSONS LEARNEDLEARNED
Emilia Nesheva, Ministry of Defense, Bulgaria
Greta Keremidchieva, Rakovski National Defense Academy, Bulgaria
MINISTRY OF DEFENSEREPUBLIC OF BULGARIA
ENGLISH LANGUAGE TRAINING ENGLISH LANGUAGE TRAINING AND TESTING – OFFICIAL AND TESTING – OFFICIAL
POLICY OF THE MODPOLICY OF THE MOD
BASED ON:BASED ON: DOCUMENTATIONDOCUMENTATION TRAINING PROGRAMSTRAINING PROGRAMS QUALIFIED TEACHERS – “SOFT POWER”QUALIFIED TEACHERS – “SOFT POWER” BOOKS AND EQUIPMENT – “HARDWARE”BOOKS AND EQUIPMENT – “HARDWARE”
REGULATING DOCUMENTSREGULATING DOCUMENTS
1. 1. Order of the Minister of DefenseOrder of the Minister of Defense №130 №130// 21.02.2011 21.02.2011 regarding regarding Strategy for Strategy for development of the English language development of the English language training systemtraining system
2. 2. Order of the Minister of DefenseOrder of the Minister of Defense №760/04.10.2011 №760/04.10.2011 regarding regarding selection of selection of candidatescandidates, , preliminary testing and preliminary testing and conducting English courses conducting English courses
3. 3. RegulationRegulation №Н-19/25.05.2010 №Н-19/25.05.2010 regarding regarding planning and development of language planning and development of language coursescourses..
LANGUAGE TRAINING CENTERSLANGUAGE TRAINING CENTERS
RAKOVSKI NATIONAL DEFENSE ACADEMY – SOFIARAKOVSKI NATIONAL DEFENSE ACADEMY – SOFIA
NATIONAL MILITARY UNIVERSITY – VELIKO TARNOVONATIONAL MILITARY UNIVERSITY – VELIKO TARNOVO
LANGUAGE TRAINING CENTER – SHUMEN - part of LANGUAGE TRAINING CENTER – SHUMEN - part of the the Individual Training and Education Programme Individual Training and Education Programme – – ITEPITEP,, NATO NATO Tier 1 Tier 1 Smart Defence 1.7 Project Smart Defence 1.7 Project
NAVAL MILITARY ACADEMY – VARNANAVAL MILITARY ACADEMY – VARNA
MILITARY UNITSMILITARY UNITS
ENGLISH LANGUAGE TRAINING IN ENGLISH LANGUAGE TRAINING IN RAKOVSKI NATIONAL DEFENSE RAKOVSKI NATIONAL DEFENSE
ACADEMYACADEMY
Training of officers in the Master’s degree Training of officers in the Master’s degree programprogram
Training of military personnel and civilians Training of military personnel and civilians in General and Specialized English in General and Specialized English coursescourses
Target audience in Rakovski National Defense Academy
Officers in the Master’s Program
Entry – now 1111; next year (2222)
Exit – now 2222; next year (3 2+3 2)
Officers, NCO’s and civilians in English language courses
NEW
Training program in Rakovski National Defense Academy
The Master’s Program starts with a preliminary block of 360 hours intensive English training prior to their first semester of military studies
Advantage: some military subjects could be studied in English
NEW
General and Specialized English language training
Course Length Books
ALC
Entry requirements
Expected exit outcome
Level I 18 weeks 1-12 - 45 points
ALCPT
Level II 18 weeks 13-24 45 points 1+ 1 1+ 1
Level III 15 weeks 25-30 1+ 1 1+ 1 2 2 2 2
Level IV 15 weeks 31-34 2 2 2 2 3 2+ 3 2+
Specialized courses
2-8 weeks 2 2 2 2 -
NEW
ASSESSMENT AND CONTROLASSESSMENT AND CONTROL
Preliminary TestingPreliminary Testing
((ALCPT before LevelALCPT before Level ІІ) ІІ) Entry TestsEntry Tests Book quizzes, Progress testsBook quizzes, Progress tests Final TestFinal Test, , ALCPT, STANAG 6001ALCPT, STANAG 6001
FACULTY MEMBERSFACULTY MEMBERS
Teachers – 10 University graduates withTeachers – 10 University graduates with
specializations in different countries and specializations in different countries and institutions; experienced; participants in institutions; experienced; participants in national & international conferencesnational & international conferences
Test Team members – 5 University Test Team members – 5 University graduates; trained; design, pilot & graduates; trained; design, pilot & administer STANAG Tests nation-wideadminister STANAG Tests nation-wide
BOOKS & EQUIPMENTBOOKS & EQUIPMENT
ALC Books 1 - 34ALC Books 1 - 34 Specialized textbooks – Campaign, Specialized textbooks – Campaign,
Military English, Army, Air Force, Navy, Military English, Army, Air Force, Navy, etc.etc.
Language Labs Language Labs Self-Access CenterSelf-Access Center Audio & multimedia programsAudio & multimedia programs
FINANCIAL RESOURCESFINANCIAL RESOURCES
MOD program provides financial MOD program provides financial resources for the accomplishment of Force resources for the accomplishment of Force GoalGoal Е 0356 Е 0356;;
for improvement of infrastructure and for improvement of infrastructure and learning technologies;learning technologies;
for conferences and seminars.for conferences and seminars.
Enhancing Enhancing Testing Effectiveness Testing Effectiveness
through through Computer TestingComputer Testing
Action PlanAction PlanMinisterial Order Ministerial Order № 857№ 857//1010.11.2011.11.2011
till 1 March, 2012 – approbation of the computer-assisted till 1 March, 2012 – approbation of the computer-assisted “writing” module;“writing” module;
till 1 July, 2012 – analysis of the results;till 1 July, 2012 – analysis of the results; since exam session Autumn 2012 – optional computer-assisted since exam session Autumn 2012 – optional computer-assisted
“writing” module; “writing” module; till 1 March, 2013 – approbation of computer-assisted modules till 1 March, 2013 – approbation of computer-assisted modules
for “reading” and “listening”;for “reading” and “listening”; till 1 July, 2013 – analysis of the results;till 1 July, 2013 – analysis of the results; since exam session Autumn 2013 – optional computer-assisted since exam session Autumn 2013 – optional computer-assisted
“writing”, “reading”, and “listening” modules;“writing”, “reading”, and “listening” modules; since exam session Autumn 2014 – all candidates are to be since exam session Autumn 2014 – all candidates are to be
tested on computers for the modules “writing”, “reading”, and tested on computers for the modules “writing”, “reading”, and “listening”.“listening”.
End of pen-and-paper STANAG 6001 tests.End of pen-and-paper STANAG 6001 tests.
Approbation Approbation
of the computer-assisted “writing” moduleof the computer-assisted “writing” module
In February 2012, two In February 2012, two groups of volunteers – groups of volunteers – military and civilian military and civilian personnel sat for the personnel sat for the approbation of the writing approbation of the writing test. The only requirement test. The only requirement for taking part in the for taking part in the approbation was to have a approbation was to have a STANAG 6001 certificate STANAG 6001 certificate issued during the exam issued during the exam session “Autumn 2011”.session “Autumn 2011”.
Comparability of handwritten and computer-Comparability of handwritten and computer-based writing assessments – some findings based writing assessments – some findings
in the specialized literaturein the specialized literature
Questionnaires (excerpt)Questionnaires (excerpt)
Did you find the font of the tasks friendly Did you find the font of the tasks friendly enough?enough?
Did you meet any technical difficulties Did you meet any technical difficulties during the computer-assisted “writing” during the computer-assisted “writing” test?test?
Do you think that your speed of typing is Do you think that your speed of typing is decisive for your success at the decisive for your success at the computer-assisted “writing” test?computer-assisted “writing” test?
Recommendations Recommendations from the Questionnairesfrom the Questionnaires
1.1. The font of typing to be Times New Roman 12 The font of typing to be Times New Roman 12 oror 1414..
2.2. Candidates to be able to go back to the Candidates to be able to go back to the previous task for revising or correcting previous task for revising or correcting mistakes.mistakes.
3.3. Word count to be available.Word count to be available.4.4. Tabulation for indicating a new paragraph to Tabulation for indicating a new paragraph to
be available.be available.5.5. Five minutes for a spell check to be added to Five minutes for a spell check to be added to
the time allotted for the writing test.the time allotted for the writing test.
Actions TakenActions Taken
The commercial company that had The commercial company that had developed the testing software was tasked developed the testing software was tasked to correct the faults noticed during the to correct the faults noticed during the approbation and to add the functions approbation and to add the functions recommended in the questionnaires.recommended in the questionnaires.
The STANAG testing team developed The STANAG testing team developed detailed administrative instructions for the detailed administrative instructions for the computer-assisted writing test.computer-assisted writing test.
Approbation ResultsApprobation Results
55%
12%
33%
sustained level
higher level
lower level
AnalysisAnalysis
Special attention was paid to the candidates who Special attention was paid to the candidates who attained higher or lower STANAG 6001 level at attained higher or lower STANAG 6001 level at the computer-assisted writing test. Analysis of the computer-assisted writing test. Analysis of the possible reasons:the possible reasons:
- was the rating of their pen-and-paper test was the rating of their pen-and-paper test subject to discussions? how many raters had subject to discussions? how many raters had assessed the papers?assessed the papers?
- additional English language studies?additional English language studies?- different “pre-writing” habits?different “pre-writing” habits?- rater effects in scoring handwritten and rater effects in scoring handwritten and
computer-based writing assessments? computer-based writing assessments?
LESSONS LEARNTLESSONS LEARNT(possible scoring biases)(possible scoring biases)
Typed essays generally appear to raters Typed essays generally appear to raters shorter than identical handwritten responses.shorter than identical handwritten responses.
Errors made are more evident when typed.Errors made are more evident when typed.
LESSONS LEARNT (cont.)LESSONS LEARNT (cont.)(the importance of the Interface)(the importance of the Interface)
Traditional text editing tools – cut, copy, and Traditional text editing tools – cut, copy, and paste – need to be designed to support less paste – need to be designed to support less experienced computer users.experienced computer users.
Space management – “word count” and Space management – “word count” and “print preview” should be available.“print preview” should be available.
Questions?Questions?
In this presentation have been used In this presentation have been used quotations by: quotations by:
Breland, Lee, &Muraki, 2005 Breland, Lee, &Muraki, 2005 Bridgeman & Cooper, 1998;Bridgeman & Cooper, 1998; Davis, Strain-Seymour, Lin, & Kong, 2008;Davis, Strain-Seymour, Lin, & Kong, 2008; Horkay et al., 2006;Horkay et al., 2006; Pearson, 2010; Pearson, 2010; Russell &Plati, 2001; Russell &Plati, 2001; Russell & Haney, 1997;Russell & Haney, 1997; Way & Fitzpatrick, 2006. Way & Fitzpatrick, 2006.