Post on 18-Dec-2015
EVALUATING A TBLT SPANISH IMMERSION
PROGRAM
COLLOQUIUM: Evaluating task-based language programs
Greg Burwell – BPA – greg.burwell@dhs.gov
Marta Gonzalez-Lloret –UHM – marta@hawaii.eduKatie Nielson – UMD (CASL) – knielson@casl.umd.edu
TBLT at the Border Patrol Academy (BPA)
BPA Training Environment Spanish at the BPA (before and after) Students, instructors, and supervisors
sought practical training Developed in 2006-2007
Needs Analysis Genuine audio/video Extensive input from actual Border Patrol
agents
TBLT at the Border Patrol Academy (BPA)
First implemented in December of 2007 First cohort of 15 students graduated in
March 2008 As of August 2009, 2,813 students have
completed the program To date, the program has started 180
classes with over 3,000 students 100% pass rate
TBLT at the BPA- Curriculum
8-week program following Long’s (1985, 2005a) stages
for designing and implementing a TBLT program
7 target tasks
Several target task
types
Multiple pedagogic tasks
TBLT at the BPA - CurriculumTarget Task Types:
Extract subject
Interview
subject
Determine
citizenship status
Inspect Vehicle
TBLT at the BPA - Curriculum
Input (video, audio)
Role-play tasks
CO
MPLE
XIT
Y
+ Lab Activities + La Chamba interactive video game
TBLT at BPA- Assessment
•Computer-based language test •Performance-based assessment where trainees interact with native Spanish speaking role-players as they complete that module’s target task•One-on-one conference with instructor
Weekly formative
assessment:
•Critical scenarios with professional role-players•Students role-play key course tasks over two days•Criterion-referenced scoring•Each scenario scored by two instructors
End-of-course
summative assessment:
TBLT at BPA- Assessment
Multiple versions of end-of-module assessments
Questions considered in each iteration: Are these elements required to complete
the task? Which are critical? How do we assess language?
Global accuracy, fluency, and pronunciation ratings
Inclusion of “using appropriate Spanish” with each required task
Removal of any mention of language
TBLT at BPA - Assessment
Instructor training Video evaluation Role-play evaluation
Reliability Rubric modification
Perceived grade inflation Language versus task accomplishment Non-obligatory steps removed Average grade changed from 94.10% to
95.48%
ParticipantsGrammar-based group (GB) (n=20)TBLT group (TBLT) (n=20)
Oral narrative based on picture storyHypotheses:
• TBLT will perform better on measures of fluency, lexical complexity and structural complexity
• GB will perform better on measures of grammatical accuracy
Pilot Study – Oral narration task
Pilot study- cont
Compared on: Fluency [syllables/second] (Gilabert, 2004)
Lexical complexity [lexical words/words] (Robinson, 1995; Gilabert,
2004) [1st time use of lexical word/words] Ratio of lexical/functional words (Gilabert, 2004)
Structural complexity [words/utterance] (Ortega, 1999)
Grammatical accuracy TLU of noun-modifier agreement (Ortega, 1999) TLU of noun-verb agreement
Pilot study- Results - Fluency
GB TBLT0.00
0.20
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00sy
llab
les/
seco
nd
Pilot study- Results – Lexical complexity
Lexical complexity 1 Lexical complexity 2 Lexical complexity 30.00
0.20
0.40
0.60
0.80
1.00
1.20
GBTBLT
Lexical words/words
1st use lexical w/words Ratio of lexical/ functional words
Pilot study- Results- Structural complexity
GB TBLT0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
8.00
GBTBLT
#of
word
s/ u
ttera
nce
Pilot study- Results –Grammatical Accuracy
Grammatical accuracy 1 -(S-N agrement) Grammatical accuracy 2 (S-V agreement)0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
GBTBLT
TLU of noun-modifier agreement TLU of noun-verb agreement
Pilot study - Discussion
Very small n-size due to difficulty
obtaining samples
Significant effect for fluency and
structural complexity (.05)
Students from TBLT course tend to produce more fluent and complex speech that is less grammatically accurate
but contains a greater variety of lexical items.
ParticipantsFirst 256 TBLT students
Performance on computerized oral proficiency interviewHypotheses:
• TBLT will improve students’ overall proficiency• TBLT will affect overall proficiency regardless
of language abilities at the start of the course
Ordinate study
Ordinate Study - continued
Ordinate (Versant) Computerized Oral Proficiency Exam Used at BPA as a placement test Scored between 20 – 80
Overall Sentence Mastery Vocabulary Fluency Pronunciation
Students took the test before and after the 8-week TBLT program
Ordinate Study - Mean Scores
Overa
ll
Sent
ence
Mas
tery
Voca
bula
ry
Flue
ncy
Pron
unciat
ion
0
10
20
30
40
50
Pre-testPost-test
Ordinate Study – Change Scores
* Overall:Mean: 7.52SD: 5.42
* Sentence mastery:
Mean: 7.55SD: 8.21
* Vocabulary:Mean: 7.16 SD: 10.99
* Fluency:Mean: 5.86SD: 7.06
* Pronunciation:
Mean 3.53SD 4.92
All change scores
significant at .01
Ordinate Study - Discussion
Hypothesis #1
Overall proficiency will improve as a result of TBLT
Yes: Post-test means improve an average of 7.5 points overall on an
unrelated, standardized proficiency test
Students’ evaluations
Electronic (anonymous) surveys 2 Groups:
Pre-experienced
Current students of the TBLT Spanish
program
n=41
ExperiencedCurrent BA Agents trained with TBLT Spanish program
n=14
In general students liked the best… The work done with the role-players. The direct application of the class to their job.
“Role playing is very helpful. I feel that it helped me learn more spanish (sic) and become more comfortable speaking it.”
“I enjoyed the hands on experience the most. Being able to read and write is important when learning a new language, but being able to incorporate it into your daily duties is beneficial.”
“I have enjoyed that we get to have a lot of hands on in this program. Also helps a lot for us to be able to apply all that we have learned in basic and build apon (sic) basic.”
Students’ evaluation
Students’ evaluations
Students suggestions: More conversational Spanish (not for the
job) More grammar in class More role-play work with Spanish speakers
Agents’ evaluation (experienced)
The Spanish program (topics, activities, lab) was interesting.
I completed most of the lab exercises.
The topics I studied are now useful in the field.
The vocabulary & expressions I studied are now useful in the field.
The class exercises are now useful in the field.
I apply most of what I learned in class in the field.
The lab exercises helped me learn Spanish.
The Spanish lessons gave me confidence to use Spanish in the filed.
The Spanish lessons reflect well what happens in the filed.
I played the video game La chamba at least twice a month.
The video game helped me learn important Spanish words & expressions.
0.00 0.50 1.00 1.50 2.00 2.50 3.00
Agents’ evaluations
What they found most useful from the Spanish course was: Commands and expressions Knowing the procedures and forms in
Spanish What they would have liked to learn:
More conversation (outside scenarios) More vocabulary Grammar and verb conjugation
TBLT at BPA
Students are all
passing
Overall student
proficiency improves after the
TBLT course
Student speech is
more fluent and complex
than in the old course
What does this
mean?
Students find the course
practical
•Difficulty of training instructors properly on methods and principles of TBLT•Instructors reluctant to adopt new approach•Lack of trained personnel to collect and analyze data•Lack of support for follow-up research
Difficulties with evaluating TBLT at the BPA