Elicited Imitation and Automated Speech

Recognition as a Placement Test:

Analysis Of JSL Learners’ Language Development Process And Differences Between Instructed And Self-Instructed Learners

NCOLTCL 2011Brigham Young University

Shinsuke Tsuchiya

OverviewO IntroductionO PurposeO BackgroundO Research QuestionsO Test DesignO ResultsO AnalysisO LimitationsO ImplicationsO Future Research

IntroductionAssessment and Testing using technology can be:O QuickO ReliableO Objective

Brigham Young UniversityO Problem: Placement test for incoming learners is

all multiple-choice questions (near-native learners, instructed and uninstructed learners)

“Can we use Elicited Imitation and Automated Speech Recognition as a speaking portion of the

placement test?”

BackgroundO Elicited ImitationO Automated Speech Recognition

1. Ohayoo gozaimasu ‘Good morning.’

2. _______________“the more you know of a foreign language, the better you can imitate the sentences of the language.” (p. 247) Bley-Vroman & Chaudron (1994)

A viable oral testing method (Erlam, 2006; Graham et al., 2008)

BackgroundEnglish EI

BYU PSST Research Group http://psst.byu.edu (Pedagogical Software and Speech Technology)O 1659 tests were administered 1279 ESL

subjects (Lonsdale & Weitze, 2009)O “overall comparisons between EI scores of

ESL learners and their scores on other measures of oral language proficiency are promising” (Hansen, Graham, Brewer, Brewer, & Tieocharoen)

O “EI scores can predict Oral Proficiency Interview (OPI) scores within two levels of a scale of ten. (Graham, 2006; Lonsdale, Graham, & Madsen, 2005).

BackgroundJapanese EI test using ASR

Over 800 JSL learners have taken the EI.

O the correlations between human grading and ASR grading was r=0.84 (p < 0.001)

O the correlation between Japanese EI score and OPI scores was r=0.77 (p < 0.3)

(Matsushita & LeGare, 2010)

Sound recording ソノナカニナニハイッテンノ ‘sono naka ni nani haitten no’

Student’s utterance ソノナカナニハイッテタショ ‘sono naka ni nani haitteta syo’

Evaluation: C CCCD C CCCCCS SI

Correct = 76.92% 10 ( 10) substitutions = 15.38% 2 ( 2) Deletions = 7.69% 1 ( 1) Insertions = 7.69% 1 ( 1) Errors = 30.77% 4 ( 4)

Research QuestionsCan elicited imitation and automated speech recognition grading system be used as a placement test?

RQ 1. Can they differentiate instructed beginning, intermediate, and advanced learners of Japanese?• If so, what EI items?

RQ 2. Can they differentiate instructed and uninstructed learners?• If so, what EI items?  

Test DesignO A total number of participants: 241 BYU JSL learnersO EI was taken in the computer lab in Fall 2010O 3-seconds pause before repeating back (Erlam, 2006)O 60 items (30 from textbooks, 30 from CSJ corpus) (15 min.)

O Gender-neutral contentsO no technical termsO Sentence length: 10-30 moraeO Randomized orderO Prompts (both male and female)

O ASR graded mean scores for each level as well as individual items were compared (t-test)

O Participants were divided into different groups for each research question

Test itemsO Beginning (20)Ex. 明日行きませんか。‘asita ikimasen ka’‘won’t you go tomorrow?’O Intermediate (20)Ex. 昨日は学生さんが手伝ってくれたんですよ。‘kinoo wa gakusee san ga tetudatte kureta n desu yo.’‘As for yesterday, a student helped (me) as a favor.’O Advanced (20)Ex. 日本がどんな国なのか知りたがっている人がたくさんいます。‘nihon ga donnna kuni na no ka siritagatte iru hito ga takusan imasu.‘There are many people who are eager to know what kind of country Japan is.’

Research Question 1Can EI and ASR differentiate instructed beginning, intermediate, and advanced learners of Japanese?

Instructed learners:O *1st year: beginning: 79O 2nd year: intermediate: 34 O **3rd year +: advanced: 21

*False beginners have been excluded**Native speakers and uninstructed learners who have been excluded.

Research Question 2Can EI and ASR differentiate instructed and uninstructed learners?

O *Advanced Instructed learners: 38O **1st semester Uninstructed learners in 3rd year

level: 40

*second half of 2nd year students were added to equalize number of participants.**uninstructed learners who had more than 2 semesters of instruction have been excluded.

Results (RQ 1)Beginning, intermediate, advanced learners’ mean scores were significantly different:Mean scores:O Beginning: 34.7O Intermediate: 45.3O Advanced: 58.8p-valuesO Beginning vs. Intermediate: p=0.000O Intermediate vs. Advanced: p=0.001O Beginning vs. Advanced: p=0.000

Results (RQ 1)Significant itemsAccuracy rate

All progressed:O Beginning-

intermediate40 itemsO Intermediate-

advanced25 itemsO Beginning-advanced58 items

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 590.00

20.00

40.00

60.00

80.00

100.00

120.00

100200300+ (Classroom)

Results (RQ 2)Advanced instructed learners and 1st semester uninstructed learners in 3rd year level were also significantly different. Mean scores:instructed learners: 56.6uninstructed learners: 63.4

p=0.034

Results (RQ 2)Significant itemsAccuracy Rate

17 significant itemsUninstructed learners > instructed learners (16 items)Instructed learners > uninstructed learners (1 item)

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 590.00

20.00

40.00

60.00

80.00

100.00

120.00

Classroom 202+RM 300

Analysis (Summary)1. EI and ASR used in this research can be used to differentiate:O beginning, intermediate, and advanced level

learners.O Instructed and uninstructed learners.2. JSL learners’ accuracy rate improved in each item in general:O beginning level items (beginning to intermediate)O advanced level items (intermediate to advanced)3. Uninstructed learners outperformed instructed learners.

LimitationsTestO Technological problem (background noise, mis-recording, data loss etc.) O Face validity O L2 interlanguage errors that are not

recognized by the present ASR systemO Discouragement SubjectsO Lack of advanced level instructed learners

ImplicationsEI and ASR could be used as a:O Placement test (listening/speaking portion)

only if it is accompanied by other tests. O Computer Adaptive Test (efficient time &

cost with reliable & consistent scores). O Program Review (inter/intraprogram

assessment and evaluation)O Second Language Acquisition Research

Future ResearchTest improvement:O Prevent technological problemsO Increase face validity (include comprehension tasks)O Improve ASR (dealing with L2 interlanguage)O Increase MotivationO More advanced level classroom students

Second Language Acquisition Research:O Item analysis

O Grammatical elements (i.e. case particles)O Sentence complexity and lengthO Vocabulary acquisition

AcknowledgementO Hitokazu Matsushita (Japanese EI/ASR

creator)O Ray Graham, Ray Clifford, Dan Dewey (BYU

Linguistics department)O Robert Russell, Paul Warnick and other BYU

Japanese professorsO BYU Japanese studentsO BYU JFSB Computer lab assistants

Thank you!Questions?

Shinsuke Tsuchiyas.tsuchiya777@gmail.com