Gillian Lord

University of Florida

glord@ufl.edu

The Promise of Online Language Learning Programs:

Myth or Reality?

• Powerful, omnipresent marketing by stand-alone online language programs

• Hyperbolic claims to be the only / fastest / easiest / best/ most ___… way to learn a language

• K-12 (and higher ed!) programs feeling threatened…

Why this topic?

Why Rosetta Stone?

SOU

RC

E: “

Cro

wd

sou

rcin

g D

rive

s La

test

Dis

rup

tive

Tec

h in

O

nlin

e La

ngu

age

Lear

nin

g.”

Forb

es. M

arch

24

, 20

15

.

“Rosetta Stone … spent $98.5 million on advertising in 2011, up from $70.5 million in 2010,

according to Kantar Media” www.nytimes.com/2012/06/20/business/media/rosetta

-stone-ads-emphasize-fun-not-efficiency.html

Why Rosetta Stone?

GOAL OF PRESENTATION:

• Provide linguistic evidence to be able to assess these kinds of marketing claims

• Offer multiple data sources to consider the advantages and disadvantages of this kind of program

• Take as a case study example Rosetta Stone for learning (Spanish as) a second/foreign language

or

1. Professional assessments

2. (Previous) Empirical studies

3. Student attitudes (a priori)

4. Perceived quality of materials

5. Affective factors

6. Outcomes: (“Does it work?”)

a) Self-perceived communicative abilities

b) Quantitative measures of communicative abilities

c) Qualitative measures of communicative abilities

How to assess the program?

1. Professional assessments

2. (Previous) Empirical studies

3. Student attitudes (a priori)

4. Perceived quality of materials

5. Affective factors

6. Outcomes: (“Does it work?”)

a) Self-perceived communicative abilities

b) Quantitative measures of communicative abilities

c) Qualitative measures of communicative abilities

How to assess the program?

University of Florida students enrolled in Beginning Spanish 1 (avg. age = 20)

• L1 English

• No other L2 proficiency (beyond h.s. requirement)

• Course designed for those with NO prior Spanish instruction

3 environments:

• Classroom (C): N=4

• Rosetta Stone (RS): N=4

• Classroom+Rosetta Stone (RS+C): N=4

My data – Participants

Original population had 20-25 participants in

each of the three groups.

Classroom

• In-tact section

• Followed regular syllabus with standard materials

My data – 3 environments

Rosetta Stone • Self-selected (required by

IRB) • Not required to attend any

regular class • Used RS package

(“Conversational Spanish”): – 16-week course designed

to cover material comparable to a face-to-face beginning class

– 6 units of Rosetta Stone® Version 4 TOTALe® Spanish, each has 4 lessons [Level 1, half of Level 2]

– Minimum of 6 Rosetta StudioTM sessions

– Minimum of 8 hours in Rosetta WorldTM

– Monitoring of program access and time on task

RS + Class • In-tact section of Beginning

Spanish class

• Same instructor as Classroom group

• Used Rosetta Stone materials as their textbook

– (including all features described for RS group)

Quantitative Data

• Portion of Spanish CLEP test (grammatical competence)

• Versant Proficiency Test in Spanish (oral proficiency)

• Attitude survey (Likert, style, pre- and post)

• Acoustic analysis of vowel production in Spanish interviews o 10 tokens of /e/ extracted from each participant’s interview.

Tokens separated for male and female

Each formant measured at midpoint using Praat

F1 and F2 extracted using a script and compared to standard native values

Data used here (1)

Qualitative Data • Fluency analysis of Spanish interviews

o Total number of words spoken o Number of Spanish words; Number of English words o Number of dysfluencies o Lexical density (number of unique Spanish words) o Number of fillers/non-lexical items

• Content analysis of English interviews o Affective factors o Reactions to instructional materials o Student perceived learning o Language communication and use o Other specific language problem or comment

Data used here (2)

PROFESSIONAL ASSESSMENTS OF THE ROSETTA STONE PROGRAM

Criterion 1:

• Godwin-Jones (2007, 2009) o Traditional computer-based training programs often informed by

technology specialists, not SLA specialists

o Rosetta Stone provides visualization feedback for specific sounds (pronunciation); mostly accurate

• Lafford & Sykes (2007) o Evaluate if programs provide the tools necessary for effective language

learning, based on features that research has shown to be important (interaction, relevant contextualization of language, etc.)

o “… these products do not incorporate a number of the [necessary] research-based insights (e.g., the need for culturally authentic, task-based activities) that informed SLA scholars might have given them.”

Professional Assessments (1)

• Santos (2011) o Lack of context

o General inability to respond to spontaneous student speech

o What Rosetta Stone calls interaction is “a rather poor and limited version of what one would encounter in a real-life conversation”

• DeWaard (2013) o Based on personal experience, professional reactions

o “Not a viable replacement of current instruction at the postsecondary level”

Professional Assessments (2)

EMPIRICAL STUDIES INTO THE OUTCOMES OF THE ROSETTA STONE PROGRAM

Criterion 2:

• Vesselinov (2009) o Commissioned by Rosetta Stone; RS beginning users

demonstrated increased knowledge of the language after 55 hours of use.

• Nielson (2011) o Self-study programs in workplace; some success among

committed users, but extreme attrition.

• Stevenson & Liu (2010) o Use of social interaction in web-based language learning tools;

lack of ability to engage learners in true interaction; users do not take advantage of network opportunities.

Empirical studies

STUDENT ATTITUDES TOWARDS THE ROSETTA STONE PROGRAM

Criterion 3:

• Why did you volunteer for the Rosetta Stone class-replacement option? o I heard a lot of good things about RS and wanted to

try it.

o Can better manage my time and schedule and move more at my own pace without dealing with class.

o Sounded more beneficial.

o I was going to use my own to supplement education anyway.

Background questionnaire

PERCEIVED QUALITY OF THE ROSETTA STONE MATERIALS

Criterion 4:

• Comments from English interviews coded for mentions of the Rosetta Stone materials o 42/181 comments in RS group

o 80/124 comments in RS+C group

o Difficult to classify as categorically positive or negative

• Main themes: o Technology [Flexibility, Ease of use, Glitches]

o Approach to language learning

Student Impressions of Materials

• Like you’re able to kind of do it like on your own time, you know, I’m not like restricted.

• It’s nice not to go to class.

• I have always preferred to learn language, like, on my own.

• It’s more flexible with my schedule.

Sample student comments (Flexibility)

• Because it’s a lot of visual stuff, and I feel like I’m a visual learner.

• It’s pretty easy to use. • World is good, I do the games. • Oh, and you can do it on your

iPad … so I’d do that a lot, like take it to people’s houses, and they’ll all be like watching TV, and I would be like, doing my Rosetta Stone.

• I like how it’s like uh, more like a game, so I’m more willing to actually do it…

Sample student comments (Ease of Use)

• It was just kind of a lot harder [to use] than I expected.

• I just didn’t show what words I needed to use before it.

• I [don’t like] the lack of human interaction.

• Sometimes it’ll show the person speaking, and sometimes it’ll say like he or she, and sometimes it’ll be I. And I couldn’t tell the difference.

• …but it is tedious, a lot of it.

• I’m still having problems with the computer.

• I can’t get the microphone to work…

• I was doing my Studio session and… I had no audio, like, I could hear them, but they couldn’t hear me the whole time.

Sample student comments (Tech glitches)

• I feel like it’s more like how you naturally learn the language instead of like, “These are … your vocabulary words this week”.

• I think the Studio session was the most interesting thing.

• Speaking it for sure [is hardest], because when you see it on paper it’s a little bit… but when it gives you four options, it’s easier to be like, it’s one of these four.

• The program is really good with like teaching like vocabulary.

• With like vocabulary, it’s like really good, and you get by.

Student Impressions (Language learning in general)

• It’s just like the grammar, and how to like, put it together.

• You can’t ask questions. • You don’t get any writing, and then all

of a sudden there’s one writing thing. • You have that whole grammar and

conjugation issue on Rosetta, because they don’t really explain it.

• You really need to have communication with a real person.

• I would enjoy getting more grammar lessons … just to get a foundation of knowledge, then building up on that.

• Rosetta Stone doesn’t like, I don’t know, hammer it down for you like they would in a normal grammar class.

AFFECTIVE FACTORS AND EMOTIONAL REACTIONS TO THE ROSETTA STONE PROGRAM

Criterion 5:

• Comments from English interviews coded for any mention of affect/emotion o Related to using the materials, learning, etc. and how

the learners felt in that respect

• 33/181 comments in RS group o 8 positive towards RS, 8 negative, 17 unrelated

• 24/123 comments in RS+C group o 2 positive towards RS, 9 negative, 13 unrelated

Affective factors

• I don’t have a problem with staying focused on my computer at all.

• I like not having a textbook to worry about.

• I like it a lot.

Student reactions/emotions

• I’m kind of struggling. • I didn’t like the stories. • I’m always just frustrated

because I’m like, I don’t really understand it.

• I feel like it should be more structured.

LINGUISTIC OUTCOMES OF THE ROSETTA STONE PROGRAM a) Self-perceived communicative abilities

Criterion 6:

• Comments from English interviews coded for any mention of perceived learning or outcomes

• 34/181 comments in RS group o 9 positive assessments, 22 negative, 3 unrelated to

materials

• 14/123 comments in RS+C group o 8 positive assessments, 3 negative, 3 unrelated

Perceived learning

• I understood like the vocab. • I feel like I’m slowly learning

and progressing to learn the language.

• I do like the studio sessions… When I do do them, like I feel like it helps me more.

• I can understand almost always what is being said.

• Learning a language just like on your computer as opposed to like in person can actually be effective.

Perceived learning

• Rosetta Stone doesn’t give you too much, like, actual instruction so you don’t learn.

• I don’t know how to use words in Spanish.

• I’m just kind of like nervous about going back to class.

• I don’t feel as confident as I did at that last meeting, honestly.

• We didn’t ever have to like memorize the conjugations, so I don’t know them, so I can’t say that it was super effective.

• It was just kind of a lot harder than I expected.

Significant decrease among RS on item #11:

“Interacting via chat or telephone is comparable to interacting face-to-face.”

Significant increase among RS and RS+C on item #19:

“I would prefer to learn a language on my own time and at my own pace than in a group or classroom setting.”

Perceived learning (pre-post changes on relevant items from attitude survey)

LINGUISTIC OUTCOMES OF THE ROSETTA STONE PROGRAM b) Empirical measurements of communicative abilities

Criterion 6:

CLEP test Average scores (converted to %)

38.65 39.17 47.50

0.00

10.00

20.00

30.00

40.00

50.00

60.00

70.00

80.00

90.00

100.00

Classroom Rosetta Stone RS+class

p = 0.165

RS = Classroom = RS+Class

27.08 26.25 20.00

0.00

10.00

20.00

30.00

40.00

50.00

60.00

70.00

80.00

90.00

100.00

Classroom Rosetta Stone RS + Class

Versant test Average scores (converted to %)

p = 0.620

RS = Classroom = RS+Class

• No significant difference between groups or test times. • Classroom males show

slight trend toward more native-like F1 values in final interview.

Pronunciation Acoustic analysis of vowel /e/

RS = Classroom = RS+Class ?

LINGUISTIC OUTCOMES OF THE ROSETTA STONE PROGRAM c) Discourse analysis of communicative abilities

Criterion 6:

INTERVIEWER: Cuéntame, ¿qué te gusta hacer en tu tiempo libre, o los fines de semana? SL: Repitas, please. INTERVIEWER: ¿Qué te gusta hacer? SL: Qué te gusta hacer… INTERVIEWER: ¿Te gusta ir a películas? ¿Te gusta escuchar música? SL: Uh, ¿fin de semana? INTERVIEWER: Sí. SL: Uh, sí. En fin de semana, yo… yo estudio, uh, mucho. INTERVIEWER: ¿Sí? SL: Uh, para mis exámenes. Sí. Yo tengo muchos examines en química orgánica, biología, y laboratorio. Uh, sí. Mucho, uh… no, muy ocupado. So, no películas, no, uh, deportes. INTERVIEWER: ¿Cuál fue la última película que viste? SL: Cuál te… INTERVIEWER: La última vez, the last time, que viste una película. SL: Phew… Hmm. Let’s see… dos menses. INTERVIEWER: Meses, mhm. SL: Meses. Ago. ¿Cómo se dice “ago”? INTERVIEWER: Hace. Hace dos meses. SL: Hace, sí. INTERVIEWER: Wow. SL: Yo no… yo no veo muchas películas en Gainesville. INTERVIEWER: ¿Qué película fue esa, hace dos meses? ¿Cómo se llamaba? SL: Uh, el pelí—la película… ¿cómo se dice “was”? INTERVIEWER: Era, o fue. SL: Era. La película era… INTERVIEWER: ¿No te acuerdas? SL: Yo no… sí.

Final interview - Classroom

INTERVIEWER: Mhm, ¿y qué haces en Gainesville? SH: Um… you’re going to have to forgive me, my mind’s like blown… Um, yo estoy estudiar. INTERVIEWER: ¿Tú estudias? ¿Y qué más? SH: Yo trabajo en un restaurante de Dragonfly. INTERVIEWER: Y, ¿con mucha frecuencia, vas de compras? SH: Yo no entendí, repetirlo, por favor. INTERVIEWER: ¿Con mucha frecuencia, vas de compras? “Ir de compras” significa go shopping. SH: All right, say that one more time, please. INTERVIEWER: ¿Con mucha frecuencia, vas de compras? SH: Uh, no, uh, no voy a… what did you, how did you say “to go shopping”? INTERVIEWER: Ir de compras. SH: No voy de compras. INTERVIEWER: Y, ¿qué vas a hacer este verano? SH: Este verano, yo voy a visitar Brazil. INTERVIEWER: Vas a visitar Brazil, y ¿vas a estudiar en Brazil? SH: No, um, yo voy a trabajar en Brazil. INTERVIEWER: Y, em, ¿qué más vas a hacer en Brazil? ¿Vas a leer, vas a jugar deportes? SH: What am I going to do in Brazil? I thought I just answered that. INTERVIEWER: ¿Solo trabajar? SH: I don’t know, I’m going on a missions trip, I don’t know how to express that in Spanish, but… INTERVIEWER: Pues, buena suerte, muchas gracias.

Final interview – Rosetta Stone

Ratio of L1/L2 words

0.26

0.83

0.68

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

Ratio of English-to-Spanish words used, by group

Control Average

RS + class Average

Rosetta Stone Average

0 = no English words produced 1 = 1 English word produced for every Spanish word

Assistance requests

2.06

1.03

0.11

1.75

0.61

3.24

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

# Clarification requests in Spanish # Clarification requests in English

Average # of clarification / assistance requests by group

Control Average RS + class Average Rosetta Stone Average

SUMMARY

1. Professional assessments 2. Empirical studies 3. Student attitudes (a priori) 4. Perceived quality of materials 5. Affective factors 6. Outcomes:

a) Self-perceived communicative abilities b) Empirical measurements of communicative

abilities c) Discourse analysis reflecting communicative

abilities

Myth or Reality?

GROUP Completion

Rate Average

Score Total Course

Usage (hours) Total Class

Time (hours)

Classroom 96.99% 90.77% 70.00 39.00

RS+C 93.67% 98.63% 32.81 37.25

RS 97.67% 95.88% 30.69 NA

What about time on task?

But…

IN CONCLUSION…

• Further professional assessments of these types of programs, as new tools and new features are developed

• Continuing analysis of current data

• Empirical studies of outcomes and effectiveness o Attitudes and reactions

o Linguistic outcomes

o Larger, more diverse populations

o Consider culture, sociocultural aspects

1. More research is needed.

In certain circumstances: o Introduction and exposure to new languages

o Refresh skills previously acquired

o Excellent vocabulary presentation and practice

o Flexible for varied populations

o Highly motivated/diligent learners more apt to make progress

… and are undoubtedly better than nothing!

2. Online (stand-alone) programs can be effective.

Based on multiple data sources, programs like Rosetta Stone do not appear to be necessarily any more …

than other methods, such as common classroom-based approaches, immersion programs, etc.

3. BUT…

4. As language experts, then, we should…

… not believe everything the ads promise! o There is a significant divide between what the ads

promise and what the programs seem capable of

… keep an open mind. o New tools and capabilities emerge daily

o Knowledge of another language is always valuable

… work to develop criteria for assessing these programs.

o [at tomorrow’s workshop!]

Special thanks to

• UF College of Liberal Arts & Sciences

• UF CLAS Humanities Scholarship Enhancement Fund

• Laura Bradley, Lisa Frumkes (Rosetta Stone)

• Caroline Reist, Brandon Shufelt, Keegan Storrs, Diana Wade (RAs)

• Carlos Enrique Ibarra (statistics)

Thank you glord@ufl.edu

DeWaard, L. (2013). “Is Rosetta Stone a viable option for L2 learning?” Forthcoming in ADFL Bulletin.

Godwin-Jones, R. (2007). “Emerging technologies; Tools and trends in self-paced language instruction. Language Learning and Technology,” 11(2), 10-17. Retrieved 26 September 2012 from http://llt.msu.edu/vol11num2/emerging/

Godwin-Jones, R. (2009). “Emerging technologies: Speech tools and technologies. Language Learning and Technology,” 13(3), 4-11. Retrieved 26 September 2012 from http://llt.msu.edu/vol13num3/emerging.pdf

Lafford, B. & Sykes, J. (2007). “Entre dicho y hecho …: An assessment of the application of research from second language acquisition and related fields to the creation of Spanish CALL materials for lexical acquisition.” CALICO Journal, 24(3), 427-529.

Nielson, K. B. (2011). “Self-study with language learning software in the workplace.” Language Learning and Technology, 15(3), 110-129. Retrieved 26 September 2012 from http://llt.msu.edu/issues/october2011/nielson.pdf

Santos, V. (2011). “Review of Rosetta Stone Portuguese (Brazil) levels 1, 2, & 3.”CALICO Journal, 29(1), 177-194.

Stevenson, M. P. & Liu, M. (2010). “Learning a language with web 2.0: Exploring the use of social networking features of foreign language learning websites.” CALICO Journal, 27(2), 233-259

Vesselinov, Roumen. Measuring the Effectiveness of Rosetta Stone. http://resources.rosettastone.com/CDN/us/pdfs/Measuring_the_Effectiveness_RS-5.pdf.

Works Cited

Sample CLEP test items

Versant proficiency test

Instrument has been tested for reliability and validity

Versant proficiency test

Rosetta Stone interface

Rosetta Stone interface (vocabulary)

Rosetta Stone interface (grammar)

Rosetta Stone interface (pronunciation)

Rosetta Stone interface (World – “play”)

Rosetta Stone interface (World – “talk”)

Rosetta Stone interface (World – “explore”)

Rosetta Stone interface (Studio)

Attitude survey