Twenty Years of MALL Project Implementation: A Meta-analysis of Learning Outcomes

Jack BurstonHonorary Research FellowCyprus University of Technology

CALICO 2014

MALL Learning Outcomes: 1994-2012

• Few statistically reliable measures of learning outcomes

• Over 40% of MALL publications unrelated to language learning applications

• Pervasive methodological shortcomings

MALL Research Problems

• Sample size and treatment duration

Figure 1: MALL Project Duration

MALL Research Problems

• Sample size and treatment duration

Figure 2: MALL Student Numbers

MALL Research Problems

Minimal requirements for statistical analysis:

• 4 weeks duration (20 class days)

• 10 experimental subjects

= 109 MALL studies

MALL Research Problems

• Lack of quantifiable data:

74 studies contain no quantitative learning outcome data

= 35 MALL studies with objective learning outcomes from implementations that lasted at least a month and involved ten or more experimental subjects

MALL Research Problems

• Research design shortcomings– Student usage of mobile devices not tracked (3) – Inadequate description of the control group (3)– Uncontrolled variables (4)– Confounding variables (3) – Inadequate statistical analysis (3)

= 19 valid research studies

VALID MALL Studies: General Characteristics

Figure 3: General Characteristics of MALL Studies

VALID MALL Studies: Duration of Study

Figure 4: Duration of Study

VALID MALL Studies: Group Size

Figure 5: Group Size

VALID MALL Studies: Year of Study

Figure 6: Year of Study

VALID MALL Studies: Research Findings

Vocabulary Acquisition: 11/19L2 English8/11 SMS/MMS on basic mobile phones3/11 PDAs and smartphones

VALID MALL Studies: Research Findings

5/11 = Positive MALL advantage 3 SMS (Iran) 2 MMS (Turkey, Taiwan)

3/11 SMS Mixed results 2 Delayed post-tests contradict

initial results (China, Iran) 1 No significant difference (Iran)

VALID MALL Studies: Research Findings

3/11 Non SMS/MMS projects2 PDA-based projects positive

results (Taiwan)1 Smartphone negative

results (Japan)

VALID MALL Studies: Research Findings

Reading Competency: 4/19 All PDA-based All positive MALL results

2 L1 Spanish (Chile) 2 L2 English (Taiwan)

VALID MALL Studies: Research Findings

Listening/Speaking Skills: 3/19 Mobile devices used as audio recording tools All positive MALL results

L2 English (Taiwan)L2 Indonesian (Australia)L2 English (Cyprus)

VALID MALL Studies: Research Findings

Writing Skills: 1/19 Unspecified mobile device L2 English (Taiwan) Positive results

VALID MALL Studies: Conclusion

• Only 35 MALL studies meet minimal conditions of duration and sample size

• Only 19 MALL studies provide statistically reliable outcome data

• 15 Unequivocally positive results (80%): reading, listening, speaking and writing

• 4 No significant difference: vocabulary

MALL Studies: Fundamental Problems

• Inadequate research design• Technocentricity

Thank you for your attention!

Any questions?

MALL Learning Outcomes: 1994-2012

Burston, J. (2014). 20 years of MALL project implementation: A meta-analysis of learning outcomes. ReCALL Journal, 27(1), xxx-xxx. (available online)

Burston, J. (2014). The reality of MALL project implementations: Still on the fringes. CALICO Journal, 31(1), 43-65.

Burston, J. (2013). Mobile-Assisted Language Learning: A selected annotated bibliography of implementation studies 1994-2012. Language Learning & Technology, Special volume on Mobile-Assisted Language Learning, 17(3), 157–224.

MALL Learning Outcomes: Bibliography