Mobiles for Literacy in Developing Countries: An ...
Transcript of Mobiles for Literacy in Developing Countries: An ...
University of Pennsylvania University of Pennsylvania
ScholarlyCommons ScholarlyCommons
Journal Articles (Literacy.org) Literacy.org
3-2014
Mobiles for Literacy in Developing Countries: An Effectiveness Mobiles for Literacy in Developing Countries: An Effectiveness
Framework Framework
Daniel A. Wagner University of Pennsylvania, [email protected]
Nathan M. Castillo University of Pennsylvania, [email protected]
Katie M. Murphy University of Pennsylvania
Molly Crofton
Fatima T. Zahra University of Pennsylvania
Follow this and additional works at: https://repository.upenn.edu/literacyorg_articles
Part of the Curriculum and Instruction Commons, Educational Administration and Supervision
Commons, Educational Assessment, Evaluation, and Research Commons, Educational Methods
Commons, Instructional Media Design Commons, International and Comparative Education Commons,
and the Language and Literacy Education Commons
Recommended Citation Recommended Citation Wagner, Daniel A.; Castillo, Nathan M.; Murphy, Katie M.; Crofton, Molly; and Zahra, Fatima T., "Mobiles for Literacy in Developing Countries: An Effectiveness Framework" (2014). Journal Articles (Literacy.org). 30. https://repository.upenn.edu/literacyorg_articles/30
This paper is posted at ScholarlyCommons. https://repository.upenn.edu/literacyorg_articles/30 For more information, please contact [email protected].
Mobiles for Literacy in Developing Countries: An Effectiveness Framework Mobiles for Literacy in Developing Countries: An Effectiveness Framework
Abstract Abstract In recent years, the advent of low-cost digital and mobile devices has led to a strong expansion of social interventions, including those that try to improve student learning and literacy outcomes. Many of these are focused on improving reading in low-income countries, and particularly among the most disadvantaged. Some of these early efforts have been called successful, but little credible evidence exists for those claims. Drawing on a robust sample of projects in the domain of mobiles for literacy, this article introduces a design solution framework that combines intervention purposes with devices, end users, and local contexts. In combination with a suggested set of purpose-driven methods for monitoring and evaluation, this new framework provides useful parameters for measuring effectiveness in the domain of mobiles for literacy.
Keywords Keywords literacy, mobiles, technology, developing countries, success, evidence, teachers, quality of education
Disciplines Disciplines Curriculum and Instruction | Education | Educational Administration and Supervision | Educational Assessment, Evaluation, and Research | Educational Methods | Instructional Media Design | International and Comparative Education | Language and Literacy Education
This journal article is available at ScholarlyCommons: https://repository.upenn.edu/literacyorg_articles/30
Prospects, March 2014
Open File
Mobiles for literacy in developing countries: An effectiveness framework Daniel A. Wagner, Nathan M. Castillo, Katie M. Murphy, Molly Crofton and Fatima Tuz Zahra D. A. Wagner
International Literacy Institute
University of Pennsylvania/Graduate School of Education
3700 Walnut St.
Philadelphia, PA 19104-6216
United States E-mail: [email protected]
N. M. Castillo
K. M. Murphy
M. Crofton
F. Tuz Zahra
International Literacy Institute
University of Pennsylvania/Graduate School of Education
3700 Walnut St.
Philadelphia, PA 19104-6216
United States
ABSTRACT
In recent years, the advent of low-cost digital and mobile devices has led to a strong expansion of
social interventions, including those that try to improve student learning and literacy outcomes.
Literacy and mobiles, 2
Many of these are focused on improving reading in low-income countries, and particularly
among the most disadvantaged. Some of these early efforts have been called successful, but little
credible evidence exists for those claims. Drawing on a robust sample of projects in the domain
of mobiles for literacy, this article introduces a design solution framework that combines
intervention purposes with devices, end users, and local contexts. In combination with a
suggested set of purpose-driven methods for monitoring and evaluation, this new framework
provides useful parameters for measuring effectiveness in the domain of mobiles for literacy.
Key words: literacy, mobiles, technology, developing countries, success, evidence, teachers,
quality of education
Literacy and mobiles, 3
New technologies are of growing importance around the world, and in many facets of everyday
lives and livelihoods. These information and communications technologies (ICTs), especially
mobile devices, may have special benefits for learning, both in and out of schools. At the same
time, major claims are often made about the success of particular devices, before substantial
research has been undertaken. In this article, drawing on Wagner, Murphy, and deKorne (2012)
and Wagner (2013), we explore the current state of literacy and mobiles, and recommend ways
to incorporate improved monitoring and evaluation (M&E) plans for the future.
According to a recent report by UNESCO (2013), this is the first time in history that the world
has more connected mobile devices than people. Despite the ubiquity of mobile technologies and
their increased use in educational settings, little empirical evidence supports their use for
learning. More common are anecdotal accounts that do not necessarily indicate real learning
gains or broader contextual impacts.
At the same time, available evidence from developing countries reveals that significant progress
has been made toward international goals for education. Enrolment is up on average, and in
many countries gender parity is approaching the Education for All (EFA) target of equity
(UNESCO 2012). In addition, the types of educational resources found within many schools
have evolved. ICT applications to promote learning within schools are increasingly apparent,
even in poor schools, leading to concerns about equitable access to Internet connections,
computers, and mobile phones. Such transformations in infrastructure offer a sense of future
possibility to many students and teachers in poor regions who may have earlier felt cut off from a
globalizing world.
Literacy and mobiles, 4
A major challenge remains in educational development: how best to produce quality learning
experiences. Put more directly, access to schooling is no longer sufficient, and merely being
enrolled in school does not guarantee that a student is learning (Wagner et al. 2012; also see
Brookings (2011).
Early efforts around reading and literacy instruction relied on available resources that were
limited to standard classroom inputs, such as texts, chalkboards, and workbooks—and the
teachers who implemented them. Data on outcomes were collected through observation or paper-
and-pencil assessments, and analyses were conducted at a central level with dissemination
limited to bureaucrats and policy-makers. With advances in ICT design and manufacturing,
available devices have proliferated, and they cost a fraction of earlier versions. In the
development context, mobile phones, and increasingly tablet devices, are outpacing other forms
of technology at an exponential rate (see Figure 1).
Figure 1 Growth of mobile phones, 1998–2008
Source: Adapted from Wagner et al. (2012) and ITU (2009), p. 1.
Literacy and mobiles, 5
This new reality translates into several possible benefits for education planners. First, it is now
easier to access, and experiment with, multimedia and other electronic content for subject-matter
instruction, both in and outside the traditional classroom setting. Second, data collection and
feedback take significantly less time and effort. Third, ICTs can empower communities by
making information more accessible. Given these and other appealing benefits of ICT
innovations for literacy, coupled with the diminishing costs of procurement, it is no surprise that
the interest in, and associated trends in spending levels, for such interventions have steadily
increased around the world (see Figure 2).
Figure 2 Growth of ICT spending by region, 2004–2013, in US$ trillions.
Source: Adapted from WITSA (2010, p. 14)
Literacy in the context of ICTs
Although development agencies worldwide have a variety of agendas, a child’s ability to read is
commonly regarded as the principal indicator for assessing the quality of basic education, for
both OECD member states and lesser-developed countries. In fact, the three bilateral
Literacy and mobiles, 6
development agencies that gave the most aid in 2012 according to the OECD (2013) have written
literacy into their official education strategies, either explicitly or implicitly. The United States
Agency for International Development (USAID) is focused on improving literacy among 100
million children by 2015. Similarly, the UK Department for International Development and the
German Society for International Cooperation (GIZ) prioritize quality of teaching and learning
for basic literacy and numeracy within the EFA framework.
This focus seems appropriate given the increasing need for literacy today. In most countries,
inadequate reading ability in primary school is among the most prominent predictors of future
disadvantage in terms of educational, social, and economic outcomes. For example, Levy and
Murnane (2004) found that the skills required for jobs in the United States shifted between 1969
and 1998: by 1998 they required fewer routine manual labor skills and correspondinglymore
skillsrelatedtoproblem-solvingandcommunication.Furthermore, the global shift toward
knowledge-based economies requires more literate workforces, and reading provides a crucial
foundation for continuing education.
Historically, reading and writing skills have been at the centre of all definitions of literacy. Most
definitions of reading refer to the understanding of meanings of written words and texts that
allow them to communicate messages across time and distance. For example, an early UNESCO
definition of reading states that a functionally literate person “has acquired the knowledge and
skills in reading and writing which enable him to engage effectively in all those activities in
which literacy is normally assumed in his culture or group” (Gray 1956, p. 19) Making sense of
written words and communicating through shared texts with interpretive, constructive and
critical thinking are central cognitive tasks that formal schooling tries to confer around the world
Literacy and mobiles, 7
(Wagner, Venezky, and Street 1999). To better assess reading skills, analysts have determined
the key component skills which include the alphabetic principle, phonemic awareness, oral
reading fluency, vocabulary, and comprehension (August and Shanahan 2006; USNRP 2000).
The measurement of these core skills has gained increased attention, as exemplified by the Early
Grade Reading Assessment (EGRA) administered in numerous developing countries (Gove and
Cvelich 2010).
Another approach to reading acquisition focuses more on the social practices of literacy: how
people use literacy in formal and informal contexts (Street 1999). From a socio-cultural
perspective, literacy practices must be understood in their own particular cultural context. Thus,
when a child holds a book or a smartphone with the purpose of reading, this common “practice”
may lead to diverse outcomes with particular implications for reading. That is, each specific tool
may be utilized differently depending on the user’s intent and the device’s design and capability.
A child sitting with a book might be reading a short narrative text; if she is sitting with a
smartphone, she might be reading that same text, or she might be sending a text message (SMS).
What might look like similar activities may represent different literacy practices, with quite
different impacts on learning. A key distinction here is between learning how to read, with its
emphasis on the acquisition of component skills, and reading to learn, which mainly involves
comprehension of textual materials.
Evidence is growing that the ways that ICTs are utilized are also changing the nature of learning
processes themselves. Observational studies indicate that young learners in wealthier
communities engage actively with websites, message boards, social media, and so on. If they
have a choice, they often prefer to interact socially on the Internet or a mobile rather than listen
Literacy and mobiles, 8
passively to an instructor or read a textbook (Tolani-Brown, McCormac, and Zimmermann 2009).
Others have found that students’ continuous interaction with web-based literacy activities is
having a significant impact on reading skills themselves (Leu, O’Byrne, Zawilinski, McVerry,
and Everett-Cacopardo 2009). Some have described the reciprocal nature of learning using
mobiles, such as the impact on, and use of, metacognitive skills (Terras and Ramsey 2012) and
the ways that mobiles have affected human discourse (Traxler 2009). In other words, ICTs are
changing both what gets learned and how learning takes place, and not just standard reading
outcomes. With the advent of e-books and social media, it is clear that young children with
access to mobiles already have many new opportunities and ways to learn to be literate. In
addition, given that mobiles lower the barrier of opportunity to students creating texts themselves,
students have new ways to produce texts, rather than simply consume them.
Mobiles for literacy: An effectiveness framework
Our framework considers six main purposes through which mobiles can be used to support
literacy learning. These purposes intersect with devices, and with types of end users, to create a
variety of design solutions. When combined, these form an effectiveness framework for
assessing utility and impact.
The purposes of mobile intervention
One way to conceptualize the nature of mobile interventions is to consider their intended purpose
as stated by programme designers. Sometimes the purposes of interventions are explicitly stated;
at other times they are merely implicit in the available reports. Naturally, many interventions do
Literacy and mobiles, 9
not fall exclusively within a single purpose and some combine two or more purposes within a
single intervention. In this section six primary purposes of intervention are described, drawing on
a landscape review of 51 projects worldwide (Wagner 2013).
Formal learning and instruction
This type of intervention is best characterized as intentional or directed learning. It incorporates
guided inquiry through clearly planned learning objectives in a structured format, which the end
user usually experiences through modules or units. While project documents for this type of
approach generally refer to the end user as a student, the formal learning process need not occur
in a school classroom setting; the person could also learn on a mobile phone at home or in the
community. Projects within this intervention purpose may focus on the acquisition of new
knowledge and skills, such as phonemic awareness or vocabulary.
For instance, the Bridges to the Future Initiative (BFI) in South Africa, run by Molteno (2013)
and supported partly by All Children Reading’s (2011) Grand Challenge initiative, promotes
reading acquisition through supplementary instruction using mobile tablets, as well as instruction
supported by desktop computers. Similarly, Ustad Mobile in Afghanistan (Paiwastoon 2013), is
focused on reading comprehension and listening, as well as learning numeracy. Ustad Mobile is
an application that runs offline on simple feature phones and provides material in both Dari and
Pashto from the first letter of the alphabet through grade 3 literacy and numeracy; it also includes
hours of narrated instruction, reading comprehension exercises, quizzes, educational games, and
video clips for visual learners.
Informal learning
Literacy and mobiles, 10
Informal learning may cover a range of non-directed learning objectives, such as encouraging
students to use SMS to practice reading or providing digital materials on mobiles in the local
language. Informal learning interventions offer learning materials or practices that will most
likely engage the learners outside of formal learning settings, and may simply be a by-product of
other activities on their mobile devices (Jack and Suri 2010). In other words, this intervention
purpose includes a fairly wide variety of innovations that attempt to engage the end user in
behaviors that may have additional consequences for learning.
For example, the BBC Janala (2013) project, in Bangladesh, aims to teach English language
listening and reading skills using a combination of interactive audio and mobile technology
along with print and other ICT-based materials. Three-minute audio English lessons and quizzes
are accessible through text messages using cell phones, TV programmes, newspapers, textbooks,
and CDs. In India, VideoKheti enables low-literate and novice users to find and watch
agriculture-related videos through a text-free, multimodal interface with speech, graphics, and
touch, delivered through tactile smart phones and tablets (Cuendet, Medhi, Bali, and Cutrell
2013).
Content
Some mobile interventions begin with the premise that the lack of textbooks, and reading
materials in general, leads to low-literacy environments that fail to support useful literacy
practices, or that few text materials are available in local written languages. For these reasons,
interventions that provide e-books pre-loaded with stories at varying levels of difficulty are now
becoming more commonplace.
Literacy and mobiles, 11
One such example is Worldreader, which (as of late 2012) distributed over 400,000 Kindle e-
books to students and teachers in sub-Saharan Africa (Worldreader, ILC Africa, and USAID
2012). Though the e-readers have been deployed widely, critics point out that only minimal
content as yet is available in local languages. Another project, Sesame Workshop’s M is for
Mobile initiative (Stewart 2013), has begun to focus on the potential of existing mobile
technologies to reach educators and families that may have limited access to traditional media
channels in India. Sesame Workshop’s approach, which targets children aged 2 to 8 in migrant
labor communities, allows listeners to call in to the local station and receive a call back with a
prerecorded message that provides instructions on how to listen to up to seven previous episodes
covering a variety of subjects.
Training
Training interventions contain direct or guided instruction for adults who serve as intermediaries
for children’s learning. They are usually part of efforts at teacher professional development, but
can also include facilitators or parents. The main purpose of supported training is to enhance the
pedagogical skills of the teachers, most often by introducing them to new curricular or
instructional content, such as lesson plans.
For instance, in South Sudan, Across Radio uses solar-powered digital audio players to provide
teaching resources and content in the local languages, Dinka and Bari, supported by the All
Children Reading (2011) initiative. Recordings can also function as a supplemental literacy tutor
for children in school or for children who are not in school but who listen to the recordings in
community group meetings. Also, in Mali, the PHARE programme (EDC 2011) uses Internet-
enabled mobile phones to offer primary teachers access to resources and suggested lesson plans
Literacy and mobiles, 12
for French literacy instruction. School supervisors observe the lessons based on these resources
and provide feedback as a form of ongoing professional development.
Data collection
Given the increasing emphasis on accountability in development work, some mobile projects
focus on rapidly assessing individual and cluster-level learning gains in reading (Wagner 2011).
Such technologies aim to provide real-time monitoring of what a student does and does not know,
and to show teachers where to focus their instructional efforts so they can remediate the
weaknesses they observe. Overall, relatively few projects seem to have invested in measuring
learning outcomes, or to have reported them as an impact variable. Some projects have back-end
systems that can collect information on how the learning devices are used, and can feed that
information back into analyses and planning. Such systems usually collect a broad range of data
from ICTs, such as the keystrokes used, and participation time—both of which are difficult to
deploy for real-time purposes.
For example, e-EGRA (EDC 2011–12), piloted in Mali and the Philippines, and Tangerine (RTI
International 2011), piloted in Kenya and Indonesia, have allowed for real-time, student-level
literacy assessment to identify individual strengths and weaknesses. Adapted from the paper
version of the EGRA assessment for use on laptop computers and tablets, Tangerine provides
classroom teachers and headmasters with quick summaries of students’ reading progress, and
frees them from the burden of sending off completed assessments to a central office and waiting
for data entry, analysis, and evaluation.
Communication
Literacy and mobiles, 13
Finally, a set of innovations attempts to harness the power of mobile solutions to engage in social
exchange and quick delivery of information. Though not directly focused on literacy instruction,
these efforts provide end users with the opportunity to engage in active dialogue where literacy
and language skills are practiced informally.
For example, the Pink Phone Revolution project in Cambodia uses mobile phones to empower
women, by allowing them to participate fairly in the marketplace and to improve their safety and
security (Mendoza and Vergel de Dios 2012; Yardley 2011). By using SMS messages, women
can receive current information on agriculture trends, market prices, and relevant disaster
preparedness. Incorporated into the project’s objectives is an active network of mobile users who
share knowledge and report on emergencies and cases of domestic violence. Similarly, the Yoza
Project (2009–2013) in South Africa incorporates the use of mobile phones to communicate
learning content and m-novels to create social media communities that foster literacy.
The six purposes described above are by no means exhaustive, nor are they guaranteed to persist
in relevance as the field continues to expand. Still, without a sense of purpose, there is little that
can help a project find its ‘vision’ and thus an appropriate design solution, the focus of the
following section.
Towards an effectiveness framework: Devices, end users, and design solutions
In addition to the preceding purposes, one must think in terms of two additional parameters when
making a particular design choice. First, the devices themselves must be available, adaptable,
and affordable. Second, end-user characteristics must be thoroughly understood. The design
Literacy and mobiles, 14
solution, therefore, should lie at the intersection of the purposes and these two key parameters, as
well as the context of deployment (see Figure 3). An effective design solution needs to keep each
of these dimensions in mind.
Figure 3 Mobiles for literacy: An effectiveness framework
Devices
As ICT-based approaches in learning and schooling have become more prominent, several
device-based implementations have been promoted and publicised. One Laptop per Child (OLPC
2013) sought to capitalize on the advantages of relatively low-cost and portable laptops (with
some other features) to provide one-to-one computing for children. The design was mainly the
laptop device itself; it seemed to be highly attractive and brought a great amount of media (and
fiscal) attention. A second example is the Hole-in-the-Wall (HIWEL 2013) initiative. Internet
Literacy and mobiles, 15
access was seen as the primary design approach, and little effort was made to create more than
informal and user-friendly kiosks. Its purpose was to engage with children, and the project may
or may not have succeeded in the goal of increasing Internet access, as credible data is scarce.
But a problem persists: mere access to devices and the Internet is rarely a sufficient design
solution if the learning is to have formal or informal consequences.
If local language content can be placed on a low-cost tablet, then it may help promote greater
access to the practice of reading to learn. If students are using SMS messages to communicate
about their lives, then it seems reasonable that cell phones could offer an avenue that students,
teachers, and volunteer facilitators could use to share information on school assignments or to
provide advice on math or reading questions. This application is being piloted in more recent
innovations. Worldreader-Phone, a variation of the Kindle version mentioned above, takes
advantage of content provided in feature cell phones that have the greatest penetration in poor
countries, making it available to those who do not have sufficient reading materials in their own
local languages. These are just three examples of how the device itself is a prominent component
of the implementation. In this present era of incredibly rapid change, devices are clearly very
alluring to developers, and to the public, in the broad domain of international development.
End users
Designs may fail when they turn out to be incompatible with particular population characteristics.
And, conversely, they may be hugely successful when they are designed with a particular
population in mind. Two of the projects mentioned above—OLPC and Hole in the Wall—were
targeted to young children in some obvious ways. OLPC used brightly colored laptops to appeal
to young children, while Hole in the Wall created kiosks that comfortably accommodate the
Literacy and mobiles, 16
smaller stature of a child. But other aspects of each project had limitations. OLPC offered only
limited professional development for teachers, and at relatively high cost. Hole in the Wall
provided only limited content in local languages and was not well integrated with the schools.
Regarding gender equity, several of the projects included in Wagner’s (2013) landscape review
were found to focus on participation, though they often provided little detail on how gender
influenced the design of the programmes themselves. More can certainly be done to tailor the
design of programmes than to simply distribute devices to girls and women. In contrast, some
interventions target specific ethno-linguistic groups by providing reading content in locally
appropriate languages. Strikingly, none of the projects reviewed had designs that explicitly
focused on learners who have mental or physical disabilities.
Context
End-user approaches are inevitably linked to cultural contexts. Thus, it is somewhat surprising
that most projects provide relatively few details on the take-up (in empirical terms) of the
intervention within their particular contexts. For example, if a project distributed mobile phones
to individuals to use at home outside of class hours, in contrast to using such phones in a
completely flexible way, would those specific contexts lead to differing results? Although
designers usually kept contexts in mind as they chose the setting to implement an intervention,
they rarely used information about the social ecology of a context to inform the intervention
design. One counter-example is the Bridges to the Future Initiative in South Africa; that
programme was developed in four languages, runs on a multilingual platform, and is deployed
for specific linguistic groups.
Literacy and mobiles, 17
In sum, those creating appropriate design solutions—whether they employ device-based
approaches or they focus on a specific set of end users—face most of the usual problems of any
development intervention. More specifically, educational specialists in the mobiles domain
confront the significant challenge of designing different devices for different skills for different
people at different ages in different contexts. This, of course, is what makes this domain both
challenging and rewarding, and an effectiveness framework, built around design solutions, helps
to maintain a focus on the main elements that determine utility and impact.
M&E in the domain of mobiles for reading
Much of the robust evidence available on the impact that ICT innovations have on subject matter
content is limited to OECD countries (Wagner, Day, James, Kozma, Miller, and Unwin 2005;
Wagner et al. 2012). Therefore more research is necessary to move the evidence in the
development context beyond anecdotal output findings to discussions about real outcomes for
students, teachers, and classrooms, and the broader learning environment.
Purpose-driven methods
Overall, the purposes inherent in the approach and design of an ICT intervention should drive the
methods for monitoring and evaluation. The above framework can be used to interpret how the
varied approaches to generating evidence lend themselves to a spectrum of purpose-driven
methods. On one end of the spectrum, there is an opportunity for experimental and quasi-
experimental design through randomized controlled trials (RCTs). On the other end,
ethnographic inquiry or other qualitative or observational methods are crucial for interpreting
Literacy and mobiles, 18
how technology is actually used in context.
With formal learning and instruction, methods should emphasize data collection around learning
outcomes. For example, under USAID’s current education strategy, the ability to demonstrate
reading and comprehension skills at grade level as defined by national country standards is the
outcome of interest (USAID 2012). In this context, the most appropriate research technique may
be to use experimental and quasi-experimental designs to establish causal inference between the
intervention and the impact it has on specific learning skills.
Alternatively, interventions whose primary purpose is informal learning may need to measure the
habits and behaviors of individuals as related to literacy but outside of the formal learning
environment. Here, the most appropriate data collection methods would focus on how learners
incorporate mobile devices into their daily lives to practice reading and writing. A relevant
outcome indicator would attempt to capture a child’s level of interaction with mobile content
delivered outside of the classroom.
Where content delivery is the primary purpose, indicators related to use and engagement with the
material may be most relevant. New technologies that capture data usage information and
magnitude of engagement with the material are now available and can provide rich content for
analysis (Streitfeld 2013). This is already happening with the Worldreader-Phone project, which
is collecting data on the content that users explore.
With regard to professional development, training interventions are most concerned with how
facilitators are changing their pedagogy and other learning practices, including incorporating
Literacy and mobiles, 19
student-centred and interactive lesson plans. Therefore, useful methodologies would include
those that capture changes in lesson preparation, style of instruction, and innovative uses of
materials.
Interventions that incorporate data capture technologies have been particularly beneficial within
formal learning and instruction projects for identifying real-time outcomes at the level of
students and clusters, as mentioned earlier in describing the Tangerine project.
Finally, within projects where communication is the primary purpose, research methods should
be focused on gathering dissemination practices and the ways information is used within and
between communities. Indicators would include the amount of social networking among
individuals that may enhance the practice of literacy skills, as well as transparent efforts by
communities and education planners to discuss findings and feedback opportunities.
Incorporating focus groups or survey data would be very relevant within this category.
Arguably, one of the more difficult aspects of impact investigations is deciding how much data
to collect, and what kind. Ideally, researchers collect just the right amount to assess impact, but it
is much easier to state this than to effectively put it into practice. Not surprisingly, this data
collection process is referred to as “right-sizing”, or more recently, “evidence-centred design”
(Braun and Kanjee 2006, p. 16).
Pro-equity approaches to evaluation, evidence, and effectiveness
Literacy and mobiles, 20
Despite good intentions, a disproportionate number of those on the wrong side of the educational
and digital divide are women, members of ethno-linguistic minorities, and people with special
needs. ICT mobile interventions can work to narrow that divide, through more intentional pro-
equity M&E activities (Wagner 2005). To do so, approaches should be transparent in terms of
what is being measured by demographic breakdown, while findings should be incorporated into
policy formation and expansion efforts.
Further, though many of the current efforts to use mobiles for reading and broader ICT for
literacy have succeeded in reaching diverse end users, they do not focus sufficiently on learning
within marginalized populations in low-income countries, partly because government ICT
policies tend to use a ‘one-size-fits-all’ strategy that limits attention to diversity (Hinostroza,
Isaacs, and Bougroum 2012). For example, the vast majority of software and web content is
offered in the major languages, including English, Chinese, Russian, French, Arabic, and
Spanish—making such content of limited value to many millions of people who speak (or are
literate in) only other languages.
ICTs have been used in education for much longer than is usually recognized (for a review, see
Hinostroza et al. 2012). But rigorous evaluation methods (such as RCTs; e.g., Miguel and
Kremer 2004) have only recently been utilized in the mobiles and literacy domain. Some of the
best-known initiatives—such as OLPC or Hole in the Wall—have not been supported by
adequate empirical research; or, where they were tested, researchers found they had no
significant impact on learning gains (Cristia, Ibarrarán, Cueto, Santiago, and Severín 2012;
DeBoer 2009; Hinostroza et al. 2012; Nugroho and Lonsdale 2010). It is also important to
remember that the rapid changes in ICTs over the past decade have made evaluation unusually
Literacy and mobiles, 21
difficult, and prone to what some simply term failure (Dodson, Sterling and Bennett, 2012).
Often, by the time an evaluation study can be completed or even started, the ICT platform (phone,
tablet, or software) may have changed so much that the study is no longer relevant. Such rapid
change has made it harder to design and understand how to use mobiles for high-quality reading
interventions. To date, ICT for learning resembles other areas of educational reform: a fairly long
period of trial and error, followed by slow and incremental changes as the research base develops.
Furthermore, if a rigorous evaluation is to yield robust findings, the treatment or intervention
must first be guided by preliminary evaluations in order to efficiently measure the potential
impact. Consider how development initiatives often involve multi-stage programme evaluation
strategies, including community mapping and needs assessments, followed by formative
evaluation, full implementation, summative evaluation, and evaluations of the impact and/or
cost-benefit ratio. In this process, once a beta product is ready for implementation and adequate
infrastructure is in place, a formal field test would be undertaken. As noted, rapid changes in
ICTs often lead to a realization that new devices or communications applications make a multi-
year assessment no longer relevant.
Nonetheless, the credibility of evidence remains the key for interpreting claims of effectiveness,
and what can be done to make progress in any field. As noted above, what is most important is
aligning intervention purposes with appropriate research methods. However, collecting data
merely for the sake of doing so is clearly the wrong approach, though it remains common in
many projects that require program evaluation. Rather, it is crucial to invest substantial time and
resources in the initial design so as to create a “right-size” data management and evaluation
system. Also, improved practices for coding demographic information will provide better
Literacy and mobiles, 22
opportunities to understand the impact on marginalized populations. Finally, realistic
expectations of the implementation lifecycle can ensure that an intervention is “impact ready,”
while providing a more credible basis for claims of success.
Conclusions: What can we call success?
The long-term implications for the use of new technologies are profound, both for the delivery of
literacy education and for a new vision of what it means to be literate in a world fundamentally
transformed by technology. The prospect exists for technological developments to offer
innovative tools that help meet the goals of improving education, reducing poverty, reaching
gender equity, and improving child and maternal health. Or, those same developments could
create new disparities between rich and poor countries and communities.
The advent of mobile technologies offers an unprecedented opportunity, given their tremendous
reach into populations across the globe: young and old, urban and rural, rich and poor, women
and men. The challenges are many and inevitably complex. In this review, we have proposed an
effectiveness framework that will begin to rectify this situation by refocusing attention on the
parameters of appropriate and measureable design solutions – one that adds to other proposed
frameworks on mobile learning, such as those of Koole, McQuilkin, and Ally (2010) and
Pachler, Cook, and Bachmair (2010).
In the domain of mobiles and literacy today, there has been an extraordinary amount of publicity
about the potential and the “success” of new technologies in developing countries. Some of these
efforts are bearing fruit, but most remain in their infancy. It is important to take stock of what
Literacy and mobiles, 23
these claims consist of, and what they do not.
Let us first consider what success is not. Success should be more than mere discussions about
access. Nor does it consist of photo opportunities with smiling children who have just received a
smartphone or tablet. The days when photos equal data are over. Nor is success a matter of
anecdotes about the way a few children or teachers appear much more motivated. Success is also
not about the potential capability of reaching people in need. It is not about the good ideas
designed by good people and well-meaning institutions and agencies. Simply put, success is not
the body of subjective statements that represent the views of interested parties.
What should success look like? We have argued that claims of success should be focused on
outcomes: cognitive, social, and economic impact. The use of appropriate indicators is key to
measuring and communicating the impact on student learning gains, teacher pedagogy, and
accountability through community participation. A claim of success need not require
groundbreaking impact across all, or even many, aspects of our effectiveness framework. Rather,
interventions with very specific purposes can lead to real and meaningful advances in the body
of research—in what we might call success. Indeed, embarking on a project with a clear
understanding of its intended purpose, and including the means to appropriately evaluate its
outcomes, is a modest success in itself.
There is little doubt that the use of mobiles for improving literacy and reading will continue to
grow, along with mobile ubiquity around the world. Developing measureable, sustainable, and
scalable design solutions, as part of a framework for effectiveness, will be an important way to
advance the field in the coming years.
Literacy and mobiles, 24
References
All Children Reading (2011). Project website. http://allchildrenreading.org
August, D., & Shanahan, T. (Eds.) (2006). Developing literacy in second language learners: Report of the National Reading Panel on Language Minority and Youth. Mahwah, NJ: Lawrence Erlbaum Associates.
BBC Janala (2013). Project website. http://www.bbc.co.uk/mediaaction/where_we_work/asia/bangladesh/bbcjanala.html
Braun, H., & Kanjee, A. (2006). Using assessment to improve education in developing nations. In Cohen, J. E., Bloom, D. E., & Malin, M. (Eds.), Improving education through assessment, innovation, and evaluation (pp. 1–46). Cambridge, MA: American Academy of Arts and Sciences.
Brookings Center for Universal Education (2011). A global compact on learning: Taking action on education in developing countries. Washington, DC: Brookings.
Cristia, J. P., Ibarrarán, P., Cueto, S., Santiago, A., & Severín, E. (2012). Technology and child development: Evidence from the One Laptop per Child Program. IDB Working Paper IDB-WP-304. Washington, DC: Inter-American Development Bank.
Cuendet, S., Medhi, I., Bali, K., & Cutrell, E. (2013). VideoKheti: Making video content accessible to low-literate and novice users. Paper presented at CHI 2013, April 27–May 2, 2013, Paris. http://research.microsoft.com/pubs/183489/Cuendet-CHI2013-VideoKheti-Final.pdf
DeBoer, J. (2009). The relationship between environmental factors and usage behaviors at ‘Hole-in-the-wall’ computers. International Journal of Educational Development, 29(1), 91–98. doi:10.1016/j.ijedudev.2008.02.005
Dodson, L., Sterling, S. R., & Bennett, J. R. (2012). Considering Failure: Eight Years of ITID Research. Information Technology and International Development, 9(2), ICTD2012 Special Issue, 19–34.
Literacy and mobiles, 25
EDC [Education Development Center] (2011). PHARE project. http://idd.edc.org/projects/mali-usaidphare-program-programme-harmonisé-dappui-au-renforcement-de-leducation
EDC [Educational Development Center] (2011–12). eEGRA: Usable reading results—Right now! http://eegra.edc.org/
Gove, A., & Cvelich, P. (2010). Early reading: Igniting Education for All. A report by the Early Grade Learning Community of Practice. Research Triangle Park, NC: Research Triangle Institute. http://www.uis.unesco.org/Education/Documents/early-reading-report_gove_cvelich.pdf
Gray, W. S. (1956). The teaching of reading and writing. Paris: UNESCO.
Hinostroza, J. E., Isaacs, S., & Bougroum, M. (2012). Information and communications technologies for improving students’ learning opportunities and outcomes in developing countries. Draft report. Washington, DC: Brookings GCL.
HIWEL [Hole in the Wall Education Limited] (2013). Project website: http://www.hole-in-the-wall.com
ITU [International Telecommunications Union] (2009). Information statistical profiles 2009: Africa. Geneva: ITU.
Jack, W., & Suri, T. (2010). The economics of M-Pesa. Nairobi: Central Bank of Kenya. http://www.mit.edu/~tavneet/M-PESA.pdf
Koole, M., McQuilkin, J. L., & Ally, M. (2010). Mobile learning in distance education: Utility or futility? Journal of Distance Education, 24, 59–82.
Leu, D. J., O’Byrne, W. I., Zawilinski, L., McVerry, J. G., & Everett-Cacopardo, H. (2009). Expanding the new literacies conversation. Educational Researcher, 38(4), 264–69.
Levy, F., & Murnane, R. J. (2004). Education and the changing job market. Educational Leadership, 62(2), 80–83.
Mendoza, D., & Vergel de Dios, B. (2012). The Pink Phone Revolution in Cambodia: A case study. Phnom Penh: OXFAM.
Literacy and mobiles, 26
Miguel, E. & Kremer, M. (2004). Worms: Identifying impacts on education and health in the presence of treatment externalities. Econometrica, 72(1), 159–217.
Molteno (2013). Organizational website. http://www.molteno.co.za/our-programmes
Nugroho, D. & Lonsdale, M. (2010). Evaluation of OLPC programs globally: A literature review. Version 4. Canberra: Australian Council for Education Research.
OECD (2013). Aid to poor countries slips further as governments tighten budgets. OECD Newsroom. http://www.oecd.org/newsroom/aidtopoorcountriesslipsfurtherasgovernmentstightenbudgets.htm
OLPC [One Laptop per Child] (2013). Project website: http://one.laptop.org/
Pachler, N., Cook, J. & Bachmair, B. (2010). Appropriation of mobile cultural resources for learning. International Journal of Mobile and Blended Learning, 2, 1–21.
Paiwastoon (2013). Project website. http://svr1.paiwastoon.net/mliteracy/
RTI [Research Triangle Institute] International (2011). Tangerine project website. http://www.tangerinecentral.org/node/973
Stewart, A. (2013). “M” is for mobile: Sesame Workshop’s mobile initiatives in India. Presentation at Connected Living Asia Summit mEducation, Shanghai, 25 June. http://www.gsma.com/connectedliving/wp-content/uploads/2013/03/MAE-Connected-Living-Summit_Sesame.pdf
Street, B. V. (1999). The meanings of literacy. In D. A. Wagner, R. L. Venezky, & B. V. Street (Eds.), Literacy: An international handbook (pp. 34–40). Boulder, CO: Westview Press.
Streitfeld, D. (2013, April 8). Teacher knows if you’ve done the e-reading. New York Times. http://www.nytimes.com/2013/04/09/technology/coursesmart-e-textbooks-track-students-progress-for-teachers.html?pagewanted=all
Tolani-Brown, N., McCormac, M., &. Zimmermann, R. (2009). An analysis of the research and impact of ICT in education in developing country contexts. Journal of Education for International Development, 4(2), 1–12.
Literacy and mobiles, 27
Terras, M. M., & Ramsey, J. (2012). The five central psychological challenges facing effective mobile learning, British Journal of Educational Technology, 43, 820–832.
Traxler, J. (2009). Learning in a mobile age. International Journal of Mobile and Blended Learning, 1, 1–12.
UNESCO (2012). Education for All global monitoring report. Youth and skills: Putting education to work. Paris: UNESCO.
UNESCO (2013). Policy guidelines for mobile learning. Paris: UNESCO. http://unesdoc.unesco.org/images/0021/002196/219641e.pdf
USAID (2012). USAID Education Strategy: Implementation guidance. Washington, DC: USAID. http://pdf.usaid.gov/pdf_docs/pdact461.pdf
USNRP [United States National Reading Panel] (2000). Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction. NIH Pub. No. 00-4769. Washington, DC: National Institutes of Health.
Wagner, D. A. (2005). Pro-equity approaches to monitoring and evaluation: Gender, marginalized groups and special needs populations. In D. A. Wagner (Ed.), Monitoring and evaluation of ICT in education projects: A handbook for developing countries (pp. 55-64). Washington, DC: World Bank/InfoDev.
Wagner, D. A. (2011). Smaller, quicker, cheaper: Improving learning assessments in developing countries. Washington/Paris: EFA Fast Track Initiative and UNESCO-IIEP.
Wagner, D. A. (2013). M4R: A landscape research review of mobiles for reading. Draft technical report. Philadelphia: University of Pennsylvania. http://literacy.org/sites/literacy.org/files/publications/wagner_m4r_ilitechrep_sept2013_copy.pdf
Wagner, D. A., Day, R., James, T., Kozma, R. B., Miller, J. & Unwin, T. (2005). Monitoring and evaluation of ICT in education projects: A handbook for developing countries. Washington, DC: World Bank/InfoDev. http://www.infodev.org/en/Document.9.pdf
Wagner, D. A., Murphy, K. M. & de Korne, H. (2012). Learning first: A research agenda for improving learning in low-income countries. Washington, D.C.: Brookings Institution.
Literacy and mobiles, 28
Wagner, D. A., Venezky, R. L., & Street, B. V. (Eds.) (1999). Literacy: An international handbook. Boulder, CO: Westview Press.
WITSA [World Information Technology and Services Alliance] (2010). The digital planet (2010 edition). http://www.witsa.org/v2/media_center/pdf/DP2010_ExecSumm_Final_LoRes.pdf
Worldreader (2014). Organization website. http://www.worldreader.org/.
Worldreader, ILC Africa, & USAID (2012). IRead Ghana Study: Final evaluation report. Washington, DC: USAID.
Yardley, H. (2011). International Women’s Day: Cambodia’s pink tech revolution. Oxfam Southern Africa Blog. ttp://www.oxfamblogs.org/southernafrica/?p=2246#sthash.Ip6bbFzf.dpuf
Yoza Project (2009–2013). Project website. http://yozaproject.com/about-the-project/
Author Biographies
Nathan M. Castillo (United States) is a research fellow at the International Literacy Institute and a Ph.D. student at the Graduate School of Education at the University of Pennsylvania. Molly Crofton (United States) holds a master’s degree from the Graduate School of Education at the University of Pennsylvania. Katie M. Murphy (United States) is a research fellow at the International Literacy Institute and a Ph.D. student at the Graduate School of Education at the University of Pennsylvania. Daniel A. Wagner (United States) is UNESCO Chair in Learning and Literacy, Professor of Education, and Director of the International Literacy Institute at the University of Pennsylvania. Fatima Tuz Zahra (Bangladesh) is a research fellow at the International Literacy Institute and a Ph.D. student at the Graduate School of Education at the University of Pennsylvania.