From Research Laboratory to the Market:the Experience of TypeInBraille

Sergio MascettiUniversity of Milan and

EveryWare Technologiesmascetti@dico.unimi.it

Cristian BernareggiUniversity of Milan and

EveryWare Technologiesbernareggi@dsi.unimi.it

Andrea GerinoUniversity of Milan and

EveryWare Technologiesgerino@dico.unimi.it

1. INTRODUCTIONWhile the research for assistive technologies is an activefield since the middle ′70, new challenges and opportuni-ties arose in the last years, due to the diffusion of accessi-ble mobile and pervasive devices. Among the others, thereare new challenges in the design of interaction paradigms forthese devices that are characterized by small physical dimen-sions and different interaction hardware (e.g., input throughtouchscreen versus keyboard and mouse). Concerning theopportunities, the pervasiveness and portability of these de-vices allow the users to benefit from assistive technologieseach time of the day. The research conducted in the lastyears leads to the design of brilliant solutions, both to ad-dress some of the challenges and to take advantages fromthe opportunities.

Despite a significant research effort was devoted to developa number of working prototypes, only a few of them becameactual products from which final users could take advantage.This is due to two intertwined reasons: first, the legitimategoal of researchers in this field is to devise solutions, ratherthan develop them for distribution to end users. The secondreason is that a great effort is required to deploy a final prod-uct starting from a research prototype and that this processrequires economical investment as well as some competencesthat are not always available in a typical research team, liketechnologists for the implementation of the application andits maintenance, experts in communication, to enhance thediffusion of the application, and translators to localize theapplication in different languages.

In this contribution we want to share the experience of ourteam that, starting from the scientific research conductedin the university laboratory “EveryWare Lab” 1, developsapplications and commercializes them through the spin-offcalled “EveryWare Technologies” 2. In particular, we will

1http://everywarelab.dico.unimi.it2http://www.everywaretechnologies.com

focus on an application called TypeInBraille, that allows vi-sually impaired persons to type on touchscreen devices usingBraille encoding [3, 4]. We first briefly describe the researchand development steps that the application has undergone(Section 2) and then we discuss some problems that we en-countered and lessons that we learned from this fascinatingexperience (Section 3).

2. RESEARCH AND DEVELOPMENTDesign methodology. TypeInBraille was designed anddeveloped according to a participatory user-centered designmethodology, following the star life cycle model of an in-teractive system [2]. In the star life cycle, the results ofeach design and development stage must be evaluated byend users and by the designers before passing to anotherstage (e.g., evaluation of the software prototype may leadto revise requirements specification). Hence, the productgradually evolves thanks to the continuous interaction be-tween users and designers. It is worth observing that ourteam includes a blind designer proficient in Braille and ex-pert user of iOS devices that was involved both in the designand the evaluation of TypeInBraille since its early stages. Aconsequence of this methodology is that the design and de-velopment of the product do not end with its first release,since it evolves according to new user needs emerging fromevaluation sessions and product use.

Identification of a typing technique. In order to testdifferent typing techniques we implemented an iPhone ap-plication to record the user gestures so that the resultingoutput could be interpreted off-line with the help of a semi-automatic application. During this stage we involved twoblind users in the testing of a large number of different inputtechniques. The advantages of the semi-automatic interpre-tation of the result resides in its flexibility and easiness toadapt to different input strategies hence making it possibleto experiment very different solutions with a small coding ef-fort. At the end of this phase, thanks to the users’ feedback,we identified a set of three possible typing techniques.

The proof of concept. In order to decide which tech-nique to adopt, we implemented a proof-of-concept appli-cation for each of them. The only functionality providedby each application was typing. The gestures interpretationwas performed “on-line” so that the applications could giveaudio feedback to the user. A group of 7 blind users evalu-ated these applications. Users were proficient in writing andreading Braille and well acquainted with the on screen QW-

ERTY keyboard. The aim was to evaluate the typing speedand error rate comparing the three solutions both amongthemselves and with the default iPhone QWERTY. Theanalysis of the result highlighted that one of the proposedsolutions outperformed the other two as well as the defaultQWERTY keyboard. The idea of this solution, which is atthe basis of the TypeInBraille application, is to enable usersto type a character on the touchscreen through its Braillerepresentation by inserting, one by one, the three rows thatcompose each Braille character (see Figure 1). Since each

Figure 1: Subdivision of each Braille character intothree pairs of dots.

row is composed by two dots, four gestures are defined torepresent each possible combination of a row. These ges-tures are based on a subdivision of the screen in two parts:one tap on the left part corresponds to the left dot raisedand the right dot flat, one tap on the right part correspondsto the left dot flat and the right dot raised, a two-fingertap corresponds to two raised dots and a three-finger tapcorresponds to two flat dots (see Figure 2).

(a) Left dot (b) Right dot (c) Both dots (d) No dot

Figure 2: Gestures defined to enter a pair of dots.

From users tests we also collected useful feedbacks that leadto the implementation of new features, including: a vibra-tional effect upon symbol insertion and additional gesturesto speed up the typing of frequent symbols, like the blankspace or the new line.

The application prototype. After identifying the mostpromising typing solution we developed a prototype appli-cation that included additional features, like text navigationand selection. We conducted two tests. The former was con-ducted by a blind user for a long period (about one month)and was intended to estimate the typing performances of atrained user. As a result we observed that the trained userwas able to type about 2.5 times faster and with a muchhigher accuracy with respect to the on-screen QWERTY.

The latter test involved 9 blind users for about 2 hours eachin two real-world contexts: at the desk and on a tramcar.In each context, TypeInBraille was compared with the on-screen QWERTY keyboard according to typing speed andaccuracy. The results highlight that TypeInBraille outper-forms the on-screen QWERTY in both contexts and that thedifference between the two techniques is even more remark-able in the tramcar scenario since TypeInBraille can also beused with partially audible feedbacks [4]. Also in this case,we collected valuable user feedback that allowed us to im-prove the prototype. For example the original application,according to the iPhone UI best practices, adapted itself tothe device orientation. However, as many users pointed out,while using the application in an uncomfortable environmentlike the tramcar, it happened relatively frequently that thedevice was accidentally rotated, hence resulting in an hin-drance for the user. For this reason, we decided to keep therotation of the application fixed in portrait mode.

The application deploy and improvement. The deployof TypeInBraille on the AppStore was delayed due to limita-tions with iOS in its former versions. The problem was thatit was not possible for an application to directly capturescreen touch events when VoiceOver was enabled. Conse-quently, it was not possible to detect the gestures and, atthe same time, to give speech feedback via VoiceOver. Theprototype application had overcame this problem by usingundocumented APIs that, however, cannot be used in appli-cations deployed on the AppStore. This limitation has beenremoved since iOS version 5.0 hence making it possible todeploy the first public version of the app. Since its publi-cation, at the end of October 2011, three major updates ofthe application were released introducing new features andcustomizations.

3. PROBLEMS AND LESSONSThe research, design, development and deploy activities re-lated to TypeInBraille posed many different problems. Ad-dressing these problems, we learned many valuable lessonsthat we believe can be applied in the research and develop-ment of other pervasive assistive technologies.

3.1 The importance of customizationThe first set of problems we run into concerns the crucialrole of customization for a pervasive assistive applicationas TypeInBraille. We can identify two kinds of customiza-tions: those based on the user profile and those based on thecontext of use. The former category includes language lo-calization as well as other special needs by particular classesof users. Concerning localization, in addition to translatingthe application messages into several languages, a not ne-glectible effort was required to localize the Braille code foreach locale. Note that there exist at least one Braille code foreach language and, for some languages (e.g., English), thereare different character mappings in different countries (e.g.,US and United Kingdom). Moreover, two Braille represen-tations are in use in English: contracted and uncontracted.Furthermore, some non standard Braille representations ofcertain characters are widely used by some Braille readers(e.g., by those who read on a Braille display).

Concerning the customization for particular classes of users,TypeInBraille has been extended with many features sug-

gested by users themselves. For example, many commentsfrom users required a feature to show the Braille dots and en-tered text in order to facilitate the communication betweenvisually impaired and sighted people. Figure 3 is a screen-shot of the application showing the entered text and the dotscomposing the character being entered. Another feature ad-

Figure 3: Screenshot of TypeInBraille while enteringthe letter ’d’.

dresses the reading needs of partially sighted people, thatcan benefit from receiving magnified graphical feedbacks.Similarly, we received requests from users with multiple dis-abilities, for example related to the hindrances that mayemerge in the use of the “rotor” gesture for partially motorimpaired.

The latter category of customizations is related to the con-text of use. For example, users reported different needs whilewriting in a quiet or noisy environment. Indeed, in the for-mer case the vibration effect is disturbing, while in the lattercase it is helpful to notify the user that a character is entered.Similarly, on one side, some users requested an input tech-nique that is less mentally and physically demanding (e.g.,a “one finger” input technique that requires six gestures toenter each Braille cell). This is intended to be used in un-comfortable environments, in which, for example, one handonly can be used to type. On the other side, other usersasked for a more requiring technique, like BrailleTouch [1],that is designed to be used with two hands and is intuitivelymore suitable in a comfortable environment.

From these various requests we learned that, although wedesigned the application mainly for blind people, the ac-tual set of users is much broader and it is necessary to con-

sider other different user needs. Consequently, users withdis-homogeneous characteristics should be employed in thetests. Moreover, we designed the input technique with theidea that it could fit many different contexts. However, thereis not a single technique that best suits every situation. Forthis reason, it is necessary to give the user the opportu-nity to choose the preferred typing technique depending onthe context. Clearly, the mental effort to switch among thetechniques should be limited. In this view, the underlyingadoption of Braille is a common feature in potential differenttyping techniques and can leverage the user from learningmany different encodings.

3.2 The interaction with usersThe maintenance and improvement activity after the de-ploy was driven by user feedbacks collected from two mainsources: direct communication with the support team anddiscussions among the users on forums and social networks.In order to increase the number of user feedbacks, a very sim-ple but effective functionality is to allow the users to writeto the support team directly from inside the application.

Overall, after four months from its release, there are about50 threads of discussion about TypeInBraille in English andwe received about 100 messages from distinct persons. Con-sequently, a great number of different remarks and sugges-tions was collected and analyzed by the design team, lead-ing to the improvement of the application. Among the mostrequested updates, users asked for localization in differentlanguages, the implementation of new functionalities (e.g.,direct publishing on social networks) and the adaptation ofthe application to special user needs and contexts.

The users have also proved to be proactive in the creationof new content related to the application. Indeed, manyusers in the discussion forums provided descriptions of theapplication, comparison with other solutions and supportfor other users. Among the other, some users shared screen-casts or audio-descriptions of the application, while otherusers proposed to participate in the process of applicationlocalization, by providing the localized Braille tables and bytranslating the application messages, the description and thehelp. The difficulty in this process was that it was managedmanually, by using one-to-one communication media, mainlyemail hence resulting to be a time-consuming task.

Another problem that we solved with the help of the users’community is the dissemination of information about the ap-plication. Since the resources for an advertising campaignwere very limited, we started publishing information on ourwebsite, on some social networks and in some forums. Weobserved that the users’ contribution in the diffusion of theinformation is fundamental. Indeed, users are active in re-porting the information on other forums and local discussiongroups, also translating the information when necessary.

From this experience, we realized that the community of vi-sually impaired persons is very active and ready to providesupport, feedback and cooperation. While, on one side thesecontributions are of paramount importance in the improve-ment of the application, on the other side, the managementof the interaction activities can be time consuming and re-quires that a member of the team devotes part of his/her

time to this activity. For these reasons, we are planning toimplement a platform to facilitate the collection of contri-butions by the users. For example, one functionality of thisplatform will be to create the application localization in acollaborative manner by presenting the English version ofthe text to the users and allowing them to contribute withthe translation in their own language.

3.3 The importance of system support for as-sistive technologies

The existence of accessible devices and operating systems isa necessary condition for the development of assistive appli-cations. However, this condition is not sufficient in manycases. Indeed, it is necessary that the operating system issufficiently flexible to make the OS assistive functionalitiesavailable at application level. For example, TypeInBraillerelies on the operating system method that “reads” custommessages via VoiceOver. This is very useful to read boththe single characters while they are inserted, and the en-tire words and sentences when required by the user. It isclear that the possibility to call this method from a customapplication significantly simplifies the application develop-ment and also makes the application more consistent withthe operating system and the other applications.

Note that there is a continuous improvement of the API re-lated to accessibility. On one side, this enhances the devel-opment of assistive applications. For example, TypeInBraillewas deployed after the introduction, in iOS 5.0, of the speechmethod in the accessibility API. On the other side, the fre-quent updates also pose some maintainability problems. Forexample, in the recent update from iOS 5.0 to iOS 5.1, someaccessibility-related APIs were changed, introducing a bugin TypeInBraille. Nevertheless, the improvement of the ac-cessibility APIs is a necessary process, that is dictated bythe new necessities arising in the research and developmentof accessibility applications. Indeed, TypeInBraille wouldtake advantage from some APIs that, however, have notbeen implemented yet. One example is related to an issuewith VoiceOver’s focus mechanism: the OS forwards touchevents to the application only when they occur in a focusedview. However, the focus can only be changed by the userand there is no API that makes it possible to change it fromcode. As a consequence, when a view is presented, the firsttouch is always used to acquire the view focus and hencecannot be used by the application itself. This may be incon-venient, like in the case of TypeInBraille.

The lesson that we learned is that the mobile OSs are fre-quently updated and many new release introduce new ac-cessibility related APIs. These improvements are rarely pre-sented as important new feature, and it is necessary to gotrough the “change log” to discover them. This process istime consuming but it can unveil new enabling features.

4. CONCLUSIONS AND FUTURE WORKUsers’ contribution is a central aspect in the design of as-sistive technologies. While many useful comments and per-formance data can be collected while testing the applicationprototypes, the number of users that take part in the testsis generally limited to few units, or tens. Vice versa, whenapplications are distributed as final products through the

market, the number of users that can be reached is orders ofmagnitude higher, even with a small marketing campaign.

As we have motivated in this contribution, a significant ef-fort is required to transform a prototype into a final productthat can be distributed on the market. However, this effortis justified by two major results that can be reached. Theformer is that many users in the world can benefit from theresults of the research. The latter is that, as we have experi-enced, users are eager to be involved in the improvement ofthe solution, hence they are proactive in providing feedback,highlighting problems and proposing new features.

From the scientific point of view, deploying application onthe market enables the collection of experimental data froma much wider audience, in many different contexts of use aswell as from users with different disabilities and needs. Forthese reasons, we are planning to introduce new features inTypeInBraille so that the application will automatically col-lect performance-related information, like, for example, thetyping speed and the number of undo/delete operations. Forthe same reason, in order to perform a comparison amongdifferent typing techniques, it will be necessary to implementother typing solutions to be included in the same applicationso that users can be able to rapidly switch among the typingtechniques according to the context and their needs.

5. REFERENCES[1] Brian Frey, Caleb Southern, and Mario Romero.

Brailletouch: mobile texting for the visually impaired.In Proceedings of the 6th International Conference onUniversal Access in Human-Computer Interaction:Context Diversity, UAHCI’11. Springer-Verlag, 2011.

[2] Deborah Hix and H. Rex Hartson. Developing userinterfaces: ensuring usability through product & process.John Wiley & Sons, Inc., New York, NY, USA, 1993.

[3] Sergio Mascetti, Cristian Bernareggi, and MatteoBelotti. Typeinbraille: a braille-based typingapplication for touchscreen devices. In In Proceedings ofthe 13th International ACM SIGACCESS Conferenceon Computers and Accessibility. ACM, 2011.

[4] Sergio Mascetti, Cristian Bernareggi, and MatteoBelotti. Typeinbraille: Quick eyes-free typing onsmartphones. In Proceedings of the 13th InternationalConference on Computers Helping People with SpecialNeeds. Springer, LNCS, 2012, to appear.