2-Thumb Gesture: The Design & Evaluation of a Non-Sequential Bi-manual Gesture Based Text Input Technique for Touch Tablets
Khai N. Truong1, Sen Hirano2, Gillian R. Hayes2, Karyn Moffatt3
1Department of Computer Science
khai@cs.toronto.edu
University of California, Irvine Irvine, CA 92697
{shirano,gillianrh}@ics.uci.edu
karyn.moffatt@mcgill.ca
ABSTRACT We present 2-Thumb Gesture (2TG), a non-sequential bi-manual gesture-based text input technique for touch tablets, which enables the user to enter text with both hands by using each thumb to draw small strokes over the keys on their respective sides of the keyboard without waiting for their turn in the letter sequence of a word. The results of a study comparing 2TG to Swype (a 1-finger word drawing method), suggest that the learning and use of the 2TG technique to perform text input is comparable with the commercial Swype technique by those who had no prior experience with either. Furthermore, participants were able to hold and use the tablet with both hands without experiencing the substantial fatigue that results from a one-handed approach. Only 60 minutes after being introduced to the technique, participants were able to use the 2- Thumb Gesture keyboard to enter text at 24.43 wpm, with an uncorrected error rate of 0.65%.
Categories and Subject Descriptors H5.2 [Information interfaces and presentation]: User Interfaces. – Input devices and strategies.
General Terms Design, Experimentation, Human Factors
Keywords Text entry, soft keyboard, gesture, bi-manual interaction
1. INTRODUCTION As mobile devices have grown in popularity, an increasing number of text-input techniques have been introduced. One such technique is shape writing—a method that allows the user to input text by drawing a stroke with a near unique shape containing points close to all letters found sequentially in a word. Drawing shapes enables faster input than traditional typing techniques that require the individual pressing of each letter in a word. As a result, it is now found commercially in a variety of products, including ShapeWriter [36], Swype [28], and SlideIT [26].
More recently, sales of large mobile touchscreen devices (i.e., those with screens four inches and larger) have increased. The size and weight of these devices make them hard to hold steady with only one hand while interacting with the other for long tasks, especially text input. As a result, commercial products such as Keymonk [10] and research efforts such as Bi et al.’s investigation of bimanual gesture keyboards [2] have begun to explore how words can be drawn on mobile touchscreen devices using both hands. In contrast to previous shape writing techniques, which support text input as the drawing of a single stroke containing points near the keys found sequentially in a word, these methods break the input of a word into
two smaller strokes that can be drawn by both thumbs over the keys sequentially on their respective sides of the keyboard.
In this paper, we demonstrate that the two-stroke gestures for entering English words are highly unique and hence each stroke can be drawn over the letters non-sequentially by the two thumbs. We describe our design and evaluation of a non-sequential bi-manual gesture based text input technique for touch tablets, called 2-Thumb Gesture or 2TG (see Figure 1). Through a controlled experiment, we establish the feasibility of learning and using the 2TG technique to perform text input alongside the commercial Swype keyboard (a 1-finger word drawing software). As expected, those with prior knowledge of Swype were faster using Swype than 2TG; however, their learning and performance rates with our two-thumb version were not statistically different from those with no prior Swype knowledge. Furthermore, by the end of the study, participants had begun to learn the gestures for many short words and were able to perform almost one fifth of the input by simultaneously gesturing with both thumbs and touching keys out of order with respect to their turn in the actual letter sequence. These results suggest that as participants begin to remember more gestures over time, 2TG
Figure 1. The 2-Thumb Gesture keyboard. The user enters text by using both thumbs to perform drag gestures across the letters in a word on their respective half of the keyboard.
can potentially extend similar performance benefits found in the traditional Swype technique to large-size touchscreen tablets, while enabling the user to hold and use the tablet with both hands to eliminate the substantial fatigue that results from a one handed approach.
2. RELATED WORK The 2-Thumb Gesture keyboard was designed to allow the user to hold the tablet with both hands and draw two short stroke gestures with both thumbs simultaneously, touching keys non-sequentially. Bi-manual word level gestures for entering text on a soft-keyboard has been previously introduced in commercial products such as Keymonk [10] and research efforts such as Bi et al.’s bimanual gesture keyboard [2]. Bi et al. [2] showed that a bi-manual gesture- based method can reduce the length of the stroke used to draw a word into two shorter strokes which are drawn by the thumbs. Our work builds upon these systems by allowing both thumbs to draw small strokes over the keys on their side of the keyboard without waiting their turn in the letter sequence of a word. Though Bi et al. mention the possibility of concurrent input [p.144, 2], this was not explored; the user touches the keys in a word sequentially with both Bi et al.’s bimanual keyboard and Keymonk.
Below, we discuss how our method builds upon other works. We review prior research in gesture-based text input, thumb-based text input, and two-handed text input methods for mobile devices.
2.1 Gesture-Based Text Input Virtual keyboards can enable text-input on mobile devices that do not have a physical mini-QWERTY keyboard. Methods that support text-entry using gestures—either through touch or stylus— enable users to enter text either at the character-level or at the word- level. A detailed review of such techniques can be found elsewhere [19].
Character-level input techniques allow users to type a word by making multiple stroke gestures that correspond to different letters, one a time. The most common character-level technique is Graffiti, created by Palm, Inc. The Graffiti alphabet resembles the Roman letters, which makes it easy for users to recognize and learn [4]. Likewise, unistroke maps a single stroke to a single character [6]. Unistroke gestures do not resemble Roman letters, but are designed to be well distinguishable from one another. One key distinction of Graffiti and unistroke from other character-level input techniques is that the user does not need to perform target selection as a part of their text-entry. Castellucci and MacKenzie’s evaluation [4] comparing Graffiti and unistroke shows that participants were able to input 15.8 wpm using unistroke and 11.4 wpm using Graffiti; however, there were no statistical differences in input speed between the two. EdgeWrite uses a character set that resembles the Roman letters, but only requires the user to touch corners of a box in a particular sequence as they draw these gestures [33]. This enables for users with motor problems to type using a lower keystroke per character (KSPC) than with Graffiti. The Vector keyboard [11] and MessagEase [21] combine unistroke gestures with target selection to simplify the gesture set. Although no numbers were given within the paper [11], it can be derived from the presented graph that 9 participants were able to input text with the Vector keyboard at ~10.3 wpm. Using a Fitt’s Law analysis of the MessagEase design, Nesbat [21] concludes that a soft-key version of the keyboard potentially can support expert text input performance between 30 and 37 wpm.
Word-level gesture-based input techniques, such as Cirrin [20] and Quikwriting [23], allow users to enter a word with a single stroke. Mankoff and Abowd [20] reported that one user was able to achieve
20 wpm with Cirrin after using the technique for over 2 months; Isokoski and Raisamo [9] showed that participants were able to reach 16 wpm with Quikwriting using a stylus after 20 15-minute sessions. In 2003, Zhai and Kristensson began to explore shape writing as a way to allow users to perform shorthand gestures to input words on an optimized keyboard layout [12][13][34][35]. This concept has since been adopted by several commercial products for small mobile touchscreen devices, such as ShapeWriter [36], Swype [28], and SlideIT [26]. These virtual keyboards have been used on a variety of platforms, including small devices, tabletops and tablets. Expert performance of shape writing using a QWERTY layout on tabletops have been computed to reach 40.7 wpm [24]; but have not been computed for the tablet form factor. In Bi et al.’s study [2], participants reached an average of 30 wpm using the unimanual shape writing technique on a tablet. However, tablet users often describe concerns of fatigue in the hand holding the device as well as in the finger performing the gestures back and forth across the large touchscreens [5][15].
2.2 Thumb-Based Text Input Although very few techniques have been altered or designed specifically to support thumb-based text-input, users often are able to effectively employ their thumbs to perform input. For example, Silfverberg et al.’s study shows that two-handed index finger and one-handed thumb use of multi-tap are ~27 wpm and ~25 wpm respectively [25]; their study also shows that two-handed index finger and one-handed thumb of T9 is ~46 wpm and 41 wpm. One notable exception is Gonzalez et al.’s work [7] which explored different ways of supporting text-entry while the user is driving and her hands are on a steering wheel. Their work showed that EdgeWrite [33] can be adapted to work on a Synaptics Stampad that is embedded on steering wheel. The focus of these previous works has been to examine how use of the thumb to support one- handed interaction affects performance. Results from these works show that performance is minimally affected, but provides other benefits (such as it frees up the other hand, and it requires less visual attention because the targets are in fixed location relative to the thumb’s anchor position). An important implication when designing thumb-based interactions, however, is to make target sizes large enough that they can be accurately selected by the thumb [22].
2.3 Two-Handed Text Input Because typing on a desktop QWERTY keyboard is typically performed as a two-handed task, it is not surprising that users can effectively employ both hands with many existing text-entry methods. For example, MacKenzie and Soukoreff [17] studied two handed (specifically thumb) use of physical mini QWERTY keyboards and predicted that expert typing speed can reach ~61 wpm. Novel techniques for chording on a standard 12-key keypad also have been developed to leverage two-handed input, such as ChordTap [30].
On large touchscreen devices, the 1Line Keyboard is a technique that supports touch-typing on a tablet with a reduced layout [16]. Most other work, such as LucidTouch [31], SwiftKey [27], Thumb Keyboard [29], and Samsung’s S1 tablet support typing by dividing the keyboard interface into two halves, allowing users to press the keys with the hand closest to each. The 2-Thumb Gesture keyboard presented in this paper, along with the previously mentioned Keymonk [10] and Bi et al.’s bimanual gesture keyboard [2], differ from these other works in that users employ their thumbs to draw stroke gestures across the keys on each side of the keyboard instead of performing individual key-presses. Bi et al. [2] investigated two modes for completing the entry of a word: finger-release and space-
required. They showed that after 1.5 hours, participants were able to enter text at 26 wpm and 21 wpm with the finger-release and space-required methods respectively. Although participants were slower with the bimanual techniques, they perceived that it was more comfortable and less physically demanding than a unimanual gesture-based technique. The benefits and challenges associated with two-handed input have been well studied [1][3][8][14], the research question that this paper seeks to answer specifically is how to extend previous work to enable the user to perform simultaneous word drawing gestures on large touchscreen tablets while holding the device in both hands.
3. DESIGN & IMPLEMENTATION The 2-Thumb Gesture soft keyboard is a non-sequential bi-manual gesture based text input technique for touch tablets. We developed 2TG as an input method editor (IME) for Android OS version 2.3 or higher. In contrast to prior work designed to optimize input with the tablet placed on a table (e.g., the 1Line keyboard [16], we designed 2TG for use while holding the tablet (i.e., with the device cupped in both hands and using the thumbs for input). In particular, we are hoping to enhance experiences in which users must stand and hold the tablet (e.g., a nurse taking notes on her rounds, a coach taking notes on the field, a user typing on a bus). Similar to Bi et al.’s bimanual keyboard [2] and Keymonk [10], the user draws two separate strokes that together touch all the letters in a word. Because Bi et al. have previously showed that the finger-release and space-required gesture completion modes resulted in comparable entry speed, with finger release being slightly faster,
we only explore finger-release in this work. Furthermore, we explore allowing the user to draw the strokes non-sequentially, that is, the thumbs do not need to wait until their turn in the word’s letter sequence to continue drawing their respective stroke. By allowing the two thumbs to non-sequentially draw short stroke gestures on their side of the screen to touch all the letters in a word, it is possible to also reduce the amount of time required to enter a word.
3.1 Key Layout & Size Because the size and weight of a tablet make it hard to hold single- handed for long tasks, we devoted special attention to enabling the user to hold it steady with both hands during text input. Given that both hands would be holding the device, we designed the keyboard’s layout and size specifically to support comfortable thumb-based input.
The 2-Thumb Gesture keyboard preserves the QWERTY layout to avoid the overhead associated with learning a new letter arrangement. To prevent the user from straining to reach any key with the thumbs—which happens most often with keys towards the middle of the keyboard (T, G, B, Y, H, and N), we enlarged these center keys, similar to the Microsoft Natural keyboard. Thus, the layout of the keys on both halves remains visually similar to traditional desktop keyboards and allows easier access of these frequently used keys (see Figure 2).
3.2 Input Language Users input text by drawing stroke gestures over the letters sequentially (see Figure 3). However, because the thumbs only
Figure 2. The 2-Thumb Gesture keyboard interface. The design draws inspiration from the Microsoft Natural keyboard; it enlarges the keys at the center (T, G, B, Y, H, N) to split the layout into two distinct halves while preserving the familiar QWERTY layout. Keys on each side are intended to be swiped only by the thumb on that side of the screen.
Figure 3. Typing love as a novice user. The user starts by placing her right thumb on the L key and then swipes it to O. Next, she places her left thumb on V and then swipes it to E. Once she lifts both thumbs from the keyboard, the keyboard interprets the gesture into a list of candidate words and automatically selects the highest frequency word.
draw on their side of the screen, the stroke gestures are inherently different from the unistroke gestures supported by counterpart systems. First, although a novice user may still swipe her thumbs to different keys in the sequential order defined by the respective letter’s appearance in the word, the strokes themselves no longer visually capture the full sequential order of the keys across both sides of the keyboard. This property of the gesture language results in a feature that allows expert users who are familiar with the gestures to input the strokes on each side of the keyboard non- sequentially and even simultaneously (i.e., without needing to wait for that side’s turn in the word’s letter sequence). Figure 4 illustrates how an expert user would type love using both thumbs.
3.3 Gesture Recognition Our method performs a dictionary lookup based on the gestures drawn using each hand. However, because keys for letters not found in a word are crossed while the user swipes between the letters in the word, gestures must be interpreted as a set of likely candidate words. As mentioned above, we removed all entries in COCA that contained non-alphabetic characters as well as those entries not found on dictionary.com. We then extracted the list of keys that must be entered sequentially on the left side of the keyboard as well as its right side equivalents. We next built an SQLite database holding the word, its frequency, and the left and right key sequences for entering it.
To determine the word the user entered through the thumb gestures, the keyboard computes the list of key points from each stroke, including the first and last points. Key points are points in a gesture where the thumb changes its direction either vertically (between up and down) or horizontally (between left and right). We used these strict criteria to lower the false detection of key points. However, this approach then only extracts from the strokes a set of key sequences that partially matches those in the database for the word that the user wants to enter. The input key sequence entered may also partially match with the key sequences for a few other words. The system retrieves a full list of the possible candidates, sorts the responses based on the frequency of the words in the COCA corpus, and displays the top four as “candidate words” for the gestures, with the top one automatically selected and shown in the editor field.
To evaluate the coverage of the words in COCA supported by our algorithm, we computed the percentage of words that appear as the most likely candidate for the associated gesture sequences, as well as the percentage of words that appear as the 2nd, 3rd, and 4th most likely candidates for those sequences. Again, these calculations assume a perfect gesture that travels through the center of each key in the word’s letter sequence. The results showed that in the top
10,000 words, 97.19% can be typed without the need for disambiguation and 99.96% appear in the top 4 most likely words for any gesture sequences. Overall, 99.51% of the words in the entire corpus appeared in the top 4 most likely words for its gesture sequences (see Figure 5).
We note that other approaches for performing gesture recognition can be adopted, such as an adaptation of the shape writing algorithm [13] or machine learning. In this paper, we investigate the user’s ability to learn and perform the strokes, without introducing system support for inferring the user’s intent; this allows us to then analyze how much effect adopting a different gesture recognition method and relaxing the requirement for the users to accurately and precisely draw a stroke could potentially have on the user’s text input performance.
4. EVALUATION METHOD We evaluated the 2-Thumb Gesture technique in a laboratory study focused on the learnability and usability of the technique as a text input method on tablet computers, with Swype [28] as a reference technique. As previously noted, in Bi et al.’s study [2], participants reached an average of 30 wpm using the unimanual shape writing technique on a tablet, while reaching between 21 and 26 wpm using the bimanual approach. Similarly, we included Swype as a reference technique to develop a rich understanding of the advantages and drawbacks of both one-finger and two-thumb shape writing on touch tablets.
We note that our decision to compare against a fully implemented commercial unimanual shape writing technique has advantages as well as drawbacks. We, of course, did not expect to outperform a fully implemented Swype; however, we hoped to demonstrate that our implementation’s performance is promising in relation to a fully implemented system that has error-handlings. A drawback to this decision is that it does not allow us to fully compare the 2- handed approach in which keys do not have to be touch sequentially against a 1 finger approach implemented using the same gesture recognition method. That issue and controlling for other design parameters (e.g., support for word prediction, typo detection/auto- correction, and so forth) are outside of the scope of this specific investigation and can be points for future studies.
4.1 Study Design & Procedure A 3-factor mixed design was used with input technique (1-finger Swype or 2TG) and session (1, 2, or 3) as within-subjects factors, and prior experience (No Experience, Prior Swype Experience) as
Figure 5. Coverage of the gesture recognition algorithm. Each line shows the coverage of the words returned within the first, second, third and fourth most likely candidate for its gesture sequences.
Figure 4. Typing love as an expert user. The user starts by placing her right thumb on L and her left thumb on V, and then simultaneously drags her right thumb to O and her left thumb V.
a between subjects factor. Presentation order was fully counterbalanced and participants were randomly assigned to a condition.
During the experiment, we asked participants to enter short phrases presented on the screen as fast and accurately as possible using the given text entry method. We prepared phrases based on MacKenzie and Soukoreff‘s set [18]. We randomized the order of the phrases and grouped them into six blocks of phrases. No phrase repeated within a block.
Each person participated in three sessions. Consecutive sessions were scheduled 2–72 hours apart around the participants’ schedules. At the first meeting, participants also learned about and practiced each text entry method.
Each session consisted of two 20-minute half-sessions (one 20- minute half-session for each technique). Each half-session started with three practice phrases, followed by the phrases pre-arranged for that block. Participants had 20 minutes to type as many phrases as they could. We counter-balanced the presentation order of the text entry methods across the participants for the first session and alternated their presentation for subsequent sessions with the same participant.
4.2 Participants We used word of mouth, flyers, and online posts to recruit right- handed participants. Because of procedural inconsistencies, data for 2 participants were discarded from analysis, leaving 10 participants (6 female and 4 male; with a median age of 29 years, SD=5.3) for this experiment. All participants had at least a high- school level of English literacy. All participants reported no motor disability in their arms and hands, and reported no visual disability. Half of the participants reported having prior knowledge of how to use Swype (and either currently used it or had used it in the past); the remaining 5 participants had no prior experience with the technique.
4.3 Apparatus We used an Asus Eee Pad Transformer for the text entry interfaces. We asked participants to hold the tablet during the study. For the Swype condition, we used the Beta 3.26 version of the Swype soft keyboard. The height and width of the keys in the Swype soft keyboard are the same as those in the 2-Thumb Gesture keyboard (with the exception of t, a, g, h, and n keys, which are wider for 2TG as shown in Figure 2).
5. RESULTS To analyze collected data, we modified Wobbrock and Myer’s StreamAnalyzer software [32] to account for word-level operations like word deletion and word corrections. We removed any data point over 3SD from the average typing time and the average error rate for each participant in each session as an outlier. In total, 3.0% of the data points were removed.
We ran a 232 (technique session expertise) repeated measures ANOVA on words-per-minute, and the corrected and uncorrected error rates. In the presentation below, where df is not an integer, we have applied a Greenhouse-Geisser adjustment for non-spherical data. All pairwise comparisons were protected against Type I error using a Bonferroni adjustment. Along with statistical significance, we report partial eta-squared (η2), a measure of effect size. A preliminary analysis, which included presentation order as a between-subjects factor, yielded no significant main or interaction effects involving order, giving us confidence that counterbalancing the techniques sufficiently accounted for any learning or fatigue effects.
5.1 Performance Overall, text input rate was impacted by input technique, indicating that overall the more familiar 1-finger Swype method outperformed the novel 2-Thumb Gesture technique (F1,8 = 47, p = .0001, 2
= .855). At the 3rd session, participants averaged 32.4 wpm (SD: 8.51) with Swype and 24.43 wpm (SD: 4.29) with 2TG. However, there was also a significant interaction between technique and expertise (F1,8 = 16.29, p = .004, 2
= .671) Pair-wise comparisons revealed that the effect of technique only held for those with prior Swype experience (p < .001). For those with no prior experience with the techniques, no significant difference was found (p = .081). There was, however, no main effect of expertise (p =.079). Figure 6 shows words-per-minute for each technique by level of expertise.
A main effect of session on words-per-minute (F1.22,9.72 = 44.02, p < .0001, 2
= .846), confirmed by pair-wise comparisons (Session 1–2, p = .001; Session 2–3, p < .001), showed that performance improved with each session. This effect was consistent across techniques and expertise, as shown in Figure 7 and indicated by the absence of interaction effects (session technique: p = .271, session expertise: p = .716, and session technique expertise: p = .62).
The average uncorrected error rate (the rate of errors remaining in the transcribed text, including insertion, substitution, and deletion errors [32]) over all sessions was 0.4% (SD=1.8%) and 0.7% (SD=3.2%) for the Swype and 2-Thumb Gesture techniques, respectively. These low rates are not surprising once we look at their corrected equivalents. On average, participants had a corrected error rate of 9.8% (SD: 15.4%) with the Swype technique and 14.8% (SD: 14.7%) with the 2-Thumb Gesture technique, indicating few errors remained in the input, because participants were correcting their errors in this study.
As with input rate, expertise and technique impacted the corrected error rate, yielding a main effect of technique (F1,8 = 21.36, p = .002, 2
= .727) and an interaction effect between technique and expertise (F1,8 = 12.98, p = .007, 2
= .619). Pairwise comparisons revealed that those with prior Swype experience had higher corrected error rates with 2TG (p < .001), suggesting a possible negative transfer effect. There was no difference between the techniques for those without Swype experience (p = .492), as shown in Figure 8. There was also a significant interaction between session and technique (F1,8 = 5.69, p = .014, 2
= .416). Pairwise comparisons revealed that the main effect of technique only held in sessions 1 (p < .001) and 3 (p = .001), but not in session 2 (p = .11). In session 2, the error rate spiked for the Swype technique, but remained consistent in 2TG. For uncorrected error rate, no significant differences were found, which is not surprising given the low rates observed.
5.2 Non-Sequential Bi-Manual Input We designed the 2-Thumb Gesture technique so that both thumbs could non-sequentially draw stroke gestures. The challenge with being able to perform text input with the 2-Thumb Gesture keyboard in this manner is being able to identify or recall the correct left and right gesture sequences for a word, behavior that we expect to develop with practice.
As expected, participants initially found the idea of performing 2- Thumb Gesture to be daunting.
“It seemed hard at first to kinda cut the words and type them in halves. I had to be more conscious of what each thumb was supposed to do. But it’s a cool idea and technique. I got used to it.” –P12 (22, F)
Thus, when entering a word for the first time, the thumbs would each draw a stroke over the keys on their side of the keyboard to sequentially spell each letter in a word. However, an interesting outcome of the initial mental and motor requirements associated with learning the 2-Thumb Gesture method is how participants began to remember the gestures, more so than with Swype. Participants commented that they began to remember gestures for short words such as “the” and “this.”
“I liked the two hand method better for shorter words! I know the gestures for words like ‘the’ and ‘water.’” –P2 (23, F)
Not surprisingly, as participants began to learn the gestures for short words and common letter sequences, they also began to draw both strokes simultaneously, touching keys out of order with respect to their turn in the actual letter sequence. Participants perceived that doing so improved their input rate.
“I don’t have to do things simultaneously, but I feel like I would be slower [sequentially].” –P2 (23, F)
At the 3rd session, 19.0% (SD: 8.0%) of the input happened in this manner. We expect that as participants begin to remember more gestures over time, they will continue to improve their text-input rates. For example, participants commented that for longer words, they had begun to remember the gestures for portions that made up of common letter sequences, such as “ing” and “pre.”
5.3 User Feedback Although mentally dividing a word into a two stroke gesture seemed difficult to the participants initially, participants started to describe it as being “intuitive” and even fun to use as they continued to learn the technique,. For example, after initially struggling to use the 2-Thumb Gesture keyboard for the first 5 minutes, P7 (38, M) declared “I got it.” As a person without prior knowledge of Swype, P7 was able to learn both at the same rate; however, he preferred the 2-Thumb approach.
“Dude, this (Swype) sucks in comparison to the new school way (2TG).”—P7 (38, M)
“The two hand method seems more intuitive. Even though two hand is less accurate, the user experience feels enhanced.” – P5 (30, M)
Furthermore, although a few participants suggested ways of improving the comfort of 2TG (e.g., by changing the placement and size for some of the keys), participants uniformly reported discomfort when using Swype. In particular, participants noted that holding the tablet with one hand can be tiring on the hand and the wrist.
“Oh my g-d, my hand!” –P2 (23, F)
“This gets really heavy after a while! My hand…the one that holds the tablet is dead. The fatigue is definitely less with the two hand method.” –P4 (31, F)
“After 5 minutes or so, my finger started hurting and my wrists were cramping!” –P11 (20, F)
However, all participants described the Swype technique as being “more forgiving” and some indicated that with time, the two handed technique might improve.
“Two hand seems more intuitive, but the technology is not quite there.” –P5 (30, M)
Figure 6. Words per minute for 2 Thumb Gesture and Swype, by prior experience with Swype. Error bars show 95% confidence intervals (N=10).
Figure 7. Words per minute across each session, by technique and experience with Swype. Error bars show 95% confidence intervals (N=10).
Figure 8. Corrected error rate for 2 Thumb Gesture and Swype, by prior experience with Swype. Error bars show 95% confidence intervals (N=10).
Participants noted that the commercial Swype keyboard always tried to guess what they were gesturing. For example, the participants could miss some keys (instead only touching their neighboring keys) when performing a gesture and the system may still return what was intended as a potential candidate word. Furthermore, they described being able to continue with a gesture and correct it even if they messed up. Although the gesture recognition approach that we adopted also allows the participants to adjust their gesture if it is incorrect, it requires the participants to accurately touch all the keys in the word.
By developing and evaluating a keyboard without any tolerance for input error, we were able to gain an understanding of the effectiveness of the 2-Thumb Gesture technique by itself alone. Thus, the results of the study demonstrate the feasibility of the method. Though 2TG did not outperform Swype, adding tolerance for input error to our method (similar to that already provided by the commercial Swype technique used as a comparison) can only improve its user performance.
6. DISCUSSION In this section, we discuss some of the factors that could have affected the text input rate observed in the study. Additionally, we discuss some possible ways to improve upon this work.
6.1 Input Errors Participants made two types of errors when using 2TG that limited their input rates. First, the system rejected some of the participants’ gestures. Whenever the system was unable to recognize a gesture, the participant had to perform that gesture again. Second, much like typing on a normal keyboard, participants sometimes entered typos, which resulted in the keyboard returning the wrong words. The design of our recognition algorithm did not recognize and fix typos automatically. The low uncorrected error rate observed indicates that participants corrected these typos, thereby reducing the input rate.
6.1.1 Rejection Errors The system rejected 5.9% (SD: 2.6%) of the gestures inputted by participants. Although 5.9% is a low rate, it has a noticeable effect on the participants.
“I re-enter text more in 2TG because when it's wrong, there aren't any choices.” –P3 (31, F)
In contrast, participants did not report experiencing any rejection errors when using the Swype technique. All participants commented that they wished the technique would be as forgiving as Swype, which did not require them to directly touch all the keys but only be in proximity of those keys. Future versions of the keyboard can relax the requirement that the users must accurately touch the keys in the gesture sequence and apply a proximity threshold against which it accepts user input.
A common problem that many participants encountered was touching keys towards the middle of the keyboard (T, G, B, Y, H, and N) with the opposite hand. Our implementation only allows users to touch keys from one side of the keyboard with the respective thumb. However, participants mentioned that they typically do not abide by such a strict input model when using desktop keyboards. As a result, while entering text with the 2- Thumb Gesture keyboard, there was a tendency for them to reach for some keys with the opposite thumb.
“I have to think of the halves of the keyboard. My biggest problem was that I had to divide the keyboard in my head and I couldn’t go over.” –P2 (23, F)
Because our system disallowed this behavior, participants’ input gestures were rejected by the recognizer when they included keys from the other half of the screen. To address this problem, the gesture database can be built to treat the middle keys as ones that can be input as a part of the gestures from either side of the screen. Alternately, the keyboard could be further split, leaving a space with invisible keys in between, as on the iOS5 split keyboard and Bi et al.’s bimanual keyboard [2].
6.1.2 Typos Overall, the average corrected error rate for 2TG was 14.9%. In this work, the corrected error rate reflects the rate at which two types of actions could have occurred: 1) the manual deletion of incorrectly typed text and re-entering correct text, and 2) the automatic correction of incorrect text that was returned by the gesture recognizer through the selection of another word from the candidates list (i.e., the user performed a disambiguation action). As mentioned earlier, for the 10,000 highest frequency words in the COCA, 97.19% can be typed with the 2-Thumb Gesture keyboard without the need for disambiguation and 99.96% appear in the top 4 most likely word for any gesture sequences. When we examine the disambiguation rate, at the 3rd session, participants on average performed disambiguations for only 2.7% (SD: 1.4%) of the text that they entered. This means that the remaining amount of the corrected errors (~12.2%) resulted from correcting typos.
By developing and evaluating a keyboard without any tolerance for input error, we were able to gain an understanding of the effectiveness of the 2-Thumb Gesture technique by itself alone. The results from our study establish that the 2-Thumb Gesture keyboard is a feasible method that can be learned and used. Adding tolerance for input error to the 2-Thumb Gesture keyboard can only improve its user performance. With feasibility established, an obvious next step is to collect the data necessary to model common input errors. For example, participants sometimes touch keys from the same side of the keyboard in the wrong order (e.g., gesturing EV instead of VE for LOVE), or miss the last key by a few pixels (undershooting the target). Using such data, we can develop an error model that can be used to automatically correct typos.
6.2 Learnability Participants commented that the visual feedback provided by our keyboard implementation currently does not help them remember the gesture patterns for long words.
“I think one of the problems is that for long words, you start losing track of what (the letters in) the words are.” –P5 (30, M)
Currently, our keyboard implementation paints the full gestures until both thumbs are lifted. In future versions of the software, we will design the keyboard to better highlight parts of the gestures in which high-frequency letter sequences were inputted. As participants learn the common gestures, we can begin to adapt the visualization to continue to teach them the lower frequency letter sequences as well.
7. CONCLUSION In this paper, we presented the design and evaluation 2-Thumb Gesture, a technique that breaks the input of a word into two smaller stroke gestures that can be performed non-sequentially by both thumbs on each side of the keyboard. Our evaluation results show that after 40-60 minutes of use, participants were able to use the 2- Thumb keyboard to enter text at 24.43 wpm, with an uncorrected error rate of 0.65% and a corrected error rate of 14.9%. These
overall average performance results were similar to those reported by Bi et al. [2], who also showed that participants reported a difference in comfort and physical demand between the unimanual and bimanual input approaches; similarly, our study confirmed participants were able to hold and use the tablet with both hands without the substantial fatigue that results from the one-handed approach.
Beyond those results, our implementation and study show that at the third session, participants had begun to learn the gestures for many short words and were able to perform on average 19.0% (SD: 8.0%) of the input by simultaneously gesturing with both thumbs and touching keys out of order with respect to their turn in the actual letter sequence while only needing to disambiguate 2.7% of their input. As participants begin to remember more gestures over time, they will continue to improve their text-input rates.
Additionally, our study shows that the learning and use of the 2- Thumb Gesture technique to perform text input was comparable to that of the commercial Swype technique by those who had no prior experience with either method. Although input by those with prior knowledge of Swype was faster using Swype than 2TG, their learning and performance rates with our two-thumb version were not statistically different from those with no prior Swype knowledge. Furthermore, the current 2-Thumb Gesture keyboard does not support erroneous input in the same manner as the commercial Swype keyboard, against which it was evaluated. It is important to note that these results were achieved despite the fact that at the 3rd session, 18.1% of the participant’s input were errors, which required the user to re-input the text (5.9% were gestures rejected by the recognizer; 12.2% were typos not recognized by the system).
For future work, we will improve the interface to address issues that prevented the participants’ input rates from being closer to the predicted value. First, we aim to help users learn how to input many common letter sequences by modifying the way that the strokes are drawn on the screen to highlight those gestures. Additionally, we will modify the interface to give users more freedom for how to input letters towards the middle of the keyboard and to remove the requirement that users must directly touch the keys (to make the technique more “forgiving” like Swype). Finally, we will develop a model capturing how users incorrectly type different words in order to automatically correct typos.
8. REFERENCES [1] Balakhrishnan, R., Hinckley, K. Symmetric bimanual interaction. In
Proc. of CHI 2000, ACM Press, 33-40. [2] Bi, X., Chelba, C., Ouyang, T., Partridge, K., Zhai, S. Bimanual
Gesture Keyboard. In Proc. of UIST 2012, ACM Press, 137-146. [3] Buxton, W., Myers, B. A study in two-handed input. In Proc. of CHI
1986, ACM Press, 321-326. [4] Castellucci, S.J., MacKenzie, I.S. Graffiti vs. unistrokes: an empirical
comparison. In Proc. of CHI 2008, ACM Press, 305-308. [5] Enterprise Efficiency. Swype Makes Tablets Interesting.
http://www.enterpriseefficiency.com/messages.asp?piddl_msgthread id=231470
[6] Goldberg, D., and Richardson, C., Touch-typing with a stylus. In Proc. of INTERCHI 1993, ACM Press, 80-87.
[7] González, I.E., Wobbrock, J.O., Chau, D.H., Faulring, A., Myers, B.A. Eyes on the road, hands on the wheel: thumb-based interaction techniques for input on steering wheels. In Proc. of Graphics Interface 2007, 95-102.
[8] Guiard, Y. Asymmetric division of labor in human skilled bimanual action: the kinematic chain as a model. Journal of Motor Behaviour, 19(4), 1987, 486-517.
[9] Isokoski, P., Raisamo, R. Quikwriting as a multi-device text entry method. In Proc. of NordiCHI 2004, ACM Press, 105-108.
[10] Keymonk. http://keymonk.com/ [11] Klima, M., Slovacek, V.: Vector Keyboard for Touch Screen Devices.
In Proc. of HCII 2009: Ergonomics and Health Aspects of Work with Computers, 250–256.
[12] Kristensson, P.O. Discrete and Continuous Shape Writing for Text Entry and Control. Ph.D. Dissertation, Department of Computer and Information Science, Linköping University (Linköping, Sweden), 2007.
[13] Kristensson, P.O., Zhai, S. SHARK2: a large vocabulary shorthand writing system for pen-based computers. In Proc. of UIST 2004, ACM Press, 43-52.
[14] Leganchuk, A., Zhai, S., Buxton, W., Manual and Cognitive Benefits of Two-Handed Input. TOCHI, 5(4), 1998, 326-359.
[15] Le Pan Life. Multi-touch Screen. http://lepanlife.com/forum/showthread.php?tid=266
[16] Li, F.C.Y., Guy, R.T., Yatani, K., Truong, K.N. The 1line keyboard: a QWERTY layout in a single line. In Proc. of UIST 2011, ACM Press, 461-470.
[17] MacKenzie, I.S., Soukoreff, R.W. A model of two-thumb text entry. In Proc. of Graphics Interface 2002, 117-124.
[18] MacKenzie, I.S., Soukoreff, R.W. Phrase sets for evaluating text entry techniques. In Ext. Abs. of CHI 2003, ACM Press, 754-755.
[19] MacKenzie, I.S., Soukoreff, R.W. Text entry for mobile computing: Models and methods, theory and practice. Human-Computer Interaction, 17, 147-198.
[20] Mankoff, J.C., Abowd, G.D. Cirrin: A Word-Level Unistroke Keyboard for Pen Input. In Proc. of UIST 1998, ACM Press, 213-214.
[21] Nesbat, S.B. A system for fast, full-text entry for small electronic devices. In Proc. of ICMI 2003, ACM Press, 4-11.
[22] Parhi, P., Karlson, A.K., and Bederson, B.B. Target size study for one- handed thumb use on small touchscreen devices. In Proc. of MobileHCI 2006, ACM Press, 203-210.
[23] Perlin, K. Quikwriting: Continuous Stylus-Based Text Entry. In Proc. of UIST 1998, ACM Press, 215-216.
[24] Rick, J. Performance optimizations of virtual keyboards for stroke- based text entry on a touch-based tabletop. In Proc. of UIST 2010, ACM Press, 77–86.
[25] Silfverberg, M., MacKenzie, I.S., Korhonen, P. Predicting text entry speeds on mobile phones. In Proc. of CHI 2000, ACM Press, 9-16.
[26] SlideIT. http://www.mobiletextinput.com/ [27] SwiftKey. http://www.swiftkey.net/ [28] Swype. http://www.swype.com/ [29] Thumb Keyboard.
http://www.beansoftapps.com/archives/products/thumb-keyboard [30] Wigdor, D., Balakrishnan, R. A comparison of consecutive and
concurrent input text entry techniques for mobile phones. In Proc. of CHI 2004, ACM Press, 81-88.
[31] Wigdor, D., Forlines, C., Baudisch, P. Barnwell, J., Shen, C. Lucid touch: a see-through mobile device. In Proc. of UIST 2007, ACM Press, 269-278.
[32] Wobbrock, J.O., Myers, B.A. Analyzing the input stream for character- level errors in unconstrained text entry evaluations. TOCHI, 13(4), ACM Press, 2006, 458-489.
[33] Wobbrock, J.O., Myers, B.A., Kembel, J.A. EdgeWrite: a stylus- based text entry method designed for high accuracy and stability of motion. In Proc. of UIST 2003, ACM Press, 61-70.
[34] Zhai, S., Kristensson, P.O. Shorthand writing on stylus keyboard. In Proc. of CHI 2003, ACM Press, 97-104.
[35] Zhai, S., Kristensson, P.O. The Word-Gesture Keyboard: Reimagining Keyboard Interaction. Communications of the ACM, 55(9), 2012, 97-104.
[36] Zhai, S., Kristensson, P.O., Gong, P., Greiner, M., Pen, S., Liu, L., & Dunnigan, A. Shapewriter on the iPhone: from the laboratory to the real world. In Ext. Abs. of CHI 2009, ACM Press, 2667-2670.
9. VIDEO