SHORTHAND ALPHABETS AND PEN COMPUTING …ctappert/cs631-02/shorthand.pdfShorthand Alphabets and Pen...

Shorthand Alphabets and Pen Computing 1/1

SHORTHAND ALPHABETS AND PEN COMPUTING

C.C. TAPPERT CSIS, Pace University, 861 Bedford Road, Pleasantville NY 10570

E-mail: [email protected]

This paper reviews the history of shorthand alphabets, focusing on the Roman alphabet, and analyzes the shorthand alphabets developed for machine recognition in pen computing. We begin with a description of a sequence of historical shorthand alphabets, then discuss handwriting recognition, both OCR and online recognition, and finally analyze the use of shorthand alphabets in pen computing. We find that the Graffiti recognizer in the popular Palm OS PDA devices, notably the Palm Pilot and Handspring models, cleverly combines known technology to produce a highly useful system. It uses a shorthand alphabet of easy to learn symbols that must be written in a prescribed way and has separate writing areas for alphabetic and numeric input.

1 Introduction

We review the history of shorthand alphabets, focusing on the Roman alphabet and not covering numerics or punctuation, and describe three early alphabets that we later relate to online alphabet systems. We then discuss handwriting recognition, both OCR and online recognition, but focus on the latter and especially on the online recognition difficulties and solution methodologies. Finally, we analyze the use of shorthand alphabets developed for machine recognition in pen computing.

2 History of Shorthand Alphabets

Shorthand is “a method of writing rapidly by substituting characters, abbreviations, or symbols for letters, words, or phrases” and can be traced back to the Greeks [11]. We focus on orthographic shorthands that use conventional spelling and have one symbol for each letter of the Roman alphabet. Here, we present and describe three documented shorthand alphabets that are relevant to this study: the Tironian, Stenographie, and Moon alphabets.

Marcus Tullius Tiro, Cicero’s secretary, devised the first extensive Latin shorthand system to record speeches in the Roman Senate. Many Romans, including Julius Caesar, favored shorthand and this system remained in use for over a thousand years. The Tironian alphabet [11] (63 BC, Fig. 1) consists of 22 symbols (no J, U, W, or Y) and 10 of them resemble in some respect the corresponding current Roman alphabet symbols: C and I completely; K, V, and Z closely; A, B,


and T partially (in one stroke); and D (lowercase delta) and H are somewhat like their lowercase counterparts.

Figure 1. Tironian Alphabet, 63 BC

During the Middle Ages shorthand became associated with witchcraft and fell

into disrepute, and it was not until the late 12th century that King Henry II revived the use of Tironian shorthand. It is interesting that many famous writers throughout history preferred shorthand – Cicero’s orations, Martin Luther’s sermons, and Shakespeare’s and George Bernard Shaw’s plays were all written in a style of shorthand.

The next alphabet presented is particularly interesting because it uses a systematic graphical design approach of eight basic shapes in different orientations to denote the whole alphabet: Stenographie [4] (1602, Fig. 2).

Figure 2. Stenographie Alphabet, 1602

Note that the letters C and K are represented by the same symbol and only a few of the symbols resemble their corresponding current Roman alphabet symbols. In


contrast to cursive shorthand, this is an example of geometric shorthand, which is based on geometrical figures such as circles, ovals, straight lines and combinations of these [5]. The basic shapes (Fig. 3) are I, L, U, (, α, V, Z and +.

Figure 3. Stenographie Alphabet Basic Shapes and Orientations

Several shorthand systems arose in the 19th century spawning the effective

Pitman (1837) that uses stroke thickness to differentiate some letters and Gregg (1885) phonetic alphabets, and the Braille (1824) system for the blind. Several cursive shorthands such as the 1834 Gabelsberger system [5] also originated during this time. None of these are covered here.

The next alphabet presented is Moon Type [7] (1894, Fig. 4), named for its English inventor William Moon. It is a system of embossed letters for the blind that was designed to require less finger sensitivity than Braille targeting those blinded late in life. It consists of eight basic shapes derived from the Roman capital letters and used in varying orientations to denote the whole alphabet – the basic shapes (arranged by column in the figure) are V, J, C, L, I, Z, an angle shape, and O. Over half of these symbols resemble in some respect their corresponding current Roman alphabet symbols – eight completely and perhaps seven partially.

Figure 4. Moon Alphabet, 1894


3 Offline (OCR) Handwriting Recognition

Offline handwriting recognition, a subset of optical character recognition (OCR), uses optical scanning equipment to convert handwriting images into bit patterns for machine recognition. Although most OCR work has been on machine-printed characters, considerable effort has also been devoted to handwriting. A handwritten document can be scanned at any time after it is written, even many years later. No special equipment is necessary during the writing process, in contrast to online recognition. However, the information captured is static and does not contain any of the handwriting dynamics.

Recognizing handprinted characters is considerably more difficult than recognizing machine printed characters due to the great variation among writers in drawing the characters. In order to reduce these variations and increase recognition accuracy, the OCR Committee of the American National Standards Institute (ANSI) developed a standard in 1974 [15] (Fig. 5). The extra strokes on the O and S were introduced to reduce ambiguity. Other countries have also developed standards. Unfortunately, however, most writers of the documents processed by these systems do not use these standards.

Figure 5. ANSI Standard Alphabet for OCR

4 Online Handwriting Recognition

4.1 What is it?

Online or real-time handwriting recognition means that the machine recognizes the writing while the user writes. In contrast to offline, online devices capture the temporal or dynamic information of the writing. This information consists of the number of strokes, the order of the strokes, the direction of the writing of each stroke, and the speed of writing within each stroke. A stroke is the writing from pen down to pen up. In English, uppercase handprint averages about two strokes and lowercase about one stroke, while cursive averages less than one stroke per letter.

This dynamic information can be helpful – for example, in distinguishing the statically similar two-stroke 5 from the one-stroke S. It can also complicate the recognition process with having to handle many variations of a character, some of which, like the one-stroke 5, may be similar to other characters. As an example of the large number of possible variations, the letter E can be written with one (in


cursive fashion), two, three, or four strokes, and with various stroke orders and directions, and many variations appear the same when completed. The four-stroke E consisting of a vertical and three horizontal strokes (or, for that matter, any four-stroke character) has 48 variations (24 different stroke orders and two stroke directions for each stroke).

Unlike offline recognition, special equipment is required during the writing process. Tablet digitizers (electronic tablets), available since the 1950’s, allow the capture of handwriting and drawing by accurately recording the x-y coordinate data of pen-tip movement. The advent of pen computers in the 1980’s combined digitizers and flat displays bringing input and output into the same surface to provide immediate electronic ink feedback of the digitized writing and mimicking the famil iar pen and paper paradigm to provide a "paper-like" interface [12, 14, 16].

With a pen computer, or a pen-enabled computer, users can not only use the pen (writing stylus) as a mouse but also write or draw as they do with pen and paper. Keyboard entry can be mimicked by touching sequences of buttons on a "soft" keyboard displayed on the screen or, alternatively, handwriting can be automatically converted to ASCII code.

Researchers have worked on many pattern recognition problems for handwriting and drawing on tablet digitizers. The more common problems for English include the recognition of handprinted (discrete) characters and the recognition of cursive writing. Available handwriting recognition products are highly accurate on careful handprinting and some products are available that recognize cursive script with accuracy dependent on the writing style and the regularity and clarity of the writing. In this paper we deal only with handprinting.

4.2 History of Recognition Strategies

There are many tradeoffs in designing a handwriting recognition system. For example, at one extreme, the designer puts no constraints on the user and attempts to recognize the user's normal writing. At the other extreme, the writing is severely constrained, requiring the writer to write strokes in a particular order, direction, and graphical specification.

Because computers and tablet digitizers were rather primitive in the late 1950s and 1960s, much of the early work employed simple character segmentation and recognition techniques and required the user to write the characters in a prescribed manner with respect to their stroke number, order, and direction. Later systems tended to be more flexible in that they attempted to handle most of the ways that the alphabet characters are usually written.

Some systems allowed the user to train the system with a user’s particular way of drawing each character. Many of these were actually hybrid systems, allowing a user to train the system, but also recognizing the common variations in the untrained or walk-up mode.


4.3 Handprint Recognition Difficulties and Solution Methodologies

4.3.1 Segmentation

One difficulty of handprint recognition is character segmentation (separation). Although this problem is extreme in cursive writing where several characters can be made with one stroke, it remains a significant problem with handprinted characters because they can consists of one of more strokes and it is often not clear which strokes should be grouped together. Segmentation ambiguities include the well-known character within character problem where, for example, a hand printed lowercase d might be recognized as a cl if drawn with two strokes that are somewhat apart from one another. This problem is alleviated somewhat, however, by the fact that character separation can occur only at stroke ends – not at the end of every stroke but at the end of those that end a character.

Most segmentation methods are applied prior to recognition, either under writer control or during preprocessing (machine processing prior to recognition). The writer-controlled techniques include an explicit signal from the user, temporal separation usually characterized by a time-out signal, writing characters in predefined boxes, and the use of a penlift either alone or in combination with a time-out [16]. The preprocessing techniques typically involve spatial separation, which actually is also under writer control.

Other methods involve recognition. The stroke code method [3, 8, 16] performs stroke-by-stroke recognition prior to segmentation. That is, each stroke is recognized immediately upon pen lift. Then in the general case of multiple stroke characters the resulting stroke codes are combined to obtain the recognized characters.

Another method performs character segmentation in conjunction with recognition. In this method, each stroke end (pen lift) is a potential segmentation point and all combinations of strokes, up to the maximum number of strokes per character, is sent to the recognition component and the results sorted to obtain a final output [13, 16]. This run-on method can recognize characters that run together – those that touch or overlap or even those written on top of one another.

Perhaps the simplest segmentation method is to use an alphabet of only single-stroke characters. Burr [2] uses this method of using a pen lift to separate characters by instructing his writers to write each character with a single stroke. This method can also be viewed as the trivial case of either the stroke code method where each stroke code is a character or the run-on method where only single strokes are sent to the recognizer.

4.3.2 Uppercase versus Lowercase versus Digits

What makes handwritten communication possible is that differences between different characters are more significant than differences between different drawings


of the same character, and this might be considered the fundamental property of writing. This property holds within the subalphabets of uppercase, lowercase, and digits, but not across them. Fig. 6 shows an example of the uppercase I, lowercase l, and number 1 all drawn identically with a single vertical stroke, and the O and 0 drawn identically with an oval.

Figure 6. Different Characters with the Same Shape

The most general solution to this problem is to handle it the way humans do by

using context. This is usually done in a postprocessing (after recognition) phase that uses syntax and possibly semantics to resolve ambiguities.

Simpler solutions are to use different regions of the writing surface for different subalphabets or to use special symbols to put the system into uppercase, lowercase, or digit mode [16].

4.3.3 Dynamic Writing Variation

Dynamic writing variation is the various ways – in stroke number, order, and direction – that characters can be drawn. In some countries, like Japan, there is more homogeneity in writing dynamics because children are taught to write each character in a specific way. In other countries, li ke America, this writing variation is greater. As we saw earlier with the various ways of writing E, dynamic variation can be a serious problem, and it becomes even more complex when combined with the segmentation problem.

The simplest way to handle wide dynamic variation is not to allow it. Thus, many systems, especially the early ones, prescribe the limited ways each character can be written and the writer must conform to these variations.

When allowing many variations, one solution is to reorder the strokes into a normalized form, for example from left to right and top to bottom, and writing direction could be handled similarly. For example, a four stroke E would be reordered into the vertical stroke followed by the three horizontal ones from top to bottom. Another method is to incorporate the more common variations and either inform the users as to what they are or simply let them adapt to the system by trial and error.

Other techniques, like stroke codes or single stroke alphabets mentioned earlier, can also be used to solve this problem.


4.3.4 Sloppiness of the Writing

Perhaps the most difficult problem is sloppy writing, and size and slant variation can also be included here.

The easiest solution, and that used in many systems, is to use an alphabet in which the symbols are quite different from each other, and that is the reason many of the shorthand alphabets are based on a small number of shapes in different orientations. The Roman alphabet, unfortunately, does not have this property since many letter pairs, such as the U and V, are similar.

Solution to the more general problem involves sophisticated recognition algorithms like dynamic programming, Markov models, and Affine transformations [12, 14, 16].

4.4 Future

The ideal recognition system is one that supports handwritten input of Unicode characters and does not impose any constraints on what and where the user writes. The user should also be able to intermingle text with graphics and gestures, and the recognizer should be able to distinguish these kinds of input. Even though recent advances in handwriting recognition are promising, further research is warranted since the above goals have yet to be achieved.

Similar to the ANSI alphabet standard of constrained fonts for OCR handwriting recognition, standards for online drawing of characters has been suggested as appropriate for certain applications due to the higher recognition rate that could be achieved [16].

5 Shorthand Alphabets in Pen Computing

Shorthand in the field of handwriting recognition is well known. Some of the earliest instances were in the field of CAD/CAM applications where symbols were used to represent various graphical items and commands. Later, shorthand was used to represent scientific symbols and notations, and Pitman shorthand was also implemented. Other systems used special alphabets and symbols for online character recognition and we present and discuss several of these in this section.

Any of the three historical alphabets presented above could be used for machine recognition and all of the symbols of those alphabets are drawn with a single stroke except for the K in Tironian and the + in Stenographie. In addition to shape and orientation, online systems can also use stroke direction to differentiate among symbols. We present and discuss four alphabet systems: Organek, Allen, Goldberg, and Graffiti. Stroke direction of online symbols is usually indicated either with a dot at the start of the stroke or with an arrow at the end.


The Organek system uses straight lines in eight different directions and three different lengths [9] (Fig. 7). Each of these alphabet symbols is drawn with a single stroke (starting at the origin in the figure) and sequences of the symbols can be connected to form words. This alphabet has only one basic shape, a straight line, in four orientations (the same orientations used in other geometric shorthands), three lengths, and two stroke directions to represent the first 24 letters of the alphabet. The y and z are represented in a second wheel along with numerals and other characters. These symbols do not resemble those of the Roman alphabet.

Figure 7. Organek Alphabet

The Allen alphabet [1] (Fig. 8) consists of four basic shapes in various

orientations and stroke directions to denote the whole alphabet, and each of these alphabet symbols is drawn with a single stroke.


Figure 8. Allen Alphabet

The four basic stroke shapes (Fig. 9) are a straight line, and three two-line-

segment strokes with angle changes of 45, 90, and 135 degrees. The straight stroke has four orientations and two possible writing directions for a total of eight possibili ties. Each of the angle change strokes has eight orientations but the writing direction is not used as a differentiator – that is, both writing directions are used to represent the same letter. Therefore, a total of 32 symbols can be represented with this alphabet.


Figure 9. Allen Alphabet Basic Shapes and Orientations

Although there can be little similarity of these symbols to their Roman

counterparts, given the symbol constraints, there is some attempt in that direction - the I matches exactly and the F and T match partially (the same as in Graffiti, see below), as does A (but not as closely as Graff iti). The vowels, except for A, are made from straight strokes presumably for speed of input since vowels are high frequency letters.

The Goldberg alphabet [6] (Fig. 10) is designed for accurate machine recognition and speed of entry.

Figure 10. Goldberg Alphabet

Goldberg’s alphabet symbols come from five basic shapes (I, L, tight U, Z, and α) each rotated in four different orientations (Fig. 11), and each capable of being drawn with two stroke directions. Thus, with the five shapes, four orientations, and two stroke directions, up to 40 different output symbols can be represented.

Figure 11. Goldberg Alphabet Basic Shapes and Orientations

The symbols are graphically well separated from each other for ease of machine

recognition and, within the design constraints, several of the alphabet symbols are similar to their Roman, mostly lowercase, counterparts. While the specially drawn symbols of this alphabet, and to a lesser extent the two preceding alphabets, are easy to recognize automatically, the disadvantage is that one must remember the unique way to draw the symbols and consistently draw each symbol accurately.


The basic shapes are simple so they can be drawn quickly, and to optimize for writing speed the alphabet is chosen to assign the simplest shapes to the frequently used letters. For example, straight strokes are used for the common letters a, e, i, r, and t.

The symbols of this alphabet are single stroke lexical symbols and, as with the Burr system [2], the single stroke per symbol recognition method avoids the common recognition difficulties.

The Graffiti alphabet [10] (Fig. 12) is used in the popular Palm OS devices, notably the Palm Pilot and Handspring models. However, it is not composed of a small number of basic shapes but rather has a high correspondence with the Roman alphabet – twenty letters match exactly (19 with uppercase and one with lowercase Roman alphabet symbols) and six match partially. Of the partial matches, the symbol for A (a typical first stroke of the Roman A) is that of the Tironian, Stenographie, and Moon alphabets; the F (again the first stroke of the Roman F) and K (basically the second stroke of the Roman K) are close to those of the Moon alphabet; T is from Tironian; and Y is a one-stroke way to draw the Roman Y. This high correspondence with the Roman alphabet makes it easier to learn than the basic-shape alphabets described above, but at the sacrifice of graphical separabili ty and speed of entry.

Figure 12. Graff iti Alphabet

This alphabet has one symbol that requires two strokes, the X (although there is

an acceptable one-stroke variant), and the extended alphabet of this system has other multiple stroke characters. This system uses the stroke code recognition method and separate writing areas for the alphabetic and numeric symbols to avoid the common recognition difficulties.

6 Conclusions

Shorthand alphabets are frequently designed for use in pen computing because their symbols are drawn quickly with few strokes and can essentially avoid or at least simplify the common recognition diff iculties. The Graffiti system uses symbols similar to the Roman alphabet for ease of learning. The few symbols that


do not correspond exactly to their Roman counterparts have one stroke in common or other similarities, and several appear to have evolved from the historical alphabets. This system uses the stroke code recognition method and separate writing areas for subalphabets to simpli fy the recognition process. We conclude that the designers have cleverly combined known techniques to create a highly useful system.

In spite of the clever design of Graff iti, we hear that most Palm users prefer the “soft” keyboard to handwriting input, indicating that the devices are popular more for their form factor and other attributes than for their handwriting capability. Finally, we note that the general unconstrained handprint recognition problem is as yet unsolved and will li kely require advanced syntactic and semantic processing capabili ty.

7 Acknowledgements

I thank my colleagues of the graphonomics and handwriting recognition communities for interesting and educational interactions over the years I have worked in this area.

References

1. Allen, G., “Data input grid for computer,” U.S. Patent 5,214,428, 1993. 2. Burr, D.J., “Designing a handwriting reader,” IEEE Transactions Pattern

Analysis Machine Intell igence, Vol. PAMI-5, 1983, pp. 554-559. 3. Crane, H.D., “Process and apparatus involving pattern recognition,” U.S. Patent

4,718,102, 1988. 4. Daniels P.T. and Bright W. (eds.), The World’s Writing Systems, Oxford Press,

1996. 5. Glatte, H., Shorthand Systems of the World, Philosophical Library, 1959. 6. Goldberg, D., “Unistrokes for computerized interpretation of handwriting,”

U.S. Patent 5,596,656, 1997. 7. Gove, P.B. (ed.), Webster’ s Third New International Dictionary, 1986. 8. Loh, S-C, “On-line handwritten character recognition apparatus with non-

ambiguity algorithm,” U.S. Patent 5,034,989, 1991. 9. Organek Technology Ad, “penput – Beyond character recognition!” 1991. 10. Palm Computing, “Palm Pilot: Graffiti Reference Card.” 11. Panati, C., “The Browser's Book of Beginnings,” Houghton Mifflin, 1984. 12. Schomaker L., “From handwriting analysis to pen-computer applications,”

Electronics and Communication Eng. J., 1998, pp. 93-102. 13. Sklarew, R., “Handwritten keyboardless entry computer system,” U.S. Patent

4,972,496, 1990.


14. Subrahmonia J. and Zimmerman T., “Pen computing: challenges and applications,” Proc. 15th Intl. Conf. Pattern Recognition, Vol. 2, 2000, pp. 60-66.

15. Suen, C.Y., “Automatic recognition of handprinted characters – The state of the art,” Proc. IEEE, Vol. 68, No. 4, 1980, pp. 469-487.

16. Tappert C.C., Suen C.Y., and Wakahara T., "The state-of-the-art in on-line handwriting recognition," IEEE Transactions Pattern Analysis Machine Intell igence, Vol. PAMI-12, 1990, pp. 787-808.

SHORTHAND ALPHABETS AND PEN COMPUTING …ctappert/cs631-02/shorthand.pdfShorthand Alphabets and Pen...

Documents

Transcript of SHORTHAND ALPHABETS AND PEN COMPUTING …ctappert/cs631-02/shorthand.pdfShorthand Alphabets and Pen...