Post on 13-Feb-2016
description
PLT 2007
CSISCSIS
Shorthand Handwriting Recognition for Pen-Centric Interfaces
Charles C. Tappert1 and Jean R. Ward2
1 School of CSIS, Pace University, New York, USA2 Pen Computing Consultant, Massachusetts, USA
PLT 2007
CSISCSIS
Thesis: Pen-Centric, Chatroom-Like Shorthand Interfaces
• Will provide critical infrastructure for many pen-centric applications
• Will provide fast text input• Will have greatest impact on applications
running on small mobile devices
PLT 2007
CSISCSIS
Agenda• Handwriting
– Fundamental Property of Writing– Handwriting Recognition Difficulties
• Historical Shorthand Alphabets• Online (Pen-Centric) Handwriting Recognition
– Online more accurate than Offline Recognition– Online Info Can Complicate Recognition Process– Design Tradeoffs/Decisions
• Pen-Centric Shorthand Alphabets• Pen-Centric Word/Phrase Shorthand• Allegro/Chatroom Experimental Shorthand System
PLT 2007
CSISCSIS
Fundamental Property of Writing
• Differences between different characters are more significant than differences between different drawings of the same character
• This makes handwritten communication possible
PLT 2007
CSISCSIS
Fundamental Property of Writing
• Property holds within subalphabets of uppercase, lowercase, and digits, but not across them
• “I”, “l”, and “1” written with single vertical stroke• “O” and “0” written similarly with an oval
PLT 2007
CSISCSIS
Handwriting Recognition Difficulties
• Shape, size, and slant variation• Similarly shaped characters – U and V • Careless writing
– in the extreme, almost illegible writing• Resolving difficult ambiguities requires
sophisticated recognition algorithms, syntax/semantics
PLT 2007
CSISCSIS
Historical Shorthand Alphabets(prior to pen computing)
• Famous writings were written in shorthand – Cicero’s orations– Martin Luther’s sermons– Shakespeare’s and George Bernard Shaw’s plays
• We focus on shorthand appropriate for PDAs• Two main types of shorthand
– Non-geometric shorthand– Geometric shorthand
• Small number of basic shapes• Shapes reused in multiple orientations
PLT 2007
CSISCSIS
Tironian Alphabet, 63 B.C.
PLT 2007
CSISCSIS
Stenographie Alphabet, 1602
PLT 2007
CSISCSIS
Stenographie Alphabet, 1602
Geometric shorthand – basic shapes/orientations
PLT 2007
CSISCSIS
Moon Alphabet, 1894• Geometric shorthand – basic shapes/orientations
PLT 2007
CSISCSIS
Other Historical Shorthand Systems
• Phonetic alphabets– Pitman (1837)– Gregg (1885)
• Systems for the blind – Braille (1824)
• Cursive shorthands – Gabelsberger (1834)
PLT 2007
CSISCSIS
Online (Pen-Centric) Handwriting Recognition
• Machine recognizes the writing as the user writes • Digitizer equipment captures the dynamic
information of the writing – Stroke number, order, direction, speed – A stroke is the writing from pen down to pen up
PLT 2007
CSISCSIS
Online (Pen-Centric) more accurate than Offline (Static) Recognition
• Can use both dynamic and static information• Can often distinguish between similarly
shaped characters – E.g., 5 versus S where the 5 is usually written
with two strokes and the S with one stroke
PLT 2007
CSISCSIS
Online Information Can Complicate Recognition Process
• Segmentation ambiguities– Character-within-character problem – cl versus d
• Large number of possible variations – E can be written with one, two, three, or four strokes, and with various
stroke orders and directions – Four-stroke E has 384 variations (4! stroke orders x 24 stroke directions)
PLT 2007
CSISCSIS
Design Tradeoffs/Decisions
• No constraints on the user– Machine recognizes user's normal writing
• User severely constrained– Must write in particular style such as handprint – Must write strokes in particular order, direction,
and graphical specification• Simplest is one stroke per character, one stroke
direction, one shape
PLT 2007
CSISCSIS
Pen-Centric Shorthand Alphabets
• Some of the earliest were for CAD/CAM• Others developed for text input on PDAs • We review geometric and non-geometric
shorthands appropriate for small devices• Historical alphabets presented above could be
used for machine recognition• In addition to shape and orientation, stroke
direction can differentiate among symbols
PLT 2007
CSISCSIS
Allen Alphabet
PLT 2007
CSISCSIS
Allen AlphabetBasic Shapes and Orientations
PLT 2007
CSISCSIS
Goldberg Alphabet
PLT 2007
CSISCSIS
Goldberg AlphabetBasic Shapes and Orientations
PLT 2007
CSISCSIS
Graffiti Alphabet(non geometric )
PLT 2007
CSISCSIS
Allegro Alphabet (non geometric)
PLT 2007
CSISCSIS
Simplified Design Tradeoffs/Decisions for Graffiti and Allegro PDA Alphabets
• Small alphabet– one case rather than both upper and lowercase
• Small number of writing variations per letter– preferably only one
• One stroke per character (character = stroke)– allows machine to recognize each character upon pen lift
• Separate writing areas for letters and digits– avoids confusion of similarly shaped letters and digits
PLT 2007
CSISCSIS
Graffiti and AllegroCommercially Successful Shorthands
• High correspondence to Roman alphabet– Easier to learn– Graffiti used in Palm OS devices
• notably the Palm Pilot and Handspring models – Allegro used in Microsoft Windows devices
• Geometric alphabets not successful
PLT 2007
CSISCSIS
Pen-Centric Word/Phrase Shorthande.g., Chatroom Shorthand
• Further increase speed of text entry• Potential applications
– Where input speed important– Where word/phrase abbreviations occur
frequently – e.g., email
PLT 2007
CSISCSIS
Allegro/Chatroom Shorthand System
• Developed for M.S. dissertation– Student was hearing impaired– Developed as output component of
communication system• Handwriting to text to speech
• Two input writing areas– One for Allegro (all-purpose)– One for chatroom-like words/phrases (e.g., CUL, F2F)
PLT 2007
CSISCSIS
Allegro/Chatroom Shorthand SystemStroke acquisition GUI
allegro strokerecognition
alphabet
sentence accumulator
Sentence display and spoken output
allegro strokelibrary
user-defined stroke library
a single stroke
other strokerecognition
word/phrasecharacter
done? no
yes
meaning
is it
PLT 2007
CSISCSIS
Allegro/Chatroom Shorthand System
PLT 2007
CSISCSIS
Allegro/Chatroom Shorthand System Preliminary Experimental Results
• Allegro/Chatroom pen-centric shorthand input faster than typing text and comparable to typing text and chatroom shorthand characters
PLT 2007
CSISCSIS
Conclusions: Pen-Centric, Chatroom-Like Shorthand Interfaces
• Will provide critical infrastructure for many pen-centric applications
• Will provide fast text input• Will have greatest impact on applications
running on small mobile devices