Pen Research Jay Pittman Development Lead Tablet PC Handwriting Recognition Microsoft Corporation...
-
Upload
vanessa-wilkins -
Category
Documents
-
view
224 -
download
0
Transcript of Pen Research Jay Pittman Development Lead Tablet PC Handwriting Recognition Microsoft Corporation...
Pen ResearchPen Research
Jay PittmanDevelopment LeadTablet PC Handwriting RecognitionMicrosoft Corporation
Jay PittmanDevelopment LeadTablet PC Handwriting RecognitionMicrosoft Corporation
TDNNTDNN
Time-Delayed Neural Network
Ink is cut into segments via simple algorithmFor Latin script, we just cut at bottoms
For Arabic or Hindi, we may need a different algorithm
Features are computed per segmentChebychev coefficients
Variety of size metrics: width, height, offsets to neighbors, etc.
Each supported letter or character has an output nodeMany have 2 outputs, for beginning of character and continuation of character
Very large training sets are collected from tens of thousands of native writers
Time-Delayed Neural Network
Ink is cut into segments via simple algorithmFor Latin script, we just cut at bottoms
For Arabic or Hindi, we may need a different algorithm
Features are computed per segmentChebychev coefficients
Variety of size metrics: width, height, offsets to neighbors, etc.
Each supported letter or character has an output nodeMany have 2 outputs, for beginning of character and continuation of character
Very large training sets are collected from tens of thousands of native writers
Beam SearchBeam Search
Very similar to speech recognition systems
System lexiconSimple list of words (like a spellchecker)
Stored as a trie
Plus regular expressions for numbers, dates, times, currency, phone numbers, etc.
Low-scoring sequences are “culled” from the trie as we go
Converts character recognizer into word recognizerWe can recognize a word even if you mangle one of the letters
Supports sloppier (and therefore faster) handwriting
Very similar to speech recognition systems
System lexiconSimple list of words (like a spellchecker)
Stored as a trie
Plus regular expressions for numbers, dates, times, currency, phone numbers, etc.
Low-scoring sequences are “culled” from the trie as we go
Converts character recognizer into word recognizerWe can recognize a word even if you mangle one of the letters
Supports sloppier (and therefore faster) handwriting
88 8 682263574 4461575723
923151 9 4720
711252 8 7913
5318792857 6
……
…
1381 8 2 14 3
1717 5 7 4390
7 1657914415
Output Matrix
dog 68
clog 57
dug 51
doom 42
divvy 37
ooze 35
cloy 34
doxy 29
client 22
dozy 13
Ink Segments
Top 10 List
d 92
a 88
b 14
c 86
o 77
a 73
l 76
t 5
g 68
t 8
b 6
o 65
g 57
t 12
TDNN
SpaceTDNN
a
b
do
g
ab
t
tc
l
og
t
Lexicon
e
a
…
…
…
…
… Beam Search
a*a
d*d
g*g
o*o
TDNN + Beam SearchTDNN + Beam Search
PersonalizationPersonalization
New In Vista
Shape adaptationCollect samples from you
Simple idea: continue same training we do at Microsoft, but only on your samples
Implicit
Explicit
Text adaptationCollect your personal words from Word and outgoing emails
New In Vista
Shape adaptationCollect samples from you
Simple idea: continue same training we do at Microsoft, but only on your samples
Implicit
Explicit
Text adaptationCollect your personal words from Word and outgoing emails
Capital I?
dev RTM KKOMO dogfooding
Qi trie featurize Herry
Open SystemOpen SystemRecognizer API is published
Any recognizer may support this API
A non-Microsoft recognizer can be installed, and it will be invoked by the inking platform
Non-Microsoft recognizers available now in Japan, China, and Russia
Compete with my group
Or cover languages we don’t cover
Or cover non-text (music, math, chemistry)Sorry: recognition result API is very text oriented
Strings in Unicode
Perhaps your text might be XML
Recognizer API is publishedAny recognizer may support this API
A non-Microsoft recognizer can be installed, and it will be invoked by the inking platform
Non-Microsoft recognizers available now in Japan, China, and Russia
Compete with my group
Or cover languages we don’t cover
Or cover non-text (music, math, chemistry)Sorry: recognition result API is very text oriented
Strings in Unicode
Perhaps your text might be XML
http://www.microsoft.com/downloads/details.aspx?FamilyID=69640b5c-0ee9-421e-8d5c-d40debee36c2&displaylang=en
Microsoft Tablet PC / Tablet PC Platform / Programming the Tablet PC / Creating a RecognizerStart / All Programs / Microsoft Tablet PC Platform SDK / Microsoft Tablet PC Platform SDK Documentation
Latin OrthographyLatin Orthography
XP has U.S. English, U.K. English, German, French, Spanish, and Italian
Vista adds Dutch and Brazilian Portuguese
Currently working on Swedish, Danish, Norwegian (Bokmal), Finnish, Polish, Czech, Portugal Portuguese, Catalan, Romanian, Croatian, and Serbian
No ship estimates are available
Serbian is written in both Latin and CyrillicMaking plans for the next batch: Probably Bahasa Indonese (Indonesian), Hungarian, Turkish, Slovak, Slovene, Lithuanian, Estonian, Latvian, Vietnamese, Tagalog (Filipino), others TBD
World’s largest orthographyLargest count of languages
Largest combined count of literates
XP has U.S. English, U.K. English, German, French, Spanish, and Italian
Vista adds Dutch and Brazilian Portuguese
Currently working on Swedish, Danish, Norwegian (Bokmal), Finnish, Polish, Czech, Portugal Portuguese, Catalan, Romanian, Croatian, and Serbian
No ship estimates are available
Serbian is written in both Latin and CyrillicMaking plans for the next batch: Probably Bahasa Indonese (Indonesian), Hungarian, Turkish, Slovak, Slovene, Lithuanian, Estonian, Latvian, Vietnamese, Tagalog (Filipino), others TBD
World’s largest orthographyLargest count of languages
Largest combined count of literates
East Asian OrthographiesIdeographic OrthographiesEast Asian OrthographiesIdeographic Orthographies
Completely different code baseFocus on large count of characters
XP has Japanese, Chinese (Simplified), Chinese (Traditional), and Korean
Vista adds personalization, and improves cursive recognition
Completely different code baseFocus on large count of characters
XP has Japanese, Chinese (Simplified), Chinese (Traditional), and Korean
Vista adds personalization, and improves cursive recognition
Cyrillic and Greek OrthographiesCyrillic and Greek Orthographies
Same TDNN + beam search technology works equally well in Greek and Cyrillic
But we must collect new training sets
Cyrillic has more upper/lowercase confusion
Russian Collection completedCurrently working on recognizer
No estimate on shipping date
Some work also underway in SerbianCyrillic: Russian, Ukrainian, Bulgarian, Serbian, Byelorussian, Macedonian, Kazakh
Other former-Soviet Turkic republics are in a state of transition back to Latin scripts
Serbian is written in both Latin and Cyrillic
Same TDNN + beam search technology works equally well in Greek and Cyrillic
But we must collect new training sets
Cyrillic has more upper/lowercase confusion
Russian Collection completedCurrently working on recognizer
No estimate on shipping date
Some work also underway in SerbianCyrillic: Russian, Ukrainian, Bulgarian, Serbian, Byelorussian, Macedonian, Kazakh
Other former-Soviet Turkic republics are in a state of transition back to Latin scripts
Serbian is written in both Latin and Cyrillic
GreekGreek
ΕλληνικήέδωσανμουγλώσσαΤη
Ink by Dr. John Drakopoulos
I was given a language that is Hellenic
RussianRussian
Ink by Vladimir Smirnov
Кремль. Кремль, Ото слышал всех я про
Kremlin, Kremlin. I’ve heard from everyone about [it] Kremlin, Kremlin. I’ve heard from everyone about [it]
Bi-Directional OrthographiesArabic and HebrewBi-Directional OrthographiesArabic and Hebrew
Text written right-to-left, but numbers written left-to-rightArabic has its own digits
Hebrew uses “western” digits
No uppercase / lowercase distinction
Arabic is cursive-onlyUp to 4 forms per letter (initial, medial, final, isolated)
Hebrew is print-only5 letters have a separate final form
Abjabs (consonant alphabets)
We have collections underwaySome initial experimentation in both languages
No estimate on shipping dateArabic script variations used for Farsi (Persian), Urdu, Kurdish, Azeri (in Iran), Pashto, Darwi, Baluchi, Sindhi (in Pakistan), Kashmiri (in Pakistan)
Text written right-to-left, but numbers written left-to-rightArabic has its own digits
Hebrew uses “western” digits
No uppercase / lowercase distinction
Arabic is cursive-onlyUp to 4 forms per letter (initial, medial, final, isolated)
Hebrew is print-only5 letters have a separate final form
Abjabs (consonant alphabets)
We have collections underwaySome initial experimentation in both languages
No estimate on shipping dateArabic script variations used for Farsi (Persian), Urdu, Kurdish, Azeri (in Iran), Pashto, Darwi, Baluchi, Sindhi (in Pakistan), Kashmiri (in Pakistan)
Bi-Directional ExampleArabicBi-Directional ExampleArabic
١٢٣123
عمره[his] age
� عًامًاYear[s]
Ink by Ahmed Kamal
Bi-Directional ExampleHebrewBi-Directional ExampleHebrew
.מבבל
Plimpton 322 [is] from Babylonia.
322 פלימפטון
Ink by Ethan Zoller
Brahmic OrthographiesBrahmic OrthographiesLeft to right, no uppercase / lowercase
Abugidas (syllabic alphabets)Default vowel (short “Ə” or “uh” sound) is not written
All other vowels require a vowel sign added to the consonantThis includes a “no-vowel sign”
Called halant (“choked”) in Hindi, virama in other Indic languages
Vowel sound follows consonant sound
There are also independent vowels
Hindi Devanagari collection in progressSome initial experimentation
No estimate on shipping date
Brahmic scripts: Devanagari (Hindi, Marathi), Thai, Bengali, Gujarati, Gurmukhi (Punjabi), Tamil, Telugu, Kannada, Malayalam, Oriya, Sinhala, Khmer, Lao, Tibetan, Myanmar, Sindhi (in India), Kashmiri (in India)
Left to right, no uppercase / lowercase
Abugidas (syllabic alphabets)Default vowel (short “Ə” or “uh” sound) is not written
All other vowels require a vowel sign added to the consonantThis includes a “no-vowel sign”
Called halant (“choked”) in Hindi, virama in other Indic languages
Vowel sound follows consonant sound
There are also independent vowels
Hindi Devanagari collection in progressSome initial experimentation
No estimate on shipping date
Brahmic scripts: Devanagari (Hindi, Marathi), Thai, Bengali, Gujarati, Gurmukhi (Punjabi), Tamil, Telugu, Kannada, Malayalam, Oriya, Sinhala, Khmer, Lao, Tibetan, Myanmar, Sindhi (in India), Kashmiri (in India)
Vowel SignsVowel Signs
“KAA” “KU” “KUU”“KO” “KOW”
“KE” “KEE”“KA” “KAI”“KAY”
“KR”
“K”
Consonant ClustersConsonant Clusters
S T halant R I N Ghalant
Unicode buffer:
A transliteration of the English word “string”
Brahmic ComparisonBrahmic Comparison
K Kh G Gh C Ch
Devanagari: क ख ग घ च छBengali: ক খ গ ঘ চ ছGurmukhi: ਕ ਖ ਗ ਘ ਚ ਛGujarati: ક ખ ગ ઘ ચ છTamil: க சTelugu: క ఖ గ ఘ చ ఛKannada: ಕ ಖ ಗ ಘ ಚ ಛ
K Kh G Gh C Ch
Devanagari: क ख ग घ च छBengali: ক খ গ ঘ চ ছGurmukhi: ਕ ਖ ਗ ਘ ਚ ਛGujarati: ક ખ ગ ઘ ચ છTamil: க சTelugu: క ఖ గ ఘ చ ఛKannada: ಕ ಖ ಗ ಘ ಚ ಛ
Demo TitleDemo Title
NameTitleGroup
NameTitleGroup
Early FeedbackEarly Feedback
As for Vista, I am in love with the B2 Tablet implementation! I can see where the effort is going! The TIP is awesome and the handwriting recognition is vastly improved. There is hardly a time where I have to correct my input - no matter how sloppy I am.
Unsolicited feedback from a Vista Beta II user whose identity I do not know
© 2006 Microsoft Corporation. All rights reserved.Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation.Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft,and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation.MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
Backup SlidesBackup Slides
Devanaguri ExampleDevanaguri Example
प“PA”
[R sign]period[abbreviation]
headlineheadline
ल“LA”
period[abbreviation]
headline
म“MA”
ि�[vowel sign]
“I”
� [vowel sign]
“AA”
ि�[vowel sign]
“I”
Pr. Ltd. [Private Limited]Pr. Ltd. [Private Limited]