Natural Input On Mobile PC Systems Bodin Dresevic Development Manager Tablet PC Group, Mobile PCs...
-
Upload
shanon-nash -
Category
Documents
-
view
221 -
download
0
Transcript of Natural Input On Mobile PC Systems Bodin Dresevic Development Manager Tablet PC Group, Mobile PCs...
Natural Input On Mobile PC Systems
Bodin DresevicDevelopment ManagerTablet PC Group, Mobile PCs Divisionbodind @ microsoft.comMicrosoft Corporation
Session OutlineSession Outline
Applications of Natural Input
Speech Input
Pen Input
Touch Input
Summary
Call to ActionCall to Action
Microsoft will continue investing in making Natural Input more ubiquitous and easier to use
OEMs and IHVs should take advantage of the software innovations in Natural Input
It takes both to bring the best experience to our customers and to succeed in today’s competitive marketplace!
Applications of Natural InputApplications of Natural Input
Text InputDictation & handwriting recognition
Natural annotation and capture Ink or audio annotations on text
Ink or audio notes, recordings or emails
Natural Navigation and ActivationSpeech commands
Pen & Touch used as mouse and for gestures
AuthenticationBiometric: signature, fingerprint etc.
Hardware Buttons to provide access tosystem features
Ctrl-Alt-Delete, change resolution, switch orientation, scroll wheel etc
SpeechSpeech
User benefits
Software support in Windows
Hardware considerations
Recommended solutions
Call to Action
Speech – End User BenefitsSpeech – End User Benefits
Speech Input offers distinct advantages in certain scenarios / situations
For users with repetitive stress injuries
For devices without a keyboard: slate Tablet PCs, PDAs, smart phones
Solutions exist for vision impaired
Audio capture: voice as voice record a meeting, attach voice note to email message, audio sticky
Text to SpeechAccessibility applications: only US and CHS engines offeredin Windows codenamed “Longhorn”
Speech Input or Text to Speech on the serverMultiple applications in telephony, e.g. caller ID
Software Support for Speech in Windows XPSoftware Support for Speech in Windows XP
Support built in Tablet PC Edition 2005 (Windows XP SP2)
Speech input integrated in Tablet Input Panel
Mode switch required between Dictation and Command & Control
Accuracy in Commanding mode superior to that in Dictation mode
Training the speech recognition engine highly recommended
Planned Software Support for SpeechPlanned Software Support for Speechin Longhornin Longhorn
Focused improvement on user experience
Dedicated UI for speech input on desktop
Integrated Dictation and Commanding experience
On Tablet PC the new UI lives side by side with Pen TIP
Engine training is being replaced by interactive tutorial
25 minutes in training and learning speech commands greatly improves overall experience
Accuracy of the speech engines is greatly improved
More languages: US, UK, German, French, Spanish, Chinese Traditional, Chinese Simplified and Japanese
Software support for microphone array technology
Audio Hardware ConsiderationsAudio Hardware Considerations
Command & Control: constrained recognition outputWorks well with almost any reasonable microphone
Dictation requires high-quality audio stream 16KHz / 16-bit, better than 20db signal-to-noise ratio
Usually headset is recommendedAnalog or USB microphones meet the dictation requirement
Bluetooth 2.0 headsets are likely to meet the dictation requirement
Microphone arrays can meet dictation requirementsFor < 2 feet distance and normal noise conditions comparable to close-talk microphone
Use three or more in non linear arrangement at uneven distances
Inexpensive: $5 manufacturing cost (processing in software)
Today’s hardware limited with stereo input paths(2 input channels)
Speech Input DemoSpeech Input Demo
Tablet PC running Longhorn
Email dictation with correction and editing commands
Deleting word, making it bold or italic, etc.
Call to ActionCall to Action
Improve the sound capturing quality: integrate microphone arrays in your devices
For more info please attend the following session:
Microphone Array Support in Longhorn
Pen InputPen Input
User benefits
Software support in Windows
Hardware Considerations
Recommended solutions
Call to Action
Pen Input – User BenefitsPen Input – User Benefits
Handwriting capture as Ink for note takingand drawing
Handwriting recognition for text entry through TIP
Simple Pen gestures for basic navigationTap, double-tap, drag
press & hold for right-click
Gestures for advanced navigation shortcutsFlicks for scrolling up and down, browser backand forward
Additional flicks for copy, paste, delete, undo
Scratch out in TIP, etc.
Pen Input DemosPen Input Demos
Note taking in OneNoteink “flows” from the pen in real time
Search ink notes, apply bullets to text outline
Text Input via recognition: Longhorn TIPEdge targeting: bring TIP from the edge of the screen
Auto-complete in the Run dialog
Auto-complete in IE address bar
Auto-complete stock quotes in IE
Scratch out improvements: x gesture, horizontal line over a word, old scratch out
For password input fields TIP switches to soft keyboard and back to lined mode for user name (Hotmail)
Pen Demo - Shell ImprovementsPen Demo - Shell Improvements
New visual feedback for basic Pen gesturesTap: a drop in the water, concentric circles
Barrel Button: blue circle around cursor that flashes upon leaving the surface
Press and Hold: circle grows and flashes whenleaving surface
Cursors switch between new pen cursor and traditional mouse cursor
Checkboxes allow us to do multiple selection without doing ctl + (multiple mouse clicks)
Pen Demos – FlicksPen Demos – Flicks
Flicks can be configured in 2 modes:Basic: N = page down, S = page up, E = forward, W = back
Advanced: NE = paste, SE = delete, NW = copy, SW = undo
Control Panel applet allows the flicks to be remapped, their sensitivity adjusted
Notepad demo: open new document, delete it (SE) and undo (SW)
Outlook demos: Flick up and down through a long email
Flick emails out of the inbox (SE), occasional undo (SW)
Copy (NW) / paste (NE) emails between two folders
Flick back / forth between Inbox and Calendar
Pen Input – Software Support in WindowsPen Input – Software Support in Windows
In Windows XPWindows XP Tablet PC Edition 2005
is a superset of Windows XP SP2 Professional
Contains support for pen input, inking and handwriting recognition
In Longhorn, planning to take Tablet functionality mainstream
Pen Input and inking available on hardware units with digitizer
Handwriting recognition and ink viewing available even on desktop
Improved end user and developer experience
Pen Input – Hardware ConsiderationsPen Input – Hardware Considerations
For best recognition accuracy use a high-resolutionEM digitizer
Recommended resolution: > 9K x 6K, sampling rate > 133Hz
User’s top pain points when using penAccurate targeting of small objects: check boxes, icons or buttons
Accidental right click activation when stylus button is depressed
Today’s solutions require a user to calibrate the digitizerThe calibration utility is hard to discover
Calibration has to be done in landscape and portrait modes
Calibration does not remove the problem entirely
Calibration is required becauseDigitizer’s EM field is non-linear especially near the edges
Parallax caused by:the high position of the coil inside the stylus
Glass thickness and optical properties
Pen Input – Call to ActionPen Input – Call to Action
Continue innovating and improving digitizers built into Tablet PC units
Suggested ideas to reduce parallax and need for end user calibration
Better OEM calibration
“Tuck” the digitizer under the screen edges
Lower the position of the coil in the stylus
Or look for alternative technology to enable Hover and Stylus tracking above the digitizer surface
Use thinner glass with better optical properties
Recommendations (con’t)Recommendations (con’t)
Provide innovative stylus designs – remove possibility of accidental button activation
Adjust screen surface smoothness to provide paper-like writing experience
The Input experience is inextricably tied to the good screen readability
Build great LCD screens…With wide viewing angles
That enable outdoor readability – use low reflection glass and have auto-adjustable display brightness
TouchTouch
User benefits
Software support in Windows
Hardware Considerations
Recommended solutions
Call to Action
Touch – End User BenefitsTouch – End User Benefits
In Longhorn we are planning to add Touch as a new mode for Navigation and Control
Great complement to Pen and Speech Input
All navigation with Pen also works with Touch but is more efficient and intuitive - no pen required
Mouse operations: Tap and Double tap with finger
Advanced gestures: Finger flicks
Longhorn plans to enable easy precision targeting with finger
Stylus still preferred for ink input
Touch Input DemoTouch Input DemoUI is Under Construction – It May Change!UI is Under Construction – It May Change!
Demo on retrofitted Tablet PC with N-Trig digitizer
Left click/double click on larger targets is performed simply by tapping on the target
For small targets Longhorn provides intuitive Targeting Ring UI
To right click on a targeteither depress the Right Click button on the Ring when the pointer is above the desired target
Or simply press & hold the target with a finger
Finger visual metaphors are very similar to the pen ones
Touch is disabled when Stylus is in proximity (<1cm)
Palm rejection – important when pen is not in proximityBlunt, low-pressure contact is ignored
Existing Pen and Touch Digitizer TechnologiesExisting Pen and Touch Digitizer Technologies
Digitizer Type
EM(Tablet V1,V2) Capacitive Resistive
Input Device
“pointy stick” Gloved finger
Finger EM stylus “”
Longhorn Software Architecture for TouchLonghorn Software Architecture for Touch
For touch to work correctly on Longhorn system
HID Layer (HIDCLASS.SYS)
EM digitizer mini-driver Touch digitizer mini-driverEM-Touch digitizer mini-driver
If the unit has an integrated dual mode digitizer (such as N-Trig), HID layer still needs to enumerate Touch and EM digitizers as two virtual independent HID devices
If a unit has two independent digitizer devices (Touch screen and EM digitizer), each one needs an independent HID mini-driver (could be the same but listed twice in HID layer)
Software Support (con’t) OEM / IHV GuidelinesSoftware Support (con’t) OEM / IHV Guidelines
Technical Requirements for Touch DigitizerHID usage page is 0X0D (misc. devices) and the Usage is 4 (touch screen)
Packet Properties – still evolving requirementsRequired: x, y
Optional: area of contact, pressure
Highly recommended: confidence (that this is a finger as opposed to palm)
Confidence is very important for correct mode auto-switching decisions.
Longhorn does the final arbitration when to turn Touch on or off
The OS can take advantage of the context knowledge (the active application, buttons, scroll bar etc positions)
Start button can always be touched even when pen is close
Application Behavior and TouchApplication Behavior and Touch
Current applications – pre-LonghornPen aware: treat Finger and Stylus as same – show OneNote
Pen non-aware (ordinary windows apps): Touch generates mouse messages and facilitates navigation
New applications can take advantage of TouchTablet Platform APIs (Ink Collector and RTS) are extended:
New method to inquire about the types of digitizer devices Stylus, Touch or mouse
propagate digitizer information (x,y, area) with pen events
Enables building of modal apps, Stylus vs. TouchDemo: app that uses stylus for inking and finger for erasing
Avalon Inking Platform supports Touch at the same level as COM / Winforms Tablet PC Platform
Shipping Digitizer Mini-Driver with LonghornShipping Digitizer Mini-Driver with Longhorn
Digitizer device manufacturer has a responsibility to provide the HID mini-driver(s) for its device(s)
Microsoft is building Windows Logo Program Test Suite for (EM and touch) digitizer mini-drivers
Examples of what the Test Suite may checkScreen/digitizer bounds aligned
Linearity / calibration (edges and hover)
Noise: Hold the pen still and ensure x, y change minimally
Minimal lag / distance between pen and cursor
Sample rate and digitizer resolutions sufficient
Button state permutations register correctly
Basic system gestures tap, double tap, and press & hold work
To ship “in the box” must pass the Test Suite
The Test Suite spec not available yet, stay tuned
Call to ActionCall to Action
Add Touch support to your Tablet PC units and greatly enhance user’s navigation experience
Build in a single integrated dual mode digitizer into your Tablet PC
Or build a traditional Touch digitizer and a traditional EM digitizer into your Tablet PC unit
In either case Longhorn will do the arbitration for you – so long as the touch mini-driver meetsthe specifications
Community ResourcesCommunity Resources
Windows Hardware & Driver Central (WHDC)www.microsoft.com/whdc/default.mspx
Technical Communitieswww.microsoft.com/communities/products/default.mspx
Non-Microsoft Community Siteswww.microsoft.com/communities/related/default.mspx
Microsoft Public Newsgroupswww.microsoft.com/communities/newsgroups
Technical Chats and Webcastswww.microsoft.com/communities/chats/default.mspx
www.microsoft.com/webcasts
Microsoft Blogswww.microsoft.com/communities/blogs
Additional ResourcesAdditional Resources
Related SessionsMicrophone Array Support in Windows LonghornFull Client Usability for the Highly Mobile Market
Building digitizer mini-driver for LonghornWindows Logo Program Test Suite for digitizers
Email: burtp @ microsoft.com
N-Trig Web site: http://www.n-trig.comTouch support in Tablet API for Longhorn
Longhorn Platform SDK (at the moment Touch not represented yet)http://msdn.microsoft.com/longhorn
Support for Tablet functionality and Touch in Avalon API (WAP1 spec)
http://www.microsoft.com/downloads/details.aspx?FamilyID=c8f904e1-b4ca-402b-accf-aaa2bd60da74&displaylang=enhttp://msdn.microsoft.com/longhorn/understanding/pillars/avalon/default.aspx
Summary Summary
Continue innovating and improving hardware that enables Natural Input
Microsoft is working on complementing your advances with software advances in the ease of use of Natural Input and and core improvements in the recognition engines
Only together we can bring the best Natural Input experience to our customers
© 2005 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.