1

Keyboard, Speech, and Pen Input in Your Controls

Kevin GjerstadLead Program Managerkevingj@microsoft.com

Session Code: CLI351

Robert ChambersDevelopment Managerrobch@microsoft.com

2

AgendaAgenda

Why Input Matters To Your AppAvalon Input Goals and ArchitectureEvents, Routing, HandlingCode DemosFeature DemosCustomizing Speech in Your AppSummary

Why Input Matters To Your AppAvalon Input Goals and ArchitectureEvents, Routing, HandlingCode DemosFeature DemosCustomizing Speech in Your AppSummary

3

Tools

Client Application Model

Avalon Windows Forms

Web & Service Application Model

ASP.NET / Indigo Win FSCompact

FrameworkYukon Mobile PC Optimized

System.HelpSystem.Help

System.DrawingSystem.Drawing

System.NaturalLanguageServicesSystem.NaturalLanguageServices

Data Systems Application Model

Presentation Data

Mobile PC & Devices Application Model

Communication

Command Line

NT Service

DataSetDataSet

MappingMapping

ObjectSpacesObjectSpaces

ObjectSpaceObjectSpace

QueryQuery

SchemaSchema

ItemItem

RelationshipRelationship

MediaMedia

AudioAudio

VideoVideo

ImagesImages

System.MessagingSystem.Messaging System.DiscoverySystem.Discovery

System.DirectoryServicesSystem.DirectoryServices

System.RemotingSystem.Remoting

System.Runtime.RemotingSystem.Runtime.Remoting

ActiveDirectoryActiveDirectory

UddiUddi

System.Web.ServicesSystem.Web.Services

Web.ServiceWeb.Service

DescriptionDescription

DiscoveryDiscovery

ProtocolsProtocols

System.MessageBusSystem.MessageBus

TransportTransport

PortPort

ChannelChannel

ServiceService

QueueQueue

PubSubPubSub

RouterRouter

System.TimersSystem.Timers

System.GlobalizationSystem.Globalization

System.SerializationSystem.Serialization

System.ThreadingSystem.Threading

System.TextSystem.Text

System.DesignSystem.Design

Base & Application Services

Fundamentals

System.ComponentModelSystem.ComponentModel

System.CodeDomSystem.CodeDom

System.ReflectionSystem.Reflection

System.EnterpriseServicesSystem.EnterpriseServices

System.TransactionsSystem.Transactions

Security

System.Windows.TrustManagementSystem.Windows.TrustManagement

System.Web.SecuritySystem.Web.Security

System.MessageBus.SecuritySystem.MessageBus.Security

AuthorizationAuthorization

AccessControlAccessControl

CredentialsCredentials

CryptographyCryptography

System.Web.ConfigurationSystem.Web.Configuration

System.MessageBus.ConfigurationSystem.MessageBus.Configuration

System.ConfigurationSystem.Configuration

System.ResourcesSystem.ResourcesSystem.ManagementSystem.Management

System.DeploymentSystem.Deployment

System.DiagnosticsSystem.Diagnostics

Configuration Deployment/Management

System.WindowsSystem.Windows System.WindowsSystem.WindowsSystem.Windows.FormsSystem.Windows.Forms

System.ConsoleSystem.Console

System.ServiceProcessSystem.ServiceProcess

System.Windows.FormsSystem.Windows.Forms System.WebSystem.Web System.StorageSystem.Storage System.Data.SqlServ

erSystem.Data.SqlServer

AnimationAnimation

ControlsControls

ControlControl

DesignDesign

PanelPanel

ControlsControls

DialogsDialogs

SideBarSideBar

NotificationNotification

System.WindowsSystem.Windows

DocumentsDocuments

Text ElementText Element

ShapesShapes

ShapeShape

InkInk

UI ElementUI Element ExplorerExplorer MediaMedia

System.Windows.FormsSystem.Windows.Forms

FormsForms

ControlControl

Print DialogPrint Dialog

DesignDesign

System.Web.UISystem.Web.UI

PagePage

ControlControl

HtmlControlsHtmlControls

MobileControlsMobileControls

WebControlsWebControls

AdaptorsAdaptors

DesignDesign

PortsPorts

InteropServicesInteropServices

System.RuntimeSystem.Runtime

System.IOSystem.IO

System.CollectionsSystem.Collections

GenericGeneric

System.SearchSystem.Search

AnnotationsAnnotations

MonitoringMonitoring

LoggingLogging

RelevanceRelevance

System.DataSystem.Data

SqlClientSqlClient

SqlTypesSqlTypes

SqlXMLSqlXML

OdbcClientOdbcClient

OleDbClientOleDbClient

OracleClientOracleClient

CoreCore

ContactContact

LocationLocation

MessageMessage

DocumentDocument

EventEvent

System.StorageSystem.Storage

System.WebSystem.Web

PersonalizationPersonalization

CachingCaching

SessionStateSessionState

System.XmlSystem.Xml

SchemaSchema

SerializationSerialization

XpathXpath

QueryQuery

PermissionsPermissions

PolicyPolicy

PrincipalPrincipal

TokenToken

System.SecuritySystem.Security

System.CollaborationSystem.Collaboration

RealTimeEndpointRealTimeEndpoint

TransientDataSessionTransientDataSession

SignalingSessionSignalingSession

MediaMedia

ActivitiesActivities

HttpWebRequestHttpWebRequest

FtpWebListenerFtpWebListener

SslClientStreamSslClientStream

WebClientWebClient

System.NetSystem.Net

NetworkInformationNetworkInformation

SocketsSockets

CacheCache

System.WebSystem.Web

AdministrationAdministration

ManagementManagement

NavigationNavigation

Peer GroupPeer Group

PolicyPolicy

SerializationSerialization

CompilerServicesCompilerServices

RecognitionRecognition

System.SpeechSystem.Speech

SynthesisSynthesis

System.Windows.InputSystem.Windows.Input

WinFxWinFx

4

No Longer Just a Desktop WorldNo Longer Just a Desktop World

Keyboard, Mouse not enough anymoreInnovation is accelerating

Form FactorsInput DevicesNatural Language Technologies

International requirements are greatAsian markets are fastest growingWorld markets continuing to globalize

Keyboard, Mouse not enough anymoreInnovation is accelerating

Form FactorsInput DevicesNatural Language Technologies

International requirements are greatAsian markets are fastest growingWorld markets continuing to globalize

5

New Forms, Devices, ScenariosNew Forms, Devices, Scenarios

Media Center PC

Tablet PC

Smart Phone

Smart Display

Pocket PC

6

New Input MethodsNew Input MethodsTablet and Ink

Shipping 3rd generation of Tablet technologies

SpeechBuilt-in dictation and commanding support. Managed Speech API for extensibility and customization

KeyboardA suite of keyboard services exploiting new input framework

New input devices Including new navigation hardware, mice, keyboards, microphones, etc.

Tablet and InkShipping 3rd generation of Tablet technologies

SpeechBuilt-in dictation and commanding support. Managed Speech API for extensibility and customization

KeyboardA suite of keyboard services exploiting new input framework

New input devices Including new navigation hardware, mice, keyboards, microphones, etc.

7

Last ten years have seen a great deal of input innovation on Windows but…

Last ten years have seen a great deal of input innovation on Windows but…

8

MouseKeyboardDOS

WinNLS PenWinWin3xC

J

IMM32 SAPI

AIMM

TSF SAPI5

CUAS Ink SDK

Win95

Win98

Win2k

XP

Tablet

MSAA

Single binary

mouse wheel

MouseKeyboard

WinNLS PenWinC

JK

IMM32 SAPI

Windows key

HID

MSAA

AIMM mouse wheel

Single binary

TSF SAPI5 WISP

CUAS Ink SDK

Windows Input API HistoryWindows Input API History

9

Our Goals for AvalonOur Goals for Avalon

Promote Integration and Ease of UseKey functionality automatically enabledFewer scenarios requiring device-specific code

Promote Consistency Across Devices and Languages

Promote rich devices, innovationProvide unprecedented language support

Give developers (you) flexibility by making it extensible, pluggable

Promote Integration and Ease of UseKey functionality automatically enabledFewer scenarios requiring device-specific code

Promote Consistency Across Devices and Languages

Promote rich devices, innovationProvide unprecedented language support

Give developers (you) flexibility by making it extensible, pluggable

10

Input ArchitectureInput Architecture

Hwnd Source

RawInput

stream

StagingArea

Filters- Mouse- Keyboard- Stylus- User-defined

Monitors- Mouse- Keyboard- Stylus- User-defined

Event Routing

Win32Queue

Devices- Mouse- Keyboard- Stylus- User Defined

11

Event routingEvent routing

TunnelingFirst notify the grandparent, then the parent, then the child

Bubbling Opposite of Tunneling - First notify the child, then the parent, then parent’s parent, etc.

Typical patternPreviewFoo event (tunnel) followed by Foo event (bubble)We often talk about these pairs as a single event

TunnelingFirst notify the grandparent, then the parent, then the child

Bubbling Opposite of Tunneling - First notify the child, then the parent, then parent’s parent, etc.

Typical patternPreviewFoo event (tunnel) followed by Foo event (bubble)We often talk about these pairs as a single event

12

Events – Routing ExamplesEvents – Routing Examples

SpacebarTab in buttonTab in editAlt-FUnrecognized key

SpacebarTab in buttonTab in editAlt-FUnrecognized key

Window

Handled

HandledHandled

Handled

Not Handled

Panel

Button Edit

13

Event PromotionEvent Promotion

Route low-level events first, higher level events nextRetain ability for developers who want to get to the device events, keep it simple for those who don’t

Route low-level events first, higher level events nextRetain ability for developers who want to get to the device events, keep it simple for those who don’t

14

Basic Event Handlers

15

Tunnel Handler

16

Stylus & Mouse Handlers

17

Text Input - Integrated TSF SupportText Input - Integrated TSF Support

TSF is an API particularly for input methods requiring access to the document (for context, composition, conversion and correction purposes) such as IMEs, Speech, Handwriting and others. Avalon TSF Support consists of

TextBox implements the text store APIsA TextInput event for controls that don't implement their own text store

TSF is an API particularly for input methods requiring access to the document (for context, composition, conversion and correction purposes) such as IMEs, Speech, Handwriting and others. Avalon TSF Support consists of

TextBox implements the text store APIsA TextInput event for controls that don't implement their own text store

18

TextInput EventTextInput Event

Like other input events in Avalon, Text input events follow the low-level event promoted event patternAvalon will first route the appropriate KeyDown/KeyUp events, and if those are not handled, then a TextInput event.

1. KeyDown EventKeyEventArgs.Key == Key.TextInputKeyEventArgs.Key == Key.ImeProcessed

2. TextInput Event

TextInput event is device and language independent

Like other input events in Avalon, Text input events follow the low-level event promoted event patternAvalon will first route the appropriate KeyDown/KeyUp events, and if those are not handled, then a TextInput event.

1. KeyDown EventKeyEventArgs.Key == Key.TextInputKeyEventArgs.Key == Key.ImeProcessed

2. TextInput Event

TextInput event is device and language independent

19

Integrated Text Input

20

The Open Command

21

CommandingCommanding

Cut, Copy, Paste, Open – these are commands

A command is a form of inputThe “Invoke” event

Command Invoke is often the result of other input

Eg, ctrl-C invokes Copy command“ctrl-C = Copy” is an input bindingCan also call RaiseCommand(…) directly

Cut, Copy, Paste, Open – these are commands

A command is a form of inputThe “Invoke” event

Command Invoke is often the result of other input

Eg, ctrl-C invokes Copy command“ctrl-C = Copy” is an input bindingCan also call RaiseCommand(…) directly

22

Benefits of CommandsBenefits of Commands

Centralized handling logicManage command status (eg, disabling menu items)Leverage existing commandsHandle more input devicesCommand routing

Finding someone to handle the command invoked

Centralized handling logicManage command status (eg, disabling menu items)Leverage existing commandsHandle more input devicesCommand routing

Finding someone to handle the command invoked

23

Command LibraryCommand Library

A default set of common commands like Cut, Copy, Paste, Open, etc

Or define your own command

Each standard command includes:Key bindingsIconPen gestureSpeech infoFuture devices

A default set of common commands like Cut, Copy, Paste, Open, etc

Or define your own command

Each standard command includes:Key bindingsIconPen gestureSpeech infoFuture devices

24

Speech Functionality Built InSpeech Functionality Built In

Speech Recognition engineUser configuration, TrainingMicrophone control + Feedback UICommand and Control

For all on screen elements (buttons, links etc.)For starting or switching applicationsCommanding Control Pattern

DictationDictate into any text fieldDictation commands to correct/edit textStandard Correction User Experience

Speech Recognition engineUser configuration, TrainingMicrophone control + Feedback UICommand and Control

For all on screen elements (buttons, links etc.)For starting or switching applicationsCommanding Control Pattern

DictationDictate into any text fieldDictation commands to correct/edit textStandard Correction User Experience

25

Extending Speech SupportExtending Speech Support

Strengths of speech as an input modalityCan refer to things that are not on the screen

“Start Microsoft Excell”Can be parameterized

“Schedule a meeting tomorrow at 2”Efficient

“Schedule an hour long meeting with my staff and Joe tomorrow afternoon at 2 in the main conference room”

Strengths of speech as an input modalityCan refer to things that are not on the screen

“Start Microsoft Excell”Can be parameterized

“Schedule a meeting tomorrow at 2”Efficient

“Schedule an hour long meeting with my staff and Joe tomorrow afternoon at 2 in the main conference room”

26

System.Speech namespaceSystem.Speech namespaceDesktop-related functionality of COM

API (SAPI) exposed in managed codeObject-centric rather than interface-centricAdditions so application can easily integrate into the system-wide speech user experienceDynamic grammar generation using W3C’s SRGS-derived object modelSDK sample that demonstrates main functionality including semantic properties and recognition alternates

Desktop-related functionality of COM API (SAPI) exposed in managed codeObject-centric rather than interface-centricAdditions so application can easily integrate into the system-wide speech user experienceDynamic grammar generation using W3C’s SRGS-derived object modelSDK sample that demonstrates main functionality including semantic properties and recognition alternates

27

Extending Speech

28

New Device (“XEEL”)New Device (“XEEL”)

View full specifications at View full specifications at http://www.microsoft.com/hwdev/platform/pcdesign/HIG.asp

29

XEEL Navigation

30

SummarySummary

Avalon InputSimpleConsistentExtensible

Avalon InputSimpleConsistentExtensible

31

Community ResourcesGet Your Questions Answered!Community ResourcesGet Your Questions Answered!

Client Lounge: middle of the Exhibit HallConnect with Microsoft client product teams, and PDC 2003 Speakers

Ask The ExpertsTuesday 7pm – 9pm in Hall G, H

Newsgroups:Microsoft.public.windows.developer.winfx.avalon

PDC Weblogshttp://pdcbloggers.net

Longhorn DevCenter:http://msdn.microsoft.com/longhorn

Now! Questions? Comments?

Client Lounge: middle of the Exhibit HallConnect with Microsoft client product teams, and PDC 2003 Speakers

Ask The ExpertsTuesday 7pm – 9pm in Hall G, H

Newsgroups:Microsoft.public.windows.developer.winfx.avalon

PDC Weblogshttp://pdcbloggers.net

Longhorn DevCenter:http://msdn.microsoft.com/longhorn

Now! Questions? Comments?

32© 2003-2004 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.