Developing Multi-touch Applications

Reed TownsendProgram ManagerMicrosoft Corporation

Anson TsaoProgram ManagerMicrosoft Corporation

PC03

Multi-touch Is Here!

Hardware Multi-touch capable machines in

market today, multi-touch appears in a broad set of form factors

Software Windows 7, PDC partner

demonstrations & Touch platform announcement, .Net 4.0

Consumers New scenarios, next wave of user

experience, high “WOW” factor

Multi-touch in Action

Reed TownsendProgram ManagerWindows

demo

Agenda

Introduction Touch scenarios, “Good, Better, Best” model,

and developer roadmap Platform details:

Native Win32 gesture and touch support Manipulations and Inertia WPF support

UX guidelines for touch applications Where to get started - SDK, drivers, etc Wrap up

Touch Scenarios And Windows 7

We’ve identified several key scenarios for multi-touch:

Navigating and consuming the web

Reading & sorting email

Viewing photos

Playing casual games

Consuming music and video

Navigating files & arranging windows

Using Office applications

All focused on consumption

That led to four areas of investment: Developer Platform: At the root is the

touch developer platform that exposes touch APIs for any application

UI Enhancements: Focusing on the core scenarios, many parts of the core UI have been optimized for touch experiences.

Gestures: Multi-touch gestures have been added to enable consistent panning and zooming in most applications.

Applications: A set of multi-touch focused applications that demonstrate the power of touch. These will ship separate from Win7.

A Structured Approach To Adding Touch Support To Your Application

Which application scenarios get the biggest benefit from touch? Content consumption, mobile, media

and entertainment

Decide which tier of investment is appropriate: Good – Free support, UX tweaks Better – Gesture support, touch-friendly UX Best – Touch-optimized experience

Surface Hardware

Windows 7

Surface Hardware

Windows VistaWindows 7

NativeWin32

Application

Touch Development Roadmap

WPF 3.5

Surface SDK1.0

Managed Wrapper and Interop

Managed Wrapper and Interop

WinForms Application

Windows 7 ReleaseNET 4.0 / Surface 2.0 Release

WPF 3.5 SP1

WPF 4.0

Surface SDK 2.0

WPF Application

Surface Application

Multi-Touch Controls

Multi-Touch API

Surface Multi-Touch

Controls & API

Multi-Touch API

Multi-Touch API and Controls

Touch Platform Overview

Good Better BestAPIs For Free!

•Panning/zoom gestures•Right click gesture

•Gesture notifications•Pan/zoom/rotate/etc

•Raw touch data•Manipulation and

Inertia processors

Native Win32

•Controls with standard scrollbars

•WM_GESTURE message •WM_TOUCH•COM based

Manipulation and Inertia Processors

WPF •WPF 4.0 pan support in ScrollViewer

•Gesture events• Inertia configuration

• Touch events•Manipulation and Inertia

Processors

WinForms •Controls with standard scrollbars

•WM_GESTURE message• P/Invoke

•Manipulation and Inertia Processors in Microsoft.Ink.DLL•Real-time Stylus or Ink

Collector

Windows 7 Gestures & WM_GESTURECOMMAND

Reed TownsendProgram ManagerWindows

demo

WM_GESTURECOMMAND

WM_GESTURECOMMAND Better: Good gesture support is key Window message conveying gesture notifications

and information to applications Why use it? Richer than legacy notifications,

smoother more responsive scrolling and zooming

Compound gestures Coalescing Message bubbling and capture Legacy handling & SFP

WM_GESTURECOMMAND

BOOL WINAPI GetGestureCommandInfo( __in UINT uMsg, __in WPARAM wParam, __in LPARAM lParam, __in LPARAM lExtraInfo, __inout PGESTUREINFO pGestureInfo

);

typedef struct _GESTURECOMMANDINFO {UINT cbSize;DWORD dwFlags;DWORD dwCommand;DWORD dwArguments;POINTS ptsLocation;

} GESTURECOMMANDINFO, *PGESTURECOMMANDINFO;

WM_GESTURECOMMAND

Interpreting gestures:dwCommand dwArguments ptsLocation

Pan Distance between contacts

Current center of gesture

Zoom Distance between contacts

Current center of gesture

Rotate Absolute angle on rotate start, delta on updates

Current center of gesture

Two-finger tap NA Current center of gesture

Right click gesture NA First finger down

WM_GESTUREFuture Windows 7 Build

Minor API surface changes: Naming: WM_GESTURE, GESTUREINFO,

GetGestureInfo(), etc Additional fields in GESTUREINFO

Added gesture configuration support: WM_GESTURENOTIFY GESTURECONFIG struct Set/GetGestureConfig()

Single-finger Panning Future Windows 7 Build

Single-finger panning (SFP) provides an even easier way to pan where it is natural

How to control SFP in your application: SetGestureConfig() for GC_PAN Specify GC_PAN_WITH_SINGLE_FINGER_VERTICALLY

and/or GC_PAN_WITH_SINGLE_FINGER_HORIZONTALLY

If you aren’t getting the SFP default: Consider standard scrollbars SBM_GETSCROLLINFO (remarks here are helpful) Implement WM_GESTURE support

WM_TOUCH

Reed TownsendProgram ManagerWindows

demo

WM_TOUCH

WM_TOUCH* family of messages Best: WM_TOUCH enables touch-optimized

experiences WM_TOUCHDOWN, WM_TOUCHMOVE,

WM_TOUCHUP Semantically similar to mouse messages Conveys raw touch data to Win32 apps Scenario examples:

Finger painting, custom gestures, feeding higher level controls, etc

Capture Coalescing

WM_TOUCH

BOOL WINAPI RegisterTouchWindow( __in HWND hwnd, __in ULONG ulFlags);

BOOL WINAPI GetTouchInputInfo( __in HANDLE hTouchInput, // input event handle __in UINT cInputs, // number of elts in the array __out_ecount(cInputs) PTOUCHINPUT pInputs, // array of touch inputs __in int cbSize); // sizeof(TOUCHINPUT)

typedef struct tagTOUCHINPUT { LONG x; LONG y; HANDLE hSource; DWORD dwID; DWORD dwFlags; DWORD dwMask; DWORD dwTime; ULONG_PTR dwExtraInfo; DWORD cxContact; DWORD cyContact;} TOUCHINPUT, *PTOUCHINPUT;typedef TOUCHINPUT const * PCTOUCHINPUT;

BOOL WINAPI CloseTouchInputHandle( __in HANDLE hTouchInput); // input event handle

WM_TOUCH Future Windows 7 Build

WM_TOUCH consolidated to a single message

Manipulations & Inertia

Reed TownsendProgram ManagerWindows

demo

Manipulations

Best: Manipulations are a great foundation for touch-optimized experiences

Examples of higher level APIs/controls Manipulations are:

2D affine transformations (translate, scale, rotate) Superset of supported gestures Supports multiple concurrent manipulations

Need a source of raw data: WM_TOUCH or RTS Similar to Surface APIs

Interfaces: IManipulationProcessor IManipulationEvents

Inertia

Provides basic physics Works hand in hand with Manipulations

Interfaces: IInertiaProcessor IManipulationEvents – Same event interface as

Manipulations

Multi-touch In .Net Framework

For WPF: 3.5 update near Windows 7 launch

Multi-touch, gesture APIs 4.0 release

Multi-touch specific new controls

For WinForms (Windows 7 launch): Interop to native Win32 APIs – see sample Use managed Manipulations/Inertia wrappers

Multi-Touch In WPF

UIElement and UIElement3D changes Gesture events (tracking) Touch system gesture events (single) Raw touch events Touch and touch digitizer devices

Multi-touch support in controls: ScrollViewer update to accept pan gestures Base controls updated to be multi-touch aware Multi-capture support New multi-touch specific controls

Compatible with Surface SDK 2.0

Multi-touch WPF Application

Anson TsaoSenior Program ManagerWPF

demo

Gestures In WPF

Gesture events on UIElement & UIElement3D TouchGestureStarted, TouchGestureChanged,

TouchGestureEnded NOT routed, to enable multi-capture scenarios

Gesture in WPF based on Manipulations Enable Inertia at TouchGestureStarted

Inertial movements as TouchGestureChanged Events

Touch system gestures

Gestures in WPF

Anson TsaoSenior Program ManagerWPF

demo

Touch Events In WPF

Touch Events on UIElement and UIElement3D TouchDown, TouchMove, TouchUp TouchEnter, TouchLeave Preview variants

Promotion: Touch -> Stylus -> Mouse Multi-Capture

CaptureTouchDevice(TouchDevice) ReleaseTouchDeviceCapture(TouchDevice) ReleaseAllTouchDeviceCaptures() TouchDeviceCollection TouchDevicesCaptured { get; } TouchDeviceCollection TouchDevicesCapturedWithin { get; } IsAnyTouchDeviceCaptured(); IsAnyTouchDeviceCapturedWithin();

Touch Events In WPF

Anson TsaoSenior Program ManagerWPF

demo

Tablet Platform Extensions

Real-time Stylus (RTS) RTS is an excellent raw touch data provider IRealTimeStylus3::MultiTouchEnabled()

Getting multi-touch data via InkCollector is a little trickier. E.g.:

ATOM m_atom = ::GlobalAddAtom(MICROSOFT_TABLETPENSERVICE_PROPERTY);m_dwProperty = TABLET_ENABLE_MULTITOUCHDATA;::SetProp(m_hwnd, (LPTSTR)m_atomPenService, (HANDLE)m_dwProperty);

UX Considerations

General guidelines: Big targets Whitespace Avoid hover See UX guidelines

Be aware of hardware: Form factor Avoid on-hover UI Edges, jitter, etc

Gesture guidelines: Use common gestures Gestures need to be

intuitive and natural Moving away from

shortcut style gestures to manipulations and direct manipulation

Now Is The Time!

ISVs are enthusiastic about multi-touch! We are working with several ISVs today on

multi-touch applications, including: AutoDesk Avoco Secure Corel Cyberlink Fuel Industries Identity Mine Sonic

Call To Action

Decide which application scenarios make sense for touch

Decide which tier of investment is appropriate

Build something amazing!

Getting Started

What do you need to start coding? Hardware:

Multi-touch beta drivers for in-market devices:HP TouchSmart 2 Dell Latitude XT

Software: Window 7 M3 build – On the drive Touch Platform SDK and samples – On the driveNot in the Windows Platform SDK you received earlier!

WPF Multi-touch .NET 4.0 Preview Releases in 2009 .NET 3.5 update to be announced

Additional Resources

Related PDC Sessions: PC46 – WPF Roadmap PC17 – Developing for Microsoft Surface

Hands on Lab: PCHOL18: Developing Multi-touch Applications in Windows 7

PDC Whitepaper – Windows 7: Reach Out and Touch It!Product Pavilion – In the Windows boothAsk the Experts & Presentation and Client track lounge

Contact us: wintouch@microsoft.com

Evals & Recordings

Please fill

out your

evaluation for

this session at:

This session will be available as a recording at:

www.microsoftpdc.com

Please use the microphones provided

Q&A

© 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market

conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Backup Slides

Now Is The Time!

Multi-touch offers clear differentiation with a high “WOW” factor for consumers

Multi-touch enhances on-the-go use and opens up new scenarios for business

90% of people in focus groups claimed they would pay a 20-30% premium for a PC with multi-touch

Seeing is believing: Consumer purchase intent doubles after hands-on experience

Now Is The Time!ISVs are investing in multi-touch

ISVs are enthusiastic about multi-touch! We are working with several ISVs today on

multi-touch applications, including: AutoDesk Avoco Secure Corel Cyberlink Fuel Industries Identity Mine Sonic

Multi-Touch Capable PCs Are Available Today

HP TouchSmartAll-in-one PC with 2-touch digitizer

from NextWindow

Dell Latitude XTLaptop with multi-touch and pen

digitizer from N-Trig

Multi-touch & Windows 7

Multi-touch is a major investment area and strong reason for customers to upgrade to a Windows 7 PC

MT scored very favorably in Concept Value testing

Consumer awareness and interest in touch is high Customers perceive touch as the next stage of computing Touch offers a sense of personal connection and freedom Touch would help users be more efficient and be fun to use

Media response to MT disclosure has been very positive:

>130 unique articles, 1M video views in 72hrs, continued monthly coverage

WM_GESTURE Future Windows 7 Buildtypedef struct tagGESTUREINFO { UINT cbSize; // size, in bytes, of this structure

// (including variable length Args field) DWORD dwFlags; // see GF_* flags DWORD dwID; // gesture ID, see GID_* defines HWND hwndTarget; // handle to window targeted by this gesture POINTS ptsLocation; // current location of this gesture DWORD dwInstanceID; // internally used DWORD dwSequenceID; // internally used ULONGLONG ullArguments; // arguments for gestures whose arguments fit // in 8 BYTES UINT cbExtraArgs; // size, in bytes, of extra arguments, if // any, that accompany this gesture} GESTUREINFO, *PGESTUREINFO;

BOOL WINAPI GetGestureInfo( __in HGESTUREINFO hGestureInfo, __out PGESTUREINFO pGestureInfo);

WM_GESTURENOTIFY Future Windows 7 Buildtypedef struct tagGESTURECONFIG { DWORD dwID; DWORD dwWant; DWORD dwBlock; } GESTURECONFIG, *PGESTURECONFIG;

BOOL WINAPI SetGestureConfig( __in HWND hwnd, __in DWORD dwReserved, __in UINT cIDs, __in_ecount(cIDs) PGESTURECONFIG pGestureConfig, __in UINT cbSize);

BOOL WINAPI GetGestureConfig( __in HWND hwnd, __in DWORD dwReserved, __in DWORD dwFlags, __in PUINT pcIDs, __inout_ecount(*pcIDs) PGESTURECONFIG pGestureConfig,__in UINT cbSize);

SFP & SMB_GETSCROLLINFO

To help ensure SFP can control the scrollbars:

SBM_GETSCROLLINFO, SBM_SETSCROLLINFO, SBM_SETPOS and SBM_GETPOS

They also only have to do this if their scrollbar is not a system scrollbar (WS_VSCROLL and WS_HSCROLL styles I believe). The APIs we actually use would be the following which internally send these messages to the application: GetScrollInfo, GetScrollPos, GetScrollbarInfo, SetScrollInfo (may not need to use this one), SetScrollPos.

See the remarks section of this topic to see what developers need to do to support these messages. http://msdn.microsoft.com/en-us/library/bb787583(VS.85).aspx

We typically detect these types of scrollbars today, it’s just we don’t control them since they don’t pay attention to the WM_VSCROLL and WM_HSCROLL message params since these messages only support 16 bit scroll values. So they don’t need to do anything else for us to recognize their scrollbars (like set WS_VSCROLL). This won’t help windowless controls though since we cannot target portions of a window with this. Those apps would have to implement WM_GC to get their support.

Tell Me More!

What is Multi-touch? Multi-touch is two or more independently

trackable contacts It is not gesture support grafted on top of a

touch or pointing device nor pen and touch together

Choose the right input for the task at hand Touch and multi-touch are good for quick input

tasks, content consumption vs. creation, transactional input