Developing Windows 7 based Multi-Touch Application

Ramaprasanna ChellamuthuMicrosoft India

Agenda

Touch HardwareKey Touch ScenariosWindows 7 Touch InvestmentsTouch Application Development Guidance

GestureManipulation and Inertia

DemoResources

Multi-Touch…

…creates new opportunities.Software that is intuitive for users to walk up & masterSoftware that is collaborative with multiple users

…is not new.Researchers have been exploring this stuff for decades

…is now going mainstream.1. Hardware: Robust touch-capable hardware from Microsoft and OEMs

1. Dozens of Win7 touch devices now available2. OS: Windows 7 has great touch support baked in3. SDKs: Microsoft is making touch easy to leverage4. You: Developers & designers creating innovative apps that take

advantage of the technology

Windows Touch is hereWindows Touch is defined as two or more independently traceable touch points.

ConsumersTouch interfaces have proven successful Multi-Touch offers clear differentiation with a high “WOW” factor Multi-touch enhances on-the-go use and enable new scenarios

HardwareMulti-touch capable machines in market today in a broad set of form factors

Navigating and Consuming the Web

Playing Casual Games

Reading and Sorting Email

Viewing Photos

1

2

3

4

5

6 Navigating Files and Using Applications

Consuming Music and Video

Key Touch Scenarios Are Consumptive

Windows InvestmentInternet Explorer®

Pan and zoom gestures, flick forward/back“Open link in new tab” gestureAddress bar drag menu

Windows® Media Center:Live gallery panningOn-screen keyboardLarger transport controls & seekNew Now Playing & details pages

Photo Viewer (& Windows Live Photo Gallery)Multitouch Zoom and rotation

XPS ViewerStretch zoom centered around gestureSmart zoom

Touch Pack for Windows 7

Windows 7 Principle: Be Great at “Look” and “Do”

Users have expectations for how an experience will “Look” and what it should “Do”Direct Manipulation demands more from “Look” and “Do”Touch Pack for Windows 7

Surface Globe: Immersive, 2-D and 3-D touch world exploration:

Zoom in on locationsPinpoint areas of interestGet local informationPersonalize map views

Development TiersBest

Better

Good

Raw data points from WM_TOUCH for custom gestures. Manipulation/inertia APIs

Multi-touch support – zoom, finger pan, rotate, panning with inertia, and 2 finger tap. WM_GESTURE

Maps to common Windows messages – pan, zoom, right click, and double click

Gestures: Consistency is key

Gestures: Touch actions that are one step removed from direct manipulations

Two finger tapPress and tap Pinch/stretch (zoom)Circular motion (rotate)Single and two-finger panningFlicks

Useful for shortcut-like efficiencyConsistent operations like zoom in, or navigateFocus on content : to surface extra information or invoke contextual action UI

Design ConsiderationsConsistency

Should work in the same way across Windows applicationsKeep it simple – users won’t discover or remember too many abstractions

Windows 7 Touch Gestures

WM_GESTURE

Windows Message and APIs:

WM_GESTURE, WM_GESTURENOTIFY

GESTURECONFIG struct

Set/GetGestureConfig()

GESTUREINFO

GetGestureinfo()

Setting Gestures and WM_GESTURE case WM_GESTURENOTIFY: { BOOL bResult;

GESTURECONFIG gcAll[] = {0,GC_ALLGESTURES,0}; // Enable all gestures bResult = SetGestureConfig(hWnd, 0, 1, gcAll, sizeof(GESTURECONFIG));

} case WM_GESTURE: {

GESTUREINFO gi; gi.cbSize = sizeof(gi); bResult = GetGestureInfo((HGESTUREINFO)lParam, &gi); switch (gi.dwID) { case GID_BEGIN:

break; case GID_END:

break; case GID_ZOOM: //GID_PAN, GID_ROTATE, GID_TWOFINGERTAP, GID_PRESSANDTAP

break; }

WM_TOUCH

WM_TOUCH* family of messagesEnables touch-optimized experiencesProvides standard Win32 messages

WM_TOUCHDOWN, WM_TOUCHMOVE, WM_TOUCHUPSemantically similar to mouse messagesConveys raw touch data to Win32 apps

Scenario examples:Finger painting, custom gestures, feeding higher level controls, etc

.NET 3.5 Multitouch library

Multi-touch for Windows 7

NativeWindows SDK

.NET 4WPF

SilverlightSilverlight3

Gestures Available via manipulation

Raw Touch

Manipulation

Inertia

Manipulation Container

Canvas

Manipulation Container

IsManipulationEnabled=true

IsManipulationEnabled=true

Update RenderTranform

Handle ManipulationDelta Event

• Coordinates relative to Manipulation Container

Handle ManipulationStarting Event• Set Manipulation

Container

Manipulation Events

Starting Started Delta Inertia Starting Delta Complete

d

StartInertia() StartInertia()

Touch Down (Initial) Touch Move Touch Up

(All)

Initialize:•Mode• Container• Pivot

Completed()

WPF 4 Multi-Touch APIs Overview

WPF Controls Styles with Panning Enabled

ScrollViewer Panning Support

Manipulation Events

Touch Events

Extensible Touch Device

Win7 Touch Device Surface Touch Device

Beta 1

Beta 2

Release Candidate

Surface V2

Silverlight 3 Touch APIs - Listening

public partial class MainPage : UserControl { public MainPage() { ... // listen to touch events from the system Touch.FrameReported += new TouchFrameEventHandler(OnFrame); }

void OnFrame(object sender, TouchFrameEventArgs e) { // enumerate and respond to touch events }}

Silverlight 3 Touch APIs - Processing

void Touch_FrameReported(object sender, TouchFrameEventArgs e) { TouchPointCollection touchPoints = e.GetTouchPoints(LayoutRoot);

foreach (TouchPoint tp in touchPoints) { if (tp.Action == TouchAction.Down) { // a new touch has come down } if (tp.Action == TouchAction.Move) { // a previously down touch has moved } if (tp.Action == TouchAction.Up) { // a touch has been removed } }}

Inertia BehaviorsManipulationInertiaStartingEventArgs

Properties Units

TranslationBehavior InitialVelocity 1/96th DIP per millisec

DesiredDeceleration 1/96th DIP per millisec^2

DesiredDisplacement 1/96th DIP

RotationBehavior InitialVelocity Degrees per millisec

DesiredDeceleration Decrees per millisec^2

DesiredRotation Degrees

ExpansionBehavior InitialVelocity 1/96th DIP per millisec

InitialRadius 1/96th DIP

DesiredDeceleration 1/96th DIP per millisec^2

DesiredExpansion 1/96th DIPS

• Deceleration and Displacement/Rotation/Expansion mutually exclusive• Deceleration – useful to simulate friction• Displacement – useful on page flips• Default – no inertial movements

Demo !! Demo !! Demo !!

Surface is not just multi-touch

Windows 7 & WPF 4 are a great baseline for touch

Surface creates unique opportunities with specialized HW

Robust topHorizontal form factorInfrared camera array

Surface: Beyond Multi-Touch

Massive multi-touch

Multi-user experiences

Object recognition

Optical engineering

Surface: Multi-User

Trend: Software continues to reduce the need for face-to-face human interactions

Problem: Happiness and creativity depend on social interactions

Surface brings people togetherIndustrial design encourages users to gatherInput capabilities allow simultaneous useApps facilitate collaboration / competition

Surface: Massive Multi-Touch

Surface can efficiently track 50+ simultaneous fingers and objects

Surface: Objects

Uniquely identify & react to billions of tagged objects

Byte Tags256 unique values

Identity Tags>340,282,366,920,938,000,000,000,000,000,000,000,000

unique values

Tags include orientation indictors dots

Surface: Optics

Determine the size and shape of every inputSee into and project onto physical objects

>>FUTURE

What’s next for Surface hardware?

Things we’re exploring:Cheaper ThinnerVertical

Stay tuned…

ResourcesSoftware Design

Windows User Experience Interaction GuidelinesMulti-Touch in Windows 7Developer Enhancements to Windows Touch and Tablet PC(User Perceptions of Response Time) Performance UX Patterns

Developer ResourcesCode Magazine: Multi-touch capabilities in Windows 7Windows Touch on MSDNWindows Touch Developer ResourcesTablet and Touch SDKIntroducing the Microsoft Touch Pack for Windows 7

Resources

www.microsoft.com/teched

Sessions On-Demand & Community Microsoft Certification & Training Resources

Resources for IT Professionals Resources for Developers

www.microsoft.com/learning

http://microsoft.com/technet http://microsoft.com/msdn

Learning

Thank you !!