The Renci Multitouch Framework
description
Transcript of The Renci Multitouch Framework
The Renci Multitouch Framework
Because touching things is just good fun.
Overview
• Camera Capture• Filtering• Touch Detection• Touch Camera to Stitching Transformation• Tracking• Stitching to Screen Transformation• Gesture Detection
General Structure
Touch Detection & Tracking• Camera Stack
– Filter Graph• Camera• Background• High Pass• Threshold
– Blob Detection– Camera to Stitching Transform
• Tracker– Tracking– Stitch to Screen Transform
Gesture Engine• Acquire New Touches• Test against In-Flight Gestures
& Cull• Cluster Remaining Touches• Compute
– Contraction / Dilation– Rotational Offset
• Clip to registered touch regions• Push To Clients or Emit Mouse
Events
Camera Capture
• Video for Windows• Prosilica• Point Grey• Sony PS3 Eye
Filtering
Blob Detection
• Sample Screen Space given minimum blob size• Utilize a flood-fill style approach to blob region
detection• Back-fill with black to avoid multiple
detections• Compute Centroid & Bounding Box• Projective Transform from Camera to Stitched
Coordinate System
Tracking
• KD-Tree Nearest Neighbor to track a blob frame to frame– Find set of nearest neighbors from KD-Tree– Search along line from previous blob’s gradient to
find nearest blob from given set– Transform from Stitched to Screen Coordinate
System
What Are All These Transforms?
• Math on all the Pixels Is Expensive– So save time by only doing the expensive mathy-
bits on the touches• Optimally we want One Thread per Camera
with Filter Graph & Blob Detection & Transform• Basically were mapping quadrilaterals to other
quadrilaterals: – Camera System to Stitched System– Stitched System to Screen System
So where do we get the Transformation Matrices?
• Calibration and More:– Camera Calibration to remove radial distortion– Stitching Calibration– Screen Calibration– Touch Calibration ( Determine Filter graph Values )
Camera Calibration: There is an App For That
Stitching Calibration: Yep, Another App
Stitching Results on Full Images
Even More Apps: Touch & Screen Calibration