Multi-Touch Navigation Engine

Presented by Team Extra Touch:Chris Jones

Shuopeng YuanNathan Wiedeback

Detailed Design Review

1

THE PLAN

2

Make it interesting…

Challenge• Phantom touches• “[40K] ought to be enough

for anybody!”• Need “tap points” for

intermediate data• We’re not really

programmers…

Solution• Compare to previous frame• Low resolution (240 x 180)

for proof-of-concept• Make each stage’s data

available at the output• Get help!

3

Serial Input

4

Serial Stream Parser

•74 byte “packets” from the screen

•Interrupt-driven I/O on ARM -- until an entire packet received•Then process it to get 32 sensor values, average, etc.

•Normalize by subtracting the lowest value from each •Variations of ~100-200

5

(Little-endian)Average: 0x488B = 18,571First sensor: 0x4886 = 18,566

Interpolation

6

Interpolator

•Input: 32 x 2-byte sensor levels•Output: “Image” frame -- n x m x 1 bit pixel array

•Proof-of-concept: 240 x 180 (memory constraints)

•Find intersection points of sensors (16 x 16)•Interpolate linearly based on distance – first in X, then in Y (bilinear)•Set a threshold and discretize at the end

•Each pixel ends up as a 1 or a 0

7

Image source: Wikipedia

Phantom Filtering

8

Phantom Filter

•Input: Frame•Output: Filtered frame (240 x 180 x 1 bit)

•Intersections method -> phantom touches •Remember what was touched first – i.e., save and compare to the last frame•Proof of concept -- only the simple case covered

9

Pattern Recognition

10

Pattern Recognizer: Flowchart

11

Input: Whole frame (60*20)

12

First step: Blob classification

13

(Visio files provided separately)

14

Output: the outline of each single blob

15

The X-Y Coordinates in separate arrays x y

7 312 36 414 45 515 56 615 66 715 76 814 86 914 96 1014 106 1114 117 1213 127 1314 137 1413 147 1513 157 1613 168 1713 179 1812 18

x y

32 334 331 435 431 534 5

x y

39 1140 1138 1241 1239 1341 13

16

Voronoi Skeleton Algorithm 1

17

The block diagram

18

Voronoi Skeleton Algorithm 2 (The ideal case)

19

Voronoi Skeleton Algorithm(The actual result)

6 5 4.5 6.5 2.5 7.5 -0.5 8.5 -4.5 9.5 -9.5 10.5 8 5 9.5 5.5 10 6.5 10.1053 6.65789 10 7.5 9.5 9.5 7.5 10.5 5.5 12.5 3.5 13.5 0.5 14.5 -3.5 15.5 10 8.5 10 12.5 10.0833 13 10 14.5 8.5 15.5 10 13.5 10 9.33333 10.3 15.5 10 14.5 10 14.75 10.3333 15.6667 10.5 16 11.5 16.5 11.75 6 10 8.5 10 9.33333 10.5 9.5 12.5 10.5 14 12 13 13 14.5 14.5 16.5 15.5 19.5 16.5 13.5 5.5 12.5 6 13.5 6.5 15.5 8.5 17.5 9.5 20.5 10.5 26.5 12

20

Shape recognition

21

Slope algorithm

If a skeleton is linear, according to geometry, angle a1 = a2 = a3 = a4 Therefore, only linear shaped skeleton can pass this filter. To the touch panel application, it means only the side of a palm instead of a palm shape can pass.

22

Channel Assignment

23

Channel AssignmentData In: Center coordinates (X,Y)

Processing:Data gets transformed into a 2d array, and the compared against each channel frame to determine its channel. Channel frames are defined before compile.

Data Out:(X,Y) coordinates with channel data attached.

Details:Aiming for 2 channels initiallyShould be simple to add channels in the final iteration.

24

Channel Assignment – Flow Chart

25

Channel Assignment - Visual Representation

0 0 0 0 0

0 0 1 1 0

1 1 0 1 0

1 1 0 0 0

1 1 0 0 0

Channel 0 Frame0 0 1 1 1

0 0 0 0 1

0 0 0 0 1

0 0 1 1 1

0 0 1 1 1

Channel 1 Frame

Channel 2 Frame1 1 0 0 0

1 1 0 0 0

0 0 1 0 0

0 0 0 0 0

0 0 0 0 0

Output to PC: Coordinates 3,3 Channel 2

Input: Coordinate Data (3,3) touched

We have a match at Channel 2!

26

Questions?

27