Byron HoodComputer Systems Lab Project 2007-08

Applications of Image Processing to Sign

Language Recognition

Byron HoodComputer Systems Lab Project 2007-08

Overview Average person typing speed

Composing: 19 wpm Transcribing: 33 wpm

Sign speaker Full sign language: 200—220 wpm Spelling: est. 50 wpm

Byron HoodComputer Systems Lab Project 2007-08

Purpose and Scope Native signers faster “typing”

Benefits: Hearing & speaking disabled Sign interpreters

Just letters & numbers

Byron HoodComputer Systems Lab Project 2007-08

Research Original Related projects

Using mechanical gloves Tracking body parts

Image techniques Edge detection Line detection

Byron HoodComputer Systems Lab Project 2007-08

Program Architecture (1)

Webcam interface Using Video 4 Linux 1 Saves images to disk

Edge detection Robert’s Cross

Line detection Hough transform

Byron HoodComputer Systems Lab Project 2007-08

Program Architecture (2)

Line processing Image data dropped AI to find “best fit”

Letter matching Load hand data from XML files Try to match hand

Byron HoodComputer Systems Lab Project 2007-08

Sample Code From edge_detect.c : int row,col;// loop through the imagefor(row=0; row<rows; row++) { for(col=0; col<cols; col++) { // avoid the problems from trying to calculate on an edge if(row==0 || row==rows-1 || col==0 || col==cols-1) { image2[row][col]=0; // so set value to 0 } else { int left = image1[row][col-1]; // some variables int right = image1[row][col+1]; // to make the final int top = image1[row-1][col]; // equation seem a int bottom = image1[row+1][col]; // little bit neater image2[row][col]=(int)(sqrt(pow(left – right , 2) + pow(top - bottom, 2))); } }}

Byron HoodComputer Systems Lab Project 2007-08

Testing (1) General testing model

bhood@bulusan: ~/syslab-tech $ \> ./main images/hand.pgm in.pgm

[DEBUG] Edge detect time: 486 ms[DEBUG] Wrote image file to `in.pgm’

Errors: 0 Warnings: 0

Byron HoodComputer Systems Lab Project 2007-08

Testing (2) Results:

Of edge detection

Similar for cropping or line finding

Automated testing difficult and impractical

Original image(800 x 703)

Final image(800 x 703)

Byron HoodComputer Systems Lab Project 2007-08

The Mysterious Future Finish line detection

Detection and parameterization Build AI to interpret lines

Finish camera-computer interaction

OPTIMIZE!!!