Download - Lip Feature Extraction Using Red Exclusion

Lip Feature Extraction Using Red Exclusion

Trent W. Lewis and David M.W. Powers

Flinders University of SA

VIP2000

01/12/2000 Lip Feature Extraction Using Red Exlucsion

2

Overview

• Context

• Lip Feature Extraction– Related Work (greyscale, horizontal edges, red and hue colour spaces)

– Red Exclusion

• AVSR: Results and Issues

• Summary


3

Context

• Audio Speech Recognition (ASR)

• Psycholinguistic Research

• Audio Visual Speech Recognition (AVSR)


4

Context - ASR

• Up to 99% word accuracy

• However,– limited context– limited vocabulary– trained on individual– close microphone, cannot handle noise


5

Context - Psycholinguistic

• McGurk Effect– A[ba] + V[ga] [da]

• Viseme– visual phonemes– form complementary sets

• Demonstrates vision can assist the perception of speech AVSR


6

Context - AVSR

• Acoustic Features• Visual Features

– width

– height

– oral cavity

• Integration– Early

– Late


7

Lip Feature Extraction

• Pixel-Based Model


8


• Pixel-Based Model– raw pixels or minimal processing– retain linguistically relevant data– large amounts of data, time– shift and lighting variant– normalisation and PCA


9


• Pixel-Based Model– reduced input to set of hand-crafted features– width, height, average intensity, etc.– less features, time– model fitting, time– lose linguistically relevant features


10


• Pixel-Based Model– feature extraction– Steps

• preprocess to enhance contrast

• locate mouth edges

• identify corners, height, and other key features

• train recogntion engine

Our Approach


11


• Database

1 2 3


12


• Preprocessing Techniques– Grey-scale– Horizontal Edges– Red Analysis– Hue, Saturation, and Value (HSV)– Red Exclusion


13


• Grey-scale– vertical position of mouth

• minimum row sum

– threshold minimum row• average of min and max of row

– search for above threshold pixels


14


• Grey-scale


15


• Grey-scale


16


• Horizontal Edges– high horizontal edge content– 3x3, DY Prewitt operator

111

000

111

DY


17


• Horizontal Edges

“Found” Corners Binary Image


18


• Red Analysis– overcome bearded subjects– used for face location

limlim UG

RL


19


• Red Analysis


20


• HSV– disentangles illumination from colour Illumination > Hue

otherwise

whhw

hhhf o

o

,0

,)(

1)( 2

2


21


• HSV


22


• Red Exclusion– needed extraction method for AVSR– similar to Red Analysis

• face predominantly red

• variations occur in the blue and green colours

B

Glog


23


• Red Exclusion


24


• Corners found using Red Exclusion


25


• ComparisonAlgorithm

Subject 1 female

Subject 2 male bearded

Subject 3 male thin lips

Reliable Corners

Other Features

Grey-scale

Edge

Red Analysis

HSV Red Exclusion


26

AVSR: Results and Issues

• Application for red exclusion

• Used in finding lip features– Width– Height– Key pixels


27


• Visual Speech RecognitionStatic (%) Dynamic (%)

Voicing 32.2 30.8

Viseme 54.7 51.3

Phoneme 14.7 13.6


28


• AVSR - IntegrationEarly

Static

Early

Dynamic

Late

Voice/Vis

Late Error

Voice/Vis

Phoneme20.1 18.0 29.0 19.5


29

Summary

• Vision can help ASR– AVSR

• Needed good extraction technique– Red Exclusion

• AVSR is difficult when both signals degraded


30

Questions?

Trent W. Lewis

BSc (Cognitive Science)

Flinders University

[email protected]