Face Recognition Using Symmetry and other Face Features Project Lipreading
Advanced Image Processing Student Seminar: Lipreading Method using color extraction method and...
-
Upload
arianna-cantrall -
Category
Documents
-
view
213 -
download
0
Transcript of Advanced Image Processing Student Seminar: Lipreading Method using color extraction method and...
![Page 1: Advanced Image Processing Student Seminar: Lipreading Method using color extraction method and eigenspace technique ( Yasuyuki Nakata and Moritoshi Ando.](https://reader035.fdocuments.in/reader035/viewer/2022062620/551ac4e555034656628b5b55/html5/thumbnails/1.jpg)
Advanced Image ProcessingStudent Seminar:
Lipreading Method using color extraction method and eigenspace technique ( Yasuyuki Nakata and Morito
shi AndoFujitsu Laboratories Ltd., Atsugi, 243-0197 Japan )
MSc in RSIP
Jie Zhang
s0567061
10/02/06
![Page 2: Advanced Image Processing Student Seminar: Lipreading Method using color extraction method and eigenspace technique ( Yasuyuki Nakata and Moritoshi Ando.](https://reader035.fdocuments.in/reader035/viewer/2022062620/551ac4e555034656628b5b55/html5/thumbnails/2.jpg)
Background Why Lip reading?
Human computer interaction Automatically recognize speech contents by
processing lip movement images Potential application for disable facilities
What is the principle? Dealing with frame images sampled from video Lip Feature extraction Lip Feature recognition Various methodology
![Page 3: Advanced Image Processing Student Seminar: Lipreading Method using color extraction method and eigenspace technique ( Yasuyuki Nakata and Moritoshi Ando.](https://reader035.fdocuments.in/reader035/viewer/2022062620/551ac4e555034656628b5b55/html5/thumbnails/3.jpg)
System overview
Lip colour extraction algorithm Colour characteristics of lip fe
ature Roughly mouth detection with
colour algorithm Precise mouth position detecti
on Eigen image Eigen template Matching algorithm
Eigenwaveform recognition Mouth status Dictionary data Thresholding
Training Lip images
Test Lip images
Colour extraction
Eigenvector template build
Colour extraction
Eigentemplate Detection
Eigenwaveform
recognition
![Page 4: Advanced Image Processing Student Seminar: Lipreading Method using color extraction method and eigenspace technique ( Yasuyuki Nakata and Moritoshi Ando.](https://reader035.fdocuments.in/reader035/viewer/2022062620/551ac4e555034656628b5b55/html5/thumbnails/4.jpg)
Colour extraction algorithm
Colour components of face featureStrongly brightness dependence !
Colour system HSV or RGB Evenly luminance Brightness normalization
Lip region colour characteristics Three classes: lip, skin, tee
th Looking for the separation
of spectrum componentsFigure 1
![Page 5: Advanced Image Processing Student Seminar: Lipreading Method using color extraction method and eigenspace technique ( Yasuyuki Nakata and Moritoshi Ando.](https://reader035.fdocuments.in/reader035/viewer/2022062620/551ac4e555034656628b5b55/html5/thumbnails/5.jpg)
Colour extraction algorithm
Figure 2: Brightness dependent face feature colour distribution of normalised RGB image.
Well distinguished in R &G!
![Page 6: Advanced Image Processing Student Seminar: Lipreading Method using color extraction method and eigenspace technique ( Yasuyuki Nakata and Moritoshi Ando.](https://reader035.fdocuments.in/reader035/viewer/2022062620/551ac4e555034656628b5b55/html5/thumbnails/6.jpg)
Colour extraction algorithm
Lip Extraction with colour distribution Threshold function of R an
d G to separate skin and mouth
Label and Extract largest bright area to appropriate size as it is deem to be mouth
Remove teeth and lips with second R and G threshold function
Position determination with four key points of the lips
Figure 3
Lip edge is the combination of teeth and mouth cavity.
![Page 7: Advanced Image Processing Student Seminar: Lipreading Method using color extraction method and eigenspace technique ( Yasuyuki Nakata and Moritoshi Ando.](https://reader035.fdocuments.in/reader035/viewer/2022062620/551ac4e555034656628b5b55/html5/thumbnails/7.jpg)
Eigentemplate method Aim
----- precisely detect the location of lip! Creation of eigen image
Use appropriate colour extracted image (trimmed image)
PCA + one dimension vector for a single trimmed image
Form image series matrix with x vectors
![Page 8: Advanced Image Processing Student Seminar: Lipreading Method using color extraction method and eigenspace technique ( Yasuyuki Nakata and Moritoshi Ando.](https://reader035.fdocuments.in/reader035/viewer/2022062620/551ac4e555034656628b5b55/html5/thumbnails/8.jpg)
Eigentemplate method
Calculate eigenvector
Convert eigenvectors to eigenimage
![Page 9: Advanced Image Processing Student Seminar: Lipreading Method using color extraction method and eigenspace technique ( Yasuyuki Nakata and Moritoshi Ando.](https://reader035.fdocuments.in/reader035/viewer/2022062620/551ac4e555034656628b5b55/html5/thumbnails/9.jpg)
Eigentemplate method
Figure 4
![Page 10: Advanced Image Processing Student Seminar: Lipreading Method using color extraction method and eigenspace technique ( Yasuyuki Nakata and Moritoshi Ando.](https://reader035.fdocuments.in/reader035/viewer/2022062620/551ac4e555034656628b5b55/html5/thumbnails/10.jpg)
Eigentemplate method
Eigentemplate Trimmed test image
Recover eigenimage to template image
Similarity calculationpkp yb
ky
Searching and trimming the image, compare trimmed image will template. Largest similarity gives the exact location.
![Page 11: Advanced Image Processing Student Seminar: Lipreading Method using color extraction method and eigenspace technique ( Yasuyuki Nakata and Moritoshi Ando.](https://reader035.fdocuments.in/reader035/viewer/2022062620/551ac4e555034656628b5b55/html5/thumbnails/11.jpg)
Eigentemplate method
Figure 5
Figure 6
![Page 12: Advanced Image Processing Student Seminar: Lipreading Method using color extraction method and eigenspace technique ( Yasuyuki Nakata and Moritoshi Ando.](https://reader035.fdocuments.in/reader035/viewer/2022062620/551ac4e555034656628b5b55/html5/thumbnails/12.jpg)
Eigenwaveform recognition
Brief introduction to preprocessing A threshold method to detect mouth states and henc
e recognize particular mouth shape Define mouth state: i.e. open, wide open, close, tight
closed… Aspect ratio: distance b between upper and lower ed
ge points over distance a between left and right edge points
Projection vector components: project the image of mouth and lip into eigen space through time scale.
Thresholding
![Page 13: Advanced Image Processing Student Seminar: Lipreading Method using color extraction method and eigenspace technique ( Yasuyuki Nakata and Moritoshi Ando.](https://reader035.fdocuments.in/reader035/viewer/2022062620/551ac4e555034656628b5b55/html5/thumbnails/13.jpg)
Eigenwaveform recognition
Dictionary matching
Figure 7
Small difference sw recognize the test utterance as the same as the template utterance
![Page 14: Advanced Image Processing Student Seminar: Lipreading Method using color extraction method and eigenspace technique ( Yasuyuki Nakata and Moritoshi Ando.](https://reader035.fdocuments.in/reader035/viewer/2022062620/551ac4e555034656628b5b55/html5/thumbnails/14.jpg)
Summary & Analysis
Lip reading system Colour extraction method roughly classify lip and other face feat
ure Eigentemplate method precisely detect the location Eigenwaveform algorithm recognize the utterance
Analysis Widely used image processing technique Hard to get high precision Difficult for language different, culture different, appearance diff
erence user Potential problems of algorithm: brightness dependence; overla
p between lip & skin; similar words; uncommon mouth shape; various speak speed and so on.
Need more improvement!
![Page 15: Advanced Image Processing Student Seminar: Lipreading Method using color extraction method and eigenspace technique ( Yasuyuki Nakata and Moritoshi Ando.](https://reader035.fdocuments.in/reader035/viewer/2022062620/551ac4e555034656628b5b55/html5/thumbnails/15.jpg)
That’s all!Thank you very much!
Question?