PISAProduction, Indexing and Search

of Audio-visual Material

Image ProcessingTinne Tuytelaars, IBBT – PSI – K.U.Leuven

61

Computer Assisted Analysis

! Intelligent analysis = reverse engineering

! Shot cut detection

! Scene segmentation

! Video reuse detection

! Face detection

! Face recognition

! Audio classification

demo

demo

demo

demo

demo

62

Shot cut detection

= Split the video stream in atomic units, corresponding to a

continuously moving camera

! Distinguish between abrupt and smooth shotcuts

! Experimented with different methods

! Using color histograms

! Using affine motion compensation

! Using motion estimation within the compressed domain

68

Scene segmentation

69

Video reuse detection

The same video material is often reused

! can be detected automatically

! Robust to post-processing

! Efficiency

! Based on spatio-temporal local features and locality

sensitive hashing

70

Face Detection

• Face candidates selection:" Candidate regions have skin color # region-based skin

segmentation

" Personalized chrominance skin boundary

• Verification based on cues:" Shape of ellipse

" Ellipse-filling percentage

" Gray-tone smoothness

" Corners of the facial features

71

Face detection results

• 92% good face detections

"Comparable to state-of-the-art face learning of ‘Viola & Jones’ obtains 93%

"adaptation to lighting conditions and personal face looks

72

Face recognition

! Based on a 3D morphable model

! 3D model is fitted to 2D image

! Shape and texture parameters used as face descriptor

! Robust to

! viewpoint changes,

! illumination changes,

! partial occlusions.

73

Face recognition

74

Face recognition

75

Face recognition

76

Face recognition

77

Face recognition

79

Future work

! Include facial expressions in face recognition

! Multi-modal scene segmentation

! Feedback loop from Trouvaille

! Object or scene recognition

! Thesaurus-based speech recognition