CCVC L

Sensor and Vision Research for Potential Transportation Applications

Zhigang ZhuVisual Computing Laboratory

Department of Computer ScienceCity College of the City University New York

Research Initiatives for Nano and Hi-tech Research January 18, 2006

2

CCVC L Multimodal Human/Vehicle Signatures

• Collaboration with– AFRL– Prof. Tom Huang at UIUC– Prof. George Wolberg at CCNY

• Goal– Remote sensing/hearing– Multimodal signatures

• Challenges– Noisy LDV signals– Low-res, non-front faces– Target detection??– Transportation??? Figure 1. Multimodal remote hearing by human and machine: system diagram

H u m a n / F a c e D e t e c t i o n & T r a c k i n g

L D V T a r g e t i n g a n d F o c u s i n g

V i s u a l L i p R e a d i n g

L D V V o i c e R e a d i n g

M u l t i m o d a l F u s i o n f o r E n h a n c e m e n t & R e c o g n i t i o n

T a l k i n g H e a d / F a c e S y n t h e s i s

M u l t i m o d a l ( A / V / T ) A l i g n m e n t

A u d i o

V i s i b l e / I R v i d e o

V i d e o

S p e e c h & f e a t u r e s L i p m o t i o n f e a t u r e s

F a c e & f e a t u r e s E n h a n c e d s p e e c h

( a n d t e x t ) T e x t

H u m a n i n c a r

P T Z c a m

I R c a m e r a

A c o u s t i c s i g n a l s

M u l t i m o d a l D e t e c t i o n ( S e n s i n g )

M u l t i m o d a l R e c o g n i t i o n ( A n a l y s i s )

M u l t i m o d a l P r e s e n t a t i o n ( S y n t h e s i s )

S e n s o r s , H u m a n & E n v i r o n m e n t

H u m a n U s e r

V i b r o m e t e r ( L D V )

M i c r o p h o n e

O u t p u t 1

O u t p u t 2 T a l k i n g F a c e ( A v a t a r )

c o n t r o l c o n t r o l

3

CCVC L Major Sensor Acquisition

• Polytec Laser Doppler Vibrometer (LDV)– OFV-505 Sensor Head, OFV-5000 Controller – Tip-Tilt Stage VIB-A-P05 with Telescope

• FLIR ThermoVision Camera– ThermoVision® A40M Infrared Camera

• Canon Color/IR PTZ Camera– Canon VC-C50i Low Light IR PTZ Camera

• Other video cameras – Omnidirectional cameras, stereo head– Camcorders, webcam, CMOS sensors

4

CCVC L Polytec Laser Doppler Vibrometer

• Sensor Head OFV-505– HeNe (Helium-Neon) laser, =632.8 nm, W<1 mW– OFV-SLR lens (f=30mm) 1.8 m – 200+ m– “Any” surfaces, Automatic focus

• Controller OFV-5000– Low pass (5, 20,100 kHz), high pass (100Hz)– RS-232 interface for computer control

• Velocity Decoder VD-06– Ranges: 1, 2, 10 and 50 mm/s/V– Resolution 0.02 m/s under 1mm/s/V range (2mv/20V)– 350 kHz bandwidth analog output

– 24 bit, 96 kHz max. digital output on S/P-DIF interface

5

CCVC L Dynamic Pushbroom Stereo Mosaics

Innovative ClaimDescription of Research

Objectives Expected Contribution to Research Area

Video Registration, representation and 3D static/dynamic content extraction from video sequences of dynamic urban scenes taken from aerial or ground vehicles.

Results: Content-Based 3D Mosaics Concept: Dynamic Pushbroom Stereo

(1) Rapid panoramic stereo mosaic construction (2) Accurate 3D reconstruction and parametric rep. (3) Robust moving target extraction and estimation

(1) Pushbroom stereo mosaics for 3D dynamic scenes (2) Natural stereo matching primitives for 3D urban scenes and moving targets (2) Content-based 3D mosaic rep for accurate 3D and motion

An efficient and more accurate representation for large scale scenes; A new natural matching approach for both urban structures and moving targets

Sponsored by: AFRLPIs: Zhigang Zhu, George Wolberg - CUNY City College; Robert Haralick – CUNY Graduate Center

y

yyd

SBFZ

Fixed disparity

Adaptive baseline

Accumulated motion

Accurate depth

Stereo mosaics

Depth map Moving target

video

Moving target

Height H

GPS By

dy

F

Sy

Sx

“Right” Mosaic “Left” Mosaic

6

CCVC LKey Results

Accurate 3D reconstruction and motion detection on mosaics from real world video

Full Depth Map of mosaics from 1000+ frames. Sharp depth boundaries are obtained for further target recognition and motion detection

Portion of the depth map of ground and buildings; moving target are “outliers”” (small white areas)

Detected moving targets – moving vehicles are identified by a 2D search in matching (from blue to red)

7

CCVC L Under-Vehicle Inspection

• Car drives over a 1D array of cameras…

Joint Work with UMass Computer Vision Lab

Video

Stereo Mosaics revealing occlusions

8

CCVC L Gamma-Ray Cargo Inspection

Images from SAlC Mobile Vehicle and Cargo Inspection System (VACIS®)

3D measurements from 2 pushbroom gamma-ray images

3D View

9

CCVC L Summary

• Novel Sensors– Color, thermal, vibration

• Novel Algorithms– Stereo mosaicing & 3D/ motion detection

• Emerging 3D Applications– Security, surveillance & safety – in transportation