Dynamic 3D Scene Analysis from a Moving Vehicle
description
Transcript of Dynamic 3D Scene Analysis from a Moving Vehicle
Dynamic 3D Scene Analysis Dynamic 3D Scene Analysis from a Moving Vehiclefrom a Moving Vehicle
Young Ki Baik (CV Lab.)Young Ki Baik (CV Lab.)2007. 7. 11 (Wed)2007. 7. 11 (Wed)
Dynamic Scene Analysis from a Dynamic Scene Analysis from a Moving VehicleMoving Vehicle
ReferencesReferencesDynamic 3D Scene Analysis from a Moving Vehicle
• Bastian Leibe, Nico Cornelis, Kurt cornelis, Luc Van Gool• Awarded the best paper prize (CVPR 2007)
Fast Compact City Modeling for Navigation Pre-Visualization• Nico Cornelis, Kurt cornelis, Luc Van Gool (CVPR 2006)
Pedestrian detection in crowded scene• Bastian Leibe et. al. (CVPR 2005)
Putting Objects in Perspective• Derek Hoiem et. al. (CVPR 2006)
Dynamic Scene Analysis from a Dynamic Scene Analysis from a Moving VehicleMoving Vehicle
Why?Why?… were they received the best paper prize?
• They completed the impressive real application with only toy computer vision algorithm.
• They showed that the field of vision will be a key of future technique to the public.
Dynamic Scene Analysis from a Dynamic Scene Analysis from a Moving VehicleMoving Vehicle
Demo (Final result)Demo (Final result)
Dynamic Scene Analysis from a Dynamic Scene Analysis from a Moving VehicleMoving Vehicle
What?What?…is the purpose of this paper?
… is the challenges of this paper?
• Detect object in real environment (city road)• Localize them in 3D• Predict their future motion
• We are moving• Objects can be moving• Ground may not be planar
Dynamic Scene Analysis from a Dynamic Scene Analysis from a Moving VehicleMoving Vehicle
What methods?What methods?… are used to accomplish their purpose?
• Structure from motion• 2D object detection• 3D trajectory estimation
Dynamic Scene Analysis from a Dynamic Scene Analysis from a Moving VehicleMoving Vehicle
Overall flowOverall flow
• 3D structure info.3D structure info.• Ground planeGround plane
1. SfM1. SfM
Stereo cameraStereo camera Aligned stereo Aligned stereo imageimage
2. Object 2. Object detectiondetection
• 2D and 3D Object2D and 3D Object• 3D trajectory3D trajectory• OrientationOrientation
3. Tracking3. Tracking
Dynamic Scene Analysis from a Dynamic Scene Analysis from a Moving VehicleMoving Vehicle
3D structure and ground plane3D structure and ground plane3D Structure from Motion
• Visual odometry (David Nister)• Use pre-calibrated stereo camera• Use rectified stereo images• Parallel processing
→ Extrinsic camera parameters → 3D camera trajectory (in real time)
Nico Cornelis et. al. (CVPR 2006)
Dynamic Scene Analysis from a Dynamic Scene Analysis from a Moving VehicleMoving Vehicle
3D structure and ground plane3D structure and ground planeGround plane estimation
• Known ground positions of wheel base points
Nico Cornelis et. al. (CVPR 2006)
Dynamic Scene Analysis from a Dynamic Scene Analysis from a Moving VehicleMoving Vehicle
3D structure and ground plane3D structure and ground planeGround plane estimation
• Compute normal locally• Average over spatial window
Nico Cornelis et. al. (CVPR 2006)
Dynamic Scene Analysis from a Dynamic Scene Analysis from a Moving VehicleMoving Vehicle
SfM DemoSfM Demo
Nico Cornelis et. al. (CVPR 2006)
Dynamic Scene Analysis from a Dynamic Scene Analysis from a Moving VehicleMoving Vehicle
Object detectionObject detection2D/3D Interaction method
• Likelihood of 3D object hypothesis H → Given image I and a set of 2D detections h:
h
IhpIhHpIHp |,||
h
IhpHpHhp ||~
Dynamic Scene Analysis from a Dynamic Scene Analysis from a Moving VehicleMoving Vehicle
Object detectionObject detection2D object detection
h
IhpHpHhp || 2D recognition2D recognition
ISM detectors
Leibe et. al. (CVPR 2005)
Dynamic Scene Analysis from a Dynamic Scene Analysis from a Moving VehicleMoving Vehicle
Object detectionObject detection ISM detectors (Leibe et al., CVPR’05, BMVC’06)
• Battery of 5(car)+1(human) single view detectors• Each detectors based on 3 local cues
• Harris-Laplace, Hessian-Laplace, DoG interest regions• Local Shape Context descriptors
• Result: detections + segmentations
Leibe et. al. (CVPR 2005)
Dynamic Scene Analysis from a Dynamic Scene Analysis from a Moving VehicleMoving Vehicle
Object detectionObject detection2D/3D transfer
h
IhpHpHhp ||
2D/3D transfer2D/3D transfer
• Two image-plane detections are consistent if they correspond to the same 3D object.
→ Cluster 3D detections → Multi-viewpoint integration
Dynamic Scene Analysis from a Dynamic Scene Analysis from a Moving VehicleMoving Vehicle
Object detectionObject detection3D prior
h
IhpHpHhp || 3D prior3D prior
• By Using 3D structure and ground plane constraint… → Distance prior (Distance from the ground plane) → Size prior (Gaussian)
Hoiem et. al. (CVPR 2006)
• Significantly reduced search space and outlier
Dynamic Scene Analysis from a Dynamic Scene Analysis from a Moving VehicleMoving Vehicle
Quantitative results of detectionQuantitative results of detection
Detection performance on 2 test sequences• Stereo and Ground plane constraints significantly improves
precision
Dynamic Scene Analysis from a Dynamic Scene Analysis from a Moving VehicleMoving Vehicle
Detection DemoDetection Demo
Dynamic Scene Analysis from a Dynamic Scene Analysis from a Moving VehicleMoving Vehicle
Object trackingObject trackingLocalization and Trajectory estimation
• By using detection results• Obtain orientation of objects
Space-time trajectory analysis• By using the concept of a bidirectional Extended Kalman Filter
Dynamic Scene Analysis from a Dynamic Scene Analysis from a Moving VehicleMoving Vehicle
Object trackingObject tracking3D Localization for static objects (car)
• Location• Mean-shift search to find set of 3D detection hypotheses
• Orientation• Cluster shape and detector output
Dynamic Scene Analysis from a Dynamic Scene Analysis from a Moving VehicleMoving Vehicle
Object trackingObject trackingDynamic model
• Holonomic motion (Pedestrian)• Without external constraints linking its speed and turn rate
• Nonholonomic motion (Car)• Only move along its main axis• Only turn while moving
urepathcurvatK
v
t
yx
c :
velocity:
step time:
norientatio andposition :),,(
tK
tvyy
tvxx
ctt
tt
tt
1
1
1
sin
cos
tvK
tvyy
tvxx
ctt
tt
tt
sin
cos
1
1
1
Dynamic Scene Analysis from a Dynamic Scene Analysis from a Moving VehicleMoving Vehicle
Object trackingObject trackingTrajectory growing
• Collect detection in time space
Dynamic Scene Analysis from a Dynamic Scene Analysis from a Moving VehicleMoving Vehicle
Object trackingObject trackingTrajectory growing
• Collect detection in time space• Evaluate under trajectory
• Bi-directionally• Static assumption
Dynamic Scene Analysis from a Dynamic Scene Analysis from a Moving VehicleMoving Vehicle
Object trackingObject trackingTrajectory growing
• Collect detection in time space• Evaluate under trajectory
• Bi-directionally• Static assumption
• Adjust trajectory• Weighted mean
• Predicted position• Supporting observations
Dynamic Scene Analysis from a Dynamic Scene Analysis from a Moving VehicleMoving Vehicle
Object trackingObject trackingTrajectory growing
• Collect detection in time space• Evaluate under trajectory
• Bi-directionally• Static assumption
• Adjust trajectory• Weighted mean
• Predicted position• Supporting observations
• Iteration
Dynamic Scene Analysis from a Dynamic Scene Analysis from a Moving VehicleMoving Vehicle
Object trackingObject trackingTrajectory growing
• Collect detection in time space• Evaluate under trajectory
• Bi-directionally• Static assumption
• Adjust trajectory• Weighted mean
• Predicted position• Supporting observations
• Iteration• Location and orientation
Dynamic Scene Analysis from a Dynamic Scene Analysis from a Moving VehicleMoving Vehicle
Demo (Final result)Demo (Final result)
Dynamic Scene Analysis from a Dynamic Scene Analysis from a Moving VehicleMoving Vehicle
ConclusionConclusionSummary
• Exact value of 3D information • help to propose the new concept of detection algorithm• raise the performance of detection algorithm.
• Better detection results• Give more reliable tracking results• Good orientation estimation
Contribution• New detection algorithm using 3d information• Good integration and visualization of application system