GM-Carnegie Mellon Autonomous Driving CRL Structured Hough Voting for Vision- based Highway Border...

GM-Carnegie Mellon Autonomous Driving CRL

Structured Hough Voting for Vision-

based Highway Border Detection

1

Zhiding YuCarnegie Mellon University


Autonomous Driving: Not If, But When

2


GM-CMU Collaborative Research


Sensors Setup on SRX Platform

Images from: Junqing Wei et al., “Towards a Viable Autonomous Driving Research Platform,” IEEE Intelligent Vehicles Symposium (IV), 2013


Sensors: Price vs Information

Price

Info

rmat

ion

Radar

Lidar

Camera


Computer Vision Applications

Object detection (pedestrian, vehicle, bicycle…) Road parsing (lane/border detection, road segmentation,

vanishing point estimation…) Localization and tracking Driver status monitoring Many other applications……


Motivation, Description and Goal

7

Goal

– Development for future driving assistance system and autonomous driving system

– Robust detection within 0.5 to 6 meters detection range. Achieve near 100% accuracy in daytime and over 90% in nighttime on the right most lane

– Handling various scenarios including highway entrance and exit

– Extend to the joint system with front view


Concrete Barrier Guard Rail Soft Shoulder

High-Level Idea: Learning based Method

Guard Rail

Soft Shoulder

ConcreteBarrier

Lane Marking

Densely Fired scanning windows

Returned Voting Points

Border / lane marking hypotheses

Structured Hough Voting


Overall 1592 training images:1. Concrete Barrier (839 images)

2. Guard Rail (300 images)

3. Soft Shoulder (453 images)

Overall 2638 testing images:

Dataset Collection


Training Patch Alignment

Negative Samples:

Positive Samples:

Concrete Natural Steel Lane Marker


Filter Bank

Patches that are discriminative to HOG

Patches that are discriminative to filter banks

Concatenated Filter Bank Feature

Feature Extraction

HOG

Concatenated HOG Feature


Extract features from all training patches (based on previous page)

Perform Fisher discriminant analysis

Train an RBF kernel SVM

Scanning window detection (Deliberately having a lot of positive firing)

Classification & Detection

Guard Rail

Soft Shoulder

ConcreteBarrier

Lane Marking


Hough Voting


Structured Hough Voting: Intuitions Basic philosophy: A model that assumes voting results are correlated rather

than independent

Inter-frame structural info on hypotheses (Temporal smoothness)

Intra-frame structural info (Geometric relationship)

Multiple candidate hypotheses generation (Proposals with diversity)

1. Constrained Hough Voting on detected voting points (Detection + Tracking)

2. Arbitrary Hough Voting on detected voting points (Detection)

3. Constrained Hough Voting on image gradients (Pure Tracking)


Deals most of the frames where hypotheses from consecutive frames have strong correlation.

Purpose of Candidate 1


Automatically corrects result through searching for “much better” voting configurations (This is the power of detection, avoids error from tracking)



In the worst case where Type 1 voters fail, perform tracking by gradients from previous pose configuration.



Modeling under CRF: Background

A Conditional Random Field (CRF) discriminatively defines the joint posterior probability as the product of a set of potentials

The potentials are functions with hypotheses Hi being the variables. They are modeled in such a way that a larger potential value generally indicates a better hypothesis configuration.

CRF inference seeks to find the joint hypothesis configuration H that maximizes

H1

X1

H2 HN…

Unary Potential Pairwise Potential

X2 XN


Modeling under CRF: Intuition

What are the hypothesis Hi?

E.g.: image pixel labels (FG/BG, Object Class, etc.), if it is a segmentation problem.

In our problem, Hi is the Hough Voting hypothesis: Hi = (r, θ).

X is the observation of voting point coordinates and their weights.

The unary potential corresponds to the exponential of Hough voting weights: exp(v(Hi)).

The pairwise potential corresponds to the inter-frame smoothness (tracking) constraint.

H1

X1

H2 HN…

X2 XN


No Structural Information

Hbd,1…

Simplest Case: frame-wise independent Hough voting

Hbd,2 Hbd,N

X1 X2 XN

Hln,1…Hln,2 Hln,N

X1 X2 XN


Adding Inter-frame Structural Info.

Hbd,1…

Adding temporal smoothness: Hough voting constrained by neighboring frames

Hbd,2 Hbd,N

X1 X2 XN

Hln,1…Hln,2 Hln,N

X1 X2 XN


Adding Intra-frame Structural Info.

Hbd,1…

Adding Geometric Constraint: Hough voting constrained by both neighboring frames and intra-frame hypotheses

Hbd,2 Hbd,N

X1 X2 XN

Hln,1…Hln,2 Hln,N

X1 X2 XN


The Structured Hough Voting Model

• • •

• • •

Candidate Hypotheses Generation Unit

Mode Selection Potential

Coupled Structure Potential


The Structured Hough Voting Model


Candidate Hypotheses Generation Unit


Use decision tree to guide the mode selection.

The mode selection basically forces the output to be one of the candidate hypotheses, but allows discrepancy with the decision tree prediction with a penalty.

Mode Selection Potential


The coupled structure potential captures two most important relations between a border hypothesis and a lane hypothesis

Parallelism

Distance

Coupled Structure Potential


Inference

Conducting a whole inference each time given a new frame is computationally infeasible.

Relaxation: Initialize with the inferred state variable configuration of the previous t-1 frames and infer the current state variables, updating in an incremental way.

Inference procedure at t = 1:1. Perform Hough voting for both border and lane marking2. Perturbate hypotheses if geometric relationship violated (optional)

Inference procedure at t > 1:1. Generate the 3 candidate hypotheses for both border and lane marking2. Use decision tree to help selecting the best candidate3. Perturbate candidate hypotheses if geometric relationship violated (optional)4. Re-select the best candidate


Experiments: Adding Coupled Structure


Experiments: Qualitative Results

Ground Truth and Baseline methods:1. Ground Truth2. Independent Hough voting in each frame using the fired detector voting points3. Hough voting using the triggered detector voting points constrained by previous frame 4. Adding gradient tracking to Baseline 2.5. Kalman filter.6. Proposed Method


Experiments: Quantitative Results


Highway Entrance Detection and Lane State Tracking


Summary

Proposed the Structured Hough Voting Model

The proposed model can be theoretically formulated under a CRF

Fast real-time feature extraction and online inference

Achieves very robust and good performance under challenging scenarios

and low quality inputs from production camera


Thank You!Q & A

GM-Carnegie Mellon Autonomous Driving CRL Structured Hough Voting for Vision- based Highway Border...

Documents

Transcript of GM-Carnegie Mellon Autonomous Driving CRL Structured Hough Voting for Vision- based Highway Border...