1

ForeSight: Mapping Vehicles in Visual Domain and Electronic

Domain

Dong Li, Zhixue Lu, Tarun Bansal, Erik Schilling and Prasun SinhaDepartment of Computer Science and Engineering

The Ohio State University

2

Need for Targeted Communication

OK, but who are you?

What’s in front?

Are you talking to

me?

Hey you at the back -- Your lights are off!

I am overtaking you, don’t change

lane!

3

Today’s Solutions

Unicast: Hand Gestures, Eye ContactRequires parties to see each other

Broadcast: Honk, ShoutDisturbs othersAgitates/annoys both parties

4

Tomorrow’s Technology: One Possibility

Broadcast using Smartphone/DSRCHonking/shouting in the electronic domainWould cause sensory overload for drivers

5

Fundamental Problem in Targeted V2V Communication

Who is the sender/receiver?

Sender: What is the receiver’s unique address?Receiver: Which vehicle sent message to me?

6

To match vehicles in visual and electronic domains.

Objective

At the same time• Decrease matching time• Increase accuracy• Generate less network traffic

VID: Visual ID assigned by camera (e.g., red/yellow/blue box)

EID: Electronic ID of the vehicle (e.g., IP/MAC address)

7

Available Features

Features Accuracy & UniquenessVehicle Color Not always uniqueGPS Not accurate enoughVehicle Image Not unique, environment dependentPlate number Unique, but hard to readRelative Speed May not be accurate…. ….

A unique set of features known to both vehicles is desired.

8

Main Idea

If single feature is unreliable, can we use multiple features to do matching?

System requirement: Camera, GPS Receiver and Radio

Radio: communicationSmartphone, DSRC

Camera: identify vehiclesSmartphone, Vehicle Security Driving Recorder Camera

9

Challenges

Feature InaccuracyE.g., A blue vehicle might be observed as black.

Heterogeneous CapabilityVehicles may not have smartphone, camera, radio, or may not be running our solution.

Distributed in NatureEach vehicle only knows limited information

10

Vehicle Matching Process

Matching vehicles based on similarity

Estimate similarity between vehiclesWeight Features Cluster {VIDs, EIDs}

Obtain VIDs & EIDsGet VIDs from camera Get EIDs from radio

11

Visual Matrix(from Video-Analysis)

VID N Features

V1 f11 f12 … f1N

V2 f21 f22 … f2N

... … … …. …VM fM1 fM2 … fMN

Vehicles Observed through Camera

VID : Visual ID (camera assigns visual IDs to the observed vehicles)

VID only has local meaning (cannot be used by neighbors)

12

Electronic Matrix(from Electronic Messages)

EID N Features

E1 f11 f12 … f1N

E2 f21 f22 … f2N

... … … …. …EK fK1 fK2 … fKN

IDs received through WiFi/DSRC

EID: Electronic ID (IP address, MAC address, etc.)

13

Create Similarity Matrix

E1 E2 ... EK

V1 … 0.84 ... …

V2 … … ... ...

... ... ... ... ...

VM ... ... ... ...

VID Features

V1 f11 f12 … f1N

V2 f21 f22 … f2N

... … … …. …VM fM1 fM2 … fMN

EID Features

E1 f11 f12 … f1N

E2 f21 f22 … f2N

... … … …. …EK fK1 fK2 … fKN

Electronic Matrix E Visual Matrix V

Similarity Matrix S

S = V ET

14

Adaptive Weight (AW) Algorithm

The Problem:How to combine different features to get the similarity value between two cars?

The Intuition: Features with diversity values are important.

E.g., color provides no information if the cars have the same color

The Solution:Define Feature Distinguishability: the probability that any two observed vehicles are different based on this featureSimilarity of two vehicles: weighted mean of the feature distinguishability values.

15

different lane, different color

Eb

Ea

Matching with Similarity Matrix

V1

me

V2

Visual Domain Electronic Domain

0.99

0.50.5

0.01

Steps• Assign VIDs• Receive EIDs• Calc. Similarity• Remove low

similarity linksdifferent lane, similar color

same lane, different color

same lane, similar color

16

Matching with Similarity Matrix

Greedy Matching Maximal Matching

Weighted Bipartite Graph Matching Problem

Ea

Eb0.9

0.50.5

VIDs EIDs

V2 Ea

Eb0.9

0.50.5

VIDs EIDs

V2

Greedy matching is preferred.

V1 V1

17

Clustering Vehicles

Global distinguishability not requiredNearby cars need to be distinguished

Cluster the cars into smaller groups based on feature distance.

Apply the AW algorithm within clusters

Clustering

18

Experiment

Driving in freeway & local drive with 3 carsUsing smartphone to collect GPS, videoExperiment result

Vehicles with same color leads to low precision

19

SimulationUsing SUMO + NS3Modeled the visibility of neighboring carsModeled car detection prob., color detection accuracy, etc.

ForeSight significantly improves the matching performance!

0.230.18

20

Case Studies: Improve GPS

Each vehicle estimates its location withIts own GPS measurementNeighbors’ estimation of its location (assistance from Nbrs.) Interesting Observations:

• When a car’s GPS error low, it is more likely to be matched by more neighbors.

• The match error increases as the number of neighbors increases: dense traffic makes matching more unreliable.

High vehicle density

21

Case Studies: Reduce Disturbance

Application: Send message to vehicles that are in front but has a slower speedCompare Broadcast, GPS and ForeSight

GroundTruth

Broadcas

tGPS

ForeSig

ht0

50010001500200025003000

# of

Car

s Noti

fied

29× Schemes Notified Vehicles

Ground Truth 1021Broadcast 29 x 1021GPS 2706 (95% recall)ForeSight 1141 (95% recall)

22

Eb

E1 E3Ea

E1 E2

Future Work: Conflict Resolving

Conflicts may AppearMatching result computed by different vehiclesMatching result at different time

Possible SolutionCollaboration between neighbors

Eb

E1 E3

E1

Ea

E2

EID

VID