Recognition of Traffic Lights in Live Video Streams on Mobile Devices Jan Roters Xiaoyi Jiang Kai...
-
date post
19-Dec-2015 -
Category
Documents
-
view
217 -
download
2
Transcript of Recognition of Traffic Lights in Live Video Streams on Mobile Devices Jan Roters Xiaoyi Jiang Kai...
![Page 1: Recognition of Traffic Lights in Live Video Streams on Mobile Devices Jan Roters Xiaoyi Jiang Kai Rothaus 2011 IEEE Transactions on CSVT.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649d2e5503460f94a0585c/html5/thumbnails/1.jpg)
Recognition of Traffic Lights in Live Video Streams on Mobile
Devices
Jan Roters
Xiaoyi Jiang
Kai Rothaus
2011 IEEE Transactions on CSVT
![Page 2: Recognition of Traffic Lights in Live Video Streams on Mobile Devices Jan Roters Xiaoyi Jiang Kai Rothaus 2011 IEEE Transactions on CSVT.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649d2e5503460f94a0585c/html5/thumbnails/2.jpg)
Outline
IntroductionProblemsSystem Architecture
IdentificationClassificationVideo AnalysisTime-Based Verification
Experiment ResultsEvaluationsConclusion
![Page 3: Recognition of Traffic Lights in Live Video Streams on Mobile Devices Jan Roters Xiaoyi Jiang Kai Rothaus 2011 IEEE Transactions on CSVT.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649d2e5503460f94a0585c/html5/thumbnails/3.jpg)
Introduction
People with visual disabilities are limited in mobility.
Orientate pedestrians with zebra crossings at intersections
Portable PC with a digital camera and a pair of auricular stereo
Present a system for mobile devices to help sightless people cross roads.
![Page 4: Recognition of Traffic Lights in Live Video Streams on Mobile Devices Jan Roters Xiaoyi Jiang Kai Rothaus 2011 IEEE Transactions on CSVT.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649d2e5503460f94a0585c/html5/thumbnails/4.jpg)
Problems
Program usage
Real world conditionsCamera resolutionDifferent appearances
![Page 5: Recognition of Traffic Lights in Live Video Streams on Mobile Devices Jan Roters Xiaoyi Jiang Kai Rothaus 2011 IEEE Transactions on CSVT.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649d2e5503460f94a0585c/html5/thumbnails/5.jpg)
Problems
The scale of traffic lightsMany traffic lightsOccludedIlluminationRotation
![Page 6: Recognition of Traffic Lights in Live Video Streams on Mobile Devices Jan Roters Xiaoyi Jiang Kai Rothaus 2011 IEEE Transactions on CSVT.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649d2e5503460f94a0585c/html5/thumbnails/6.jpg)
Pedestrian Lights in Germany
1) Installation
2) Shape
3) Color arrangement
4) Circuitry
5) Background
![Page 7: Recognition of Traffic Lights in Live Video Streams on Mobile Devices Jan Roters Xiaoyi Jiang Kai Rothaus 2011 IEEE Transactions on CSVT.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649d2e5503460f94a0585c/html5/thumbnails/7.jpg)
Mobile Device & Databases
Nokia N95330MHZ ARM processor18Mb RAM320240
2 publicly available databaseGround truth segmentation was made manually
![Page 8: Recognition of Traffic Lights in Live Video Streams on Mobile Devices Jan Roters Xiaoyi Jiang Kai Rothaus 2011 IEEE Transactions on CSVT.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649d2e5503460f94a0585c/html5/thumbnails/8.jpg)
System Architecture
1.
2.
3.
4.
![Page 9: Recognition of Traffic Lights in Live Video Streams on Mobile Devices Jan Roters Xiaoyi Jiang Kai Rothaus 2011 IEEE Transactions on CSVT.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649d2e5503460f94a0585c/html5/thumbnails/9.jpg)
1. Localization
![Page 10: Recognition of Traffic Lights in Live Video Streams on Mobile Devices Jan Roters Xiaoyi Jiang Kai Rothaus 2011 IEEE Transactions on CSVT.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649d2e5503460f94a0585c/html5/thumbnails/10.jpg)
Red and Green Color Filter(1/3)
1. Analyze the data
![Page 11: Recognition of Traffic Lights in Live Video Streams on Mobile Devices Jan Roters Xiaoyi Jiang Kai Rothaus 2011 IEEE Transactions on CSVT.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649d2e5503460f94a0585c/html5/thumbnails/11.jpg)
Red and Green Color Filter(2/3)
2. Design the filter rules (ex : red traffic light)
The Gaussian distribution of the red cluster is defined by its mean color = (0.48,0.06,0.07) and has three eigenvectors
A color c = (r, g, b) is a red traffic light color when
![Page 12: Recognition of Traffic Lights in Live Video Streams on Mobile Devices Jan Roters Xiaoyi Jiang Kai Rothaus 2011 IEEE Transactions on CSVT.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649d2e5503460f94a0585c/html5/thumbnails/12.jpg)
Red and Green Color Filter(3/3)
3. Optimize parameters different parameter settings for each color Use 300 images to train Measure the quality of each setting by TP, FP, FN
Recall = , Precision =
![Page 13: Recognition of Traffic Lights in Live Video Streams on Mobile Devices Jan Roters Xiaoyi Jiang Kai Rothaus 2011 IEEE Transactions on CSVT.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649d2e5503460f94a0585c/html5/thumbnails/13.jpg)
Size/Circuitry Filter
Assume the traffic light is 4 to 24 meters awayFixed camera focal length and possible aspect
ratios
1. Filter out regions that are too small or too large
2. Vertical neighbor should not have different color
![Page 14: Recognition of Traffic Lights in Live Video Streams on Mobile Devices Jan Roters Xiaoyi Jiang Kai Rothaus 2011 IEEE Transactions on CSVT.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649d2e5503460f94a0585c/html5/thumbnails/14.jpg)
Background Color Filter
Inspect the region under a red light candidate or above a green light candidate
If there are no dark pixels within search region, refuse this candidate
Search region
Search region
![Page 15: Recognition of Traffic Lights in Live Video Streams on Mobile Devices Jan Roters Xiaoyi Jiang Kai Rothaus 2011 IEEE Transactions on CSVT.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649d2e5503460f94a0585c/html5/thumbnails/15.jpg)
Validation of Localization
Validate the localization results with 201 images
Optimal Validation
recall precision recall precision
Red 76% 89.5% 71.8% 87%
green 85% 98.5% 83.3% 92.6%
Error = 33.7%
![Page 16: Recognition of Traffic Lights in Live Video Streams on Mobile Devices Jan Roters Xiaoyi Jiang Kai Rothaus 2011 IEEE Transactions on CSVT.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649d2e5503460f94a0585c/html5/thumbnails/16.jpg)
2. Classification
TLC is the broadestTLC has the smallest distance to the top of imageNo other traffic light has similar height with TLC
![Page 17: Recognition of Traffic Lights in Live Video Streams on Mobile Devices Jan Roters Xiaoyi Jiang Kai Rothaus 2011 IEEE Transactions on CSVT.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649d2e5503460f94a0585c/html5/thumbnails/17.jpg)
Performance of Classification
Red Green
Recall 86.3% 86.3%
Precision 97.4% 98.1%
![Page 18: Recognition of Traffic Lights in Live Video Streams on Mobile Devices Jan Roters Xiaoyi Jiang Kai Rothaus 2011 IEEE Transactions on CSVT.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649d2e5503460f94a0585c/html5/thumbnails/18.jpg)
![Page 19: Recognition of Traffic Lights in Live Video Streams on Mobile Devices Jan Roters Xiaoyi Jiang Kai Rothaus 2011 IEEE Transactions on CSVT.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649d2e5503460f94a0585c/html5/thumbnails/19.jpg)
3. Video Analysis(1/2)
Temporary OcclusionFalsified ColorsContradictory SceneRepeating Results
![Page 20: Recognition of Traffic Lights in Live Video Streams on Mobile Devices Jan Roters Xiaoyi Jiang Kai Rothaus 2011 IEEE Transactions on CSVT.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649d2e5503460f94a0585c/html5/thumbnails/20.jpg)
3. Video Analysis(2/2)
Find the motion vector between two framesUse KLT tracker to track feature pointsOnly search in a small area around crucial traffic light
candidate (30 pixels in each direction)Correlate the features by using SAD
Search region
Crucial traffic light
Candidate region
Feature point
𝑡𝑖 −1 𝑡𝑖
![Page 21: Recognition of Traffic Lights in Live Video Streams on Mobile Devices Jan Roters Xiaoyi Jiang Kai Rothaus 2011 IEEE Transactions on CSVT.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649d2e5503460f94a0585c/html5/thumbnails/21.jpg)
4. Time-Based Verification
Reduce the false positive detections by comparing 2 kinds of results
Use state queue with 4 scenarios1) Identification and video analysis are both successful
and the locations match with each other.
2) Identification and video analysis are successful but the locations are different.
3) Video analysis succeeds but identification fails.
4) Video analysis fails but identification succeeds.
![Page 22: Recognition of Traffic Lights in Live Video Streams on Mobile Devices Jan Roters Xiaoyi Jiang Kai Rothaus 2011 IEEE Transactions on CSVT.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649d2e5503460f94a0585c/html5/thumbnails/22.jpg)
Experiment Results
and Compute at least 5 frames per secondAt least consecutive correct detection with the
same color
![Page 23: Recognition of Traffic Lights in Live Video Streams on Mobile Devices Jan Roters Xiaoyi Jiang Kai Rothaus 2011 IEEE Transactions on CSVT.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649d2e5503460f94a0585c/html5/thumbnails/23.jpg)
Experiment Results
![Page 24: Recognition of Traffic Lights in Live Video Streams on Mobile Devices Jan Roters Xiaoyi Jiang Kai Rothaus 2011 IEEE Transactions on CSVT.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649d2e5503460f94a0585c/html5/thumbnails/24.jpg)
Evaluations
ReliabilityPrevent false positive green light detection
![Page 25: Recognition of Traffic Lights in Live Video Streams on Mobile Devices Jan Roters Xiaoyi Jiang Kai Rothaus 2011 IEEE Transactions on CSVT.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649d2e5503460f94a0585c/html5/thumbnails/25.jpg)
Evaluations
InteractivityTemporal analysis reduce the interactivityThe feedback is normally given within 2 seconds
![Page 26: Recognition of Traffic Lights in Live Video Streams on Mobile Devices Jan Roters Xiaoyi Jiang Kai Rothaus 2011 IEEE Transactions on CSVT.](https://reader036.fdocuments.in/reader036/viewer/2022062516/56649d2e5503460f94a0585c/html5/thumbnails/26.jpg)
Conclusion
The system can be helpful on driver assistance systems
Limited computational power on mobile devicesThe verification ideas can be improved