Image processing techniques for driver assistance - …files.meetup.com/1642210/driving...
Transcript of Image processing techniques for driver assistance - …files.meetup.com/1642210/driving...
![Page 1: Image processing techniques for driver assistance - …files.meetup.com/1642210/driving assistance.pdf · Image processing techniques for driver assistance ... • computer vision](https://reader031.fdocuments.in/reader031/viewer/2022021512/5aed272a7f8b9ad73f90aad8/html5/thumbnails/1.jpg)
Image processing techniques for driver assistance
Razvan Itu June 2014, Technical University Cluj-Napoca
![Page 2: Image processing techniques for driver assistance - …files.meetup.com/1642210/driving assistance.pdf · Image processing techniques for driver assistance ... • computer vision](https://reader031.fdocuments.in/reader031/viewer/2022021512/5aed272a7f8b9ad73f90aad8/html5/thumbnails/2.jpg)
IntroductionComputer vision & image processing
• from wiki: “any form of signal processing for which the input is an image”
• computer vision - subfield of artificial intelligence in computer science
• sometimes referred as “the emulation of human vision by a machine”
![Page 3: Image processing techniques for driver assistance - …files.meetup.com/1642210/driving assistance.pdf · Image processing techniques for driver assistance ... • computer vision](https://reader031.fdocuments.in/reader031/viewer/2022021512/5aed272a7f8b9ad73f90aad8/html5/thumbnails/3.jpg)
Popular libraries and software• OpenCV - most popular library,
the internet is full of examples & tutorials, good documentation
• FastCV - library for mobile only, from Qualcomm the makers of Snapdragon processors found on most of today's smartphones
• matlab - also used, especially in the research field, provides one of the best camera calibration modules, very good to quickly prototype & test algorithms
![Page 4: Image processing techniques for driver assistance - …files.meetup.com/1642210/driving assistance.pdf · Image processing techniques for driver assistance ... • computer vision](https://reader031.fdocuments.in/reader031/viewer/2022021512/5aed272a7f8b9ad73f90aad8/html5/thumbnails/4.jpg)
ADAS
• Adaptive cruise control • Lane change assistance • Collision avoidance system
(pre-crash) • Traffic sign recognition • Vehicular communication
systems • Automatic braking, parking,
etc.
ADAS - Advanced Driver Assistance Systems Systems to help the driver in the driving process, examples:
Image source: Continental Automotive Systems
![Page 5: Image processing techniques for driver assistance - …files.meetup.com/1642210/driving assistance.pdf · Image processing techniques for driver assistance ... • computer vision](https://reader031.fdocuments.in/reader031/viewer/2022021512/5aed272a7f8b9ad73f90aad8/html5/thumbnails/5.jpg)
ADAS - Computer vision research2 main categories for ADAS using image processing: • single camera = monocular • stereo cameras = binocular (2 or
more cameras) some systems use more sensors in addition to the cameras sensors used to capture proximity/distance to objects: radar(RAdio Detection And Ranging) lidar(LIght Detection And Ranging) sonar(SOund Navigation And Ranging)
Image source: Texas Instruments
Image source: Bosch
![Page 6: Image processing techniques for driver assistance - …files.meetup.com/1642210/driving assistance.pdf · Image processing techniques for driver assistance ... • computer vision](https://reader031.fdocuments.in/reader031/viewer/2022021512/5aed272a7f8b9ad73f90aad8/html5/thumbnails/6.jpg)
Monocular• not so popular, single camera = cheap, maybe
improved speed/performance (you process half the number of frames/images compared to a stereo setup)
• reduced accuracy, one camera means no depth perception/knowledge
• mostly used to: detect lanes, detect traffic signs, maybe traffic light(s)?
Image source: SAE International
![Page 7: Image processing techniques for driver assistance - …files.meetup.com/1642210/driving assistance.pdf · Image processing techniques for driver assistance ... • computer vision](https://reader031.fdocuments.in/reader031/viewer/2022021512/5aed272a7f8b9ad73f90aad8/html5/thumbnails/7.jpg)
Stereo vision
• high accuracy, the 2 cameras mimic the human vision, the two eyes add a "depth" perception of the surroundings
• good usages in detection & tracking of the obstacles
• good accuracy because of the two camera setup that adds depth
mono vs stereo: in monocular approaches there are assumptions of a flat road, a constant pitch angle or absence of a roll angle
• most usages in automotive industry, expensive systems
• use special cameras that are calibrated & set-up for the specific hardware that is being used
Image source: Subaru EyeSight System
![Page 8: Image processing techniques for driver assistance - …files.meetup.com/1642210/driving assistance.pdf · Image processing techniques for driver assistance ... • computer vision](https://reader031.fdocuments.in/reader031/viewer/2022021512/5aed272a7f8b9ad73f90aad8/html5/thumbnails/8.jpg)
Stereo vision - continued• More on depth: you can get depth information from
a pair of images of same scene
• Create a disparity map
• Disparity refers to difference in image location of an object seen by the left & right eyes => in computer vision, disparity = the difference in coordinates of similar features within a pair of stereo images
![Page 9: Image processing techniques for driver assistance - …files.meetup.com/1642210/driving assistance.pdf · Image processing techniques for driver assistance ... • computer vision](https://reader031.fdocuments.in/reader031/viewer/2022021512/5aed272a7f8b9ad73f90aad8/html5/thumbnails/9.jpg)
Stereo vision - continued
Disparity map: Mars rover uses it to navigate through obstacles!
Image source: NASA
![Page 10: Image processing techniques for driver assistance - …files.meetup.com/1642210/driving assistance.pdf · Image processing techniques for driver assistance ... • computer vision](https://reader031.fdocuments.in/reader031/viewer/2022021512/5aed272a7f8b9ad73f90aad8/html5/thumbnails/10.jpg)
Basic algorithms & approaches1. Camera calibration
• using special software and by taking repeated photographs of a known pattern
• OpenCV provides a calibration software, most used is from the matlab
• through calibration process you get to know some of the camera parameters: intrinsic & extrinsic
Image source: dreamincode.net forumsImage source: http://www.driveassistapp.com
![Page 11: Image processing techniques for driver assistance - …files.meetup.com/1642210/driving assistance.pdf · Image processing techniques for driver assistance ... • computer vision](https://reader031.fdocuments.in/reader031/viewer/2022021512/5aed272a7f8b9ad73f90aad8/html5/thumbnails/11.jpg)
Basic algorithms & approaches!
2. Traffic sign recognition
• color based: for example search the RED color to detect the STOP sign
• shape based: detect the round shape of the SPEED LIMIT sign
• machine learning based: use a set of training images that contain both good & bad examples (ex: use Viola Jones detector)
• Viola Jones detector works by sliding detection window across image
• At each position, decide if desired object inside window
• Uses Haar features
Image source: Mercedes-Benz
Image source: Siemens
![Page 12: Image processing techniques for driver assistance - …files.meetup.com/1642210/driving assistance.pdf · Image processing techniques for driver assistance ... • computer vision](https://reader031.fdocuments.in/reader031/viewer/2022021512/5aed272a7f8b9ad73f90aad8/html5/thumbnails/12.jpg)
Basic algorithms & approaches!
!
3. Lane detection
• apply filter to image, ex Canny to find edges
• apply a Hough Transform to detect lines
• filter these lines input image
Images source: http://www.transistor.io/revisiting-lane-detection-using-opencv.html
![Page 13: Image processing techniques for driver assistance - …files.meetup.com/1642210/driving assistance.pdf · Image processing techniques for driver assistance ... • computer vision](https://reader031.fdocuments.in/reader031/viewer/2022021512/5aed272a7f8b9ad73f90aad8/html5/thumbnails/13.jpg)
Basic algorithms & approaches!
!
4. Obstacle detection & tracking
Monocular appearance based methods:
• use colour and shape to detect image regions that belong to obstacles
Monocular motion based methods:
use motion of image features and optical flow
• optical flow = pattern of apparent motion of image objects between two consecutive frames caused by the movement of object or camera (definition adapted from wiki)
• you have a set of points in an image, find the same points in another image
Image source: OpenCV
Stereo methods use the disparity map to get depth of image features. => detection of the ground plane (if it is unknown) and the detection of obstacles = any feature not on the ground plane
![Page 14: Image processing techniques for driver assistance - …files.meetup.com/1642210/driving assistance.pdf · Image processing techniques for driver assistance ... • computer vision](https://reader031.fdocuments.in/reader031/viewer/2022021512/5aed272a7f8b9ad73f90aad8/html5/thumbnails/14.jpg)
Basic algorithms & approaches!
!
4. Obstacle detection & tracking - continued
A lot of research & proposed solutions, some examples:
• Feature based detection (ex: using Haar features): Haar features =(sum of pixels in black region) - (sum of pixels in white region)
• Detection by background subtraction
• Many other methods, most based on stereo imaging
Image source: behance.net Image source: atlas.web.ua.pt Image source: http://reason.cs.uiuc.edu/
![Page 15: Image processing techniques for driver assistance - …files.meetup.com/1642210/driving assistance.pdf · Image processing techniques for driver assistance ... • computer vision](https://reader031.fdocuments.in/reader031/viewer/2022021512/5aed272a7f8b9ad73f90aad8/html5/thumbnails/15.jpg)
Other tips• only process what's needed/required from an image,
ex: processing road lanes, remove the sky/horizon from the image, you only want to process the road pixels (this is called Region Of Interest = ROI)
• use pre-defined, pre-computed values, for example you calibrate the camera once, and store the obtained results, re-use them when needed
![Page 16: Image processing techniques for driver assistance - …files.meetup.com/1642210/driving assistance.pdf · Image processing techniques for driver assistance ... • computer vision](https://reader031.fdocuments.in/reader031/viewer/2022021512/5aed272a7f8b9ad73f90aad8/html5/thumbnails/16.jpg)
Drive Assist App• monocular system based on Android devices
• fast, cheap & robust object detection (why? => use a mobile phone, portable, everyone has a phone, only requirement is to use a windshield mount for the phone
• not new, or breakthrough app, but performs better than competition
• real time processing & feedback (displaying the results)
![Page 17: Image processing techniques for driver assistance - …files.meetup.com/1642210/driving assistance.pdf · Image processing techniques for driver assistance ... • computer vision](https://reader031.fdocuments.in/reader031/viewer/2022021512/5aed272a7f8b9ad73f90aad8/html5/thumbnails/17.jpg)
Drive Assist AppSome testing results & effective range
Images source: http://www.driveassistapp.com
![Page 18: Image processing techniques for driver assistance - …files.meetup.com/1642210/driving assistance.pdf · Image processing techniques for driver assistance ... • computer vision](https://reader031.fdocuments.in/reader031/viewer/2022021512/5aed272a7f8b9ad73f90aad8/html5/thumbnails/18.jpg)
Drive Assist AppPerformance test
![Page 21: Image processing techniques for driver assistance - …files.meetup.com/1642210/driving assistance.pdf · Image processing techniques for driver assistance ... • computer vision](https://reader031.fdocuments.in/reader031/viewer/2022021512/5aed272a7f8b9ad73f90aad8/html5/thumbnails/21.jpg)
Q&A
Thank you!
Razvan Itu www.driveassistapp.com
www.mobileway.net twitter.com/razvan