Vision System for Autonomous Navigation Paul Nguyen and Michael Slutskiy

ECE 533 Fall 2005 Project

Table of Contents 1. Introduction................................................................................................................. 3 2. Approach..................................................................................................................... 3

2.1 Key Issues ........................................................................................................... 4 2.1.1 Texture Noise.............................................................................................. 4 2.1.2 Simulated Sandpit ....................................................................................... 4 2.1.3 Missing Barrel............................................................................................. 4

2.2 Plan of Attack ..................................................................................................... 4 3. Work performed .......................................................................................................... 5

3.1 Noise Reduction.................................................................................................. 5 3.1.1 Gaussian Filter ................................................................................................ 5 3.1.2 Median Filter................................................................................................... 5

3.2 Color Segmentation ............................................................................................ 5 3.2.1 “Orange” Filter............................................................................................ 6

4. “Green” Filter.............................................................................................................. 6 4.1 Sobel Filter and Hough Transform ..................................................................... 7

5. Results ......................................................................................................................... 7 6. Discussion ................................................................................................................... 9 7. Conclusion .................................................................................................................. 9 8. References ................................................................................................................... 9 9. Division of Labor ...................................................................................................... 10

2

1. Introduction The purpose of this project is to develop a robust image analysis system that would

allow vision based autonomous navigation and object avoidance. This package will be

part of the control software used on the IEEE Robot Team’s entry into the Intelligent

Ground Vehicle Competition1. Motivated by the fact that humans can easily perform this

task relying entirely on vision, we wish to use digital image processing techniques to

achieve the same capability.

For the competition the vehicle must avoid any obstacles it is faced with (cones,

construction barrels, barriers and potholes) while staying within a lane indicated by solid

or dashed lines. In the past, objects and lines were distinguished from the background

simply by using adaptive threshold segmentation on the blue plane intensity of the

original image. However, this had its shortcomings, particularly if the surface material or

color changed (going from a grassy surface to a simulated sandpit, for example).

Moreover, there was no way to prevent the robot from viewing a gap in the dashed lane

marking as an acceptable path which could result in the vehicle going out of bounds.

The goal is to improve the already existing software to better use visual information

and present more reliable data to the Artificial Intelligence system, which will plan the

course of action for the robot. The scope of this project will not include the AI system,

although much improvement can be achieved there as well. Mainly we will focus on

providing a binary image that is at least as good or better to the current AI

implementation. The current strategy works fairly reliably provided clear path is

represented by black and objects/lines are represented by white in the binary image.

2. Approach Of course, “reliable” and “better” are only useful adjectives when the context is

understood: below are several images that posed problems for the adaptive threshold

system. These are some of the issues we will be attempting to address in this project.

1 Additional information about the competition is available at http://www.igvc.org/deploy

3

2.1 Key Issues

2.1.1 Texture Noise Here the grass texture and

variance in color has created

noise in the binarized image,

which will be interpreted as an

object directly in front of the

vehicle – an undesirable outcome.

2.1.2 Simulated Sandpit Clearly, the thresholding

technique fails us here since the

tarp should not be interpreted as

an obstacle and thus should

appear black in the binarized

image on the right.

2.1.3 Missing Barrel In this image the shadow

side of the construction barrel is

missed and the object for the

most part does not appear in the

final image – unreliable data.

This in fact happened last year.

2.2 Plan of Attack Using these three particular problems as a guide, we decided to implement noise

filtering as well as a color segmentation scheme to aid/replace the threshold

segmentation. Using knowledge gained in class it was determined that the Median and

Gaussian filters may be good candidates for noise removal. Also, we chose to try to

implement several methods that could be helpful in future extensions of the vision system

(for example detecting dashed lane markings or preparing images for mapping). To this

4

end, we wanted to implement (time allowing) the Sobel edge extraction, Hough transform

for line extrapolation and a transform to convert the image into an overhead view.

3. Work performed

3.1 Noise Reduction

3.1.1 Gaussian Filter Gaussian filtering was

unable to eliminate all the noise

in this image. An additional

downside of this filter is that it

tends to blur the image and does

not always preserve edges well.

This was deemed unacceptable, especially when compared with Median Filter.

3.1.2 Median Filter The same image as above is

pictured here after a 5x5 Median

filter and to the right is 5x5

followed by 3x3 filter. This gave

us a great result getting rid of all

noise in the middle of the lane.

However, an unavoidable

downside of the filtering is the

almost full disappearance of the

barrel. Clearly, something had to

be done to reliably find objects

that may be in the shadow.

3.2 Color Segmentation It was observed that much of the information in the image is contained in the color

data. As we were discarding the red and green planes for our threshold technique we were

essentially throwing away valuable knowledge of what lies ahead. We decided to take a

5

two-step approach to extracting this information. First, we would find all orange pixels

(seeing as orange inevitably means obstacle) and turn it to white. Next we would look for

green, brown and yellow hues and change them to black since those most often represent

a clear path. We decided that this “filtering” would be most convenient in HSI color

space. RGB would pose a problem since orange actually contains fairly high levels of

green so there is a potential conflict with the two filters. In HSI, however, this conflict

does not occur since orange is a distinctly different hue. The results are displayed below.

3.2.1 “Orange” Filter The left image is a typical

result of the orange extraction

algorithm. The right image is the

final result after binarizing and

Median filtering. Note how the

barrel is now clearly detected.

Although parts of it are still black, our AI only requires the base of the obstacle to be

clearly visible to function correctly.

4. “Green” Filter

Here are two good examples of cases where the dynamic threshold did not perform

well originally. The middle images are the result of applying the color segmentation

based on green, brown and yellow hue range. And the final images are the result of both

color filters, dynamic thresholding and Median filtering.

Although, the right lane marking is erased in the first example, it is an acceptable

sacrifice – the line would be detected if the robot were to move closer to it, as it stands

however, it is quite far from being a serious threat.

6

While there is a chance that through a better adjustment of the filter it would be

possible to capture the tarp entirely, the median filter makes this almost unnecessary.

4.1 Sobel Filter and Hough Transform As could be seen in several of the above examples, parts of the lane marking are

sometimes eroded by noise filtering and segmentation. Additionally, although we do not

currently have a sample image of this, there are areas of the course where a dashed line

replaces the solid demarcation of the above examples. To deal with both these situations

we sought to implement the Hough transform which would at least partially allow the

robot to “guess” where the lane boundary is based on previous data. Since we already

have a good binary image to work with, it was determined that Canny edge detection was

not necessary and a simple Sobel filter would suffice to remove the “bulk” of the objects.

Although, the Hough transform has not yet been fully integrated into the AI system, we

nonetheless fully completed its implementation. See examples in the results section.

5. Results Below is a comparison of the results of our revamped vision system to the old version.

Original Old Vision System New Vision System Hough Result (blue)

7

Original Old Vision System New Vision System Hough Result (blue)

8

6. Discussion As can be seen in the above images, the new system performs at least as well and in

many cases better than its predecessor in detecting objects, ignoring tarps and eliminating

noise. Therefore, we made good progress in making the vision processing more reliable.

Additionally, we set out with the goal of making our software robust – for vision based

autonomous navigation to be possible the analysis must happen efficiently in real time.

To achieve this we employed the Java Advanced Imaging library (our software platform

is Java based to increase Operating System independence), which employs native

hardware accelerations for imaging operations. This proved to be sufficiently fast in the

past, though it remains to be seen how the new system will perform on “live” data.

Additionally, a 3D Lookup Table was used for our implementation of the Hough

transform to reduce the necessary computations. Further modifications to the code are

being considered to improve efficiency even more.

7. Conclusion This project successfully addressed several known issues with the already existing

vision system on the IEEE Robot Team’s IGVC vehicle by employing filtering and

segmentation techniques learned in class. Additionally, we extended the software

capabilities for future extensions such as mapping by efficiently implementing the Hough

transform. For this purpose we are also looking into adding the ability to synthetically

rotate the images into a top-down view. We expect to have this feature operational in the

near future. Other work will likely focus on fine-tuning the various filters and making the

software as lean as possible to increase reaction time.

8. References 1 Gonzalez, J.P. and Ozguner, U. “Lane detection using histogram-based segmentation and decision trees,”

Intelligent Transportation Systems, 2000. Proceedings. 2000 IEEE, 1-3 Oct. 2000, pp. 346 –351.

2 Yu, B.; Jain, A.K.; “Lane boundary detection using a multiresolution Hough transform,” International Conference on Image Processing Proceedings, 26-29 Oct. 1997, pp. 748 -751 vol.2.

3 Bertozzi, M. and Broggi, A. “Real-time lane and obstacle detection on the GOLD system,” Intelligent Vehicles Symposium, 1996. Proceedings of the 1996 IEEE, 19-20 Sept. 1996, pp. 213 –218.

4 Siegwart and Nourbakhsh. “Cameras, Images, and Low-Level Robot Vision,” Introduction to Autonomous Mobile Robots. MIT Press, April 2004. Sec. 4.1.8.

9

9. Division of Labor Paul Nguyen Michael Slutskiy

Research 50% 50% Program

(Development and Testing) 50% 50%

Presentation 50% 50% Report 50% 50%

Total Contribution 50% 50% __________________________________ __________________________________

Paul Nguyen Michael Slutskiy

10