LIST OF ABBREVIATIONS...LIST OF ABBREVIATIONS IV Intelligent Vehicle AI Artificial Intelligence HI...

LIST OF ABBREVIATIONS

IV Intelligent Vehicle

AI Artificial Intelligence

HI Human Intelligence

ML Machine Learning

NN Neural Network

ANN Artificial Neural Network

GPS Global Positioning System

RADAR Radio Detection and Ranging

LIDAR Light Detection and Ranging

QVGA Quarter Video Graphic Array

USB Universal Serial Bus

RC Remote Controller

LASER Light Amplification by Stimulated Emission of Radiations

CV Computer Vision

Open-CV Open Computer Vision

ADAS Advanced Driver Assistance System

IMU Initial Measurement Unit

ITS Intelligent Transportation System

IBM International Business Machine

EBL Explanation Based Learning

ii

ii

OCR Optical Character Recognition

MRI Magnetic Resonance Imaging

3D 3-Dimensional

NHTSA National Highway Traffic Safety Administration

ACC Adaptive Cruise Control

HDMI High Definition Multimedia Interface

Wi-Fi Wireless Fidelity

GPIO General Purpose Input/output

TCP Transmission Control Protocol

IP Internet Protocol

NumPY Numerical Python

ROI Region of Interest

LDW Lane Departure Warning

MLP Multi-Layer Perceptrons

iii

iii

LIST OF FIGURES

Fig: 1.5(a) Block-diagram of Deep learning and computer vision ........................... 20

Fig: 3.1.1.1(a) Raspberry PI module .............................................................................. 20

Fig: 3.1.1.1(b) Camera port in Raspberry Pi ................................................................... 21

Fig: 3.1.1.1(c) Raspberry PI command prompt .............................................................. 20

Fig: 3.1.1.1(d) Voltage divider ....................................................................................... 22

Fig: 3.1.1.1(e) Raspberry PI voltage divider .................................................................. 23

Fig: 3.1.1.1(f) HC-SR04 pins ......................................................................................... 23

Fig: 3.1.1.1(g) Raspberry PI and HC-SR04 VCC & GND connections ......................... 24

Fig: 3.1.1.1(h) Raspberry PI and HC-SR04 ECHO pin connections .............................. 24

Fig: 3.1.1.1(i) Connections on breadboard .................................................................... 25

Fig: 3.1.1.1(j) Voltage divider on breadboard ............................................................... 25

Fig: 3.1.1.1(k) Raspberry PI & HC-SR04 complete connections ................................... 25

Fig: 3.1.1.2(a) PI-Cam module ....................................................................................... 26

Fig: 3.1.1.3(a) Arduino module ...................................................................................... 27

Fig: 3.1.1.4(a) Ultrasonic sensor HC-SR04 .................................................................... 28

Fig: 3.1.1.4(b) Component attached on RC-Car ............................................................. 29

Fig: 3.1.2.2(a) Serial interface between Arduino & RC-Controller ............................... 30

Fig: 3.2.1(a) Opto-isolator, interfacing Arduino and RC-Controller ........................... 32

Fig: 3.2.1(b) Complete anatomy of the project ............................................................ 33

Fig: 3.3.2(a) Stop-sign detection and distance measurement ...................................... 35

Fig: 3.3.3.2(a) Traffic green light detection ................................................................... 37

Fig: 3.3.3.2(b) Traffic red light detection ....................................................................... 37

Fig: 3.3.3.2(c) Flow chart of HAAR cascade training .................................................... 38

Fig: 3.4(a) Flow-chart of backpropagation ............................................................... 40

iv

iv

Fig: 4.1(a) Pycharm IDE .......................................................................................... 41

Fig: 4.1(b) Arduino IDE ........................................................................................... 41

Fig: 4.1(c) Python shell of Raspberry PI .................................................................. 42

Fig: 4.1(d) Video streaming received on Laptop ...................................................... 44

Fig: 4.1(e) Console screen showing successful connection ...................................... 44

Fig: 4.1(f) Green light detection results ................................................................... 45

Fig: 4.1(g) Forward command of green light on console screen .............................. 45

Fig: 4.1(h) Red traffic light detection results ............................................................ 46

Fig: 4.1(i) Recognition of red light on Console screen ............................................ 46

Fig: 4.1(j) Obstacle car ahead .................................................................................. 47

Fig: 4.1(k) Obstacle detected on Console screen with distance ................................ 47

Fig: 4.1(l) Stop sign is at the distance of 27.1cm ..................................................... 48

Fig: 4.1(m) Forward command on console screen ..................................................... 48

Fig: 4.1(n) Stop sign detection .................................................................................. 49

Fig: 4.1(o) Stop time calculation on console screen ................................................. 49

Fig: 4.1(p) Stop sign is in less than 25cm ................................................................. 50

Fig: 4.1(q) Forward command after 5 seconds ......................................................... 50

v

v

Table of Contents

DEDICATION.................................................................................................................. iii

CERTIFICATE ................................................................................................................ iv

ACKNOWLEDGMENT ...................................................................................................v

ABSTRACT ...................................................................................................................... vi

LIST OF ABBREVIATIONS ........................................................................................ vii

LIST OF FIGURES ......................................................................................................... ix

CHAPTER-1 INTRODUCTION ......................................................................................1

1.1 Overview ........................................................................................................................1

1.2 Motivation ......................................................................................................................2

1.2.1 Evidence of technology leading to intelligent vehicle ....................................2

1.2.2 Need of intelligent vehicle ..............................................................................3

1.3 Brief introduction to project...........................................................................................4

1.4 Aims and Objectives ......................................................................................................4

1.5 Approaches ....................................................................................................................5

CHAPTER-2 LITERATURE REVIEW .........................................................................7

2.1 Image processing ...........................................................................................................7

2.2 Machine learning ...........................................................................................................9

2.2.1 Supervised algorithm ....................................................................................10

2.2.2 Unsupervised algorithm ................................................................................11

2.3 Computer Vision ..........................................................................................................11

2.4 Road detection .............................................................................................................13

2.5 Object detection and recognition .................................................................................15

2.6 Collision detection .......................................................................................................17

vi

vi

2.7 Human error .................................................................................................................18

2.8 Summary ......................................................................................................................19

CHAPTER-3 PROJECT METHODOLOGY ...............................................................20

3.1 System design ..............................................................................................................20

3.1.1 Modules.........................................................................................................20

3.1.1.1 Raspberry PI...................................................................................20

3.1.1.2 PI-Cam ...........................................................................................26

3.1.1.3 Arduino ..........................................................................................26

3.1.1.4 Ultrasonic sensor ............................................................................27

3.1.2 Interfaces .......................................................................................................29

3.1.2.1 Socket interface using TCP ............................................................29

3.1.2.2 Serial interface ...............................................................................30

3.1.3 Data ...............................................................................................................31

3.1.3.1 Video frames ..................................................................................31

3.1.3.2 Ultrasonic sensor data ....................................................................31

3.2 Methodology ................................................................................................................31

3.2.1 System hardware ...........................................................................................31

3.2.2 System software ............................................................................................33

3.3 Algorithms ...................................................................................................................34

3.3.1 Obstacle detection .........................................................................................34

3.3.2 Stop sign detection and recognition ..............................................................34

3.3.3 Traffic light detection and recognition .........................................................36

3.3.3.1 Brightest spot .................................................................................37

3.3.3.2 Position of brightest spot ...............................................................37

3.4 Self-learning using neural network ..............................................................................38

vii

vii

CHAPTER-4 RESULTS AND DISCUSSIONS ............................................................41

4.1 System based division and results ...............................................................................41

4.2 Problems encountered ..................................................................................................51

4.2.1 Software problems ........................................................................................51

4.2.2 Hardware problems .......................................................................................51

4.3 Limitations ...................................................................................................................52

4.4 Conclusion ...................................................................................................................52

4.5 Future Work .................................................................................................................53

REFERENCES .................................................................................................................54

viii

viii

1

1

CHAPTER 1

INTRODUCTION TO INTELLIGENT VEHICLES

1.1 OVERVIEW.

Vehicles, whose functions are enriched with attributes to increase safety,

environmental awareness, effectiveness, comfort level and prestige, so that they can play a

key role in creating optimal mobility, are now being invented, planned and manufactured

for general use. Throughout the full spectrum of transport, intelligent vehicles will soon

exempt people from the routine of driving. These vehicles are also contributing to make

travelling safer.

With the rapid growth of technological development, the possibilities of utilizing

the driving style intelligence has been increased. The AI and ML algorithms are being

applied in vehicles to make them intelligent. Our project implementation provides a real-

time prediction of steering of a vehicle using image processing.

In this thesis report, we have utilized ANN for the prediction of left, right, forward

and reverse orientation of a vehicle. This IV is trained on a track manually. PI camera takes

real time video frames of the track and Raspberry PI sends these video frames to laptop. In

training, manually left, right, forward and reverse (stop) keys are pressed to drive the

vehicle on the track. Simultaneously video frames are captured by PI camera. A video

frame on an instant is labeled with the key pressed on that instant. Training creates the

trained parameter for the vehicle to learn. Neural network takes the decisions according to

the labeled video frames. This thesis, entirely focuses on the application of deep learning

where full autonomous vehicles are also possible.

2

2

1.2 MOTIVATION

1.2.1 EVIDENCE OF TECHNOLOGIES LEADING TO INTELLGENT VEHICLE

IV use a variety of techniques to detect their surroundings, such as GPS, Self-steering,

Digital maps, Image recognition, Cameras and ultra sound, RADAR and LIDAR

Technologies that give rise to intelligent vehicle are:

• GPS:

GPS is one of the technologies used by self-driving vehicles. Many people who

have used GPS navigators for years can attest to their reliability. In fact, some

people can’t go anywhere within the city without a GPS navigator. The navigator

also displays their position on the map. Chances are that your phone has a GPS chip

and you can see your position on a map change as you walk or move.

• Self-steering system

Steering systems that use cameras that watch road markings and radar and laser

sensors that track other objects.

• Digital maps

Process by which a collection of data is compiled and formatted into a virtual.

• Image recognition

Image recognition is very important for achieving a true understanding of the

environment, because it is the only way to see indicators such as traffic lights,

brake lights and turn signals.

• Camera and ultrasonic

Cameras will receive visual information to help the car’s software determine

traffic signals and cross traffic warning. Ultrasonic sensor will be used to inform

about the immediate environment for parking assistance.

• RADAR

The set of technologies that help self-driving cars detect objects around them.

How do airplanes above the sky traveling at over 600 km/hr. detect other

airplanes and mountains. RADAR is also used by self-driving cars.

3

3

• LIDAR

Using bounced pinpoints of laser light, LIDAR effectively forms a 3D model of

the world around the car, trivially determining size and distance of all things

around the car, day or night, sunny or cloudy. It is also used by self-driving cars

to detect stationary and objects in motion around them.

Each of these technologies will collect different information for the vehicle, and

based on the information collected the car will make interpretations and make the

appropriate response for that situation.

3.3.2 NEED OF INTELLIGENT VEHICLES

• Autonomous technology may save thousands of lives by making decisions on the

road faster than humans are able.

• To reduce the traffic collisions and the resulting injuries.

• The relief of travelers from driving.

• A reduction in crime.

• Enhanced mobility for children, elderly, disabled and the poor people.

• Reduce the stress of driving and allow motorists to rest and work while traveling.

• May reduce many common accident risks and therefore crash costs and insurance

premiums.

• Can drop off passengers and find a parking space, increasing motorist convenience

and reducing total parking costs.

• It can take decisions faster than the humans to avid road accidents.

• Reduce costs of paid drivers for taxis and commercial transport.

4

4

1.3 BRIEF INTRODUCTION TO PROJECT

Our project is based on the practice of deep learning through the applied theme of

developing an IV which can be utilized for safety, obedience of traffic laws and driver

relaxation system. The proposed work uses RC vehicle to test the performance of the NN

algorithms in real time. Vehicle prototype modules such as Raspberry PI, PI-Cam module,

vehicle controller module, Arduino module, ultrasonic sensor module and battery module.

Tracks are used for real time performance analysis of the prototype vehicles

1.4 AIMS & OBJECTIVES

The principle aims of this 4th year group project is to design and develop an IV

which is capable of recognizing traffic light, detecting stop sign and obstacle and performs

the task of steering according to the current consequences of road.

We are aimed to equip the vehicles with advance features that can help the driver to

take some rest during a long journey and make decisions when human intelligence is not

applicable. This application is developed on an RC car, includes a Raspberry Pi board

attached with a pi camera module and an HC-SR04 ultrasonic sensor, used to collect input

data. Two client programs run on Raspberry Pi for streaming color video and ultrasonic

sensor data to the computer via Wi-Fi connection. The Laptop handles multiple tasks:

receiving data from Raspberry Pi, neural network training and prediction(steering), object

detection (stop sign and traffic light), distance measurement (monocular vision), and

sending instructions to Arduino through USB connection. The RC car used in this project

has an on/off switch type controller. When a button is pressed, the resistance between the

relevant chip pin and ground is zero. Thus, an Arduino board is used to simulate button-

press actions. Four Arduino pins are chosen to connect four chip pins on the controller,

corresponding to forward, reverse, left and right actions respectively. Overall, the RC car

could successfully navigate on the track with the ability to avoid obstacle collision, and

respond to stop sign and traffic light accordingly.

5

5

1.5 APPROACHES

The age of autonomy is real, and it's almost here but we’re still learning how to build

intelligent cars, so there are different companies trying out different ideas. At a high-level

of abstraction, there are two types of approaches of building intelligent vehicles. Most

companies use the classical approach. This approach combines computer vision, sensor

fusion, (LIDAR, RADAR), localization, control theory, and path planning.

Google relies on a highly expensive complex remote-sensing system called

“Lidar”, which in simple terms — measures distance by pointing lasers at targets

surrounding the car and analyzing the light that's reflected."

Tesla's Autopilot relies on a combination of "cameras, radar, ultrasonic sensors and

data automatically steer down the highway, change lanes, and adjust speed in

response to traffic," and uses auto braking technology by Israeli company Mobileye

The other, second approach involves training a single deep neural network to take

sensor inputs and produce steering, throttle, and brake outputs. This approach is called

“deep learning”

Drive.ai was born for this. It was founded in 2015 by deep-learning experts from

Stanford University’s Artificial Intelligence Laboratory.

Deep-learning systems thrive on data. The more data an algorithm sees, the better

it’ll be able to recognize, and generalize about, the patterns it needs to understand

to drive safely.

Decision making through deep learning based on fused sensor data has advantages

in an autonomous vehicle context. Namely, it offers some protection against sensor

failure, since the deep-learning algorithms can be trained explicitly on perception

data with missing sensor modalities. Sensor failure is most often not a hardware or

software issue but rather a sensor that isn’t producing good data for some reason,

like sun glare, darkness at night, or (more commonly) being occluded by water.

The approach we are using for our project is deep learning because we believe that

deep learning leads to better performance and smaller systems and training data can be

collected by driving on a wide variety of places and in a diverse set of lighting conditions.

6

6

We can train the weights of the network through different algorithms to minimize the

mean-squared error between the steering command output by the network.

What does the algorithm do?

• Process the image

• Output steering command based on image content

• Also output desired speed for current road condition

• Feedback the actual speed of the car

• let a speed controller to control the throttle/brake

Deep learning has become a very powerful tool for many “intelligent” tasks. This

approach needs to feed many input images and steering angles and the resulting weights

and parameters on your layers can be then used to assign a steering angle to your input

images. In our project, we have many input images labeled with steering angles which

allowed us to drive a car around

Fig: 1.5(a) Block diagram of Deep learning and computer vision

7

7

CHAPTER 2

LITERATURE REVIEW

This chapter gives an insight to the literature survey. The major of this thesis report

as well as the similar or related works are described in this chapter. Intelligent vehicles are

like self-steered vehicles which is embedded with human intelligence. These vehicles have

eyes (camera) to see the real world, brain (NN) to think and hands (steering) to keep the

vehicle smoothly on track.

The idea of machine intelligence emerged in 1950. Nowadays, these vehicles are

being manufactured by many companies and going to be commercialized very soon.

The evolution of this technology includes many different types of technological

approaches, some of them are discussed below:

2.1 IMAGE PROCESSING

The Field of image processing is continually evolving. During the past years, there

has been a significant increase in the level of interest in image morphology, neural

networks, full-color image processing, image data compression, image recognition, and

knowledge-based image analysis systems.

Image processing is processing of images using mathematical operations by using

any form of signal processing for which the input is an image, a series of images, or a

video, such as a photograph or video frame; the output of image processing may be either

an image or a set of characteristics or parameters related to the image. Image processing

methods stems from two principal application areas: improvement of pictorial information

for human interpretation, and processing of scene data for autonomous machine perception.

One of the first applications of digital images was digitized newspaper pictures sent

by undersea cable between London and New York. Introduction of the Bartlane cable

picture transmission system in the early 1920's reduced the time required to transport a

picture across the Atlantic from more than a week to less than three hours. Pictures were

coded for cable transmission and then reconstructed at the receiving end on a telegraph

8

8

printer fitted with typefaces simulating a halftone pattern. The early Bartlane systems were

capable of coding images in five distinct brightness levels. This was increased to fifteen

levels in 1929.

Analyzing road scenes using cameras could have a crucial impact in many domains,

such as autonomous driving, ADAS, personal navigation, mapping of large scale

environments, and road maintenance. For instance, vehicle infrastructure, signage, and

rules of the road have been designed to be interpreted fully by visual inspection. As the

field of computer vision becomes increasingly mature, practical solutions to many of these

tasks are now within reach.

Important Academic Papers Regarding Deep Learning Processing

• NVIDIA - “End to End Learning for Self-Driving Cars”

Video input from a forward-facing camera is trained against steering wheel

position and deep learning networks are capable of detecting important road

features with limited additional nudging in the right direction

• Comma.ai - "Learning A Driving Simulator"

Using video input with no additional training metadata (IMU, wheel angle) auto-

encoded video was generated, predicting many frames into the future while

maintaining road features. Comma approach to Artificial Intelligence for self-

driving cars1 is based on an agent that learns to clone driver behaviors and plans

maneuvers by simulating future events in the road.

• NYU & Facebook AI - “Deep Multi-Scale Video Prediction Beyond Mean Square

Error”

Learning to predict future images from a video sequence involves the construction of

an internal representation that models the image evolution accurately, and therefore, to

some degree, its content and dynamics. Therefore pixel-space video prediction may be

viewed as a promising avenue for unsupervised feature learning.

9

9

2.2 MACHINE LEARNING

ML is a sub-set of artificial intelligence where computer algorithms are used to

autonomously learn from data and information. In machine learning computers don’t have

to be explicitly programmed but can change and improve their algorithms by themselves.

Today, machine learning algorithms enable computers to communicate with humans,

autonomously drive cars, write and publish sport match reports, and find terrorist suspects.

1950 — Alan Turing creates the “Turing Test” to determine if a computer has real

intelligence. To pass the test, a computer must be able to fool a human into believing it is

also human.

1952 — Arthur Samuel wrote the first computer learning program. The program was the

game of checkers, and the IBM computer improved at the game the more it played,

studying which moves made up winning strategies and incorporating those moves into its

program.

1957 — Frank Rosenblatt designed the first neural network for computers (the perceptron),

which simulate the thought processes of the human brain.

1967 — The “nearest neighbor” algorithm was written, allowing computers to begin using

very basic pattern recognition. This could be used to map a route for traveling salesmen,

starting at a random city but ensuring they visit all cities during a short tour.

1979 — Students at Stanford University invent the “Stanford Cart” which can navigate

obstacles in a room on its own.

1981 — Gerald Dejong introduces the concept of EBL, in which a computer analyses

training data and creates a general rule it can follow by discarding unimportant data.

1985 — Terry Sejnowski invents NetTalk, which learns to pronounce words the same way

a baby does.

10

10

1990s — Work on machine learning shifts from a knowledge-driven approach to a data-

driven approach. Scientists begin creating programs for computers to analyze large

amounts of data and draw conclusions — or “learn” — from the results.

1997 — IBM’s Deep Blue beats the world champion at chess.

2006 — Geoffrey Hinton coins the term “deep learning” to explain new algorithms that let

computers “see” and distinguish objects and text in images and videos.

2010 — Microsoft Kinect can track 20 human features at a rate of 30 times per second,

allowing people to interact with the computer via movements and gestures.

After that, the work on ML goes on. ML algorithms are organized into taxonomy, based

on the desired outcome of the algorithm. Common algorithm types include:

2.2.1 SUPERVISED MACHINE LEARNING

The majority of practical ML uses supervised learning. Supervised learning is

where you have input variables (x) and an output variable (Y) and you use an algorithm to

learn the mapping function from the input to the output.

Y = f(X)

The goal is to approximate the mapping function so well that when you have new

input data (x) that you can predict the output variables (Y) for that data.

It is called supervised learning because the process of an algorithm learning from

the training dataset can be thought of as a teacher supervising the learning process. We

know the correct answers; the algorithm iteratively makes predictions on the training data

and is corrected by the teacher. Learning stops when the algorithm achieves an acceptable

level of performance.

11

11

2.2.2 UNSUPERVISED MACHINE LEARNING

Unsupervised learning is where you only have input data (X) and no corresponding

output variables. The goal for unsupervised learning is to model the underlying structure

or distribution in the data to learn more about the data.These are called unsupervised

learning because unlike supervised learning above there is no correct answers and there is

no teacher. Algorithms are left to their own devises to discover and present the interesting

structure in the data.

Unsupervised learning problems can be further grouped into clustering and

association problems.

• Clustering: A clustering problem is where you want to discover the inherent groupings

in the data, such as grouping customers by purchasing behavior.

• Association: An association rule learning problem is where you want to discover rules

that describe large portions of your data, such as people that buy X also tend to buy Y.

2.3 COMPUTER VISION

Computer vision tasks include methods for acquiring, processing, analyzing and

understanding digital images, and extraction of high-dimensional data from the real world

to produce numerical or symbolic information, e.g., in the forms of decisions.

Understanding in this context means the transformation of visual images (the input of the

retina) into descriptions of the world that can interface with other thought processes and

elicit appropriate action.

Nowadays computer vision is used in many application areas such as OCR, Face

detection, smile detection, object recognition, vision based biometrics, face recognition,

object recognition, motion capture, Panorama stitching,3D terrain modeling, Obstacle

detection, position tracking, vision guided robots, 3D imaging MRI and image guided

surgery.

Upcoming self-driving cars are also using the same computer vision technology. In

August 2012, Google announced that their self-driving car had completed over 300,000

12

12

autonomous-driving miles (500,000 km) accident-free, typically having about a dozen cars

on the road at any given time, and were starting to test them with single drivers instead of

in pairs. In late-May 2014, Google revealed a new prototype of its driverless car, which

had no steering wheel, gas pedal, or brake pedal, and was fully autonomous.

In June 2015, Google founder Sergey Reconfirmed that there had been 12 collisions

as of that date, eight of which involved being rear-ended at a stop sign or traffic light, two

in which the vehicle was side-swiped by another driver, one in which another driver rolled

through a stop sign, and one where a Google employee was controlling the car manually.

In July 2015, three Google employees suffered minor injuries when the self-driving car

they were riding in was rear-ended by a car whose driver failed to brake at a traffic light.

This was the first time that a self-driving car collision resulted in injuries. On 14 February

2016 a Google self-driving car attempted to avoid sandbags blocking its path. During the

maneuver it struck a bus. Google addressed the crash, saying “In this case, we clearly bear

some responsibility, because if our car hadn’t moved there wouldn’t have been a collision

Google characterized the crash as a misunderstanding and learning experience

Tesla Autopilot

In mid-October 2015 Tesla Motors rolled out version 7 of their software in the U.S.

that included Tesla Autopilot capability. On 9 January 2016, Tesla rolled out version 7.1

as an over-the-air update, adding a new "summon" feature that allows cars to self-park at

parking locations without the driver in the car.

Autopilot should be used only on limited-access highways, and sometimes it will

fail to detect lane markings and disengage itself. In urban driving the system will not read

traffic signals or obey stop signs. The system also does not detect pedestrians or cyclists.

According to Tesla, starting 19 October 2016, all Tesla cars are built with hardware

to allow full self-driving capability at the highest safety level’ The hardware includes eight

surround cameras and twelve ultrasonic sensors, in addition to the forward-facing radar

with enhanced processing capabilities. The system will operate in "shadow mode"

(processing without acting) and send data back to Tesla to improve its abilities until the

software is ready for deployment via over-the-air upgrades.

13

13

2.4 ROAD DETECTION

Road detection is a fundamental issue in field navigation systems, which have

attracted keen attention in the past several decades. Road detection and tracking has been

subject of discussion since the idea of IVs came into play.

It is started with basic features like edge detection and lane detection and then grown

into a complex composite and dynamic approach for road and object detection. Road

detection and tracking are important for many intelligent vehicle applications, such as

LDW systems, Anti-sleep systems, driver assistance and safety warning system,

autonomous driving. In Road detection usually following approaches are used

1. Simple sensor based approach

2. GPS or navigation based approach

3. Vision sensor based approach

• Simple sensor based approach

Sensor based approach is the most basic and conventional approach used

for road detection or lane detection in sensor based approach multiple sensor like

proximity sensors, ultrasonic sensors color sensors etc. are used to acquire basic

knowledge of the road. Simple sensor’s based approach is dependent approach as

it provides only static picture/perspective of the road.

• GPS based approach

Global Positioning System (GPS) is a worldwide radio-navigation system

formed from the constellation of 24 satellites and their ground stations. The Global

Positioning System is mainly funded and controlled by the U.S Department of

Defense (DOD). The system was initially designed for the operation of U. S.

military. But today, there are also many civil users of GPS across the whole world.

Self-driving cars rely on optical (also radar) sensors and 3D maps to

understand precisely where they are and where the hazards are. It is believed LDW

is no longer performing perfectly. In rainy or mildly snowy weather where optical

systems degrade, or the lane markings are partially obscured, the combination

14

14

might help the car maintain lane-centering a bit longer. Eventually it would shut

down.

• Vision sensor based approach

Vision sensors play an important role in road detection for their great

potential in environmental perception. Vision sensors play an important role in road

detection. Image data captured by vision sensors contains rich information, such as

luminance, color, texture, etc. Thus, vision sensors have a great potentiality in road

detection. Moreover, vision sensors are inexpensive compared to other popular road

detection sensors, such as LIDAR and millimeter-wave radar.

For these reasons, many state-of-the-art field robot systems and intelligent

systems employ vision sensors for road detection. For example, Xu et alp resented

a mobile robot using a vision system to navigate in an unstructured environment.

The vision system consisted of two cameras; one is used for road region detection,

and the other is used for road direction estimation. Rasmussen introduced a vehicle-

based mobile robot system, which has achieved success in the DARPA Grand

Challenge. Vision sensors mounted on the top of the windshield were used to detect

the road vanishing point for steering control. Vision sensor-based road detection is

a binary labeling problem trying to label every pixel in the given road image with

the category (road or background) to which it belongs. However, vision sensor-

based road detection is still a challenging job due to the diversity of road scenes

with different geometric characteristics (varying colors and textures) and imaging

conditions (different illuminations, viewpoints and weather conditions).

The main advantages of vision-based approach are as follows:

• Vision sensors acquire data in a non-invasive way, thus not polluting the road

environment. In other words, vision sensors do not interfere with each other when

multiple intelligent vehicle are moving within the same area. By contrast, besides

the problem of environment pollution, we must carefully think about some typical

problems of active sensors, such as the wide variation in reflection ratios caused by

15

15

different reasons (such as obstacles shape of material), the need for the maximum

signal level to comply with some safety rules, and the interference among active

sensors of the same type.

• In most of ITS and IV applications, vision sensors play a fundamental role, for

example, in lane marking localization, traffic sign recognition, obstacle

recognition. Among those applications, other sensors, such as laser and radar, are

only complementary to vision sensors.

• We do not need to modify road infrastructures when using vision sensors to capture

visual information. This is extremely important in practice applications

2.5 OBJECT DETECTION AND RECOGNITION

Object detection is a computer technology related to computer vision and image

processing that deals with detecting instances of real world objects such as faces, humans,

cars and building in images and videos. Object detection and recognition are used to locate,

identify, and categorize objects in images and video.

• Electromagnetic sensor

These sensors can be used in vehicles moving slowly and smoothly towards the

obstacles. If the vehicle stops immediately upon the detection of obstacle, the

sensors continue to emit a signal of obstacle's present. If the vehicle resumes its

movement, the alarm signal becomes more prominent as the obstacle approaches.

These sensors are commonly used as they do not require drilling of holes in the

vehicle and they can be discreetly mounted on the inner side of the bumper.

• Wireless Ultrasonic Sensors

These sensors are highly sophisticated devices that create sharp radio signals for

detecting the obstacles. They use the echo time of the radio signals bouncing from

the obstacles to indicate the distance of the obstacle. Wireless ultrasonic sensors

range from four to eight, placed equally in the front and rear parts of a vehicle.

They detect objects even when the car is stationary.

16

16

• Nissan’s moving object detection

Cameras detect moving objects around the vehicle when it is in park or slowly

maneuvering; the system then alerts the driver both visually and audibly. There are

two types of systems: one uses the Around View Monitor and four cameras to the

front, back and sides of the car, while the second system uses only a single camera

installed in the rear of the car. The four-camera system can alert drivers in three

scenarios: while parked or in neutral; moving forward; and backing up. When

moving forward or backing up, the cameras to the front or back respectively detect

certain moving objects. When in park or neutral, the system detects certain moving

objects around the car using a virtual bird’s-eye view image. If a vehicle has the

single rear-view camera system it can only detect certain moving objects behind

the vehicle. The system processes video imagery from the cameras and can then

detect certain moving objects.

• Object detection by pattern recognition

An object detection system includes a sensor in communication with a controller

to identify an object within a field of view. Pattern recognition algorithms are

incorporated into the controller, and objects are predefined to minimize false

detection and sift predefined objects such as vehicles from background clutter.

Upon recognition of the object, an indicator in communication with the controller

provides an alert to the operator who can then take corrective action. By defining

another field of view the controller halts or reverses the movement of a power

window to prevent trapping an object between the closing window and the window

frame. Yet another field of view includes a vehicle entry point such as a vehicle

door. Movement of the vehicle door will be identified by the controller and will

provide an alert such as activation of the vehicle alarm system.

• Reverse Car Parking Sensors

These sensors are commonly used during parking the car in a reverse car parking

system. They get activated as soon as the car is put in reverse gear, and are usually

placed in the front side of a vehicle. Reverse car parking sensors are small and

17

17

generate ultrasonic radio waves to send and receive signals reflected from the

obstacle. Mazda have already integrated this technology into their vehicles.

• Google’s self-driving cars detect and avoid obstacles

Google’s driverless car tech uses an array of detection technologies including sonar

devices, stereo cameras, lasers, and radar. All these components have different

ranges and fields of view, but each serves a purpose according to the patent filings

Google has made on its driverless cars. Anyone who has ever seen an image of

Google’s self-driving Prius has probably noticed one of these systems poking up

above the vehicle — the LIDAR laser remote sensing system. According to Google

engineers, this is at the heart of object detection.

2.6 COLLISION DETECTION

Collision detection is a key factor in enabling the integration of unmanned vehicles

into real life use. It is the ability of the vehicle to “see” other vehicles or pedestrians,

anticipate collisions, and automatically apply the brakes or take corrective steering actions.

It is also known as a pre-crash system, forward collision warning system, or collision

mitigating system. It uses radar (all-weather) and sometimes laser (LIDAR) and camera

(employing image recognition) to detect an imminent crash and ultrasonic sensors for

prototypes.

Early warning systems have been attempted as early as the late 1950s. Cadillac for

instance, developed a prototype vehicle named the Cadillac Cyclone which used the new

technology of radar to detect objects in the front of the car with the radar sensors. It was

deemed too costly and the model was subsequently dropped.

The first modern demonstration of forward collision avoidance was performed in

1995 by a team of scientists and engineers at Hughes Research Laboratories in Malibu,

California. The project was funded by Delco Electronics. The technology was labeled for

marketing purposes as "Forewarn". The system was radar based.

In the early-2000s, the U.S NHTSA researched whether to make frontal collision

warning systems and lane departure warning systems mandatory. A 2009 study conducted

18

18

by the IIHS found a 7 percent reduction in crashes for vehicles with a basic forward-

collision warning system, and a 14 to 15 percent reduction for those with automatic

braking.

The federal National Highway Traffic Safety Administration is also on board, with

an eye to making some collision-avoidance systems mandatory, but the cost of collision-

avoidance systems can still be an obstacle.

The idea of incorporating radar systems into vehicles to improve road traffic safety

dates to the 1950s. Such systems are now reaching the market as recent advances in

technology have allowed the signal processing requirements and the high angular

resolution requirements from physically small antennas to be realized. Automotive radar

systems have the potential for several different applications including ACC and anti-

collision devices. The problem with this brand of cars is that they are expensive.

2.7 HUMAN ERRORS

According to the US Department of Transportation's National Motor Vehicle Crash

Causation Survey, 94 percent of road accidents are caused by human error, and it is said

that machine intelligence technology will drastically lower.

Experts explains that this improvement in road safety with driverless cars is simply

down to there being many tasks that robots, machines or driverless cars can do much better

than people. Experts claim that once the Autonomous vehicles become a reality, we can

expect increased roadway capacity and reduced traffic congestion due to the reduced need

for safety gaps and being able to better manage traffic flow.

Intelligent vehicles open new possibilities such as allowing those who are not legally

eligible to drive - convicted drink drivers, younger people, older people or those with

disabilities - to be mobile, it should be noted that one of the major barriers to driverless

technology is legislation.

By the end of this century, there’s good reason to believe that tens of millions of

traffic fatalities will be prevented around the world. This is not merely theoretical. There’s

19

19

already some precedent for change of this magnitude in the realms of car culture and

automotive safety. In 1970, about 60,000 people died in traffic accidents in the United

States. A dramatic shift toward safety including required seat belts and ubiquitous

airbags—helped vastly improve a person’s chance of surviving the American roadways in

the decades that followed. By 2013, 32,719 people died in traffic crashes, a historic low.

Researchers estimate that driverless cars could, by midcentury, reduce traffic

fatalities by up to 90 percent. Which means that, using the number of fatalities in 2013 as

a baseline, intelligent cars could save 29,447 lives a year.

2.8 SUMMARY

Computer vision tasks include methods for acquiring, processing, analyzing and

understanding digital images, and extraction of high-dimensional data from the real world

to produce numerical or symbolic information e.g. in the forms of decisions. Upcoming

self-driving cars are also using the same computer vision technology. Nowadays computer

vision is used in many application areas such as OCR, Face detection, smile detection,

object recognition, vision based biometrics, face recognition, object recognition, motion

capture, Panorama stitching,3D terrain modeling, Obstacle detection, position tracking,

vision guided robots, 3D imaging MRI and image guided surgery.

20

20

CHAPTER 3

PROJECT METHODOLOGIES

3.1 SYSTEM DESIGN

In this chapter we will look at the architecture, modules, interfaces, and data for the

intelligent system to satisfy specified requirements. Systems design could be the

application of systems theory to product development.

3.1.1 MODULES

In modules we will discuss each of a set of standardized parts or independent units that

can be used to construct a more complex structure. Modules of Intelligent vehicles are:

3.1.1.1 Raspberry PI

Raspberry pi is a mini computer that plug into HDMI monitor and keyboard.

It is a capable little computer which can be used in electronics projects. The

Raspberry Pi is a fully functional Linux computer. A Raspberry Pi board (model

B+), attached with a pi camera module and an HC-SR04 ultrasonic sensor is used

to collect input data. Two client programs run on Raspberry Pi for streaming color

video and/ ultrasonic sensor data to the laptop via local Wi-Fi connection.

Fig: 3.1.1.1(a) Raspberry PI module

• Raspberry PI and Camera module connections

Open Raspberry PI Camera module.

21

21

Install the Raspberry PI Camera module by inserting the cable into the Raspberry

Pi. The cable slots into the connector situated between the Ethernet and HDMI

ports, with the silver connectors facing the HDMI port.

Fig: 3.1.1.1(b) Camera port in Raspberry PI

Now boot up Raspberry PI

From the prompt, run "sudo raspi-config". If the "camera" option is not listed, it

will need to run a few commands to update your Raspberry Pi. Run "sudo apt-get

update" and "sudo apt-get upgrade".

Now Run "sudo raspi-config" again - you should now see the "camera" option.

Fig: 3.1.1.1(c) Raspberry PI command prompt

• Raspberry PI and Ultrasonic Sensor Connections

22

22

The sensor output signal (ECHO) on the HC-SR04 is rated at 5V.

However, the input pin on the Raspberry Pi GPIO is rated at 3.3V. Sending a 5V

signal into 3.3V input port could damage GPIO pins.

A voltage divider consists of two resistors (R1 and R2) in series connected

to an input voltage (Vin), which needs to be reduced to our output voltage (Vout).

In our circuit, Vin will be ECHO, which needs to be decreased from 5V to our

Vout of 3.3V.

Fig: 3.1.1.1(d) Voltage Divider

Four pins on the Raspberry Pi are used for this connection:

GPIO 5V [Pin 2]; VCC (5V Power),

GPIO GND [Pin 6]; GND (0V Ground),

GPIO 23 [Pin 16]; TRIG (GPIO Output)

GPIO 24 [Pin 18]; ECHO (GPIO Input)

23

23

Fig: 3.1.1.1(e) Raspberry-PI voltage divider

Plug four of male to female jumper wires into the pins on the HC-SR04 as follows:

Red; Vcc, Blue; TRIG, Yellow; ECHO and Black; GND.

Fig: 3.1.1.1(f) HC-SR04 pins

Plug Vcc of both Raspberry Pi and sensor into the positive rail of breadboard,

similarly plug GND into negative rail.

Plug GPIO 5V [Pin 2] into the positive rail, and GPIO GND [Pin 6] into the

negative rail.

24

24

Fig: 3.1.1.1(g) Raspberry-PI and HC-SR04 VCC and GND connections

Plug TRIG of Sensor into a blank rail, and plug that rail into Raspberry PI GPIO

23 [Pin 16].

Fig: 3.1.1.1(h) Raspberry PI and HC-SR04 ECHO pin connection

Plug ECHO into a blank rail, link another blank rail using R1 (1kΩ resistor)

Link the rail with R1 to the GND rail using R2 (2kΩ resistor). Leave a space

between the two resistors.

25

25

Fig: 3.1.1.1(i) Connections on breadboard

Add GPIO 24 [Pin 18] to the rail with R1 (1kΩ resistor). This GPIO pin needs to

sit between R1 and R2

Fig: 3.1.1.1(j) Voltage Divider on breadboard

HC-SR04 is now connected with Raspberry PI

Fig: 3.1.1.1(k) Raspberry PI and HC-SR04 connections

26

26

3.1.1.2 PI-cam

The Raspberry Pi Camera Module v2 (PI noir) is used in this project. The

v2 Camera Module has a Sony IMX219 8-megapixel sensor. It is used to take high-

definition video. For initializing pi-cam, there certain steps:

• Locate the camera port in Raspberry PI as discussed above and connect the camera

• Start up the Pi

• Open the Raspberry Pi Configuration Tool from the main menu

• Ensure the camera software is enabled. If it's not enabled, enable it and reboot your

Pi to begin.

Fig: 3.1.1.2(a) PI-cam module

3.1.1.3 Arduino

Arduino is some open-source electronics prototyping platform based on

flexible, easy-to-use hardware and software. The RC car used in this project has an

on/off switch type controller. The processing unit (computer) handles multiple

tasks: receiving data from Raspberry Pi, neural network training and prediction

(steering), object detection (stop sign and traffic light), distance measurement

(monocular vision), and sending instructions to Arduino through USB connection.

When a button is pressed, the resistance between the relevant chip pin and ground

is zero. Thus, an Arduino board is used to simulate button-press actions. Four

Arduino pins are chosen to connect four chip pins on the controller, corresponding

to forward, reverse, left and right actions respectively. Arduino pins sending LOW

signal indicates grounding the chip pins of the controller; on the other hand,

sending HIGH signal indicates the resistance between chip pins and ground remain

unchanged. The Arduino is connected to the computer via USB. The computer

27

27

outputs commands to Arduino using serial interface, and then the Arduino reads

the commands and writes out LOW or HIGH signals, simulating button-press

actions to drive the RC car.

Fig: 3.1.1.3(a) Arduino module

3.1.1.4 Ultrasonic sensor

Ultrasonic sensor emits short, high-frequency sound pulses at regular

intervals. These propagate in the air at the velocity of sound. If they strike an object,

then they are reflected as echo signals to the sensor, which itself computes the

distance to the target based on the time-span between emitting the signal and

receiving the echo.

This project uses HC-SR04 ultrasonic sensor to determine distance to an

object like bats do. It offers excellent non-contact range detection with high

accuracy and stable readings in an easy-to-use package. From 2cm to 400 cm or 1”

to 13 feet. It operation is not affected by sunlight or black material like Sharp

rangefinders are (although acoustically soft materials like cloth can be difficult to

detect). It comes complete with ultrasonic transmitter and receiver module.

28

28

Fig: 3.1.1.4(a) Ultrasonic sensor- HC-SR04

Features:

• Power Supply: +5V DC

• Quiescent Current: <2mA

• Working Current: 15mA

• Effectual Angle: <15°

• Ranging Distance: 2cm – 400 cm/1″ – 13ft

• Resolution: 0.3 cm

• Measuring Angle: 30 degrees

• Trigger Input Pulse width: 10uS

• Dimension: 45mm x 20mm x 15mm

Pins:

• VCC: +5VDC

• Trig : Trigger (INPUT)

• Echo: Echo (OUTPUT)

• GND: GND

Working in our project:

HC-SR04 ultrasonic sensor is placed on front of RC car and it is connected

to GPIO pins of Raspberry PI which is also mounted on top of RC car, ultra-sonic

sensor collects data (distance) from an obstacle if it comes in front of RC car and

sends it to Raspberry PI.

29

29

Fig: 3.1.1.4(b) Components attached on the RC-CAR

3.1.2 INTERFACES

An interface is a shared boundary across which two or more separate components

of a computer system exchange information. The exchange can be between software,

computer hardware, peripheral devices, humans and combinations of these

In this project we are using two types of interfaces

• Socket interface using TCP

• Serial interface

3.1.2.1 Socket interface using TCP

A socket represents a single connection between two network applications.

Sockets are bidirectional, meaning that either side of the connection is capable of

both sending and receiving data.

In this project we are using TCP Sockets (or virtual ports) for

communication between Raspberry PI and Laptop. Each side of a socket connection

uses its own port number, which does not change during the life of that connection.

The port number and IP address together uniquely identify an endpoint. Together,

two endpoints are considered a 'socket'.

30

30

Client-host pairing:

All TCP communication has a source and destination, so there is always a

source port and a destination port used for every socket connection.

Each computer host uses a unique IP address, and uses a unique source and

destination port pairings to identify that specific connection between the two

computers.

We have used, multithread TCP server program runs on the computer to

receive streamed image frames and ultrasonic data from the Raspberry Pi. Image

frames are converted to gray scale and are decoded into NumPY arrays.

3.1.2.2 Serial interface

After receiving the data from Raspberry Pi, the computer will process all

the data and sends commands to Arduino through serial interface i.e. RS-232, which

further sending the commands/instructions to the control unit (remote controller)

so that motors connected to vehicle can be controlled.

Fig: 3.1.2.2(a) Serial interface between Arduino and the remote controller

31

31

3.1.3 DATA

3.1.3.1 Video Frames

Video stream is taken by pi cam mounted on raspberry pi, raspberry

pi sends the video stream to PC by TCP server at the rate of 10 frames per

second. Algorithms of NN that run on Laptop decide where to move and

then sends the left, right and forward commands to Arduino, connected with

the controller of RC Vehicle.

3.1.3.2 Ultrasonic sensor data

Ultrasonic sensor is only used to determine the distance to an

obstacle in front of the RC car and provides accurate results when taking

proper sensing angle and surface condition into considerations. In fact, if

we know the corresponding number to the actual distance, we know when

to stop the RC car.

3.2 METHODOLOGY

Methodology is the system of methods employed do research activity or product/project

realization

In our case it is divided in two types

1) System hardware

2) System software

3.2.1 SYSTEM HARDWARE

Input unit:

A Raspberry Pi board (model B+), attached with a pi camera module and

an HC-SR04 ultrasonic sensor is used to collect input data. Two client programs

run on Raspberry Pi for streaming color video and ultrasonic sensor data to the

computer via local Wi-Fi connection.

32

32

Processing Unit:

The processing unit (computer) handles multiple tasks: receiving data from

Raspberry Pi, neural network training and prediction(steering), object detection

(stop sign and traffic light), distance measurement (monocular vision), and sending

instructions to Arduino through USB connection.

RC car control unit:

The RC car used in this project has an on/off switch type controller. When

a button is pressed, the resistance between the relevant chip pin and ground is zero.

Four Arduino pins are chosen to connect four chip pins on the controller,

corresponding to forward, reverse, left and right actions respectively. Arduino pins

sending LOW signal indicates grounding the chip pins of the controller; on the other

hand, sending HIGH signal indicates the resistance between chip pins and ground

remain unchanged. The Arduino is connected to the computer via USB. The

computer outputs commands to Arduino using serial interface, and then the

Arduino reads the commands and writes out LOW or HIGH signals, simulating

button-press actions to drive the RC car.

NOTE: An Optical-Isolator is a component that transfers electrical signals between

two isolated circuits by using light. It prevents high voltages from affecting the

system receiving the signal. This IC is being used between Arduino and controller

Fig: 3.2.1(a) Opto-Isolator, interfacing Arduino and Remote controller

33

33

Fig: 3.2.1(b) Complete anatomy of the project

3.2.2 SYSTEM SOFTWARE

Dependencies

Programming languages: Python, C

IDE: Python 2.4, PyCharm, Arduino

• PI-camera: a library that contains framework for PI-camera

• PI-Serial: PI-Serial is a library which provides support for serial connections

("RS-232") over a variety of different devices.

• PY-game: PY-game is a cross-platform set of Python modules designed for

writing video games. It includes computer graphics and sound libraries

designed to be used with the Python programming language.

• OpenCV-Python: OpenCV-Python is a library of Python bindings designed to

solve computer vision problems. Python is a general-purpose programming

language started by Guido van Rossum that became very popular very quickly,

mainly because of its simplicity and code readability. Our project has used

Open-CV 2.4.10

34

34

• NumPY: NumPY is a library for the Python programming language, adding

support for large, multi-dimensional arrays and matrices, along with a large

collection of high-level mathematical functions to operate on these arrays.

3.3 ALGORITHMS

3.3.1 OBSTACLE DETECTION

To achieve this goal, the ultrasonic sensor is attached in front of the car. Whenever

the car is going on the desired path the ultrasonic sensor transmits the ultrasonic waves

continuously from its sensor head. Whenever an obstacle comes ahead of it the ultrasonic

waves are reflected from an object and that information is passed to the raspberry pi. The

raspberry pi sends all the data collected from ultrasonic sensor to the laptop (main

controller) which takes the decision according to the received data, and give commands to

the Arduino, which is interfaced with the controller of the car which in results control the

vehicle.

3.3.2 STOP SIGN DETECTION AND RECOGNITION

One of the goal of this project is to detect stop signs in video streams captured by

a front-facing camera mounted on a moving car. Given a video stream captured by a car

mounted, front-facing camera, recognize the STOP sign that appear in the images and track

them over a sequence of frames. The approach we have used is HAAR cascade classifier.

Haar feature-based Cascade Classifier initially proposed by Viola and Jones classifies

objects using a series of edge, line, and center-surround features that scan throughout the

image to construct ROI features. It uses the same method of negative and positive images

of stop signs as we did in traffic light detection. Once we have created a trained xml file

with the classifier, we have used those trained parameters for further detection of stop sign.

35

35

Fig: 3.3.2(a) Stop sign detection and distance measurement

Haar-Cascade classifier:

• Positive images

The very first step into creating an object detector with a machine learning

algorithm is the data collection of positive images, hereby referred to as positives.

The positives are the images which represent Stop sign. The positives were

photographed from the real world in form of high definition images of Stop sign.

• Negative images

The negatives, are images that not representing the object that is desired to

be detected. It is a method for the algorithm to distinguish between the desired

objects and anything else in the environment that shall not be detected. Those

negatives could be collected from the same environment the object will appear in.

• Training of the algorithm

The training of the Haar Cascade Classifier was performed on the collected

positive and negative images as mentioned bove. The first phase of the training was

to identify the vectors of the positive images. In this phase, the area where the

objects are in each positive image is retrieved by cropping the traffic light from the

rest of the environment. The second phase into the training was to provide the

negative images in a separate location from the positive images. The third phase

was to provide different parameters for the Haar Classifier to perform the training.

36

36

After the training was finished the opencv_traincascade provided

Classifiers for each stage performed during the training, those in turn were provided

to another tool from OpenCV to generate a XML file with all the necessary data

retrieved from the training. The training for this research took 3 weeks to complete.

This is all done for the detection of traffic light.

• OpenCV

OpenCV was used for applying the algorithms in this project. OpenCV is

an open source library aimed for computer vision. The library is written in C and

C++ with bindings to other programming languages such as Python and Java. The

OpenCV library contains more than 500 functions which contributes to computer

vision, including camera calibration, stereo vision and robotics. OpenCV does also

features ML modules, which makes it a suitable choice for this project.

3.3.3 TRAFFIC LIGHT DETECTION AND RECOGNITION

Vehicle and Traffic Safety is a growingly important research topic among the

automotive industry and academia. Traffic lights are important in terms of traffic safety;

therefore, it is of importance to have a solution to detect them without having to spend time

to find their occurrence manually in a video analysis. This project uses Haar feature-based

cascade classifiers for traffic light detection.

• Positive images

The very first step into creating an object detector with a machine learning

algorithm is the data collection of positive images, hereby referred to as positives.

The positives are the images which represent traffic light. The positives were

photographed from the real world in form of high definition images of traffic lights.

To relate to ROI, the parameters would be the positives with visible traffic lights,

positive of traffic light from different visibility angles and positives with different

traffic light status (Red, Yellow, Green). These parameters are important to be taken

into consideration since they may affect the results.

3.3.3.1 Brightest spot

37

37

The purpose of this approach is to identify which state the detected traffic

light is in, the states refer to red light, yellow light and green light. It is done in two

steps: the first step is to find the brightest point in the ROI

3.3.3.2 Position of the brightest spot.

After finding the brightest spot in detected traffic light, red or green states

are determined simply based on the position of the brightest spot in the ROI.

Fig: 3.3.3.2(a) Traffic green light detection

Fig: 3.3.3.2(b) Traffic Red light detection

38

38

Fig: 3.3.3.2(c) Flow chart of Haar cascade training

3.4 SELF-LEARNING USING NEURAL NETWORK

Neural networks can be used to implement highly nonlinear controllers with

weights or internal parameters that can be determined by a self- learning process.

A neural network is a class of computing system. They are created from very simple

processing nodes formed into a network. They are inspired by the way that biological

systems such as the brain work. They are fundamentally pattern recognition systems and

tend to be more useful for tasks which can be described in terms of pattern recognition.

They are 'trained' by feeding them with datasets (images, raw data) with known outputs.

39

39

• Back-propagation

Back-Propagation is one of the several ways in which an ANN can be

trained. It is a supervised training scheme, which means, it learns from labeled

training data (there is a supervisor, to guide its learning). To put in simple terms,

Back-Prop is like "learning from mistakes". The supervisor corrects the ANN

whenever it makes mistakes.

An ANN consists of nodes in different layers; input layer, intermediate

hidden layer(s) and the output layer. The connections between nodes of adjacent

layers have "weights" associated with them. The goal of learning is to assign correct

weights for these edges. Given an input vector, these weights determine what the

output vector is.

In supervised learning, the training set is labeled e.g. labeled images. This

means, for some given inputs, we know (label) the desired/expected output.

• Back-propagation algorithm

Initially all the edge weights are randomly assigned. For every input in the

training dataset, the ANN is activated and its output is observed. This output is

compared with the desired output that we already know, and the error is

"propagated" back to the previous layer. This error is noted and the weights are

"adjusted" accordingly. This process is repeated until the output error is below a

predetermined threshold.

Once the above algorithm terminates, we have a "learned" ANN which, we

consider is ready to work with "new" inputs. This ANN is said to have learned from

several examples (labeled data) and from its mistakes (error propagation)

40

40

Fig: 3.4(a) Flow chart of backpropagation

• Prediction and steering

In this project we are using this back-propagation methodology for

prediction, which leads to steering mechanism. A set of progressive frames are

captured and converted to an NumPY array. This array is then paired with a label

which is basically the human input and all these images are then saved into a

lightweight database. The neural network is trained in Open CV using the back-

propagation method. Once the training is completed the weights for each input node

(labels for the captured images previously) is stored into an XML file. To see the

predictions work, the neural network is loaded with this XML file.

41

41

CHAPTER 4

RESULTS AND DISCUSSIONS

4.1 SYSTEM BASED DIVISION AND RESULTS

We are using two systems: Laptop and the Raspberry PI. In Laptop we are using 2

IDE’s: PyCharm IDE (Comprised of NN-algorithms) and Arduino IDE (Codes for Sending

commands from Laptop to RC-Controller). In raspberry PI we are running (codes for video

stream and Sensor data Collection and transmission) in Python 2.3

Fig: 4.1(a) PyCharm IDE

Fig:4.1(b) Arduino IDE

42

42

Fig: 4.1(c) Python Shell of Raspberry PI

• Codes to be executed in Raspberry PI/ (built in python 2.3 environment)

o stream_client.py: stream video frames in jpeg format to the host

computer

o ultrasonic_client.py: send distance data measured by sensor to the host

computer

• Codes to be executed in Arduino/:(in Arduino IDE)

o rc_keyboard_control.ino: acts as an interface between RC-Controller and

computer and allows user to send command via USB serial interface

• Codes to be executed in computer/ (PyCharm IDE)

o cascade_xml/

trained cascade classifiers xml files

o chess_board/

images for calibration, captured by pi camera

o training_data/

training data for neural network in npz format

o training_images/

saved video frames during image training data collection stage

(optional)

o mlp_xml/

trained neural network parameters in a xml file

43

43

o picam_calibration.py: pi camera calibration, returns camera

matrix(optional)

o collect_training_data.py: receive streamed video frames and label frames

for later training

o mlp_training.py: neural network training

o rc_driver.py: a multithread server program receives video frames and

sensor data, and allows RC car drives by itself with stop sign, traffic light

detection and front collision avoidance capabilities

• For testing purposes

o rc_control_test.py: RC car control with keyboard

o stream_server_test.py: video streaming from Pi to computer

o ultrasonic_server_test.py: sensor data streaming from Pi to computer

Step by step process:

1. Flash Arduino: Flash “rc_keyboard_control.ino” to Arduino and

run “rc_control_test.py” to drive the rc car with keyboard (testing purpose)

2. Pi Camera calibration: Take multiple chess board images using pi camera at various

angles and put them into “chess_board” folder, run “picam_calibration.py” and it returns

the camera matrix, those parameters will be used in “rc_driver.py” (optonal)

3. Collect training data and testing data: First run “collect_training_data.py” and then

run “stream_client.py” on raspberry pi. User presses keyboard to drive the RC car,

frames are saved only when there is a key press action. When finished driving, press “q”

to exit, data is saved as a npz file.

4. Neural network training: Run “mlp_training.py”, depend on the parameters chosen, it

will take some time to train. After training, model will be saved in “mlp_xml” folder

5. Cascade classifiers training (optional): trained stop sign and traffic light classifiers are

included in the "cascade_xml" folder

6. Self-driving in action: First run “rc_driver.py” to start the server on the computer and

then run “stream_client.py” and “ul trasonic_client.py” on raspberry pi.

44

44

• RESULTS

1. Video stream is taken by PI-cam, which is connected to Raspberry PI is

displayed on the Laptop screen. Raspberry Pi has sent this video streaming to

Laptop through TCP server, using Client-Host pairing.

Fig: 4.1(d) Video Streaming received on laptop

Fig: 4.1(e) Console screen showing successful connection

45

45

2. Detection and recognition of green traffic light and necessary commands of

keeping the vehicle in forward direction is appeared on the Console Screen.

Fig: 4.1(f) Green light detection results

Fig: 4.1(g) Forward command of green light on Console screen

46

46

3. Detection and Recognition of Red traffic Light, with results on the console

screen.

Fig: 4.1(h) Red traffic light detection results

Fig: 4.1(i) Recognition of red light on console

47

47

4. A car as an obstacle came into the way of our trained Vehicle. Vehicle has

detected the Obstacle ahead with the distance from the obstacle is also shown

on the Console Screen.

Fig: 4.1(j) Obstacle car is ahead

Fig: 4.1(k) Obstacle is detected on console screen

48

48

5. Stop sign is detected but it is still at a distance more than 25cm, so “Forward”

command is shown on the screen. Vehicle will keep on moving forward until

the distance become less than 25cm

Fig: 4.1(l) Stop sign is at the distance of 27.1cm

Fig: 4.1(m) Forward command on the console screen.

49

49

6. Stop sign is at the distance of less than 25cm, Vehicle stopped and started

calculating the stop the time. Results are shown on the Console screen.

Fig: 4.1(n) Stop sign detection

Fig: 4.1(o) Stop time calculation on the console screen

50

50

7. When the distance is less than 25cm then vehicle will stop and wait for 5

seconds, after 5 seconds it will move “Forward”. Results are shown below

Fig: 4.1(p) Stop sign is in less than 25cm

Fig: 4.1(q) Forward Command after 5 seconds

51

51

4.2 PROBLEMS ENCOUNTERED

This thesis also reports the typical problems encountered in utilizing machine

learning technology and developing a prototype intelligent vehicle. These problems are:

4.2.1 SOFTWARE PROBLEMS

• Installation Python library packages:

While working on Python, we needed to install some library packages from

the Python Package Index. Unfortunately, this is not as simple as it could be. Few

of the Python packages were easily installed by pip which is the preferred installer

program. It is included by default with the Python binary installers.

But some Python packages have complex binary dependencies, and could not be

easily installed by using pip directly. At this point in time, we often have preferred

to install these packages by other means rather than attempting to install them with

pip.

4.2.2 HARDWARE PROBLEMS

• Arduino- controller interfacing

Not all the RC-controllers are compatible to interface directly with Arduino.

Even many times, controllers don’t work properly. So, while interfacing we used

an external circuitry consist of opto-isolator to make the controller work properly.

• Speed

Built-in speed of RC-Vehicle was too fast to capture proper frames and act

on decisions taken by NN algorithms. So, reducing the speed of vehicle was another

challenge to complete.

• Power

RC-vehicle built-in power source was not enough to run the vehicle, so

external battery cells were used to provide enough power. We used LIPO battery

cells externally for that purpose.

52

52

4.3 LIMITATIONS

Adverse weather can be a source of major frustration for Human drivers, and the

significant factor in the number of accidents can occur.

Every technology has some limitations. The approach this project has used is based

on camera, which can’t deal with snow, rain. darkness and fog very well. Like humans,

camera also faces the similar issues of reduced visibility. However, this approach is cost-

effective but become untrustworthy in these situations. Another Limitation for this project

is a strong Wi-Fi connection, that makes the communication possible between Raspberry

PI and the Laptop.

4.4 CONCULSION

ITS have attracted increasing attention in recent years due to their great potential in

meeting driving challenges.

• Scope of the project

As technology improves, a vehicle will become just a computer with tires.

Driving on roads will be just like surfing the Web: there will be traffic congestion

but no injuries or fatalities. Advanced intelligent system and new sensing

technologies can be highly beneficial, along with large body of work on intelligent

vehicles

• Advantages

The main advantages of intelligent vehicle are: it allows driver and the other

people in vehicle to reach their destination quickly, safely and in a more relaxed

frame of mind. Above all on routine journeys, in traffic jams, on crowded

motorways with speed restrictions and at accident blackspots, an intelligent vehicle

can assist the driver and relieving them of tedious routine tasks. However, the

intention is not to deprive the driver of the experience and pleasure of doing the

driving for themselves. "Our intelligent system offer to assist and unburden the

driver.

53

53

4.5 FUTURE WORK

The reliable intelligent vehicles and safety warning systems is still a long

way to go. However, as computing power, sensing capacity, and wireless

connectivity for vehicles rapidly increase, the concept of intelligent driving is

speeding towards reality. These findings suggest that the research into intelligent

system within the ITS field is a short-term reality and a promising research area

and these results constitute the starting point for future developments. Some of the

suggestions towards extension and/or future related works are identified and are

summarized below:

• New sensory systems and sensory fusion is to be explored to plug additional

information to the control system.

• This work can be extended to include different maneuvers to make the driving

system capable of dealing with all driving environments.

• Future issues may also include an algorithm for autonomous formation of the

cooperative driving.

54

54

REFERENCES

[1] David A. Medler (1998). "A Brief History of Connectionism" (PDF). Neural

Computing Surveys. 1: 61–101.

[2] J. Weng, N. Ahuja and T. S. Huang, "Learning recognition and segmentation of 3-D

objects from 2-D images," Proc. 4th International Conf. Computer Vision, Berlin,

Germany, pp. 121-128, May, 1993.

[3] Sven Behnke (2003). Hierarchical Neural Networks for Image Interpretation. (PDF).

Lecture Notes in Computer Science. 2766. Springer.

[4] Gosman, Tim (2016-07-24). "Along for the ride: How driverless cars can become

commonplace". Brand Union. Retrieved 2016-10-29.

[5] "Preparing a nation for autonomous vehicles: Opportunities, barriers and policy

recommendations.". Transportation Research Part A: Policy and Practice. 7

[6] Scott Klement,” Tcp socket”

[7] InetDaemon,” tcp socket and client-server pairing”,April 2013

[8] P. Viola and M. Jones, "Rapid object detection using a boosted cascade of simple

features”, Proc. Computer Vision and Pattern Recognition, 2001.

[9] Luke Graham,” What is a Neural Network in simple words”

[10] “Neural Networks for Visual Recognition”; CS231n.stanford.edu

[11] Nikhil Buduma,” Inside Deep Learning: Computer Vision with Convolutional Neural

Networks”, (MIT).

[12] Hemanth Kumar Mantri, “How-do-you-explain-back-propagation-algorithm-to-a-

beginner-in-neural-network “

[13] Jim Torresen, Jorgen W. Bakke and Lukas Sekanina, “Efficient Recognition of Speed

Limit Signs”, 2004 IEEE Intelligent Transportation Systems Conference Washington,

D.C., USA, October 3-6, 2004.

55

55

[14] N. Dalal, B Triggs, “Histogram of Oriented Gradients for Human Detections”, Proc.

IEEE Computer vision and pattern recognition, 2005.

[15]. Introduction to OpenCV-Python Tutorials by Alexander Mordvintsev (GSoC-2013

mentor), Abid Rahman K. (GSoC-2013 intern).

[16] Sander Soo, “Object detection using Haar-cascade classifier”, Institute of Computer

Science, University of Tartu

[17] I. Paliy, “Face detection using Haar-like features cascade and convolutional neural

network,” in 2008 International Conference on “Modern Problems of Radio Engineering,

Telecommunications and Computer Science” (TCSET), 2008.

[18] Muhanad Nabeel and David ustarbowski “Evaluation of traffic light detection

algorithms for automated video analysis” Department of Computer Science and

Engineering Chalmers university of technology, Gothenburg, Sweden 2016.

[19] Meyer, G; Beiker, S (2014). Road vehicle automation. Springer International

Publishing. pp. 93–102.

[20] Borenstein, E., Ullman, S.: Class-specific, top-down segmentation. In: ECCV (2).

(2002)

[21] Levin, A., Weiss, Y.: Learning to combine bottom-up and top-down segmentation. In:

ECCV. (2006)

[22] Leibe, B., Seemann, E., Schiele, B.: Pedestrian detection in crowded scenes. In:

CVPR. (2005)

[23] Ferrari, V., Tuytelaars, T., Gool, L.J.V.: Object detection by contour segment

networks. In: ECCV. (2006)

[24] Kokkinos, I., Maragos, P., Yuille, A.L.: Bottom-up & top-down object detection using

primal sketch features and graphical models. In: CVPR. (2006)

[25] Zhao, L., Davis, L.S.: Closely coupled object detection and segmentation. In: ICCV.

(2005)

56

56

[26] Lee Bell (2015) Humans vs robots: “Driverless cars are safer than human driven

vehicles”

https://www.theinquirer.net/inquirer/feature/2426988/humans-vs-robots-driverless-cars-

are-safer-than-human-driven-vehicles

[27] Adrienne Lafrance (2015) Self-Driving Cars Could Save 300,000 Lives Per Decade

in America

https://www.theatlantic.com/technology/archive/2015/09/self-driving-cars-could-save-

300000-lives-per-decade-in-america/407956/

[28] Ren, X., Berg, A.C., Malik, J.: Recovering human body configurations using pairwise

constraints between parts. In: ICCV. (2005)

[29] Mori, G., Ren, X., Efros, A.A., Malik, J.: Recovering human body configurations:

Combining segmentation and recognition. In: CVPR. (2004)

[30] Srinivasan, P., Shi, J.: Bottom-up recognition and parsing of the human body. In:

CVPR. (2007)

[31] Felzenszwalb, P.F., Huttenlocher, D.P.: Pictorial structures for object recognition.

International Journal of Computer Vision 61(1) (2005)

[32] Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: CVPR.

(2005)

[33] OpenCV, “Cascade Classifier Training — OpenCV 2.4.9.0 documentation,” [Online].

Available: http://docs.opencv.org/doc/use r_guide/ug_traincascade.html.

[34] C. Robin, “Train your own OpenCV HAAR classifier,” 12 [Online]. Available:

http://codingrobin.de/2013/07/22/trainyour-own-opencv-haarclassifier.html.

[35] N. Seo, “OpenCV haartraining (Rapid Object Detection with A Cascade of Boosted

Classifiers Based on Haar-like Features),”

[Online]. Available: http://note.sonots.com/SciSoft ware/haartraining.html

https://www.theinquirer.net/inquirer/feature/2426988/humans-vs-robots-driverless-cars-are-safer-than-human-driven-vehicles

https://www.theinquirer.net/inquirer/feature/2426988/humans-vs-robots-driverless-cars-are-safer-than-human-driven-vehicles

https://www.theatlantic.com/technology/archive/2015/09/self-driving-cars-could-save-300000-lives-per-decade-in-america/407956/

https://www.theatlantic.com/technology/archive/2015/09/self-driving-cars-could-save-300000-lives-per-decade-in-america/407956/

http://docs.opencv.org/doc/use%20r_guide/ug_traincascade.html

http://codingrobin.de/2013/07/22/trainyour-own-opencv-haarclassifier.html

57

57

[36] S. Nagabhushana, “Introduction,” in Computer Vision and Image Processing, New

Age International (P) Ltd., Publishers, 2005, p. 3.

[37] T. M. Inc., “Train a Cascade Object Detector,” [Online]. Available:

http://www.mathworks.se/help/ vision/ug/train-a-cascadeobject-detector.html#btugex8

[39] “Effective traffic lights recognition method for real time driving assistance system in

the daytime.” http://www.waset.org/publications/725

[40] S. Tiwari, S. S. Rathore, and A. Gupta, “Selecting requirement elicitation techniques

for software projects,” in Software Engineering (CONSEG), 2012 CSI Sixth International

Conference on, pp. 1–10, Sept 2012.

[41] D. Zowghi and C. Coulin, Requirements Elicitation: A Survey of Techniques,

Approaches, and Tools, pp. 19–46. Berlin, Heidelberg: Springer Berlin Heidelberg, 2005.

[42] “Cascade classification — opencv 2.4.13.0 documentation.”

http://docs.opencv.org/2.4/modules/objdetect/doc/cascade_classification.html.

[43] “Colormaps in opencv — opencv 2.4.13.0 documentation.”

http://docs.opencv.org/2.4/modules/contrib/doc/facerec/colormaps.html.

[44] Chenyi Chen “Deep Learning for Self-Driving Cars”

[45] “Neural and evolutionarz computing”, D. Zaharia, West Universitz of Timisoara

[46] Daniel Morariu “Machine learning”, Lucian Blaga University, Sibiu

[47] “Artificial Intelligence”, University of Berekeley

[48] “Introduction to Artificial Intelligence”, Indian Institute of Technology

Kharagpur

[50] “Road-marking analysis for autonomous vehicle guidance”, Stefan Vacek

[51] Pawlicki, T.F., Lee, D.S., Hull, JJ., Srihari, S.N. (1988) Neural network models and

their application to handwritten digit recognition. Proceedings of IEEE International

Conference on Neural Networks, San Diego, CA.

http://www.waset.org/publications/725

http://docs.opencv.org/2.4/modules/objdetect/doc/cascade_classification.html

http://docs.opencv.org/2.4/modules/contrib/doc/facerec/colormaps.html

58

58

[52] Pomerleau, D.A., Gusciora, GL., Touretzky, D.S., and Kung, H.T. (1988) Neural

network simulation at Warp speed: How we got 17 million connections per second.

Proceedings of IEEE International Conference on Neural Networks, San Diego, CA.

[53] Dickmanns, E.D., Zapp, A (1986) A curvature-based scheme for improving road

vehicle guidance by computer vision. "Mobile Robots", SPIE-Proc, Vol. 727, Cambridge,

MA

[54] Elman, J.L, (1988) Finding structure in time. Technical report 8801. Center for

Research in Language, University of California, San Diego

[55] Jordan, M.I. (1988) Supervised learning and systems with excess degrees of freedom.

COINS Tech. Report 88-27, Computer and Information Science, University of

Massachusetts, Amherst MA.

[56] Waibel, A., Hanazawa, T., Hinton, G., Shikano, K., Lang, K. (1988) Phoneme

recognition: Neural Networks vs. Hidden Markov Models. Proceedings from Int. Conf. on

Acoustics, Speech and Signal Processing, New York, New York.

[57] P. Zador, S. Krawchuck and R. Voas Automotive Collision Avoidance System

(ACAS) Program/First Annual Report. NHTSA - National Highway Traffic Safety

Administration (http://www.nhtsa.dot.gov/) DOT HS 809 080, August 2000

[58] Y. Freund and R. E. Schapire. Experiments with a new boosting algorithm. In

Proceedings of International Conference on Machine Learning (ICML), pp. 148–156,

1996.

[59] A. Ben-Yaacov, M. Maltz and D. Shinaar , Effects of an in-vehicle collision avoidance

warning system on short- and long-term driving performance. In Hum Factors., pages 335

-342, 2002

[60] O. Mano, G. Stein, E. Dagan and A. Shashua. Forward Collision Warning with a

Single Camera., In IEEE Intelligent Vehicles Symposium (IV2004), June. 2004, Parma,

Italy.

59

59

[61] A. Ben-Yaacov, M. Maltz and D. Shinaar , Effects of an in-vehicle collision avoidance

warning system on short- and long-term driving performance. In Hum Factors., pages 335

-342, 2002

[62] G. Stein and O. Mano and A. Shashua Vision-based ACC with a Single Camera:

Bounds on Range and Range Rate Accuracy In IEEE Intelligent Vehicles Symposium

(IV2003), June 2003

[63] O. Mano, G. Stein, E. Dagan and A. Shashua. Forward Collision Warning with a

Single Camera., In IEEE Intelligent Vehicles Symposium (IV2004), June. 2004, Parma,

Italy.

[64] A. Mohan, C. Papageorgiou, and T. Poggio. Example-based object detection in images

by components. In IEEE Transactions on Pattern Analysis and Machine Intelligence

(PAMI), 23:349-361, April 2001.

[65] A. Johnson, J. Montgomery, and L. Matthies, “Vision guided landing of an

autonomous helicopter in hazardous terrain,” in IEEE Intl. Conf. on Robotics and

Automation (ICRA), 2005.

[66] S. Bosch, S. Lacroix, and F. Caballero, “Autonomous detection of safe landing areas

for an uav from monocular images,” in IEEE/RSJ Intl. Conf. on Intelligent Robots and

Systems (IROS), 2006.

[67] V. Desaraju, N. Michael, M. Humenberger, R. Brockers, S. Weiss, and L. Matthies,

“Vision-based landing site evaluation and trajectory generation toward rooftop landing,”

in Robotics: Science and Systems (RSS), 2014.

[68] S. Weiss, M. Achtelik, L. Kneip, D. Scaramuzza, and R. Siegwart, “Intuitive 3D maps

for MAV terrain exploration and obstacle avoidance,” Journal of Intelligent and Robotic

Systems, vol. 61, pp. 473–493, 2011.

[69] A. Wendel, M. Maurer, G. Graber, T. Pock, and H. Bischof, “Dense reconstruction

onthefly,” in Proc. IEEE Int. Conf. Computer Vision and Pattern Recognition, 2012.

60

60

[70] M. Pizzoli, C. Forster, and D. Scaramuzza, “REMODE: Probabilistic, Monocular

Dense Reconstruction in Real Time,” in IEEE Intl. Conf. on Robotics and Automation

(ICRA), 2014.

[71] M. Faessler, F. Fontana, C. Forster, E. Mueggler, M. Pizzoli, and D. Scaramuzza,

“Autonomous, vision-based flight and live dense 3D mapping with a quadrotor MAV,” J.

of Field Robotics, 2015.

[72] P. Fankhauser, M. Bloesch, C. Gehring, M. Hutter, and R. Siegwart, “Robot-centric

elevation mapping with uncertainty estimates,” in International Conference on Climbing

and Walking Robots (CLAWAR), 2014.

[73] D. Gallup, J.-M. Frahm, P. Mordohai, Q. Yang, and M. Pollefeys, “Real-time plane-

sweeping stereo with multiple sweeping directions,” in Proc. IEEE Int. Conf. Computer

Vision and Pattern Recognition, 2007.

[74] J. Stuehmer, S. Gumhold, and D. Cremers, “Real-time dense geometry from a

handheld camera,” in DAGM Symposium on Pattern Recognition, 2010, pp. 11–20.

[75] R. A. Newcombe, S. Lovegrove, and A. J. Davison, “DTAM: Dense tracking and

mapping in real-time,” in Intl. Conf. on Computer Vision (ICCV), 2011, pp. 2320–2327.

[77] C. Forster, M. Pizzoli, and D. Scaramuzza, “SVO: Fast Semi-Direct Monocular Visual

Odometry,” in IEEE Intl. Conf. on Robotics and Automation (ICRA), 2014.

[78] S. Saripalli, J. F. Montgomery, and G. S. Sukhatme, “Vision-based autonomous

landing of an unmanned aerial vehicle,” in IEEE Intl. Conf. on Robotics and Automation

(ICRA), 2002.

[79] P. J. Garcia-Pardo, G. S. Sukhatme, and J. F. Montgomery, “Towards vision-based

safe landing for an autonomous helicopter,” Journal of Robotics and Autonomous Systems,

vol. 38, 2002.

[80] C. Forster, M. Pizzoli, and D. Scaramuzza, “Appearance-based active, monocular,

dense depth estimation for micro aerial vehicles,” in Robotics: Science and Systems (RSS),

2014.

61

61

[81] S. Lynen, M. Achtelik, S. Weiss, M. Chli, and R. Siegwart, “A robust and modular

multi-sensor fusion approach applied to mav navigation,” in IEEE/RSJ Intl. Conf. on

Intelligent Robots and Systems (IROS), 2013.

[82] G. Vogiatzis and C. Hernandez, “Video-based, Real-Time Multi View ´ Stereo,”

Image and Vision Computing, vol. 29, no. 7, 2011.

[83] X. Bresson, S. Esedoglu, P. Vandergheynst, J.-P. Thiran, and S. Osher, “Fast global

minimization of the active contour/snake model,” Journal of Mathematical Imaging and

Vision, vol. 28, no. 2, 2007.

[84] A. Chambolle and T. Pock, “A first-order primal-dual algorithm for convex problems

with applications to imaging,” Journal of Mathematical Imaging and Vision, vol. 40, no.

1, 2011.

[85] D.B. Fogel. Blondie24: playing at the edge of AI. The Morgan Kaufmann Series in

EvolutionaryComputation. Morgan Kaufmann Publishers, 2002. ISBN 9781558607835.

URL http://books.google.ie/books?id=5vpuw8L0C AC

LIST OF ABBREVIATIONS...LIST OF ABBREVIATIONS IV Intelligent Vehicle AI Artificial Intelligence HI...

Documents

Transcript of LIST OF ABBREVIATIONS...LIST OF ABBREVIATIONS IV Intelligent Vehicle AI Artificial Intelligence HI...