1

Enhancement of Lane Departure Warning System in Short Term

Autonomous Warning using Inertial Sensors Assistance

Chan Wei Hsu

Subsection Manager Specialist, Automotive Research & Testing Center No. 6, Lugong S. 7th Rd., Lugang Town, Changhua Country, 50544, Taiwan (R.O.C)

886-4-7811222ext2351, 886-4-7811333, winsonhsu@artc.org.tw Yi Feng Su

Engineer, Automotive Research & Testing Center No. 6, Lugong S. 7th Rd., Lugang Town, Changhua Country, 50544, Taiwan (R.O.C)

886-4-7811222ext2328, 886-4-7811333, efo@artc.org.tw Kuang Jen Chang

Engineer, Automotive Research & Testing Center No. 6, Lugong S. 7th Rd., Lugang Town, Changhua Country, 50544, Taiwan (R.O.C)

886-4-7811222ext2315, 886-4-7811333, kjchang@artc.org.tw Chi Feng Hung

Engineer, Automotive Research & Testing Center No. 6, Lugong S. 7th Rd., Lugang Town, Changhua Country, 50544, Taiwan (R.O.C)

886-4-7811222ext2371, 886-4-7811333, 2737@artc.org.tw

ABSTRACT This paper devotes to design and integrate a hybrid and innovative LDWs scheme to increase its availability. The scheme presents a solution of discontinuous lane marking for markings detection using camera and sensor fusion, including image processing, dead-reckoning and vehicular signals. To extract lane markings from image catching, DSP perform the task after image calibration and coordinate transform. The vision is applied by a camera which plays the role of displaying preceding image and warning information with buzzer. When operating at failure mode, inertial signals apply and provide available track. In dead reckoning technology, hybrid sensors address a global and all weather assistant design based on odometry and inertial sensor. From inertial data and vehicular information, this paper provides an extended solution with learning parameters to do remote warning. The proposed system is carried out with theoretical application and hardware integration, and furthermore the result shows extended LDWs approach applicability.

Keywords: Lane Departure Warning, Inertial Sensor, Road Geometry Estimation.

2

1. INTRODUCTION Due to the increasing awareness of automobile safety and the progress in technology, the main international automobile manufactures focus on developing Intelligent Transportation System (ITS) and Advanced Safety Vehicle (ASV). One of these important achievements is vehicle safety integration system with image processing technology, and it has been proved to reduce accidents significantly. According to statistics data of National Highway Transportation Safety Administration (NHTSA) [1-2], the majority accidents are result from driver himself/herself who has low attention in driving. The inattentive driving may be caused by many factors, including drivers talking on cell phones (30% increased risk of danger), dial phone (3 times increased risk of danger), asleep (4 times increased risk of danger), pick up moving objects (nine times increased risk of danger) and other acts. To effectively reduce the inattentive driving caused the accident and ensure safety, many design and related products have been proposed, such as lane departure warning system (LDWs), forward collision warning system (FCWs) by image processing methods to determine the lanes and the departure possibility relative to lanes in front of camera capture information. When vehicle deviates under threshold value, system will remind driver through the warning light or buzzer. In LDWs, there have been many methods for lane markings recognition. Inverse perspective mapping (IPM) approach is adopted to generate the bird’s eye image of the road plane so as to remove the perspective effect and extract lane markings through some constraints of road geometry [3]. Hough transformation and road model is also applied to estimate the initial lane boundary and improve the availability of lane detection [4-5]. To aim on an independent self-positioning, modern delicate electronic devices may be the most feasible candidate to adopt. Since GPS is a good solution for long term navigation, however, the GPS availability cannot meet short term requirement. In most application, micro-electro-mechanical systems (MEMS) inertial sensors are adopted [6]. However, due to fabrication process, MEMS inertial sensors have large bias instability and noise. A vehicular unit (VU) could sense vehicle speed and heading by calculating odometry. Although lower accuracy inertial sensors might cause the integration error with time in speed and vehicle spatial motion, VU could provide continuous speed and heading with movement through controller area network (CAN). Both inertial sensors and VU will be mutual compensation. The proposed system uses a circuit board with embedded processor to operate lane marking identification, manipulate inertial signals. LDWs is a vision-based system which detects lane markings through a CMOS camera and DSP processing simultaneously. To enhance LDWs ability, this paper presents a new method to extend lane trajectory by extrapolation using available lane points. A calibrated method is developed and processed inertial data with suitable filter in lateral and longitudinal motion under bad image quality.

3

2. CONCEPT OF SYSTEM ARCHITECTURE The system concept for LDWs is designed with an integration of environment sensing through camera and self-positioning via inertial sensors, as shown in Fig. 1. The major input is a CMOS camera, which was installed between the front windscreen and rear-view mirror. Therefore, roadway image in front of host vehicle can be acquired via the CMOS camera and transmitted to DSP after image decoding from NTSC analog format to ITUR-656 digital format. DSP processor which has high computing performance will execute recognition and estimation algorithm to provide lane departure warning functions. In dynamic warning and information, MCU [dsPIC30F6014A] is chosen as the core controller to handle real time message and extrapolate. Time slots are used to process and measure the inertial sensors data through A/D conversion. As the needs for the system, the specification and requirement of dsPIC30F6014A are listed in the following Table 1. 2.1 LDWs Processor in Lane Recognition To get meanings of road geometry model, the deviation to lane boundary will be processed from world coordinate transformation. The relation is shown in Fig. 2 among world, camera and image coordinates. Since LDWs system has to locate lane markings in image coordinates, image information should be transformed into 3D space information through the inverse perspective mapping in order to obtain the lane positions in the real space. Parameters and their definitions in this

ECU DSP

AD

CAN

5V

CAN

DC-DC converter

Odometry

Gyro

Accelerometer

Battery

DecoderCamera MonitorGPIO

Buzzer

Figure 1. System architecture

Figure 2. World, camera and image coordinates.

4

Table 1: Requirements and specifications Criterion Required Desirable dsPIC30F6014A

On chip nonvolatile memory

Enough to eliminate need

for data storage and use to buffer the data 144kB Flash, 8kB RAM

ADC 6 channels, at least 12-bit resolution, with 100SPS

Simultaneous sampling, ADC buffer memory

16 channels, 12-bit resolution, simultaneous S/H, up to 500k SPS

I/O pins Enough I/O pins to support transmission

I/O pins available for more buttons Over 48 I/O pins

Timers At least one 16-bit timer More timers 5*16-bit timers

available Controller Area Network interface

Data Acquisition from CAN

Vehicular data available

2 channel CAN, up to 500 kbits

Table 2: Symbols definition in coordinate transform Symbol Definition

u The X axis of image coordinate system v The Y axis of image coordinate system H The height of the camera from ground

k, m, b The coefficient of road geometry mθ Tilt angle of camera W Actual lane width

Xw, Yw, Zw Global coordinates eu Pixel width of CMOS sensor ev Pixel height of CMOS sensor

transformation are shown in Table 2. In such circumstance, DSP can convert lane markings from image frame into the world coordinate by Eq. (1). According to these x, y and z position data, a quadratic road geometry model can be approximated by meanings of recursive least squares processing and then the physical position of lane boundaries with respect to the host vehicle would be estimated.

⎥⎦

⎤⎢⎣

⎡−−

⋅−

=vme

Hmevme

Heee

vmeuHZYX

v

v

v

v

u

v

vwww

θ

θ

θθ

][ (1)

2.2 IMU in Dead-Reckoning An inertial measurement unit (IMU) is a closed system that is used to detect altitude, location, and motion. It normally uses a combination of accelerometers and gyroscopes to track the vehicle motion in attitude and location. In this study, inertial information plays the autonomous role in positioning. Gyros can measure angular rate with reference to inertial space, and accelerometers measure linear acceleration with respect to vehicle’s frame. The IMU utilizes a tri-axis accelerometer and three one-axis gyroscopes as inertial measurement components. The accelerometer is measured for X-Y-Z axis; while the gyros are assigned to X-Y-Z axis correspondingly. The IMU plays a full inertial function for vehicle positioning at failure mode.

5

The ADXRS614 operates on the principle of a resonator gyroscope. The output signal of ADXRS614 is a voltage proportional to angular rate about the axis normal to the top surface of the package. With the increase of the rotation rate, the output voltage leaves the neutral point. An external capacitor is used to set the bandwidth. Use external capacitors in combination with on-chip resistors to create two low-pass filters to limit the bandwidth of the ADXRS614’s rate response. ADXL330 is made by the principle of resonant accelerometers. It can measure both dynamic acceleration (e.g., vibration) and static acceleration (e.g., gravity). The outputs are analog voltages proportional to acceleration. This sensor is capable of measuring both positive and negative accelerations to at least ±3 g. Because the signal measure from the accelerometer is analog, it may be disturbed by external noise. According to specifications, the external capacitor can be chosen to determine the bandwidth of the accelerometer, e.g., 0.47μF capacitor for 20 Hz is used in this paper. 2.3 VU from OBD II Connector The amount of electronic devices in vehicles is connected and diagnosed by CAN bus. CAN is a serial, asynchronous, multi-master communication protocol for connecting electronic control modules, sensors and actuators in automotive and industrial applications. In system platform, data is transmitted or received by CAN bus [7]. VU get signals from CAN bus, which is packet with scheduled timing from OBD II. This paper adopts odometry, brake and directional light signals to finish LDWs functions [8]. The OBD II connector usually locates near brake/throttle under steering, and the connector is D-type and 16 pins (No.6 & No.14) with CAN interface. In Fig. 3, left part is OBD II connector and right part is CAN messages. 3. PRINCIPLE OF SYSTEM ALGORITHMS Extended LDWs has two key technologies, including lane departure detection, on-line autonomous positioning. To detect whether the host vehicle is departing from its lane, the primary work is to determine positions of lane markings in image plane. About on-line positioning, attitude and vehicle motion are calculated from derived methods. LDWs will provide lane departure ability, and both of IMU and VU offers independent sensing as complementary performance. These procedures to integrate LDWs, inertial rates and vehicle states follow theoretical formulations with embedded software in DSP & MCU below.

Figure 3. OBD II connector and oriented messages

6

3.1 Lane Marking Detection Detecting lane markings use a series of featured-base approaches which are lane recognition and road geometry estimation to do lane markings recognition. Lane recognition has three steps, including image adjustment in grey value, compared with lane marking width, and the continuity of lane marking. The first procedure is grey value adjustment. Regardless of white, yellow or other lane markings appeared in the roadway image, all of them have higher grey value than road color. Therefore, system can utilize the statistics of grey scale to identify the threshold value of lane markings. Horizontal Sobel mask, which is Eq. (2), is used to detect the edge of lane markings in the image, as shown in left part of Fig. 4. I(u,v) represents the element of original image; E(u,v) is the element after edge processing. S*[ ] represents the Sobel operation, and its horizontal mask is [-1 –2 –1; 0 0 0; 1 2 1]. The actual width of lane marking will be calculated from image plane through a constant ratio conversion, First element difference of Eq.(1) shows road width of lane markings, system can use a determined width which is defined by Ministry of Transport. In continuity, lane markings usually form a lane boundary as segment closed to their neighboring ones. It is an important cue to identify weather it is a lane marking or not.

⎩⎨⎧ >∗

=otherwise

)v,u(Iif)]v,u(I[S),(E

00

vu (2)

For the purpose of finishing lane marking rapidly, a road geometry model is used to confine the basic detecting zone within the detectable range. The detecting zone was equally divided into several sections from bottom to top, and moreover, it also performed the lane searching in a manner of transverse line segment, which is defined the dynamic region of interest (DROI), as shown in upper right of Fig. 4. When applying the image processing to recognize the lane

Grey value in histogram statistics Road geometry estimation

Figure 4. Lane marking recognition

7

markings, DROI not only can reduce the noise or disturbance to locate the lane markings correctly in delicate setting, but also it can shorten the processing time. The bottom right of Fig. 4 exhibits the result after images processing for lane recognition. The left line and the right line are the desired results from detection while the middle line is the estimated center line of the lane according to the left & right lines. Thus, the subsequent estimation for the amount of lane departure and the curvature of roadway is performable. 3.2 Self-Positioning To meet extended LDWs system ability, DSP will output available points at 5 m & 15 m. Furthermore, road geometry is described by quadric equation. At failure mode, vehicle kinematics can be presented as linear mode from reduction of quadric function. Hence, self-positioning is the key technology to check departure or not w.r.t reference line in Fig. 5. In self-positioning, owing to inertial sensor misalignment and scale factor uncertainty, we also derived a vehicular calibration method to calibrate parameters using autoregressive exogenous method in Fig. 6. The vehicle plant is demonstrated vehicle, and the output is captured from IMU. In calibrated operation, this paper corrects gyro and accelerometer parameters (a). Taking gyro procedure as example, the observer device used GPS to get vehicle heading. Refer to Eq. (3)-(4), MCU integrated angle rates ( w ) and compared with headings ( H ). This method uses second-order minimal energy and gradient method to get error variation ( He ) in (5), and furthermore the relation can be derived to discrete form in (6).

∫ +++= −− dtpwpwpkH n

nn

n )ˆ...ˆˆ(][ˆ0

11 (3)

][ˆ][],[][][ ˆ kHkemkHkHke HH =−−= (4)

2ˆ ])[][(

21 kekeJ HH −= (5)

⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢

⎣

⎡

⋅−⋅+

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

=

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡−

−−

1

][][

])[][(

][ˆ

][ˆ][ˆ

][ˆ

][ˆ][ˆ

1

ˆ

0

1

0

1

MMM

kwkw

keke

kp

kpkp

kp

kpkp

n

n

HHn

n

n

n

λ (6)

YW

XW5m Lx

15m Lx

5m Rx

15m Rx

Figure 5. Autonomous detection w.r.t extended trajectory

8

Vehicular Sensor Model Vehicular Sensor Model

Observer DeviceObserver Device

Vehicle Dynamics PlantVehicle Dynamics Plantu(k)a(k), w(k)

a(k),a(k-1),…,a(k-m)

v1(k)

H(k), v2 (k)

H kˆ ( )

v k( )ˆ

w(k),w(k-1),…,w(k-m)

Figure 6. Calibration method for inertial sensors

After sensor calibration, position is the integral from speed variation. The variations of velocities are integrated from the accelerations in the local geodetic frame. However, the measurements derived from sensors are in body frame. Therefore, direct cosine metrics is used to transform the measurements from body frame into the local geodetic frame [9-10]. The transformation is shown in Eq. (7). Owing to the Earth’s rotation ( nn

ei vw ×/2 ) and gravitation ( ng ), the effect of the Coriolis force ( nn

ne vw ×/ ) and gravity need to be corrected.

nnnei

nne

nn gvwwfv +×+−= )2( //& (7)

4. VERIFICATION TESTS In system operation, hardware is installed into a proper and fixed location because of inertial sensor directionality. LDWs will be done intrinsic and extrinsic parameters (roll-pitch angles, and relative position) calibration when assembling camera into designed mechanism in left Fig. 7. The right Fig. 7 presents hardware components with CAN and power interface, and DSP operates camera input and processing for lane markings detection. ARTC LDWs functions can operate in 60 kph above. System adopts two lane markings and reasonable width for departure warning without turning lights. Fig. 8 presents possible failure cases under system operation. The first two scenarios present disappearance of the road lines in intersection comparing the images before and after. The causation of third scenario is that roadway width gradually becomes larger than standard one. The last scenario is caused due to unclear lane markings before and after 0~1 second. Hence, IMU is installed to extend prior available trajectory to solve most scenarios for keeping in dead-reckoning track.

Camera installation

System platform

Figure 7. System hardware and its blocks

9

Scenario I: road intersection

Scenario II: road intersection

Scenario III: before toll station Scenario IV: blurred lane marking Figure 8. LDWs in failure mode, taking four scenarios as example

0 2 4 6 8 10 12 140

5

10

15

20

25

30

35

sec

kph

GPS Speed

Calibrated Speed

Cali-speed per 3 sec

Acceleration test under parameters learning

0 10 20 30 40 50 6050

100

150

200

250

300

350

400

Time (sec)

Hea

ding

(de

gree

)

GPS Heading

Calibrated Heading

Turning tests in gyro’s parameters learning

Figure 9. Calibration result for inertial sensors When IMU had been set well in demonstrated vehicle, a driver drove in different speed to test straight moving and turn availability in sensor calibration. The straight driving test is used to adjust accelerometer parameters refer to GPS speed. The inertial data would be processed and filtered before calibration. Owing to integration error, the result should be calibrated and delicate processed well. The left Fig. 9 used a one-axis acceleration to get speed and the result is compared with GPS speed using Eq. (3)-(6). The parameters was learned and calibrated by parameters learning and error cancelation. Hence, the speed error is under 5kph. In the similar way, the gyro integrated angular rate into heading comparing with GPS course in right Fig. 7.Hence, the IMU ability has presented fewer than 4.0% position error. Under IMU assistant, the integrated system extends available lane markings to virtual ones. The driving route of vehicle motion is on-line compared with extending available virtual lane markings, and the distance error is less than the intersection distance (longitudinal – 7.5 m with lateral error at 0.3 m). System verification had been well-scheduled and test at ARTC proving ground, express way, and general road. Based on inertial sensor and vehicular signals assistant, failure conditions can be effectively reduced, and the proposed concept enhances LDWs ability and continuity in real operation.

10

5. CONCLUSIONS In this paper, the capability of extended LDWs is applied and verified under sensor fusion and scheduled tests. The proposed concept implemented an integrated technology to provide lane departure function with virtual lane markings, which adopted extrapolation through available dead-reckoning track. The calibrated algorithm is effectively developed into self-positioning after parameters learning of vehicle motion. In conclusion, the advantage of extended LDWs provides a feasible solution to enhance driving safety. 6. ACKNOWLEDGMENT This work is supported in research projects 102-EC-17-A-04-02-0803 by Department of Industrial Technology, Ministry of Economic Affairs, Taiwan, R.O.C., under grant number; 7. REFERENCES 1. The 100-Car Naturalistic Driving Study Phase II – Results of the 100-Car Field Experiment,

National Highway Traffic Safety Administration (NHTSA), April 2006. 2. National Highway Traffic Safety Administration (2006, April), The Impact of Driver

Inattention on Near-Crash/Crash Risk: An Analysis of 100-Car Naturalistic Driving Study Data, April 2006.

3. Bertozzi, M. and Broggi, and Cellario, M. and Fascioli, A. and Lombardi, P. and Porta, M., “Artificial Vision in Road Vehicles” in Proceedings of IEEE, Vol. 90, Issue: 7, July 2002, pp. 1258-1271.

4. Suzuki, A. and Yasui N. and Kaneko M., “Lane Recognition System for Guiding of Autonomous Vehicle” in Proceedings of the Intelligent Vehicles '92 Symposium, Jul 1992, pp. 196-201. (Detroit, MI)

5. Broggi, A. and Bertozzi, M. and Fascioli, A. and Guarino, C. and Bianco, Lo and Piazzi, A., “Visual Perception of Obstacles and Vehicles for Platooning”, IEEE Transactions on Intelligent Transportation systems, Vol. 1, Sept. 2000, pp. 164-176.

6. Hwang, D. H., Moon, S. W., Sung, T. K., Lee, S. J., “Design and implementation of an efficient tightly-coupled GPS/INS integration scheme” in Proceedings of 2000 National Technical Meeting, 2000, pp. 159-165. (Anaheim, CA)

7. Zuberl, K. M. and Shin, K. G.,“Scheduling Messages on Controller Area Network for Real-Time CIM Applications”, IEEE Transactions on Robotics and Automation, Vol. 13, No. 2, April 1997, pp.310-314.

8.ISO 9141: Road Vehicles – Diagnostic System – Requirements for Interchange of Digital information, International Standard Organization, Oct. 1989.

9.Britting, K. R., Inertial navigation systems analysis, New York: Wiley-Interscience, 1971, pp. 30-43, 44-78.

10.Biezad, D. J., Integrated navigation and guidance systems, Reston: AIAA Publications, 1999, pp. 15-35, 39-71.