1

ACTIVE PEDESTRIAN PROTECTION SYSTEM,

PROJECT REVIEW

Ho Gi Jung

Research Professor, School of Electronic and Electrical Engineering Yonsei University, 134 Shinchon-dong, Seodaemun-gu, Seoul 120-749, South Korea

TEL: +82-2-2123-4537, E-mail: hgjung@yonsei.ac.kr, Homepage: http://web.yonsei.ac.kr/hgjung

Dong Suk Kim Senior Research Engineer, MANDO Corporation Global R&D H.Q.

413-5, Gomae-Dong, Giheung-Gu, Yongin-Si, Kyonggi-Do 446-901, South Korea E-mail : greenhupa@mando.com

ABSTRACT This paper introduces active pedestrian protection system (APPS) project, which was fulfilled during last three years (Nov. 2006 – Oct. 2009) and supported by ministry of knowledge economy (MKE) of Korean government, and then summarizes its achievements. To reduce the number and severity of vehicle-pedestrian accident by active safety vehicle (ASV), false alarm rate should be minimized. For this purpose, we developed critical area-centered pedestrian detection system which recognized locations of parked vehicles along the road-side then investigated only areas behind the parked vehicles. After evaluating four different approaches, sensor fusion of near infrared (NIR) vision and scanning laser radar was selected as the optimal. Although feasibility of braking and suspension for pedestrian protection was evaluated, only braking was proved to be useful.

INTRODUCTION

Recently, pedestrian protection system attracts a great interest from governmental and academic organizations because it is directly related with vulnerable road user which holds significant portion of traffic accident fatalities (1). Pedestrian protection methods are classified into 4 categories: infrastructure design enhancement, passive safety involving vehicle design, active safety system based on pedestrian detection, and collision prediction (1). Whether a system belongs to active or passive approach is determined by when the system is activated, before or after vehicle-pedestrian collision. This paper deals with our research activities related with active pedestrian protection system development. This paper introduces the first phase of active pedestrian protection system (APPS) project,

2

fulfilled during last three years from Nov. 2006 to Dec. 2009, and then summarizes its achievements. The APPS project included three companies and two universities; it was supported by MKE of Korean government; the total budget was about 4.7 million USD. The goal of the APPS project is detection of dangerous pedestrians using sensor fusion-based technologies, risk assessment considering the motion of the subject vehicle and the pedestrians, and actuation to avoid vehicle-pedestrian collision or mitigate its damage.

From the perspective of recognition system, four different approaches, recently attracting much interest, were evaluated and the most suitable one was selected for the second phase. The approaches are 1) sensor fusion of NIR vision and range sensor, 2) time of flight (TOF) camera-based, 3) range sensor array-based, and 4) sensor fusion of far infrared (FIR) stereo and visible vision. Sensor fusion of NIR vision and scanning laser radar was selected as the optimal. From the perspective of vehicle control, it was aimed to develop a risk assessment method numerically evaluating the possibility of collision avoidance by considering the motion of the subject vehicle and pedestrians. From the perspective of actuator for pedestrian protection, the feasibility of braking and suspension was evaluated; only braking was proved to be useful.

SCENARIO-DRIVEN METHOD After reviewing the requirements of the recognition system of APPS, it was recognized that the recognition system needed far better reliability compared to systems providing only warning (4). If a recognition system evaluating all pedestrian candidates is adopted, APPS can not be practical because of high false alarm rate. For example, if false alarm rate of a system is 0.01 false/frame which could be regarded as the world best, APPS will suffer false alarm once for every 10 seconds given the operation frequency is 10 Hz. We thought that almighty one should be replaced with a combination of strict experts. A novel concept “scenario-driven method” was proposed (4), (5): a recognition system treating only the top ranked situation was developed almost without false alarm, and then the next ranked situation was treated. Consequently, the final recognition system is constructed by combining these recognition systems; Fig. 1 shows the concept. Each expert is expected to be able to successfully detect dangerous pedestrian in a specific situation, even though its general performance is not perfect. Therefore, we focused on situations in which there were frequent vehicle-pedestrian accidents and the severity was high.

3

Fig. 1. Almighty classifier is replaced with a combination of strict experts.

CRITICAL AREA-CENTERED PEDESTRIAN DETECTION The top ranked situation was when a pedestrian popped up from the behind of vehicle

parked along a road-side (1), (2). When vehicles are moving on an urban road, the most common threat that a pedestrian may pose, therefore requiring successful detection, is road crossing. Stopped vehicles on the road or on the road edges may occlude visibility, thus making the detection of the pedestrian more complex. Recognition system corresponding to the situation was designed to monitor the behind of parked vehicle; the area is called ‘critical area’. Pedestrians that are clearly visible in the frontal area of the vehicle also need to be detected, but we assume that such pedestrians would be easily detected by the driver. The first row of Fig. 2 shows some examples of situations in which the visibility of a crossing pedestrian is partly or completely occluded by stopped vehicles. The second row of Fig. 2 highlights, for each situation, the areas on which the system will perform a check for the presence of a possible pedestrian.

Fig. 2. Examples of critical area. The second row shows the critical area of situations of the first row.

4

COMPARISON OF FOUR SENSING METHODS Four different approaches, recently attracting much interest, were investigated and

compared: 1) sensor fusion of NIR vision and range sensor, 2) TOF camera-based, 3) range sensor array-based, and 4) sensor fusion of FIR stereo and visible vision. Three criteria were used for the selection of the optimal one: 1) critical area localization performance, 2) latency of popping up pedestrian detection, and 3) system cost and possibility of domestic series production. Consequently, sensor fusion of NIR vision and range sensor was selected for the next phase.

Candidate 1: Sensor Fusion of NIR Vision and Range Sensor A recognition system using sensor fusion of NIR vision and scanning laser radar was

developed to meet the first scenario (3). After critical areas, where the driver’s gaze can not reach, were detected using the range data from scanning laser radar, vision-based pedestrian verification was applied to the areas (3-5). Fig. 3 shows the brief procedure of the developed recognition system.

Fig. 3. Procedure of NIR vision-scanning laser radar-based method. An all day long pedestrian classification system was developed based on an AdaBoost

cascade meta-algorithm (6). The underlying idea is to use a Haar-features-based Adaboost together with an ad-hoc-features-based AdaBoost system in order to reach a better pedestrian classification. A specific night-time pedestrian classification was developed in order to obtain a system that can be used also in poorly illuminated environments. These classifiers are joined together using a cascade AdaBoost system that uses the output of the previous classifiers to obtain a final classification for the area. Fi.g 4 shows image examples used for the classifier learning; compared with other pedestrian databases, a large number of images were captured within critical areas.

5

Fig. 4. Image examples used for the classifier learning. Additionally, we developed a pedestrian classifier using Gabor filter bank (GFB)-based feature and radial basis function (RBF)-based support vector machine (SVM). We also investigated how publicly open database could be used for pedestrian classifier learning (7). Through experiments, four facts were found: 1) GFB-RBF SVM is competent in pedestrian recognition; 2) Performance estimator can replace cross-validation. In this case, dataset used for cross-validation can be used for learning, and increases resultant performance. Joachims’ performance estimator proves valid in this usage; 3) If input image is processed by histogram equalization and sampled by the same method as the one used for the learning database, a pedestrian recognition system constructed with a database can be used for actual application without re-training with the newly acquired database. This not only increases the importance of the public open database but also decreases repeated tedious learning procedures after any changes are made to the imaging system; 4) A posteriori probability-based post-processing enhances recognition rate while losing a little in the false positive rate. This is thought to be caused from different characteristics of pedestrian and non-pedestrian. In other words, as the pedestrian changes its appearance continuously, its classifier output tends to fluctuate as opposed to the rigid non-pedestrian. Therefore, a posteriori probability-based post-processing increases pedestrian recognition rate while slightly increasing false positive rate. Additionally, high dynamic range CMOS (HDRC) camera module sensitive to NIR and NIR

head lamp having wider horizontal field of view (FOV) were developed. To make the developed recognition system operate in real-time, core algorithms were implemented by VHDL.

(a) General headlamp (b) Newly developed headlamp

Fig. 5. NIR Lamp with horizontally wider FOV was developed.

6

Candidate 2: TOF Camera-Based Because TOF camera was supposed to provide distinctive benefit (simultaneous acquisition

of range and image data) (9), its feasibility for APPS was investigated. Although modulating light source and sensor module was developed, this approach was rejected because of several reasons: Power requirement of the modulating light source is too high to be integrated into automotive electronic system if the coverage is up to 30m; The problem also causes too large size, additional cooling mechanism, and eye-unsafe light power in the near area; Furthermore, the core component, TOF sensing element, should be imported from a foreign company and its development was not matched with our schedule.

Candidate 3: Range Sensor Array-Based Previously, there were researches to recognize pedestrian using millimeter wave radar: Prof.

Hermann Rohling’s team of Hamburg University of Technology developed distinctive pedestrian detection systems using only automotive millimeter wave radar (10), (11). As millimeter wave radar

became popular because of adaptive cruise control (ACC) system’s deployment, its usage extension to APPS would be highly appreciated by automakers.

Although we developed radar signal-based object classification (12), its direct application to APPS was thought be impractical. The major weakness of millimeter wave radar is instable localization of

vehicle corner: object information from millimeter wave radar is too sensitive to the vehicle’s pose and contains relatively large variation. Furthermore, the internal algorithm of the radar prevents rapid detection of popping up pedestrian and it causes significant latency.

Candidate 4: Sensor Fusion of FIR Stereo and Visible Vision Although FIR stereo showed good performance in night time, its usability in daytime was not

verified. Recently, Bertozzi et. al. tried to combine FIR and visible information to recognize pedestrian (13). They showed that combination of FIR stereo and visible stereo could recognize pedestrian both in night time and day time.

We also developed a similar system and realized that such a combination could be used for pedestrian recognition. However, the approach needs too many cameras and its cost is far beyond other approaches because FIR camera is relatively expensive until now. Furthermore, the approach needs to incorporate parked vehicle detection to focus on our major scenario; it is contrast to inherent capability of range sensor, particularly scanning laser radar. There is a dilemma: If temperature-based segmentation of FIR image is used, latency problem could be addressed in nigh time; However, such an approach will suffer typical problems of temperature-based segmentation.

7

RISK ASSESSMENT AND ACTIVE BRAKING A risk assessment method was developed considering motions of the subject vehicle and pedestrians, and possibility of collision avoidance by braking or steering maneuver was researched. Basically, actuations in two steps, collision warning (CW) and collision mitigation by braking (CMbB), of typical 5-step collision scenario were investigated as shown in Fig. 6. Because of danger of false actuation and driver’s acceptability, steering maneuver was rejected.

Fig. 6. Typical countermeasures according to 5-step collision scenario.

To reduce danger of false actuation, pedestrian protection measures are activated according to a hierarchical strategy. Fig. 7 shows the actuations for 3 risk levels: 1) driver warning and deceleration less than 0.3g, 2) pedestrian warning and deceleration less than 0.6g, 3) full braking with deceleration larger than 0.6g. The developed control method was proved to avoid collision by controlling the vehicle’s deceleration with active braking system according to the risk level.

Fig. 7. Hierarchical strategy of pedestrian protection measures.

8

FEASIBILITY STUDY OF PCDN FOR APPS Although the possibility whether pre-crash dipping nose (PCDN), which was developed for

vehicle-vehicle collision, could be applied to vehicle-pedestrian collision was evaluated by computer simulations, PCDN showed severity increase in vehicle-pedestrian collision situations. However, as the result revealed that vehicle-vehicle solution also required pedestrian recognition system, it is believed to make a contribution to ASV developers (8). We investigated the effect of vehicle front height lowering operation of PCDN on

pedestrians. For simulation-based investigation, a vehicle model, an air spring with PCDN actuator, a pedestrian, and active hood system (AHS) were modeled. Two vehicle models were made for a sedan and a sport utility vehicle (SUV), respectively. Three facts were found: 1) In all situations, the pedestrian injury is substantially worse because of the PCDN puncture operation; 2) Particularly, pedestrian injury by side crash is significantly worse than in a frontal crash; 3) AHS substantially reduces not only pedestrian injury but also the increased severity of the injury by PCDN puncture operation. In all crash situations (frontal crash without AHS, frontal crash with AHS, and side crash without AHS), PCDN is expected to cause worse pedestrian injury.

(a) Without PCDN and AHS (b) With PCDN and AHS

Fig. 8. Vehicle-pedestrian crash simulation. Although PCDN was developed for vehicle-vehicle side crashes, there is a possibility that

the range sensor for crash detection fail to distinguish a group of pedestrians from a side-faced vehicle. Therefore, it is very important to investigate the situation when the sensing system makes a mistake. As shown in the airbag tragedies involving children and short drivers, consideration of unintended situations is essential and important for safety-related systems. Pedestrians seem to be an unintended but important threat to PCDN. In this sense and based

9

on the simulation results, we insist that PCDN should incorporate a pedestrian recognition capability into its sensing system to exclude pedestrians from its interesting obstacles. Additionally, considering the fact that AHS significantly reduced pedestrian injury and the severity of injury caused by PCDN puncture operation, we suggest a complementary method that activates AHS when PCDN is activated.

EXPERIMENTAL RESULTS (5) The developed system was tested with two test vehicles as shown in Fig. 9, one in Korea and

the other in Italy. Used camera has 752×480 resolution and 15/9 aspect ratio. Its sensitivity covers both the visible and NIR spectra. Used scanning laser radar is SICK LMS 211-30206. The detection capabilities (scanning angle of 100°, minimum angular resolution of 0.25º, up to 80m range, and fog correction) are suitable for our goal. A compact PC (an Intel Core 2 Duo-based mini-ITX) is installed in the boot. The braking system installed on the test vehicle was replaced with MANDO’s MGH-40 ESCplus to control vehicle deceleration via CAN.

Fig. 9. (a) Hyundai Grandeur test vehicle equipped in Parma, Italy, and (b) the replica in Korea. Usually, performance assessment of pedestrian-detection systems provides the percentage

of correctly localized pedestrians and the false alarm rate. As this system is developed under a new perspective, performance figures are provided as the percentage of appearing pedestrians correctly identified and the false alarm rate. Indeed, it is not important to detect all the pedestrians present in the scene: only the dangerous pedestrians are sufficient to prevent accidents. A set of sequences (different from that used for the training) was considered for extensive

performance assessment, for a total driving time of about 10 hours in complex urban scenarios. An entire week was dedicated to the final test; a total of 236 km was driven during the day and at night, under different weather conditions (sunny, cloudy, rain, and fog). Various scenarios were included: downtown, large and narrow roads, underground car parks, highways, and rural roads. During night tests, scenarios with and without external

10

illumination were acquired. To complete the test sets, some specific situations were staged (such as very dangerous pedestrian crossings), but a number of dangerous scenarios were framed during normal driving anyway. A total of 24 suddenly appearing pedestrians were correctly detected in the tests. Only one

false positive is present (due to a misclassification of a parked scooter). Considering the actual camera frame rate of 15 frame/s and the test duration, the false positive rate is about 2×10−6 false positive/frame. Only one missed warning out of 11 was due to a pedestrian misdetection; the others are caused by either alert misses or delayed detections.

Fig. 10. Some suddenly appearing pedestrians correctly detected (a) in an underground parking, (b) in the rain, (c) behind a misaligned vehicle, (d) behind a wall, and (e) at night, and (f) a suddenly appearing pedestrian detected as a non-dangerous pedestrian (false negative).

11

CONCLUSION During our 3 year project named APPS, we found the following important facts: 1) Scenario-driven method successfully relieves severity of false actuation problem.

Particularly, critical area-centered pedestrian detection is supposed to improve practical feasibility of APPS.

2) Four different pedestrian sensing methods were investigated and compared: sensor fusion of NIR vision and range sensor, TOF camera-based, range sensor array-based, and sensor fusion of FIR stereo and visible vision. Three criteria were used for the selection of the optimal one: critical area localization performance, latency of popping up pedestrian detection, and system cost and possibility of domestic series production. Consequently, sensor fusion of NIR vision and range sensor was selected for the next phase.

3) Through actual driving experiments, the possibility of active braking-based pedestrian protection was proved.

4) Evaluation of PCDN’s effect on collided pedestrian revealed that vehicle-vehicle solution also required pedestrian recognition system. It gives important implication to ASV developers (8).

Acknowledgement

This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology (grant number: 2010-0013494).

REFERENCES (1) Tarak Gandhi and Mohan Manubhai Trivedi, “Pedestrian Protection Systems: Issues,

Survey, and Challenges,” IEEE Transaction on Intelligent Transportation Systems, Vol. 8, No. 3, Sep. 2007, pp. 413-430.

(2) David Gernóimo, Antonio M. López, Angel D. Sappa, and Thorsten Graf, “Survey of Pedestrian Detection for Advanced Driver Assistance Systems,” IEEE Trans. PAMI, Vol. 32, No. 7, Jul. 2010, pp. 1239-1258.

(3) Alberto Broggi*, Pietro Cerri, Stefano Ghidoni, Paolo Grisleri, Ho Gi Jung, “Localization and Analysis of Critical Areas in Urban Scenarios”, 2008 IEEE Intelligent Vehicle Symposium, Eindhoven University of Technology, Eindhoven, The Netherlands, June 4-6, 2008, pp. 1074-1079.

(4) Alberto Broggi, Pietro Cerri, Luca Gatti, Paolo Grisleri, Ho Gi Jung, JunHee Lee, “Scenario-Driven Search for Pedestrians Aimed at Triggering Non-Reversible Systems,” 2009 IEEE Intelligent Vehicle Symposium, 2009 IEEE Intelligent Vehicle Symposium,

12

June 3-5, 2009, Xi’an, Shaanxi, China, pp. 285-291. (5) Alberto Broggi, Pietro Cerri, Stefano Ghidoni, Paolo Grisleri, and Ho Gi Jung, “A New

Approach to Urban Pedestrian Detection for Automatic Braking,” IEEE Transactions on Intelligent Transportation Systems, Vol. 10, No. 4, Dec. 2009, pp. 594-605.

(6) Pietro Cerri, Luca Gatti, Luca Mazzei, Fabio Pigoni, and Ho Gi Jung, “Day and Night Pedestrian Detection Using Cascade AdaBoost System,” IEEE International Conference on Intelligent Transportation Systems, Madeira Island, Portugal, 19-22 Sep. 2010, accepted.

(7) Ho Gi Jung, Jaihie Kim, “Constructing a pedestrian recognition system with a public open databse, without the necessity of re-training: an experimental study,” Pattern Analysis and Applications, on-line published on 7 April 2009, DOI 10.1007/s10044-009-0153-2.

(8) Ho Gi Jung, Byung Moon Kwak, Jeong Soo Shim, Pal Joo Yoon, Jaihie Kim, “Pre-Crash Dipping Nose (PCDN) Needs Pedestrian Recognition”, IEEE Transactions on Intelligent Transportation Systems, VOL. 9, NO. 4, Dec. 2008, pp. 678-687.

(9) Basel Fardi, Jaroslav Dou'sa, Gerd Wanielik, Bjorn Elias, Alexander Barke, “Obstacle Detection and Pedestrian Recognition Using A 3D PMD Camera,” Intelligent Vehicle Symposium 2006, Jun. 13-15, 2006, Tokyo, Japan, pp. 225-230.

(10) Florian Folster, Hermann Rohling, and Marc-Michael Meinecke, “Pedestrian Recognition Based on Automotive Radar Sensors”, 5th European Congress and Exhibition on ITS and Services, 1-3 June 2005.

(11) Henning Ritter, Hermann Rohling, “Pedestrian Detection Based on Automotive Radar,” International Conference on Radar Systems, Edinburgh, UK, October 2007

(12) Seongkeun Park, Jae Pil Hwang, Euntai Kim, Heejin Lee, and Ho Gi Jung, “A neural network approach to target classification for active safety system using microwave radar,” Expert Systems with Applications, 37 (2010), pp. 2340-2346.

(13) Massimo Bertozzi, Alberto Broggi, Mirko Felisa, Stefano Ghidoni, Paolo Grisleri, Guido Vezzoni, Cristina Hilario Gómez, and Mike Del Rose, “Multi Stereo-based Pedestrian Detection by means of Daylight and Far Infrared Cameras”, Lecture Notes in Computer Science (Object Tracking and Classification Beyond the Visible Spectrum), pages 371-401.