Indoor robot navigation by landmark tracking

Pergamon Mathl. Comput. Modelling Vol. 26, No. 4, pp. 79-89, 1997

Copyright@1997 Elsevier Science Ltd Printed in Great Britain. All rights reserved

PII: SO895-7177(97)00146-5 08957177/97 $17.60 + 0.90

Indoor Robot Navigation by Landmark Tracking

JIANN-DER LEE

Department of Electrical Engineering

Chang Gung University of Medicine and Technology

Tao-Yuan, Taiwan 333, R.O.C.

(Received April 1997; accepted June 1997)

Abstract-An intelligent approach to robot navigation by landmark tracking using computer vision is proposed. Thii approach is based on the concept that a human can reach the destination by tracking the specific landmark in a previous environment. Only a monocular image of the landmark

taken by the robot is required. Techniques of image processing and pattern recognition are integrated to resolve the problems of landmark tracking and understanding. Simulated results to confirm the

feasibility of this approach is also included.

Keywords--Robot guidance, Landmark tracking, Computer vision.

1. INTRODUCTION

A great deal of work has recently been addressed to the problem of automated robot naviga-

tion in complex workspace using many different sensors. These sensors are composed of active

sensors (e.g., optical, ultrasonic) and passive sensors (e.g., camera). A lot of the mobile ro-

bot systems using various sensing approaches to indoor mobile robot guidance can be found

in [l-3]. For example, Madarssz [1] developed an autonomous wheelchair to navigate inside an

office building with the capability of path planning, corridor following, obstacle detection, and

avoidance. In this approach, the location of the vehicle cannot be obtained since it is not used in

road following. To navigate inside an office or a factory, Bell Laboratory [4] designed another ex-

perimental vehicle using an optical ranger finder and an odometer. In the approach, the odometer

is used to display the location of the vehicle during navigation and bar codes distributed in the

environment are used as a global beacon for the sake of correcting the location inaccuracy due

to the accumulated errors of the odometer. The disadvantage of this method is that it needed to

change the environment.

To equip the learning capability of a mobile robot, the recent development of mobile robot guidance is to utilize computer vision techniques to analyze the image of the environment and

plan a feasible path to the destination [4-71. The typical application using computer vision

in robot guidance is to adopt landmark features of the workspace as an external assistant tool.

Usually, in such a system, the robot uses the external sensory data to estimate its location roughly,

and verify this estimated location by extra landmark features. Since the successful detection and

recognition of the landmark feature is an essential requirement to the performance of the mobile

This research is supported in part by the National Science Council, R.O.C. under Grant NSC8P2212-E182-096.

Typ==t by AM’PEX

79

80 J.-D. LEE

robot system. The choice of landmark features and the arrangement of camera configuration is various under different consideration. However, some general rules to select suitable landmark features for real-time can be summarized as

1. the landmark features can be easily detected by inexpensive visual sensors; 2. the visual landmarks are commonly observed in an indoor environment; 3. the landmark can provide either the location information for self-localization or the next

instruction to be executed, or both.

For instance, using a single camera, Huang [5] proposed a stereo matching for the guidance of a mobile robot. Their method only works under the assumption that a large disparity exists in every two consecutive images. However, this constraint cannot be satisfied if the robot is at the turning area. Ku and Tsai [6] developed an autonomous land vehicle (ALV) for indoor navigation in a corridor of an office building. In order to watch both sides of the corridor, two cameras (one face toward leftward and the other rightward) are employed to provide the feedback information for navigation. Prior to the navigation of the ALV, a map of the environment should be built. The corridor contour is used as the map and it is composed of the baselines which are the intersections of the walls and the corridor ground. By model matching techniques and adjusting criteria for direction decision, their ALV can accomplish the navigation task. Dulimarta and Jain [7] also used two cameras to grab navigation information, but one setup for detecting ceiling lights (facing up) and the other for door number plates detection (facing sideways). In order to speed up the image processing, the perspective distortion of the door plate is neglected and the cameras must be fixed in the navigation. This constraint also limits the application of a mobile robot in a fairly complex workspace. Another disadvantage of stereo vision is that a feature visible in one camera’s image may not even be visible in the other, thus it is impossible to establish the stereo correspondence. Figure 1 illustrates the procedures of the traditional approaches for robot guidance.

Current I I position

I p5GGGq

I Optimal El path for

navigation

Figure 1. Procedures of traditional path planning for robot guidance.

Indoor Robot Navigation 81

In general, most of the above-mentioned methods of robot guidance and path planning are

under the assumption that a priori knowledge about the working environment must be provided

in advance. That is, a mobile robot usually follows predetermined routes, specified by predefined maps. While the configuration of the working environment varies frequently, the map of the

environment should also be modified so that the robot can reach its destination. This map

reconstruction task largely reduces the performance of a robot navigating system, especially

when there are a lot of mobile robots in the same environment. To resolve the problem, the best

way is to equip each individual mobile robot the capability of self-localization and self-learning.

The abilities of a robot to automatically determine its location can be achieved by recognizing the

specific landmarks located in its way to the destination and then construct the map of the whole

workspace. The self-localization ability is essential to improve the performance of the whole

autonomous mobile system. For example, it can be used to correct the inaccuracies between

the computed and effected robot motion due to wheel slippage and sensor noise. The often used

approach to self-localization is to represent the robot workspace with a state diagram. The choice

of the state description is based on the objective of the system, the complexity of the system,

sensing service of the robot. The strategy for robot navigation is then to start the system from a

known state and update the current state according to the information obtained from the path

it walks.

(Unknown environment)

Finding I(andmarks

I

I -1

I sythesiziig cl understood information

I Planning III the path

Figure 2. The cognition procedures for a mobile robot in an unknown indoor environment .

In this paper, in contrast with Dulimarta’s method, a novel approach is proposed to guide the

mobile robot to the destination by following each specific landmark seen in its navigating course.

The cognition procedures of this approach is shown in Figure 2. Only a monocular image of the

landmark taken by the robot is required. These specific landmarks for the system are designed

to supply the following information:

(1) the destination,

82 J.-D. LEE

(2) the direction, and (3) the distance to the destination.

In other words, the landmark not only provides the local information of the navigation task, but also give the global information for building the whole workspace map. One of the experimental landmarks is shown in Figure 3. From this landmark, it implies that Q is the destination, 30 is the distance, and the direction is toward the right.

2 W

4 B i ---

Image coordinate system Y

Landmark coordinate system

Robot coordinate system

Figure 3. Robot, image, and landmark coordinate systems.

The remainder of this paper is organized as follows. Sections 2-4 describe the proposed algorithm for landmark tracking, i.e., Landmark Finding, Landmark Localization, and Landmark Understanding. The experimental results are given in Section 5. Finally, the conclusion and suggestions for future research are included.

2. LANDMARK FINDING STRATEGY

The overall structure of the proposed navigation method consists mainly of three components.

(1) Landmark Finding. (2) Landmark Localization. (3) Landmark Understanding.

The Landmark Finding detects each candidate landmark in a mobile robot’s workspace. More specifically, for a monocular image taken by the robot, image preprocessing such as edge detection, thinning, and boundary tracing are used to extract the contour of the landmark and its content (symbols depicting the information to the destination). If there are multiple landmarks in the grabbed image, image segmentation techniques [8] must first be performed to separate each landmark. The Landmark Localization determines the relative location of the detected landmark with respect to the robot and provides the necessary geometrical information for inverse perspective transformation which is required for symbol recognition. The Landmark Understanding extracts each symbol in a landmark and interprets the string constructed with these symbols, such that the mobile robot can understand what to do next (i.e;, which way to follow). A more detailed description of these three procedures are given as the following.

As described above, the aim of Landmark Finding is to detect every possible landmark in robot’s navigating way. Usually, in a typical office building, door plates containing office numbers and/or name are common observed landmarks. However, the landmarks we used include the information about the destination, the direction, and the distance to the destination. Before designing the landmark, several observations concerning the real environment should be discussed.


1. Each landmark should indicate at least the information about destination, direction, and the distance. In addition, the robot needs to know its relative position with respect to each landmark. The relative position, namely local location, provides the following advantages for a mobile robot: (a) it enables the robot to adjust its location; and

(b) it can combine the local location, the real distance derived from understanding the landmark, and the speed of the robot to predict the position of the destination and the arrival time.

2. Although many different types of landmarks exist in the real environment, to enhance the landmarks for the purpose of indication, each landmark is designed with a special shape or color to contrast with the background.

3. Due to the imperfection of image processing and the noise effects, many undesired objects may be detected by the robot as landmarks. To determine whether the detected objects are the desired landmark, the content of each object detected as a candidate landmark should be analyzed and understood by the robot before they can be used to derive information for navigation.

4. Each landmark should be implemented easily.

Taking the above observation into account, we designed each landmark with the following rea- sonable assumptions.

(1) Each Landmark is designed to be rectangular or “arrow” shape. Four corner points of the landmark’s contour provide the required information for the derivation of the robot’s relative location.

(2) The content of each landmark is designed with the property of several discrete loops for convenience of analysis and understanding.

(3) Only landmarks showing simple information are considered, so that we can apply the technique of syntactic pattern recognition in the understanding process.

It is noted that the direction information is provided by the landmark shape (i.e., outer contour). The content of landmarks which depict the destination and relative distance can then be described by using finite-state grammar, namely Landmark Grammar, as below.

DEFINITION 1. The Landmark Grammar is four-tuple G = (VN, VT, P, S), where V, = {S, N} is a finite set of nonterminal capital-alphabet, VT = {a’, n, la’ stands for the alphabets designed in Landmark,andn= 1,2 ,..., 9) is a finite set of input symbols in landmark, S E (I$) is the start capital-alphabet to generate a sentence, and P is a finite set of rewrite rules defined as follows.

Rewrite rule Meaning S - a’N Destination with ‘a’” at the beginning of the name. N-n The real distance of one destination is indicated. N-nN the real distance of one destination has at least two digits. N - nS the real distance of one destination is indicated and

another destination begins to be indicated.

Considering the practical application in a real environment, the Landmark Finding includes three steps. They are:

(1) detecting whether a landmark is in the monocular image taken by the robot; (2) finding the four corresponding-vertex pairs in the landmark; and (3) extracting the symbols in each seen landmark to construct a landmark sentence for later

Landmark Understanding.

Here, for convenience, we assume that each landmark is designed and located to contrast with the background. The image processing procedures are described briefly as follows.

STEP 1. Correct the taken image. Usually geometric distortion exists in the image taken, therefore, it is necessary to correct each image by applying the geometric correction method IS].

84 J.-D. LEE

(a) The source image of an example landmark.

(b) The result image of Figure 4a after edge detection and thinning.

Figure 4.

Indoor Robot Navigation a.5

STEP 2. Once obtaining the corrected image, the Sobel edge operator is then used to detect edge

points and the fast parallel thinning algorithm proposed by Zhang and Suen [9] is employed to

thin the edge points found by the Sobel operator. The result is shown in Figure 4.

STEP 3. Detect the boundary curve of the landmark in the thinned image. Since a landmark’s

boundary consists of multiple lines (e.g., four lines for rectangular shape, five lines for “arrow”

shape, and six lines for hexagonal shape), the Hough transform [lo] is applied for line detection.

In the Hough transform, a line in the image plane is represented by parameters (p, 6) as

p = Ucose +vsine,

where u and v are the image coordinates of the points on the line, 0 is the angle of the normal of

the line with respect to the positive u-axis, and p is the distance from the origin to the line. After

Hough transformation, assume (pi, &) are the line parameters of boundary line i of a landmark,

respectively, i.e.,

LI : Pi =ui.cos&+~~~sinB~.

Since the number of the discretized regions of the parameter space is finite and Li can only be

roughly estimated, we apply the least-square-error line fitting [ll] to improve the accuracy of Li. The set of line points (denoted by {(pi, vi) 1 i = 1,2,. . . , n} to be fitted is composed of the points

near the line obtained in Step 2. The fitting line is given by p = ucos B + v sin 8. The fitting

process is to determine p and B such that the square error

E2=~(p-uic0sB-vi.sine)2 i=l

is minimum. After obtaining the five line equations of the landmark’s boundary, the desired four

vertices for Landmark Localization can then be derived from the intersection of these lines.

STEP 4. Extract the content inside the boundary curve found in Step 3 and understand the

landmark by using the Landmark Automaton described in Section 4.

The above image procedures is performed under the assumption that there is only one landmark

in the image. If there are multiple landmarks in the image and each one is not occluded by

another, we can modify Step 3 in the image procedures as below.

STEP 5. Trace the thinned picture in the counterclockwise direction to derive the boundary

curves of each landmark. Any curve that satisfies the following constraint is declared as a bound-

ary curve:

(1) it is a close contour;

(2) there are several discrete loops in the contour (i.e., the symbols ); and

(3) its length is greater than a preset threshold value t (in our case, t = 150 pixels).

3. LANDMARK LOCALIZATION

The goal of this procedure is to derive robot local location (xc, yC, z,) with respect to the

seen landmark. Three coordinates systems, as shown in Figure 3, are defined to describe the

relationship of the robot and a landmark. In the figure, O-z-y-z is the landmark coordinate

system, O’-u-v-w is the camera coordinate system with the origin at the camera lens center.

The robot location with respect to the landmark coordinate system is denoted as (zC, yC, z,.), and

u’-w’ is the image plane located at v = f of the camera coordinate system. According to [12],

a 3-D point P with landmark coordinates (2, y, z) and camera coordinates (u, v, w) is related to

its corresponding image point P’ with image coordinates (u’, w’) by

@,YJ)~ = Rs b-w~)~ + (~r~crd (1)

(u, v, w) = x (u’, f, w’) , (2)

86 J.-D. LEE

where X = O’P/O’P’ is a scale factor and R is 3 * 3 rotation matrix. From the geometric properties of the landmark in Figure 3, we obtain

(1) AB=OD, (2) m = 1 (a known constant), - (3) AB I AO.

Substituting the image coordinates of 0, A, B, D into Properties l-3, the focal length f and the scale factors X0, AA, An, AD can be derived sequentially by the following equations:

x0 = 1

- $)2 -t (KAf - f)2 + (KAt,jq - V&)2] 1’2 ’

(4) (KA&

AA = KAYO, AB = KBb, AD = KDxo, (5)

where KA, KB, KD are constants. From equation (2), the camera coordinates of points A, B,

D, 0, and the length m (denoted as Ij~ll) can be obtained. Let LA, Lo, Lo represent the distance between points A, D, 0 and the camera lens center O’, respectively. From the landmark and the camera coordinate system, we have

L~=X~(u~2+f2+v~2) =xc,2+yz+*;, (6)

L: = xi (ua’ + f2 + v;“) = x2 + yp + (zc - 1)2, (7)

L; = XL (ub” + f2 + vb”) = (ZC” - pan)’ + y,” + 2,“. (8)

Rearranging the above equations, (xc, yC, .zC) can be derived as

l (G & = 5 - t; + 12) ) (9)

zc=& (La - J% + pq2) , (10)

yc = d L; - x; + y,“. (11)

Once (zC, yC, .zJ is derived, the rotation matrix R can be solved by substituting the coordinates of points A, D, 0 into equation (1). That is,

R=A.B-I, (12)

where -xc -xc lloDj[ - xc

A = -yc -yc

-2, l-z,

1;: ] and B= [I? lz Ez] . .

4. LANDMARK UNDERSTANDING

Landmark Understanding includes symbol recognition and content understanding. Before recognizing each symbol in the landmark, the inverse perspective transformation defined as equa tion (1) is used to obtain each recovered symbol. Then these symbols are recognized by their invariant moments [13], and a landmark sentence consisting of these symbols is constructed. The sentence describes the meaning depicted by the landmark. The purpose of the recognition process is to assign an unknown input pattern to a unique class in the data base. However, in general,

Indoor Robot Navigation a7

more patterns than are demonstrated are required for a real environment. For the general case,

the tree structure of the classification procedure which selects only a few features for each stage

from the whole set of features (e.g., a binary tree), is a better classifier than the conventional

single stage maximum-likelihood classifier using the same set of features. This is because the former has better performance and much faster classification speed [14].

After each symbol in the seen landmark is recognized, a sentence which consists of the symbols

is constructed. The sentence describes the meaning depicted by the landmark. To understand

the meaning of a landmark sentence, a finite-state automaton, namely Landmark Automaton

(LMA), can be employed to interpret this sentence and guide the robot to its destination. The

LMA is described below and the state diagram of this LMA is shown in Figure 6.

* :t/ H : Letter: Room no.

‘,I D ( m I

( /

B ! C I* B.

, I

/

P K / T

I ; *

II_

j /

11 11 *:Landmark location

* ,* : Road

! 1 1 m : door / A 1 /

x : entry

i -i:i *K

Figure 5. An example environment in this approach.

Figure 6. The state diagram of the proposed LMA.

DEFINITION 2. A LMA is a four-tuple LMA= (C,n,Q,F), where C is a finite set of input

symbols, i.e., C = {a’, n, 4 / a’ = A,. . . , 2; n = 1 , . . . ,9; (b is null input symbol}; Q is a finite

set of states; i.e., Q = {qolq1,q2,q3,q4}; qo E Q is the initial state; and F is the set of terminal

states, i.e., F = {qo,qd}.

5. EXPERIMENTAL RESULTS

Experiments simulating a practical environment shown in Figure 5 are performed. The results verify that the landmark can be automatically found and the depicted information can be ex-

tracted and understood correctly by this method. The whole program including image processing

88 J.-D. LEE

and location determination process are written in C-language. More importantly, no special view

angle to the landmark is required and the alignment error can be avoided. Only simple vector

analysis is required in determining robot location, and the line and point features can be extracted

by simple well-developed image processing techniques. These merits make this approach faster

than other approaches using complicated navigation strategy. The time required to compute

location parameters and recognize the landmark are 0.2 s and 2 s using an IBM PC586 computer

system, respectively. The average error in location determination is less than 3% and all the

recovered symbols in each seen landmark can be correctly recognized. Part of simulation results

in the location determination stage are illustrated in Table 1.

Table 1. The experimental results of the local location determination with respect to different landmarks in the simulated environment.

Landmark Real Location Computed Location Error Rate (%)

No. (xc, Yc, 2,) (xc, Yc, zc) on Average

1 (110,50,107) (-112.5,49.6,106.1) ItO.9%

2 (105,70,150) (105.5,71.2,148.9) ztO.9%

3 (-20,55,90) (-19.6,54.6,91.2) zt1.2%

4 (155, -90,110) (155.8, -88.8,112.2) +zl.l%

5 (86,45,70) (87.1,44.5,71.2) f1.4%

I- ~-6 1 (90,40,152) 1 (88.7,40.4,150.2) 1 ztl.Z% I 7 1 (70,-60,79) 1 (68.7,-59.3,78.1) 1 f1.4%

8 1 f80.66.60) t (80.9.64.8.58.9) 1 f1.6%

I 9 1 (-110,46,57) 1 (-107.2,44.5,55.8) 1 zh2.6% 1 IO- ~7 (1 ~~ 70,-50,110) 1 (-71.1,-50.2,111.2) 1 z&1.0% I

Besides, to reduce the measurement error caused by quantization, the accuracy of subpixel

level has been considered. To achieve this, we use line coefficients to calculate the coordinates of

corresponding vertices. The line coefficients are derived by the least-square-error method. In real

environments, Landmarks with various shapes and symbols are needed for different purposes.

Since the shapes and symbols (i.e., isolated single-loop patterns) can be recognized by the same

method described in Section 2. An intelligent symbol extraction method will increase the feasi-

bility in constructing the sentence. In addition, more information may be required to be depicted

on each landmark. Therefore, lower level grammars and automata coupled with more intelligent

understanding algorithms may be necessary.

In this paper, only local information (including Landmark understanding and relative location

determination between the seen Landmark and the robot) has been used in robot guidance (e.g.,

path planning). However, if other intelligent techniques, which can synthesize the information

seen and understood, are developed and integrated, the robot can derive the global information

(such as build a map) about the working space. The robot, thus, can have the ability of auto

matic learning in a complex and unfamiliar environment. With this ability, searching multiple

landmarks at one time becomes possible.

In these experiments, time was largely consumed in the image processing part. This is because

the image processing was implemented by software. If it is implemented by hardware, the ap-

plication of this approach will become more feasible. To assure the stability of automation and

location accuracy, the shapes of Landmarks and should not have sharp corners.

6. CONCLUSIONS

In conventional methods, a preinstalled map or predefined path is required for a robot navi-

gating in its working space. In this approach, a novel method to robot guidance by landmark


tracking in an unknown environment where the mobile robots have no information concerning

its working place is presented. Only a monocular image of the landmark taken by the robot is

required. Techniques of image processing and pattern recognition are integrated to resolve the

problems of landmark tracking and understanding. Experiments simulating a practical environ-

ment are performed. The results confirm that the specific landmarks can be automatically found

and the depicted information can be extracted and understood correctly by this method.

REFERENCES

1. R.L. Madarasz, L.C. Heiny, RF. Cromp and N.M. Masur, The design of an autonomous vehicle for the disabled, IEEE J. Robotics and Automation HA-2 (3), 117-126, (1986).

2. J.W. Courtney and J.K. Aggarwal, Robot guidance using computer vision, Pattern Recognition 17, 585592, (1984).

3. H.L. Chou and W.H. Tsai, A new approach to robot location by house corners, Pattern Recognition 19, 439-451, (1986).

4. I.J. Cox, Blanche: An autonomous vehicle for structured environments, In Proc. of IEEE ht. Conj. on Robotics and Automation, pp. 978-982, (1988).

5. T. Tsukiyama and T.S. Huang, Motion stereo for navigation of autonomous vehicles in man-made environments, Pattern Recognition 20 (l), 105-113, (1987).

6. P. Ku and W.H. Tsai, Model-based guidance of autonomous land vehicle for indoor navigation, In Proc. of 1989 Workshop on CVGIP, R.O.C., pp. 165-174, (1989).

7. H.S. Dulimarta and A.K. Jain, Mobile robot localization in indoor environment, Pattern Recognition 80 (l), 9Q-111, (1997).

8. J.D. Lee et al., Determining 3-D location parameters of a cylindrical object, Mathl. Comput. Modelling 20 (3), 89-106, (1994).

9. T.Y. Zhang and C.Y. Suen, A fast parallel algorithm for thinning digital patterns, Commun. ACM 27 (3), 236-238, (1984).

10. R.O. Duda and P.E. Hart, Use of the Hough transformation to detect lines and curves in pictures, Commun. ACM 17, 11-15, (1972).

Il. J.D. Lee, C.H. Chen and J.Y. Lee, Determining location and orientation of a labeled cylinder using point-pair estimation algorithm, Intern. J. of Pattern Recognition d Artificial Intelligence 8 (l), 351-371, (1994).

12. J.D. Lee et al., Determination of robot location using generalized cylindrical object shapes, Computers Math. Applic. 28 (8), 87-98, (1994).

13. J.D. Lee, Recognizing one-DOF industrial tools using invariant moments, Muthl. Comput. Modelling 23 (5), 17-23, (1996).

14. P.H. Swain and H. Hauska, The decision tree classifier: Design and potential, IEEE ‘Ibuns. Geosci. Electron. GE15, 142-147, (1977).

Indoor robot navigation by landmark tracking

Documents

Transcript of Indoor robot navigation by landmark tracking