The Keystone Fire Brigade 2003

Jacky Baltes and John AndersonAutonomous Agents LaboratoryDepartment of Computer Science

University of ManitobaWinnipeg, Manitoba

Canada R3T2N2Email: jacky,andersj@cs.umanitoba.ca

Abstract

The Keystone Fire Brigade is a robotic rescue teamthat has previously competed in competitions atRobocup (Fukuoka, 2002; Padua 2003), AAAI (Ed-monton, 2002), and IJCAI (Acapulco, 2003). The keyelements of our approach are an emphasis on vision,a fully autonomy solution, and an implementation oninexpensive robot bases. This paper describes the ver-sion of the team that competed at IJCAI in 2003. Weoverview the hardware employed, methods used for vi-sual processing, map-making and victim identification.We also describe the experences we had in the test do-main and offer some recommendations on future com-petitions.

IntroductionRobotic rescue is both a worthwhile application for ar-tificial intelligence and a challenge problem that allowssolutions to be compared in a controlled setting. Be-cause of the extreme difficulty associated with the prob-lem, most entries nowadays are teleoperated. This hasbeen the case since the earliest rescue competitions, andthe current competition at IJCAI was similar.

We believe that, ultimately, autonomous process-ing will be of great importance in robotic rescue.In rescue settings, issues such as operator fatigue,lack of situational awareness of the operator, cogni-tive load on the operator, and the number of individu-als an operator can control in real time (Casper 2002;Casper & Murphy 2002) all place limitations on humancontrol.

We also believe that a focus on autonomous process-ing is important from the standpoint of truly advancingartificial intelligence: the reason that most entries inthe competitions are teleoperated is precisely becauseautonomous control is still very primitive in an areasas complex as robotic rescue, and avoiding the use ofautonomous control mechanisms does not do anythingto improve this technology. We believe that once au-tonomy has improved, teleoperation can be added tofill in the gaps where a human operator can be helpfulto the situation without being overwhelmed. We have

Copyright c© 2003, American Association for Artificial In-telligence (www.aaai.org). All rights reserved.

thus been focussing on autonomy as a key focal pointin our work in robotic rescue.

This paper describes the Keystone Fire Brigade theUniversity of Manitoba’s entry in the AAAI RoboticRescue Competition. Our approach embodies severalprinciples we believe will ultimately be important insuccessful robotic rescue problems: autonomy, a multi-agent perspective, and parsimony.

Many problems are naturally amenable to solutionthrough a collection of distributed agents. A multia-gent approach to robotic rescue is a natural one: itis the approach that is already taken by humans inthis problem; it allows the power of numbers to dealwith the geographic distribution of the problem; andthe presence of multiple agents allows any individualin this domain to be considered more expendable thanwould otherwise be the case. Because of this, we wantto design agents that are meant to operate in groups.

Parsimony is an important goal in all robotics appli-cations. Any additional feature has a cost, both finan-cially and in terms of computing power and other localresources, and reliability. If a feature is not necessaryto solve the problem, eliminating it provides more re-sources to those features that are necessary. We believe,like others (Balch & Arkin 1994) that the addition ofany component should be carefully considered. Costmust be balanced with the efficacy of the equipment tothe improvement of overall system performance. Par-simony is also one of the reasons that a multi-agentapproach is important - by taking the same resourcesand spreading them among a number of simpler agents,the interaction of these over a geographic area can dealwith the problem better than a single, highly complexagent. The more parsimonious the agent design, themore expendable any individual robot can be consid-ered as well.

The remainder of this paper details the hardwareplatforms employed in this year’s Keystone Fire Brigadethe use of optical flow for localization and mapping, anddiscusses our experiences in this year’s competition.

Description of the Robot HardwareWe have two primary motivations for hardware designin rescue robots. The first of these is reliance on ex-

From: AAAI Technical Report WS-03-01. Compilation copyright © 2003, AAAI (www.aaai.org). All rights reserved.

Figure 1: Hummer: Robot platform using a Toy Carbase

tremely simple robotic platforms. Ultimately, the taskof robotic rescue will benefit from implementation on in-expensive platforms, since larger teams can be affordedand individual robots can be viewed as being expend-able in a dangerous environment. Our motivation inusing simple hardware, however, is to force reliance onmore robust and versatile control methodologies. A sys-tem relying heavily on accurate odometry, for example,is severely crippled under conditions where odometry isinaccurate. A system that does not assume the avail-ability of accurate odometry readings, however, will stilloperate under such conditions, as well as in conditionswhere odometry can be useful.

The second major design factor is an emphasis onvision. Each of our robots employs simple low-powerCMOS cameras or webcams, and has enough local pro-cessing power for vision and robot control. Vision is theonly sense employed by all of our robots.

In the 2003 version of the Keystone Fire Brigade therobots were based on two simple platform types. Fig-ure 2 illustrates the first type, a small plastic remotecontrolled toy car (Hummer) from Nikko. While notpossessing a true four wheel-drive, it affords a reason-ably high ground clearance and stability over rough ter-rain. The hummer carries an Eyebot controller (Braunl1999), which consists of a 35 MHz 68332 processor with2 MB of static RAM. The processing speed is verylow by today’s standards (supporting approximately 1fps vision along with the processing necessary for au-tonomous control), but it is comparatively cheap andalso supports a direct connection between a CMOScamera and the processor itself. Furthermore, the Eye-bot controller provides the necessary interface to con-nect motors, servos, gyroscopes, and many other sen-sors directly to the controller.

Figure 1 illustrates our new base type, a custom de-sign built from simple corrugated posterboard, with

Figure 2: Rescue2: Robot platform using a simple cor-rugated board base (front and rear views)

two servos attached to drive the left and right wheels,and an integrated webcam for vision. The servos weremodified by cutting out the motion stop and thus pro-vide relatively nice velocity control. The platform car-ries a much more powerful processer than the otherbase, a 300MHz Intel Xscale board, with 64MB RAM.This configuration is powerful enough to support au-tonomous control sofware along with 5 fps vision. Inaddition to these autonomous robots, we also deployeda teleoperated base (A PT cruiser approximately threetimes the size of the hummer depicted above), con-trolled via a small laptop with wireless Internet access.Our focus is mainly on autonomous control, but wewanted to take the opportunity to gather some expe-rience with the some of the issues facing teleoperatorsin an environment such as RoboCup Rescue .

None of these bases would be suitable for real roboticrescue, mainly do to their fragile nature. However, theyare sufficient to demonstrate the approaches we believeare useful for future robotic rescue work, and can navi-gate within the yellow zone of the NIST testbed (Jacoff,Messina, & Evans 2001). Our intent is to demonstratethe use of vision and autonomous control this task, andto further applied research in these areas in robotic res-cue, rather than to tackle the less stable terrains in theorange and red arenas. In future, we intend to buildmore robust platforms employing the VIA mini-ITXboard.

Visual Processing in the Rescue Domain

The greatest challenge on platforms such as those em-ployed by the Keystone Fire Brigade is the design andimplementation of pragmatic algorithms for intelligentvisual processing, and the adaptation of these to the thelow frame rates that are achievable using the embeddedsystems driving the robots.

The use of vision as the only form of sensing requiresthat vision not only be used to identify victims, which isthe primary use of vision for most teams, but also to al-low the robot to localize and map the environment. Thefollowing subsections describe our methods for dealingwith each of these elements.

Ego Motion EstimationIn order for a robot using only vision to map an en-vironment, its progress through the environment mustbe measured by vision rather than by odometry, sonar,laser, or a combination thereof. This is the problem ofego motion estimation, which requires the robot to es-timate its movement in terms of distance and angle byexamining the differences between visual frames. Thisis an extremely difficult problem, mainly because of theamount of confusing or conflicting information in visualframes. Furthermore, in the rescue environment itself,there is significantly less structure than that availablein other common robotic environments. In soccer forexample, the detection of walls can be done by plot-ting lines from the bottom of the image up to a white

Figure 4: Detecting Lines in the Rescue Environment

colour change (figure 3), and then calculating the wall’slikely position based on the end points of the plottedlines (Baltes 2002). While some consideration has tobe made for the portions of the wall that are not visi-ble (i.e. obscured by the ball or other players), there isgenerally enough information in most frames to plot thewall’s position quickly. In more general environmentssuch as the rescue domain, however, the structure thatcan be assumed in simpler environments does not exist.Instead we must employ more sophisticated edge detec-tion algorithms (upper part of Figure 4) followed by aline detection algorithm (Hough transform), illustratedin the lower part of Figure 4.

While every recognized line in the rescue domain isnot a wall, we can still employ these regular features todo ego-motion estimation. Our approach uses the opti-cal flow between images to estimate the motion of therobot (Figure 5), and also to indicate a lack of motionon the part of the robot (i.e. detecting when the robotis stuck).

If a recognizable pattern (a set of lines, intersections

Figure 3: Detecting Walls in Soccer

Figure 5: Ego Motion Detection from Visual Frames

between lines, etc.) can be recognized in two differ-ent frames, we can compute the change in angle anddistance on the part of the robot that produced thatchange in visual reference point. Note that we assumethat the line is at a constant height (e.g., a line on thefloor).

Figure 6 shows the geometry of the situation. As-suming that the robot can determine the angle betweenitself and a line, then the change in orientation δθ canbe easily computed by the difference in angle.

In the case of differential drive robot, this allows oneto calculate the difference between the right and legftwheel velocities (assuming the width of the robot isknown). In the case of a rear-wheel or front wheel drivecar, the steering angle can be computed (assuming theaxle distance of the robot is known).

The change in angle of the line does not allow one tosolve for right and left wheel velocities (in the case of adifferential drive robot), or the linear velocity (in caseof a car-like robot). However, given that the robot canalso determine the distance between the robot and thewall, solutions to the kinematic equations can be foundand the motion can be recovered. The geometry andsolution is shown in Fig. 7.

To determine if the robot is blocked or otherwisestuck in one position, the image is broken up into 16equal sized sub-images. Of these, only the bottom 8sub-images need to be considered - everything else is

Figure 6: Determining the change in angle from twovisual reference points

Figure 7: Determining the distance travelled from twovisual reference points

further away and would not be likely to provide usefulfeedback regarding the motion of the robot. The sys-tem then computes the differences between the currentand the previous image for each quadrant. The colourdifference is defined as the sum of the absolute valueof the differences in the red, green, and blue channels.If the difference in a sub-image is above a threshold,the quadrant is marked. If there are more than eightmarked sub-images and the motors were turned on inthe previous time step, than the system signals that therobot is stuck or blocked. We break the image into sub-images to allow for localized changes due to the motionof some other agent or other external motion in theimage, to try to limit the number of false positives.

Simultaneous Localization and Mapping(SLAM)Just as we rely solely on vision for localization throughego-motion detection, we also rely on vision for con-structing a map while loczalizing through opticalthrough. This results in a chicken-and-egg problem:While localization becomes easier as maps are created,we must begin out of necessity with no valid initial map,making localization difficult, which in turn complicatesthe process of constructing an accurate map.

Our approach to building a map involves the con-struction of sets of local two-dimensional maps. Therobot makes a map of the immediate area around it-self (1m x 1m), storing the map as an occupancy gridsuch as that shown in Figure 8. In this map, the robothas plotted an obstacle (in black) and an open area(in green), while the white areas represent unexploredareas. These local maps are separated by longer traver-sals (a random walk in a particular direction) and arelinked together as topological maps. The distance andlength of a traversal serves as a link between maps, butas new features are detected earlier maps are studiedfor these features, allowing local maps to overlap. The

Figure 8: Local Map

smaller size of the local maps allows the local area tobe explored quickly, and the traversals between allowthe robot to map different areas without errors in onemap compounding to cause problems in later maps.

We plan in the future to extend this work to includea case-based reasoning system that employs typical sen-sor readings (especially “typical” images of the area) toidentify areas and to connect them via topologial paths.

Detection of VictimsWhile the system for awarding points is strongly ori-ented toward the use of multiple forms of sensing, mostvictims in the NIST testbed are reasonably easily iden-tified visually. While there is much current work on thevisual detection of victims, we are attempting to workwith a reasonably simple, pragmatic approach to thisdifficult problem.

Our victim detection approach uses both colour aswell as shape information. Flesh colored spots aremarked as possible victim locations (these algorithmswere trained beforehand on flesh patches of team mem-bers in the lighting used in the test arena).

We have developed a 12 parameter colour modelwhich uses Red, Green, and Blue as well as three dif-ference channels: Red - Green, Red - Blue, and Green- Blue. The differences are included in this parametermodel because of their tendency to remain relativelyconstant in different views of objects of the same colourdespite of lighting variations over a field. This approachis the same used in our Doraemon vision server (An-derson & Baltes 2002) and has proven itself in years ofrobotic soccer competition.

Currently, we use a simple blob detection scheme.The system signals that it has found a victim by cal-culating the apparent size and aspect ratio of a skincoloured blob. If these parameters are within limits, thesystem signals the detection of the victim by perform-ing a series of 360 degree turns at the current location.It then continues to search the environment.

Discussion

This was our second year in the competition, and wewere frustrated by a number of hardware failures. Morerobust platforms are required even to successfully op-erate in the Yellow arena of the NIST testbed, and weintend to develop a harder-wearing platform (with thesame goal of parsimonious construction that has char-acterized our work to date).

Beyond the issue of hardware failures, the methodswe are using resulted in low scores in general. While weintend to make every effort to improve the performanceof our team for next year, performance using these tech-niques will continue to be poor in the foreseeable future:ego-motion estimation and mapping through the anal-ysis of optical flow in a domain such as robotic rescueare both extremely hard problems, and we do not ex-pect perfect solutions to these problems to magicallyappear. However, furthering work toward the solutionof hard problems such as these is one of the goals ofrescue competitions. For next year, we intend to workon improving the frame rate, which should result in cor-responding improvements in mapping and localization.We also intend to modify our algorithms to focus morecomputational effort on those parts of the image thatare likely the most important: in front of the robot andat the horizon.

Our experiences this year and last have led us to makea number of recommendations for future competitions.First, in both the IJCAI and Robocup competitions, itwas evident that the multiplier currently included forthe use of multiple sensing methodologies has too stronga weight. It is too easy for a teleoperated team usingmultiple forms of sensing to discover a victim visually(i.e. victim detection is essentially complete at thatpoint), and then simply employ one sensor after an-other in the same position to receive additional pointsfor work that was already performed once using the op-erator’s vision. This is analogous to the kind of activitythat took place last year when there was a multiplierfor multiple robots, and serves as a similar argumentto removing this multiplier as well. Multiple sensoryapproaches should allow robots to find more victims,not gain additional points from the same victim. Thetags that were used to label victims were an interest-ing variation on previous competitions, and the use ofthese did provide some incentive to explore the areaaround victims further after visually identifying them.Rules should also be flexible enough to allow for fullyautonomous approaches to have some chance at com-peting. Revising the rules committee, which has been

inactive for some time, would be helpful in dealing withthese and other issues.

ReferencesAnderson, J., and Baltes, J. 2002. The Dorae-mon User’s Guide. Department of Computer Sci-ence, University of Manitoba, Winnipeg, Canada.http://robocup-video.sourceforge.net.Balch, T., and Arkin, R. C. 1994. Communicationin reactive multiagent robotic systems. AutonomousRobots 1(1):27–52.Baltes, J. 2002. Localization for mobile robots usingstraight lines. In Proceedings of Seventh InternationalConference on Control, Automation, Robotics and Vi-sion. to appear.Braunl, T. 1999. Eyebot: A family of autonomousmobile robots. In Proceedings of the 6th Interna-tional Conference on Neural Information Prococessing(ICONIP99), 645649a.Casper, J., and Murphy, R. 2002. Workflow studyon human-robot interaction in usar. In Proceedings ofthe IEEE International Conference on Robotics andAutomation, volume 2, 1997–2003. Washington, DC:IEEE.Casper, J. 2002. Human-robot interactions duringthe robot-assisted urban search and rescue responseat the world trade center. Master’s thesis, Universityof South Florida.Jacoff, A.; Messina, E.; and Evans, J. 2001. Experi-ences in deploying test arenas for autonomous mobilerobots. In Messina, E., and Meystel, A., eds., Proceed-ings of the 2nd Performance Measures for IntelligentSystems Workshop (PerMIS), NIST Special Publica-tion 982, 87–95. National Institute of Standards andTechnology.