26th International Symposium on Automation and Robotics in Construction (ISARC 2009)

357

Application of Appropriate Technologies to 3D Local Terrain Modeling in Real-time for Intelligent Excavating System (IES)

Byung-In Yu1, Hyun-seok Yoo2 , Young-suk Kim3, Jong-won Seo4 and Seungwoo Han5

1Graduate Student, Department of architectural Engineering, Inha University, Incheon, 402-751, South Korea; PH (82) 32-872-9757; FAX (82) 32-866-4624; e-mail: ybi2222@gmail.com 2Ph.D. Candidate, Department of architectural Engineering, Inha University, Incheon, 402-751, South Korea; PH (82) 32-872-9757; FAX (82) 32-866-4624; e-mail: hsyoo.inha@gmail.com 3Associate Professor, Department of architectural Engineering, Inha University, Incheon, 402-751, South Korea; PH (82) 32-860-7593; FAX (82) 32-866-4624; e-mail: youngsuk@hotmail.com 4Associate Professor, Department of Civil Engineering, Hanyang University, Seoul, 133-791, South Korea; PH (82) 2-2220-1482; FAX (82) 2-2220-4480; e-mail: jseo@hanyang.ac.kr5Assistant Professor, Department of architectural Engineering, Inha University, Incheon, 402-751, South Korea; PH (82) 32-860-7595; FAX (82) 32-866-4624; e-mail: shan@inha.ac.kr

Abstract Automation in construction is considered as one of solutions to fundamental problems which the

construction industry has confronted. The development of real-time 3D local terrain modeling is the core methodology of achievement of IES (Intelligent Excavating System) which is a mega-size research concerning automation in construction. Stereo vision technique was selected as an appropriate method to achieve 3D terrain modeling in real-time. The objective of this study is to suggest a conceptual method by use of the stereo vision and assess feasibility of practical application under various conditions of construction sites. Bumblebee XB-3 is adopted as the stereo vision system in a prototype of a 3D terrain modeling system. This study is to investigate fundamental theories of the stereo vision system and to conduct tests in indoor and outdoor environments for determining appropriate detection range, finding installation location of cameras, doing visual assessment of the technical appropriateness, and checking reliability of 3D visual data. These decomposed tasks present valuable information for establishment of the conceptual method and improvement on the limitations of the use of stereo vision system for 3D terrain modeling in real-time. Keywords: automation, stereo vision, 3D, terrain model

1. Introduction

Many construction planners and site engineers have regarded enhancement of construction performances by high construction productivity and low unit costs as one of crucial factors for determining a success of construction projects. The shortage of experienced laborers, raise of labor costs, and debasement of construction qualities make construction industry in Korea face serious difficulties in achievement of high construction performances. Construction industry has tried to use emerging technologies that are capable of providing excellent construction performances. These efforts are especially witnessed in a field of construction equipment. A research project titled “Intelligent Excavating System (IES)” that is one of research projects supported by Korean government started with professionals in industries and researchers in academic institutes in Korea at 2007 is currently conducting (Ahn et al. 2008; Yu et al. 2008).

Stereo vision technique was selected as an appropriate method to achieve 3D terrain modeling in real-time in previous researches (Ahn et al. 2008; Yu et al. 2008). The objective of this study is to suggest a conceptual method by use of the stereo vision and assess feasibility of practical application under various conditions of construction sites. In this study, Bumblebee XB-3, a state-of-the-art stereo vision system, is adopted as the stereo vision system which is used in a prototype of a 3D terrain modeling system. In accordance with the goal of the research, this study is to investigate fundamental theories of the stereo vision system and findings by past researches related to this study. This study is to conduct tests in indoor and outdoor environments based on theoretical reviews. Indoor test allows us to determine an appropriate detection range by the stereo vision system and gives us basic information about installation location and

installation anof a conceptutechnical apprtest, another osystem with cpresent valuablimitations of

2. Literature

Stereo visacquisition, ocameras placi(Yu et al. 20estimate of thorizontal locmatching poinpresents distacan be highly(Woodfill et a

Identifying

where, f: foc b: bas

d: di dl: thdr: th

As describthe left image

ngle of cameraual method forropriateness ooutdoor test, pcomparison ofble informatio

f the use of ste

Reviews

sion techniquobject identifiing separately,008). The pictthe distances cations and dnt of the objeance data fromy effective foral. 2007; Cho ag distance dat

z: distancecal length seline isparity betwe

d = dl - dr he distance frohe distance frobed the equatie and the right

Figu

Aut

as on a cabin or practical appof the stereo vprovides quanf the result by on for establisereo vision sys

e is currentlyication and tr, that is, thosetures taken bto objects. T

different distanect, a correspom cameras to r detecting thand Park 1997ta from the cam

z

e from the cam

en the distanc

om the left imom the left imion (1), comput image would

ure 1. Diagram

tomation and Ro

3

of an excavatoplication. Outdvision system fntified differenactual measur

shment of the stem for 3D te

y used in a bracking systeme are placed oby two camerThese camerances. These twonding pixel, athe object. Qu

he object and 7; Ahn et al. 20mera to the o

dbfz ×=

mera to the ob

ces from the le

mage to the obmage to the ob

utation of disd be a technica

m of basic ope

obot Application

358

or. It would bedoor test enabfor practical apnces of distancrement. Theseconceptual merrain modelin

broad range ms. This techon different hras are mergeas detect imagwo detected iand then creatuick measuremsegmenting t008; Yu et al. bject is derive

: Equation

bject point

eft image and

ject point bject point sparity based oal issue in stere

eration by ster

ns

e primary infobles to do visupplication to cce data of 3D e decomposed

method and imng in real-time

of various fihnique is imp

horizontal leveed and produges of an ideimages are mtes one mergement in distanthe object and2008).

ed from figure

(1)

the right imag

on the correspeo vision tech

reo vision tech

ormation for eual assessmentconstruction simages by the

d tasks mentiomprovement on

e.

ields such as plemented byel but the samuce a virtuallyentical object

merged based oed image. Thisnce yields thatd gauging its

e 1 and the equ

ge

ponding pixel hnique.

hnique

establishment t of the sites. Accuracye stereo vision

oned previousln the

terrain mody two identic

me vertical levy instantaneou

with differenon an identics merged imagt this techniqusize and shap

uation as belo

in two image

y n ly

del cal vel us nt cal ge ue pe

ow.

es,

26th International Symposium on Automation and Robotics in Construction (ISARC 2009)

359

In addition to computation of the disparity, it would also be critical issue to find an appropriate corresponding pixel in an identical object with two images. Two methods have been introduced on the market: 1) feature-based method, and 2) area-based method to resolve this difficulty.

Primary conditions for achieving the objective of this study is to detect a terrain image in real-time without any interruption of equipment operation. One cycle of excavating is composed of “scooping out” and “unloading the earth”. It generally takes 10 seconds for completing one cycle according to several equipment guide books and site observation. This information notifies us that all procedure including detecting and image processing would be completed within 10 seconds. Detecting terrain image is another limited condition in this study. Detection by cameras would finish as fast as it can so that it would not give any interruption during scooping and unloading the earth on the tray of trucks. These two limitations described previously would be resolved by technical specifications of state-of-the-art stereo vision system. The emerging technique of stereo vision system provides the technical solutions in that interpreting visual data to 3D from distance data takes generally 1 ~ 2 seconds and detecting the image takes milliseconds.

3. Conceptual Method of Stereo Vision for 3D Terrain Modeling

Appropriate conceptual method of stereo vision system requires that detection range of cameras and horizontal excavation range would be primarily determined. Careful study on installation of stereo vision cameras follows. These requirements make this study investigate the detection range of cameras by indoor tests and the excavation range by construction equipment references. The investigation allows this study to determine the appropriate installation location of the stereo vision cameras.

3.1. Estimation of detection angle Available detection range of the stereo vision camera is delivered by an indoor test. According to the

technical specification of Bumblebee XB-3, available angles are vertical viewing angle of 49.5 degree and horizontal viewing angle of 66 degree. Indoor test is required so that appropriate viewing angle for use in excavation operation. The system created 3D images based on two images captured by cameras which were located in 2 meter distance from a screen. The size of projected area in screen was 2.28 meter in width and 1.75 meter height. This result and proportional computation informed that appropriate angles by the stereo vision system were vertical viewing angle of 47 degree and horizontal viewing angle of 59 degree.

3.2. Installation of the stereo vision system An excavator which is under development as a prototype shows 8.2 meter of maximum reach and 5.0

meter of maximum depth below ground. Previous study (Seo et al. 2008) indicated that appropriate horizontal excavating range was 3.4 ~ 7.67 meters when the specification data of the same excavator were delivered.

Location and mounted angle of the stereo vision system are determined by information derived from appropriate detection angle and horizontal excavating range delivered previously. On-site observation of excavating operation informed that the stereo vision system was installed on a cabin of the excavator to obtain effective detection with appropriate coverage of the excavated area. Specification of the excavator used in this study says that that height of prospective installation location on the cabin is 2.8 meter. Figure 1 illustrates basic simulation chart of the excavator for searching appropriate installation location of the stereo vision system.

Figure 2 indicates that the stereo vision system needs to be installed on the cabin with 2.8 meter height and 30 degree incline to cover 3.4 ~ 7.67 meters of appropriate horizontal excavating area.

4. Application Analysis by Field Tests

4.1. 3D terrain modeling There are a few of specific technologies such as Triclops SDK by Point Grey Research Inc., Open GL (Open Graphic Library) by Silicon Graphics Inc., and Microsoft Visual C++ compiler, version 6.0 by Microsoft Corp. which are utilized for 3D terrain modeling. Triclops SDK allows us to control the operation of stereo vision cameras and to transmit image data. It also includes conversion function from the image data to point cloud suing SAD (Sum of Absolute Difference) algorithm. Open GL makes the converted

point clou

functions by T

The imageon the image which is compimages, and aThe conceptuterrain image after matchin

4.2. Field tests Many field

vision systemsites are detec

ud by TriclopsTriclops SDK

Figu

es detected bycaptured by thposed with 3D

a depth map illual method of on the triggerg then is auto

d tests on varim for 3D terrai

cted in field te

Aut

SDK be illusK and Open G

Figure

ure 3. Diagram

y stereo vision he camera on

D terrain modlustrated as a

f stereo vision r signal by excmatically pres

ous constructn modeling. V

ests.

tomation and Ro

3

trated in the uGL.

2. Operation

m of functions

cameras showthe right loca

del display, 2Dgrey scale indsystem condu

cavating systemsented on the

tion sites are cVarious shapes

obot Application

360

user interface.

chart of the e

s by Triclops S

w 640× 480 reation. Figure 4

D stereo imageex which is coucted by this sm automatical3D modeling

conducted to as of terrain an

ns

Figure 3 pres

excavator

SDK and Ope

esolutions in c4 shows a 3D e map created onverted fromstudy enables lly or by manudisplay viewe

assess the pracnd objects whi

sents basic dia

en GL

color and are mmodeling dispby matching t

m the computethe camera to

ually. The merer.

ctical applicatiich may exist i

agram of

merged based play viewer two captured ed distance dato detect the rged image

ion of stereo in constructio

ta.

n

The stehorizon. Tfield tests.developedconceptua

Figuresexisting inFigures 4 aexcavating

Profesrange of ocorrespon

Figure 5

26th Internatio

ereo vision syThe images is d Basically, all

d in this study.al method of ss 5 and 6 show

n constructionand 5 are asseg works. sionals agreed

operation tolernding pixels in

. 3D terrain im

onal Symposium

F

stem is moundetected with captured imag. Consequentlystereo vision sw 3D terrain i sites, respecti

essed and dete

d that most 3Drance of excavterrains while

mages of vario

m on Automati

Figure 4. 3D m

ted on a tripo640× 480 res

ges are convery, assessment ystem feasibleimages of variively. All 3D i

ermined wheth

D images showvation. This fae detecting var

ous shapes of

ion and Roboti

361

modeling displ

od of 2.8 metesolutions in corted to 3D imaof noise cond

e or improvedious shapes ofimages converher these are s

wed appropriaavorable test rrious patterns

terrain Figu

ics in Construc

lay viewer

r height and 3olor. Total 98 age via a speciditions and prd further. f terrain and wrted from the sufficiently app

ate accuracy wesult is due to in terrain.

ure 6. 3D terra

ction (ISARC 2

30 degree incliimages are capific display sys

rocessing time

with various tycaptured imagpropriate to b

with considerato easiness of fi

ain images wit

009)

ine from the ptured by totastem which w

e makes the

ypes of objectsges shown in

be applied in

tion of a broadinding

th various obje

al 5 was

s

d

ects

Unlike mo

determined nwould be resoshows the 3D

Visual asse

images createConsequentlybe compared

Figure 8 sh

As shown

meter height data by stereocamera to the

Comparisoindicates 0.05deviations anagraphs, table

As shown deviations of vision systemfrom the cam

ost 3D imagesot to be used olved by contr

D images of ter

Figure 7

essment showd by stereo vi

y, the accuracywith those byhows an accur

in figure 8, thand 30 degree

o vision systeme target is meaon of total 14

51 meter of analyzed in the 41 presents anain figure 9 anthe distance d

m has limitationmera due to dif

Aut

s, the images thdue to poor qrolling camerarrain with con

. 3D terrain im

wn in figures 4ision system ary test requires y actual measuracy test cond

Figure 8.

he accuracy tee incline detecm are compareasured manuall8 samples of t

n average devia4th and the 5th alysis results ond table 1, it ndata between tn for practicalfficulty in find

tomation and Ro

3

hat concretedquality caused a’s shutter spencrete hume p

mages with co

4, 5, and 6 showre perfectly mthat the distan

urement. ducted in cons

Accuracy test

st is conductects targets whied with those ly. the distance dation and 0.16field tests we

of deviation bynotes that morthe 3D imagesl application toding the corres

obot Application

362

d hume pipes aby not findin

eed and creatinpipes and surfa

oncrete hume

w proper resumatched with a

nce data in 3D

struction sites.

t of stereo vis

ed that the sterch placed randby actual surv

data between 368 meter of a mere illustrated iy distance fromre distance fros and actual mo detecting tersponding pixe

ns

and surface wag corresponding noise remoace water in p

pipes and surf

ult in accuracyactual terrain inD images by st

.

ion system

reo vision systdomly within veying in that

3D images andmaximum devin figure 8. In m the camera

om the camerameasurement. Irrain where is l.

ater were capting pixels. Thi

oval algorithmoor quality

face water

y. It doubts whnformation ortereo vision sy

tem which is l2 ~ 11 metersthe distance f

d actual measuviation. Graphaddition to dand the targe

a to the target It explains thalocated with l

tured in are is limitation . Figure 7

hether 3D r not. ystem need to

located in 2.8 s. The distancfrom the

urements hs where eviation

et. has bigger

at the stereo long distance

o

ce

Di

5. Conclu

This stmodel in eis normallyapplicationchallengesexperimenassessed tefollows:

• Indexcand

• As in 3whimoshometmo

• In tto vandwelon exp

The prexcavator further stu

26th Internatio

stance range

3 ~5 5 ~7 7 ~9 9 ~11

usions and Fu

tudy introduceexcavation opy limited to stn of stereo viss which are totnts using stereechnical chara

door experimeavator. Exper

d 30 degree incresult of seve

3D images. A ite hole is witnre advanced

owed that averter. These devre. this study, thevisual assessmd actual measull-designed exvibration by a

perimented. rimary findingbut also a doz

udies for achie

onal Symposium

Figure 9. D

Table 1. CMaximudeviatio

0.0830.1360.1580.168

uture Works

ed the conceperation. The stationary objecsion system totally different

eo vision systeacteristics inclu

ent conductedrimental resultcline for acquieral field tests few of imagenessed due toalgorithm forrage deviationviations were g

e conducted tement of 3D imurement. Althperiments neean excavator a

gs and conceptzer, a loader aeving accurate

m on Automati

Deviation graph

Comparison rum ons d3 6 8 8

tual method ostereo vision scts in outdoor

o constructionwith past resem in construcuding limitatio

d to find apprts inform thatiring availablein constructio

es detecting coo mismatchingr noise removn of distance dgradually bigg

ests of accuramages and comhough these ted to be contand reliability

tual method suand other conse 3D terrain m

ion and Roboti

363

hs in the 4th a

esult of deviatAverage

deviations 0.034 0.050 0.068 0.065

of stereo visiosystem has beer environment

n operation forearches. Accoction sites in oons. These exp

ropriate installt the camera oe detecting areon sites, mostoncrete hume g two images. val. The accurdata between 3ger while the d

cy of 3D imagmparison assetests providedtinuously deveand accuracy

uggested in thstruction equip

model would b

ics in Construc

nd the 5th field

tions (unit: meStandardeviatio

0.02130.03360.04310.0484

n system to ben deployed ints or steady-str creating 3D rdingly, this st

order to achievperimental res

lation of stereon the excavateas. t targeted terrpipes and surIt, however, wracy test of t3D images anddistance from

ges by stereo essment of disd basic guideleloped. Accur

y of the acquir

his study woulpment. Basede conducted i

ction (ISARC 2

d tests

eter) rd on

V

3 6 1 4

e applied to cn broad rangetate objects in terrain modeltudy conducteve appropriatesults suggest c

eo vision systetor is located i

rain is well derface water hawould be ovethe distance dd actual measuthe camera to

vision systemstance data beline to the coracy and stabired 3D images

ld be applied td on findings inin IES researc

009)

Variance

0.0005 0.0011 0.0019 0.0023

reate 3D terras of area wherindoor. The

l would face med many e applications conclusions as

em on cabin oin 2.8 meter h

etected and creave a problemercome with udata of 3D imurement was 0

o the target be

m were limitedetween 3D imonceptual metility of this sys would be fu

to not only ann this study, h project.

ain re it

many

and s

of an height

eated m that use of mages 0.051 came

d only mages thod, ystem urther

n

Automation and Robot Applications

364

Acknowledgement

This research was supported by a grant from the research titled “Development of Intelligent Excavating System”, a part of Construction Technology Innovation Program(CTIP) by Ministry of Land, Transportation and Maritime Affairs(MLTM) of Korean government.

References

[1] Ahn, J. S., Yu, B. I., Kim, Y. S., and Han, S. W. (2008). “Emerging Technologies for Real-time 3D Terrain Modeling and Automatic Soil Volume Computation in Earthmoving Operation” Proc., 12th ICCCBE &2008 ICITC, 276.

[2] Cho I. S., and Park J. G. (1997). “Extraction of Depth Information using Correspondence Analysis in Stereo Vision.” Basic Science and Engineering, 1(2), 899-905.

[3] Lu, M., Chen, W., shen, X., Lam, H., and Liu, J. (2007) “Positioning and tracking construction vehicles in highly dense urban area and building construction sites.” Automation in Construction, 16(5), 647-656

[4] Woodfill, J. I., Buck, R., Jurasek, D., Gordon, G., Brown, T. (2007). “3D Vision: Developing an Embeded Stereo Vision System.” IEEE Computer.

[5] Yu, B. I., Ahn, J. S., Lee, J. H., Yu, H. S., Kim, Y. S., and Han, S. W. (2008). “Implementation of Real-time 3D Terrain Modeling in Automatic Earthmoving Operations.” Proc., 11th Symposium on Construction Robotics in Japan, 75-80.