3-D Object Reconstruction Using Spatially Extended Voxels andMulti-Hypothesis Voxel Coloring

EckehardSteinbachandBerndGirodInformationSystemsLaboratory

StanfordUniversityEmail:

�steinb,bgirod� @stanford.edu

PeterEisertandArnulf BetzTelecommunicationsLaboratory

Universityof Erlangen-NurembergEmail:

�eisert,betz� @LNT.de

Abstract

In this paperwedescribea voxel-based3-D reconstruc-tion techniquefrom multiple calibratedcamera views thatmakesexplicit useof thefinitesizefootprintof a voxelwhenprojectedinto theimageplane. Wederivea classof compu-tationallyefficientaxis-alignedvolumetraversalordersthatensure that a processedvoxel cannot occludepreviouslyprocessedvoxels. For each view, one out of 79 differentcasesof volumetraversal is identifieddependingon therel-ativepositionbetweencamera andvoxelvolume. Viewsbe-longingto thesamevisibility classcanbeprocessedsimul-taneously. Our voxelcoloring strategy is basedon a colorhypothesistestthatensurestheconsistencyof theprojectedreconstructionwith theoriginal images.A surfacevoxellistis constantlyupdatedduring reconstructionensuringthatonly a minimumnumberof voxelshasto beprocessed.Ex-perimentalresultsthat compare the reconstructionqualityfor voxelswith and without spatial extent underscore theconclusionthat it is worthwhiletakinginto accounttheex-act footprintof theprojectedvoxels.

1. Introduction

The automaticacquisitionof photorealistic3-D com-putermodelsfrom many cameraviews is a very active re-searchareawith applicationsin virtual reality and multi-media. The large body of work devoted to this problemcan basicallybe divided in two different classesof algo-rithms. Thefirst classof 3-D modelacquisitiontechniquescomputesdepthmapsfrom two or moreviewsof theobjectandthenregistersthedepthmapsinto a single3-D surfacemodel. The depthmaprecovery often relieson sparseordensematchingof imagepointswith subsequent3-D struc-tureestimation[1, 2, 3] or is supportedby additionaldepthinformation from rangesensors[4, 5]. Other approachesarebasedon volumeintersection,andareoftenreferredtoas shape-from-silhouettealgorithms[6, 7, 8]. The objectshapeis typically computedastheintersectionof theoutlineconeswhich areback-projectedfrom all availableviews ofthe object. This requiresthe reliableextractionof the ob-

ject contourin all views which restrictstheapplicability tosceneswherethe objectcanbe easilysegmentedfrom thebackground.

Recently, techniquesfor 3-D objectreconstructionfrommultiple calibratedviews have beenproposedthat com-binetheadvantagesof thetwo aforementionedclasses.Us-ing a volumetric representationof the objects,volumeel-ements(voxels) arecoloredby comparingthe pixel colorof the projectedvoxel in all views wherethe voxel is visi-ble [9],[10],[11]. Thesetechniquesavoid imagecorrespon-denceproblemsby working in a discretizedscenespacewhoseelementsaretraversedin a fixedorderduringrecon-structionfor correctvisibility handling. In [11] voxelsareassumedto be3-D pointswithout spatialextent leadingtolow computationalcomplexity at the expenseof shining-throughartifactsin the 3-D computermodels. In [9] and[10] thefootprintof thevoxelsis approximatedby asquaremask. During volumetraversal,voxels in the samevoxellayer are assumednot to occludeeachother. The voxelcoloring stategy in [9],[10] assignsmeancolor valuestothenon-transparentvoxelswhile thevoxel coloring in [11]is basedon a multi-hypothesistestof the projectedvoxelmodel.

In this paperwe combinethe explicit considerationofthe finite extent of the volume elementswith the multi-hypothesisvoxel coloring strategy in [11]. In comparisonto [9] and [10], it is not assumedthat voxels in the samevoxel layercannotoccludeeachother, leadingto exactvis-ibility handlingduringvolumetraversal.

2. Voxel Projection

The projectionof voxels without spatialextent into theimageplaneleadsto a singlepoint. In comparison,for ex-tendedvoxels a small footprint in the imageplane,poten-tially coveringmorethanonepixel, is obtained.Theexactfootprintof avoxelhasto considerits cubicshape.Thepro-jection leadsto a convex 2-D polygonin the imageplane,either4-sidedor 6-sided. In orderto obtainthe exact out-line of theprojectedvoxel we first projecttheeightcorner

pointsinto the imageplaneandthencomputetheir closedconvex hull. Oncethe voxel footprint hasbeencomputed,

Figure 1. Voxel footprint and pix el contrib ution.

it canbe determinedwhich pixels fall inside the polygon.Fig. 1 shows anexamplewherethecontribution of a pixel,measuredasthepercentageof theareaintersectingwith thepolygon,is representedasgrayvalues.Theintersectingareafor the borderpixels is determinedusingpolygonclippingof thevoxel polygonwith respectto therectangularpixel.

3. 3-D Object Reconstruction

Thefirst stepof theproposedreconstructionalgorithmisto definea volumein the referencecoordinatesystemthatenclosesthe3-D objectto bereconstructed.Thevolumeex-tensionsaredeterminedfrom thecalibratedcameraparam-etersandits surfacerepresentsaconservativeboundingboxof theobject.Thevolumeis discretizedin all threedimen-sionsleadingto an array of voxels with associatedcolor,wherethepositionof eachvoxel in the3-D spaceis definedby its indices����������� . Initially, all voxelsaretransparent.

3.1. Hypothesis Generation

In the secondstepof the proposedreconstructionalgo-rithm, color hypothesesareassignedto eachvoxel of thepredefinedvolume. The ���� hypothesis�������� for a voxel� ����� with voxel index ����������� is

� �������� � �!��"$# �&%'#(� �*)+��"�#(��%,#(� �.-/�0"�# �&%'#&� � � (1)

with ��"$# �&%'#(� beingthepixelpositionof theperspectivepro-jectionof thevoxel center��1 � ��2 � �*3 � � into the 4(��� cameraview. � , ) , and - are the threecolor components.Theprojectionof thevoxel centerfor view 4 is obtainedas

"�# �65+7*8 1 � #3 � # �9%'# �:5+7*; 2 � #3 � # � (2)

with

�01 � #(��2 � #(�.3 � #&�=< �?>A@ �01 � �&2 � �*3 � �B<DCFE @0G (3)

>�@ and E @ are the object rotationand translationin view4 with respectto the referencecoordinatesystem.Thepa-rameters7 8 and 7 ; describethe camerageometryandthescalingthatrelatespixel coordinatesto world coordinates.

Hypothesis�������� is associatedto voxel� ����� if thepro-

jection of� ����� into at leastoneothercameraview HJI� 4

leadsto anabsolutedifferenceof thecolor channelsthat islessthanapredefinedthresholdKLNMBOQPRMTSBU,MTVW M OQP M STU M VYX L[Z\OQP]Z\STU^Z_VW Z OQP Z S�U Z V `a MBOQPRMBSBU,M�VW]M�OQPRM=STU,M�V X a ZbOQP]Z\STU^ZbVW+ZbOQP]Z*S�U^ZbV `cNMBOQPRMTSBU,MTVW M OQP M STU M VYX c[Z\OQP]Z\STU^Z_VW Z OQP Z S�U Z V dfe (4)

withg # ��"h�&%A� � � # �0"h��%��'Ci) # ��"h�&%��jCk- # ��"h�&%�� G (5)

Thenormalizationof thecolorcomponentsin (4) is usedtoincreasetherobustnessof thereconstructionalgorithmwithrespectto varying illumination conditions. For hypothesisgeneration,the finite sizeof the voxel footprint is not ex-ploited. Of courseit could be, but experimentsshow thatit is sufficient for this stepto usethe voxel center. Thisleadsto a smallernumberof color hypothesesthathave tobestored.Thevoxel neednot bevisible in all viewsduetoocclusionsandit mightnot bevisible in any view at all if itis insidetheobject.At thisstageof thealgorithmwedonotknow thegeometryof theobjectandcannotdecidewhethera voxel is visible. We thereforehave to remove thosehy-pothesesof theovercompletesetthatdo not correspondtothecorrectcolor of theobject’ssurface.

3.2. Consistency Check and Hypothesis Rejection

Thecolorhypothesesin Section3.1areextractedfrom 2or moreconsistentviews but might contradictotherviewswherethevoxel is visibleaswell. Wenow startto refineourvolumeby iteratingoverall views. We startfrom theouter-mostvoxel layerof thevolumeandremovevoxelsuntil the3-D shapeof theobjectis recovered.Thedecisionof whichcolor hypothesesto eliminatefor eachvoxel considersthefinite areaof theprojectedvoxel. Let l^m betheareaof theprojectionof a voxel obtainedwithout consideringocclu-sion and l � the areawhich is obtainedwhenconsideringocclusionby othervoxels. Thecolor testcriterion in (4) ismodifiedasfollowsn if the projectedarea l � is small, the consistency test

is notperformedsinceeitherthevoxel is very far fromthecameraor heavily occludedby othervoxels.n thethresholdK in (4) is modifiedaccordingto

K �po�q � �srt 5vut l �lwm � K (6)

which leadsto an increaseof the thresholdvalue forheavily occludedvoxelsof about50%.

Colorhypotheseshaveto begeneratedonly for thosevoxelsthatbecomevisibleduringreconstruction.Wekeeptrackofpotentiallyvisible voxelsby storingandupdatinga surfacevoxel list. Fig. 2 illustratesthe updateof the surfacelistafter removal of onesurfacevoxel. Only thosevoxels that

x y z { | } ~ z ~ � � � ~ � � � ~ � x y z { | } ~ y � � | � ~

z ~ � � � ~ | � | � � x y z { | } ~ y � � | � ~x ~ � ~ } � � � � ~ z � � � ~ �

x y z { | } ~ � � � ~ � � � � � x � � � ~ � � � ~ � z ~ � � � ~ � � � � ~ �

Figure 2. Surface list update after voxel remo val.

arevisible in a particularview have to be testedfor colorhypothesisconsistency. This requiresthe determinationoftheir visibility. Sincethe objectmodelconsistsof a struc-turedsetof voxels,a traversalordersimilar to the onede-scribedin [11] canbederived. Processingthesurfacevox-elsaccordingto this volumetraversalorderensurescorrectocclusionhandlingfor theparticularview.

We now derive the different volume traversal orderclassesthat can be identified when consideringall possi-ble mutualvoxel occlusion.Enlarging theboundingplanesof theobjectvolumein all coordinatedirectionsdividesthe3-D spaceoutsidethevolumeinto 26regions.Thesediffer-entregionsareclassifiedinto threecases:

1. Theclosestelementof thevolumeis a cornervoxel.

2. Theclosestelementof thevolumeis anedgevoxel.

3. Theclosestelementof thevolumeis a facevoxel.

Fig. 3 illustratesthesedifferentcases.Thesimplesttraver-

� � � � �� � � � �

� � � � �

Figure 3. Partitioning of the 3-D space into threediff erent cases.

salorderis obtainedfor case1 ascanbeseenfrom Fig. 4.The camerais representedby a sphere,the arrow pointsto the closestvoxel, and the numbersindicatethe traver-salorder. Thevoxel indices� , � , and runoverall definedvalues.Theclosestcornerof theboundingbox determines

Figure 4. Volume traversal order for case 1.

whether� , � or have to be incrementedor decremented.Theorderin which this hasto bedonedependson therela-tivepositionof thecamerawith respectto thecorner. Fig. 5illustratestheselectionof a particularorderin 2-D. A total

� � � �� � � � � � � � � �

� � � � � �

Figure 5. 2-D illustration of the inde x perm utationselection.

of rs� ��� differentpermutationsof the voxel indicescanbe identified. Sincethesepermutationsapply for all 8 cor-nersof theboundingvolumeweobtain ��� � �?¡ � differenttraversalorders.These48 casescorrespondto thetraversalordersin [11]. For case2 in Fig.3, theclosestvoxel is foundon anedgeof theboundingvolume.Here,

t � � tdifferent

index permutationsexist. Sincethesepermutationsapplyfor all 12 edges,we obtain

t � u t � t ¡ different traver-salorders.Thetraversalorderfor this caseis illustratedinFig. 6. Case3 in Fig. 3 leadsto thetraversalordershown in

Figure 6. Volume traversal order for case 2.

Fig. 7. For acamerathatlies in theinteriorof thevoxel vol-ume(e.g.,whenreconstructingaroom),adifferenttraversalorderhasto beused.Thisis illustratedin Fig.8 for onesliceof theobjectvolume. Thetotal numberof differenttraver-sal orderscannow be summarized.For the 8 cornersweobtain48 differentcases.The twelve edgesof the volumeadd24 casesandthe6 facesanother6 cases.Includingthetraversalorderfor thecamerain the interior of thevolumeweobtain ¡ ��C t ¡ C � C u �J¢�£ differentvolumetraversalordercases.

Figure 7. Volume traversal order for case 3.

Figure 8. Volume traversal order for a camera thatfalls inside the voxel volume .

4. Experimental Results

The following experimentillustratesthe reconstructionquality that is obtainedwhenconsideringextendedvoxelsandtheexactocclusionhandlingpresentedin this paper. A24view sequence( r�¤ t � t ��� pixels)of aplantisused.Fig.9shows two differentoriginal viewsof thesequence.Fig. 10comparesthereconstructionresultfor new viewsof theob-ject for the caseof point voxels (top) andextendedvoxels(bottom). New viewing positionsareselectedthat arenotpart of the original set of views. It can be seenthat thereconstructionquality considerablyimprovedfor extendedvoxels. Themain reasonfor this is the inherentdeficiencyin occlusionhandlingfor point voxels. For new viewingpositions,voxelsthatshouldbeoccludedmaybecomevis-ible producingtheshining-throughartifactsobservedin theupperimagesof Fig. 10. However, usingpoint voxelsde-creasesthe computationalcomplexity in our implementa-tion by anorderof magnitude.

Figure 9. Two original views of the plant sequence .

Figure 10. Reconstructed views of the plant. Theupper images sho w the results obtained when con-sidering point voxels. The lower images sho w theresults for extended voxels.

References

[1] P. Beardsley, P. Torr, andA. Zisserman,“3D ModelAcquisi-tion from ExtendedImageSequences,” Proc.ECCV’96, pp.683-695,Cambridge,UK, 1996.

[2] R. Koch,M. Pollefeys, andL. VanGool, “Multi ViewpointStereofrom UncalibratedSequences,” Proc.ECCV’98, pp.55-71,Freiburg, Germany, 1998.

[3] M. Pollefeys, R. Koch, M. Vergauwen,and L. Van Gool,“Flexible Acquisition of 3D Structurefrom Motion,” Proc.TenthIMDSPWorkshop’98, pp.195-198,Austria,1998.

[4] B. C. Vemuri, J. K. Aggarwal, “3-D Model Constructionfrom Multiple ViewsUsingRangeandIntensityData,” Proc.CVPR’86, pp.435-437,Miami Beach,1986.

[5] B. CurlessandM. Levoy, “A VolumetricMethodfor Build-ing Complex Models from RangeImages,” ACM Siggraph’96, pp.303-312,June1993.

[6] E. Boyer, “Object modelsfrom contoursequences,” Proc.ECCV’96, pp.109-118,Cambridge,UK, 1996.

[7] W. Niem, J. Wingberm̈uhle, ”Automatic Reconstructionof3D ObjectsUsinga Mobile MonoscopicCamera”,Proc.In-ternationalConferenceon RecentAdvancesin 3D ImagingandModelling, Ottawa,Canada,May 1997.

[8] R. Szeliski, “Rapid octree constructionfrom image se-quences,” CVGIP93, pp.23-32,July 1993.

[9] S.M.SeitzandC.R.Dyer, “PhotorealisticSceneReconstruc-tion by Voxel Coloring,” Proc. CVPR’97, pp. 1067-1073,PuertoRico,1997.

[10] K.N. KutulakosandS. Seitz,“A Theoryof Shapeby SpaceCarving,” Proc.ICCV ’99, pp.307-314,1999.

[11] P. Eisert,E.Steinbach,andB. Girod,”Multi-hypothesis,Vol-umetricReconstructionof 3-D Objectsfrom Multiple Cali-bratedCameraViews,” Proc. ICASSP’99, pp. 3509-3512,Phoenix,March1999.