Human Motion Reconstruction and Animation fromVideo Sequences

Fabio RemondinoInstitute for Geodesy and Photogrammetry

ETH Zurich, SwitzerlandE-mail: fabio@geod.baug.ethz.ch

Web: http://www.photogrammetry.ethz.ch

Andreas RoditakisInstitute for Geodesy and Photogrammetry

ETH Zurich, SwitzerlandE-mail: roditak@antrikos.net

Web: http://www.photogrammetry.ethz.ch

Abstract

In this paper we present a new framework forthe modeling and animation of humancharacters from monocular videos. Themodeling is performed semi-automaticallyusing a polygonal shape model obtained fromlaser scanner measurements. Most of theimage-based techniques use probabilisticapproaches to derive the poses of a characterfrom an image sequences; on the other hand weuse a determinist approach, recoveringcharacter’s poses and movement through acamera model. The recovered human modelscan be used for visualization purposes, togenerate new virtual scenes or for gait analysis.

Key words: 3D Reconstruction, Animation,Human Body Modeling

1. IntroductionThe realistic modeling and animation of humancharacters is one of the most difficult tasks inthe vision and graphic community. Inparticular, body modeling from videosequences is a challenging problem that hasbeen investigated a lot in the last decade.Recently the demand of 3D human models isdrastically increased for applications likemovies, video games, ergonomic, e-commerce,virtual environments and medicine. A completehuman model consists of the 3D shape and themovements of the body; some approachesconsider these two modeling procedures asseparate even if they are very closed. Theissues involved in creating virtual humans arethe acquisition of the body shape, the modelingof the data and the acquisition of theinformation for the animation.Different representations of virtual charactersare present in the literature. One of the mostused articulated 3D representation for animated

character is the H-Anim (Figure 1). It is anInternational Standard that defines the H-Animas an abstract representation (in VRML97language) for modeling 3D human figures; itdescribes a standard way to represent andanimate humanoids and it is a simple designfor 3D Internet applications [28].

Figure 1: Different representation of virtualcharacters (avatars) with articulated joints

according to the ISO H-Anim [28].

1.1 Related work on static shape captureA standard approach to capture the static 3Dshape (and colour) of an entire human bodyuses the laser scanner technology [8][27][32]:it is quite expensive but it can generate a wholebody geometry in ca 20 seconds. Afterwardsthe 3D shape can be animated (Figure 2),articulating the model and changing its posture[1][16][20] or dressed for garment models[2][3].

Figure 2: 3D human shape recovered with laserscanner and then animated. Sources (from left

to right): [1][16][20].

Concerning image-based approaches, theyrequire different acquisitions around a staticperson: then they can deform a 3D templatemodel to the extracted human silhouette[7][10][22] or they can recover a 3D humanmodel through a camera model and with

template matching [12]. Finally, computeranimation software [26][29][30][31] allows thegeneration of virtual humans using splines andsmoothing simple polygonal elements.

1.2 Related work on movementreconstructionPrecise information related to charactermovements is generally acquired with motioncapture techniques: they involve a network ofcameras and prove an effective andsuccessfully mean to replicate humanmovements [23][24][25]. Other approachesinstead rely on video sequences as primaryinput. Single- or multi-stationsvideogrammetry offers in fact an attractivealternative technique, requiring cheap sensors,allowing markerless tracking and providing, atthe same time, for 3D shapes and movementsinformation. The analysis of existing videoscan moreover allow the generation of 3Dmodels of characters who may be long dead orunavailable for common modeling techniques.In [11][17][18][21] computer visiontechniques, image cues, backgroundsegmentation, prior knowledge about humanmotion, pre-defined articulated body modelsand no camera model are used to recovermotions and 3D information from monocularsequences (Figure 3). Multi-camerasapproaches [4][5][19] are instead employed toincrease reliability, accuracy and avoidproblems with self-occlusions (Figure 4).

Figure 3: Monocular approach [11]: originalframe and skeleton reconstruction with a

gaussian probability model.

Figure 4: Multi-camera approach [4]: originalframes, silhouette tracking and fitting process.

In [4] smooth implicit surfaces are fitted to therecovered 3D data while in [5] taperedsuperquadratic are fitted onto the extractedhuman poses. Usually model-based approachesare more common, in particular withmonocular video streams, while deterministicapproaches are almost neglected, often due to

the difficulties in recovering the cameraparameters and because of occlusions of thebody parts. The framework that we propose isan image-based deterministic process torecover shape and motion of a moving personfrom monocular videos (Figure 5). First the camera parameters aredetermined and afterwards the human’s posesand movements are recovered and modeledusing a polygonal body model.

2. Sequence analysis for 3D posereconstructionA moving character imaged with one movingcamera represents the most difficult case forthe deterministic 3D reconstruction of itsposes. We analyse existing sport videos, toavoid copyright problems. They are usuallyacquired with a stationary but freely rotatingcamera and with a very short baseline betweenthe frames. Therefore, due to the movements ofthe character, a standard perspective approachcannot be used, in particular for the 3Dmodeling of the character.

: fully automated : semi-automated = manual

Figure 5: Workflow for character modelingand animation from video sequences

2.1 Camera calibration and imageorientationThe photogrammetric calibration andorientation of the video is performed with abundle adjustment [14]. The procedure isrequired to achieve the camera parametersnecessary for the determination of the scene’smetric information and for the human’s posesestimation.

2.2 Body Poses estimationFor man-made objects (e.g. buildings),geometric constraints on the object (e.g.perpendicularity and orthogonality) can beused to solve the ill-posed problem of the 3Dreconstruction from a monocular image. Incase of free-form objects (e.g. the humanbody), we could use probabilistic either model-based approaches or other assumptions must beprovided [13]:• the perspective collinearity model is

simplified into a scaled orthographicprojection;

• the human body is represented in a skeletonform, with a series of joints connected withsegments of known relative lengths (Figure6, left);

• further constraints on joints depth andsegment’s perpendicularity are applied toobtained more accurate and reliable 3Dmodels.

This reconstruction algorithm (Figure 6, right)is applied to the analyzed frames of thesequence, given the image coordinates of somehuman joints and the relative lengths of theskeleton segments. The image points of thejoints cannot be automatically tracked alongthe sequence, due to occlusion problems andinterlaced video; therefore the points aremeasured semi-automatically with a leastsquares matching (LSM) algorithm [6]. Foreach image, a 3D human model is generatedand afterwards transformed to the absolutereference system with a 3D-conformaltransformation. Finally the 3D coordinates arerefined using the camera parameters recoveredin the orientation process [15].

Figure 6: The human body simplified and

represented with a skeleton of 13 joints plus thehead (left). The process to recover 3D human

models from single images or monocular videos(right).

3. Human character modeling andanimationThe recovered character’s poses are used toreconstruct the 3D model of the humanskeleton. To improve the visual quality andrealism of the model, a pre-defined polygonalmodel [27] is afterwards fitted to the recoveredphotogrammetric 3D data. The scanned modelwill be the skin of our 3D virtual character.The fitting and animation processes areperformed with the animation features of Mayasoftware [30]. Other approaches where a wholebody scanned data is deformed to generateanimatable human bodies are described in[1][16][20].

3.1 Skeleton reconstruction andvisualizationThe recovered 3D coordinates of the humanjoints are given to a procedure that uses VRMLlanguage to automatically visualize thesimplified human model. This step is mainlyused to check if the recovered 3D model iscorrect. A simple skeleton is created, where allthe human joints are represented with spheresconnected with cylinders or tapered ellipsoidsin case we want to model the muscles (Figure7).

Figure 7: 3D model of the human skeletonshown in VRML using spheres, simple cylindersor different shapes for torso arms and legs [13].

3.2 Polygonal model fittingThe first step in fitting a polygonal model(Figure 8) to the extracted poses is theconstruction of a new native skeleton, whosejoints are running through the fourteencharacteristic points of the input skeleton; itwill be the structure on which the skin fittingand animation will be based.The new skeleton is then moved inside thepolygonal body, translating, scaling androtating its limbs so that they lie inside thebody. Small disagreements can be presentbecause of errors in estimating the lengths ofthe limbs from the image data.

Figure 8: The polygonal model obtained withlaser scanner [27], consisted of approximately

300 000 polygons (left). Two closed views of thewireframe model in areas subordinated to

strong deformations during the movements.

3.3 Binding the skin with the skeletonThe basic idea behind skeleton and bodybinding is that the skeleton controls the bodymovements and deformations. Thus the bodytakes the poses of the skeleton limbs. As wewant to represent any possible human pose, allthe deformations that occur when the bodyparts are moving must be taken intoconsideration. The Maya animation platform,as most of the animation software on themarket, supplies two solutions for simulatingthe interaction of a skeleton with its skin: (1)soft bind, that allows two or more skeletonjoints to influence their surrounding geometryand works very well in soft object simulation;(2) rigid bind, that allows the influence of onlyone joint per body segment and performs betterin solid body modeling. During themovements, while some human body partsremain solid, there are soft body deformationsin areas such as the arms and knees, whichmakes the application of any of the abovemethods unrealistic. Moreover, in soft binding,if we want to modify the weight values of ajoint, we need to do this on every part of theinfluenced body surface. And the process istime-consuming if we have to fine-tune all theinfluences on a soft bind skin. Therefore, acombination of rigid body modeling with 3Ddeformation lattices is required. Lattices areapplied on the sensitive areas and they are oneof the best approximations of the physical bodybehaviour. Deformation lattices are placed inthe body’s areas subordinated to strongdeformation during the movements, e.g.shoulders and feet ankles of the polygonalhuman model (Figure 9). The basic idea is thatthe skeleton should deform the lattice, whichconsequently influences the position of theaffected points of the polygonal model. In thisway the deformations follow the skeleton

movements. The number of grid cells of thelattices depends on the average number ofpoints that every cell should contain, so itdepends on the spatial resolution of thepolygonal body. Lattices with dimensionslarger than 10 in all three axes are provencomputationally expensive, while dimensionssmaller than 3 are too coarse.

Figure 9: The lattices set in areas of thepolygonal model subordinated to strong

deformation during the movements.

For joints like elbows and knees, an additionaland faster deformation procedure can be used.The flexor deformers are specially designeddeformers for human body animation that haveautomated the task of smooth skin simulationin such areas. Because of the simplicity of theirgeometry, they are used in areas that are notstressed too much by the movementsdeformations.

3.4 Reverse Feet mechanismThe animation of some human’s parts (like thefoot) is not a trivial issue. In fact, whenanimating a walking biped, the feet should stayfirmly on the ground. In addition to this,deformations occur in the ankle area and theangle between the foot and the leg is changing,leading to high modification of the polygonalarea. In order to model correctly themovements of the human feet and to simulatetheir deformations in a realistic way, thereverse foot mechanism is used [9]. A newadditional skeleton from the ankle to the toe iscreated (control skeleton) and an extra joint isadded (Figure 10).

Figure 10: The control skeleton defined toanimate and model correctly the movements of

the feet.

This last joint (control joint) is the root of theduplicate skeleton and is not connected to thehierarchy of the main skeleton. Then, usingPoint and Orientation constraints, the feet onthe main skeleton is constrained to follow theposition and orientation of the correspondingcontrol skeleton. This does not belong to themain skeleton and can be key-framedseparately.

3.5 Animation (IK handles)The movement of the human limbs iscontrolled by automated procedures calledKinematics. They are also known as handlesand work as a chain running through all thejoints. They control the rotations andtranslations among sequential skeleton jointsthat build up a limb. In this way complexmovements on multiple joints can beperformed by only adjusting the chain’sendpoint (effector) position and rotation. Inorder to achieve this effect, there are twodifferent types of kinematics: InverseKinematics and Forward Kinematics. In thefirst case the position and rotation of theeffector is enough to position all theintermediate joints; the latter demands that thetranslation and rotation of every joint is given,from top to bottom in the hierarchy, until thedesired pose is achieved. Because of theirsimple and intuitive control, inverse kinematicshandles are preferred.

3.6 Movement constraintsAlthough the handles require less input fromthe animator, the presence of more than onepossible correct solution for the intermediatejoints makes the use of skeleton constraintsnecessary. The physical movement limits of thehuman skeleton are therefore defined withangular constraints, in order to avoid falsepositions or impossible twists and get thecorrect poses of the limbs.

3.7 PerformanceOnce the human skeleton is constructed andimported in Maya, the maximum time spent forbringing the polygonal body into the correctpose is approximately 2 minutes. The lattices’definition take ca 3 minutes, as more attentionmust be given to their positioning and toinclude the correct points that must be affected.For the rendering, due to the interpolationfunctions, and only some key-frames are

imported and analyzed. We used the nativeMaya renderer and for every frame (PALresolution) approximately 1 minute is required.The animation of a sequence composed of 200frames required a little more than 3 hours to becompleted.

4. ResultsA sequence of 60 images was digitized from anold videotape, using a Matrox DigiSuitegrabber, with a resolution of 720x576 pixels(Figure 11).

Figure 11: Some original frames of a movingcharacter.

After camera calibration and image orientation,the human joints are tracked all over thesequence and afterwards the 3D coordinatesare computed. A VRML model of the imagedscene, with some camera positions and themoving human skeleton is presented in Figure12.

Figure 12: The camera poses as well as the 3Dreconstruction of the basketball court and the

moving character.

The polygonal human model is then fitted tothe skeleton and animated using the recoveredposes and following to the presentedmethodology (Figure 13).

Figure 13: Results of the fitting process onto

the photogrammetric 3D data. In this case alsothe ball is modeled.

Critical polygonal areas subordinated to strongdeformations are quite well modeled as shownin Figure 14.

Figure 14: Two examples of areas where themovements create high deformations of the

polygons. The lattices helped to create arealistic result.

In this example we chose lattices with 6 cells inall dimensions for the shoulders and 4 cells forthe feet ankles, which gave a balance betweenperformance and quality.

Figure 15: Some frames of a basketball videosequence.

Another sequence, presented in Figure 15, wasanalyzed. The camera, mounted far away fromthe scene and probably on a tripod, is rotatingand zooming to follow the moving character.The calibration and orientation process,performed with a self-calibrating bundleadjustment with frame-invariant APs sets,recovered a constant increasing of the camerafocal length [14].The 3D reconstruction of the moving characteris afterwards performed and the results areshown in Figure 16. A new virtual scene of theanalyzed sequence is also presented.

Figure 16: Results of the 3D modeling of the

moving character visualized in skeleton form.

To improve the visual quality and the realismof the reconstructed 3D human skeleton, thefitting is afterwards performed. The semi-automatic modeling process producedsatisfactory results, as shown in Figure 17. Thecamera viewpoint is also changed, to generatenew virtual views of the scene.

Figure 17: Two examples showing the results of

the fitting and the modeling process. Fiveframes of the sequence (upper image) and themiddle frame seen from 2 different viewpoints

(lower images).

Despite of the complex movements and strongdeformations, the critical areas like feet andshoulders are realistically rendered (Figure 18).

Figure 18: The feet and the shoulders of the

character modeled using the deformationlattices.

5. ConclusionsThe analysis of monocular video sequencesand the generation of 3D human models werepresented. After the sequence orientation andcalibration, the positions of some human jointsare recovered throughout the frames. This 3Dinformation is afterwards used for furthermodeling, where a whole body scanned modelis fitted and animated using photogrammetricdata.The use of existing and old videos showed thecapability of videogrammetry to provide forvirtual characters useful for augmented realityapplications, persons identification and togenerate new scenes involving models ofcharacters who are dead or unavailable forcommon modeling systems. We alsodemonstrated that in combination withcomputer animation software we could achieverealistic reconstruction from any desiredviewing angle and produce realistic reality-based animations.The proposed framework can become avaluable tool for supplying input motion datafor avatars in virtual reality environments.Currently we are trying to improve thepolygonal model, in particular in the handsarea. Furthermore we will try to back-projectthe fitted polygonal model onto the images,using the recovered camera parameters, todeeply check the quality and accuracy of thereconstruction and fitting process. Finally, weplan to fit the image-based results with asimpler human model, like. a H-Anim VRMLmodel.

References[1] B. Allen, B. Curless and Z. Popovic.

Articulated body deformation from rangescan data. Proc. SIGGRAPH, pp.612-619,2002

[2] F. Cordier, N. Magnemat-Thalman. Real-time animation of dressed virtual humans.Computer Graphics Forum, Blackwell,Vol.21(3), 2002

[3] F. Cordier, H. Seo and N. Magnenat-Thalmann. Made-to-Measure technologiesfor online clothing store. IEEE CG&Aspecial issue on ‘Web Graphics’, pp.38-48,2003

[4] N. D'Apuzzo, R. Plänkers, P. Fua, A. Gruenand D. Thalmann. Modeling human bodiesfrom video sequences. In El-Hakim/Gruen(Eds.), Videometrics VI, Proc. of SPIE,Vol. 3461, pp. 36-47, 1999

[5] D.M. Gavrila and L. Davis. 3D model-based tracking of humans in action: amulti-view approach. IEEE CVPR Proc.pp. 73-80, 1996

[6] A. Gruen. Adaptive least squarescorrelation: a powerful image matchingtechnique. South African Journal ofPhotogrammetry, Remote Sensing andCartography, Vol. 14(3), pp.175-187,1985

[7] A. Hilton, D. Beresfors, T. Gentils, R.Smith, W. Sun and J. Illingworth. Whole-body modeling of people from multiviewimages to populate virtual worlds. TheVisual Computer, Vol. 16, pp. 411-436,Springer-Verlag, 2000

[8] Horiguchi. Body Line Scanner. Thedevelopment of a new 3-D measurementand Reconstruction system. InternationalArchives of Photogrammetry and RemoteSensing, Vol.32(5), pp.421-429, 1998

[9] Learning Maya 5 - Foundation, 2003. AliasWavefront

[10] W. Lee, J. Gu and N. Magnenat-Thalmann. Generating animatable 3Dvirtual humans from photographs.Eurographics, Vol. 19, No. 3, 2000.

[11] M. Leventon, W. Freeman. Bayesianestimation of 3D human motion from animage sequence. TR-98-06, MERL, 1998

[12] F. Remondino. 3D reconstruction of statichuman body with a digital camera. SPIEProc, Vol. 5013, Videometrics VII, pp. 38-45, 2003.

[13] F. Remondino and A. Roditakis. HumanFigures Reconstruction and Modeling fromSingle images or Monocular VideoSequences. IEEE International 3DIMConference, pp. 116-123, 2003

[14] F. Remondino and N. Boerlin.Photogrammetric calibration of sequencesacquired with a rotating camera.International Archives of Photogrammetry,Remote Sensing and Spatial InformationSciences, Vol. XXXIV, part 5/W16, 2004

[15] F. Remondino. Character reconstructionand animation from monocular sequence ofimages. International Archives ofPhotogrammetry, Remote Sensing andSpatial Information Sciences,Vol.XXXV/5, in press

[16] H. Seo and N. Magnenat-Thalmann. AnAutomatic Modeling of Human Bodiesfrom Sizing Parameters", ACMSIGGRAPH Symposium on Interactive 3DGraphics, pp. 19-26, 2003

[17] H. Sidenbladh, M. Black, and D. Fleet.Stochastic Tracking of 3D Human FiguresUsing 2D Image Motion. ECCV, D.Vernon (Ed.), Springer Verlag, LNCS1843, pp. 702-718, 2000

[18] C. Sminchisescu. Three DimensionalHuman Modeling and MotionReconstruction in Monocular VideoSequences Ph.D. Dissertation, INRIAGrenoble, 2002

[19] S. Vedul and S. Baker. ThreeDimensional Scene Flow. ICCV '99, Vol.2, pp. 722-729.

[20] X. Ju, N. Werghi and J.P. Siebert.Automatic Segmentation of 3D Human

Body Scans. International Conference onComputer Graphics and Imaging (CGIM2000), pp 239-244, 2000

[21] M. Yamamoto, A. Sato, S. Kawada, T.Kondo and Y. Osaki. IncrementalTracking of Human Actions from MultipleViews. IEEE CVPR Proc., 1998

[22] J.Y. Zheng. Acquiring 3D models fromsequences of contours. IEEE Transactionon Pattern Analysis and MachineIntelligence 16(2), pp 163-178, 1994.

[23] Ascension: http://www.ascension-tech.com/ [Accessed April 2004]

[24] Vicon: http://www.vicon.com [AccessedApril 2004]

[25] Motion Analysis:http://www.motionanaly-sis.com/[Accessed April 2004]

[26] 3D Studio Max 3.1:/http://www.discreet.com/products/3dsmax/[Accessed April 2004][27] Cyberware: http://www.cyberware.com

[Accessed April 2004][28] H-Anim: http://www.h-anim.org

[Accessed April 2004][29] Lightwave: http://www.lightwave3d.com

[Accessed April 2004][30] Maya: http://www.aliaswavefront.com/

[Accessed April 2004][31] Poser, Curious Labs:

http://www.curiouslabs.com [AccessedApril 2004]

[32] Vitus:http://www.vitus.de/english/home_en.html[Accessed April 2004]