Polarization Human Shape and Pose Dataset

Shihao Zou1, Xinxin Zuo1, Yiming Qian2, Sen Wang1, Chuan Guo1, Chi Xu3,Minglun Gong4, and Li Cheng1

1 University of Alberta2 Simon Fraser University

3 School of Automation, China University of Geosciences, Wuhan 430074, China4 University of Guelph

{szou2,xzuo,sen9,cguo2,lcheng5}@ualberta.ca,yimingq@sfu.ca,xuchi@cug.edu.cn,minglun@uoguelph.ca

1 Introduction

Polarization images are known to be able to capture polarized reflected lightsthat preserve rich geometric cues of an object, which has motivated its recentapplications in reconstructing detailed surface normal of the objects of interest.Meanwhile, inspired by the recent breakthroughs in human shape estimationfrom a single color image, we attempt to investigate the new question of whetherthe geometric cues from polarization camera could be leveraged in estimatingdetailed human body shapes. This has led to the curation of Polarization HumanShape and Pose Dataset (PHSPD)5, our home-grown polarization image datasetof various human shapes and poses.

Our PHSPD dataset synchronizes four cameras, one polarization camera andthree Kinects v2 in three different views (each Kinect v2 has a depth and acolor camera). The depth and color images from three-view Kinects v2 are usedto get more accurate annotations of shape and pose in 3D space. We proposean economic yet effective approach to annotating shape and pose in 3D space.Compared with Human3.6M [1] that uses expensive Motion Capture system toannotate human poses, we do not require subjects to wear special tight clothesand a lot of sensors, which makes the acquired images restrictive and impractical.

We show some of our annotated shapes and poses in Fig. 1, where the shapesare rendered on the image plane and the poses are shown in 3D space. We can seethat our annotated shapes and poses align well with the subjects in the imageplane from four camera views.

2 Data Acquisition

Our acquisition system synchronizes four cameras, one polarization camera andthree Kinects V2 in three different views (each Kinect v2 has a depth and acolor camera). The layout is shown in Fig. 2. The main task in data acquisition ismulti-camera synchronization. As one desktop can only control one Kinect v2, we

5 Our PHSPD dataset will be released soon for academic purpose only.

arX

iv:2

004.

1489

9v2

[cs

.CV

] 2

9 Ju

l 202

0

2 S. Zou et al.

polarization image polarization image with rendered shape

color image view 1with rendered shape

color image view 2with rendered shape

color image view 3with rendered shape

pose displayed in 3D

Fig. 1. The figure shows our annotated shapes and poses. The first column is the polar-ization image for reference. The second to the fifth columns show the annotated shaperendered on the polarization image and three-view color images. The sixth columnshows the annotated pose in 3D space.

Polarization Human Shape and Pose Dataset 3

develop a soft synchronization method. Specifically, each camera was connectedwith a desktop (the desktop with the polarization camera is the master and theother three ones with three Kinects are clients). We use socket to send message toeach desktop. After receiving certain message, each client will capture the mostrecent frame from the Kinect into the desktop memory. At the same time, themaster desktop sends a software trigger to the polarization camera to captureone frame into the buffer. Practically, our synchronization system can be as fastas 15 frames per second (fps). Fig. 2 shows the synchronization performance ofthe system that we develop. We let a bag fall down and compare the positionof the bag in the same frame from four views. We can find that the positions ofthe bag captured by four cameras are almost the same in terms of its distanceto the ground.

Fig. 2. Left figure: the layout of our multi-camera system. Three Kinects are placedaround a circle of motion area with one polarization camera. Right figure: the synchro-nization result of our multi-camera system. The same frame of the three-view colorimages and one-view polarization image are displayed. Note that the layout of ourmulti-camera system has been changed to the left figure, but other settings are thesame.

group # actions

1 warming-up, walking, running, jumping, drinking, lifting dumbbells2 sitting, eating, driving, reading, phoning, waiting3 presenting, boxing, posing, throwing, greeting, hugging, shaking hands

Table 1. The table displays the actions in each group. Subjects are required to doeach group of actions for four times, but the order of the actions each time is random.

Our dataset has 12 subjects, 9 males and 3 females. Each subject is requiredto do 3 different groups of actions (18 different actions in total) for 4 times plusone free-style group. Details are shown in Tab. 1. So each subject has 13 short

4 S. Zou et al.

subject#

gender# of original

frames# of annotated

frames# of discarded

frames

1 female 22561 22241 320 (1.4%)2 male 24325 24186 139 (0.5%)3 male 23918 23470 448 (1.8%)4 male 24242 23906 336 (1.4%)5 male 24823 23430 1393 (5.6%)6 male 24032 23523 509 (2.1%)7 female 22598 22362 236 (1.0%)8 male 23965 23459 506 (2.1%)9 male 24712 24556 156 (0.6%)10 female 24040 23581 459 (1.9%)11 male 24303 23795 508 (2.1%)12 male 24355 23603 752 (3.1%)

total - 287874 282112 5762 (2.0%)

Table 2. The table shows the detail number of frames for each subject and also thenumber of frames that have SMPL shape and 3D joint annotations.

videos and the total number of frames for each subject is around 22K. Overall,our dataset has 287K frames with each frame including one polarization image,three color and three depth images. Quantitative details of our dataset are shownin Tab. 2.

3 Annotation Process

3.1 Shape and Pose Representation

We represent the 3D human body shape (mesh) using SMPL model [2], which isa differentiable functionM(β,θ) ∈ R6890×3 that outputs a triangular mesh with6890 vertices given 82 parameters [β,θ]. The shape parameter β ∈ R10 is thelinear coefficients of a PCA shape space that mainly determines individual bodyfeatures such height, weight and body proportions. The shape space is learnedfrom a large dataset of body scans [2]. The pose parameter θ ∈ R72 mainlydescribes the articulated pose, which consists of one global rotation of the bodyand the relative rotations of 23 joints in axis-angle representation. The finalbody mesh is produced by first applying shape-dependent and pose-dependentdeformations to the template body, then using forward-kinematics to articulatethe body and finally deforming the surface with linear blend skinning. The 3Djoint positions J ∈ R24×3 can be obtained by linear regression from the outputmesh vertices.

Polarization Human Shape and Pose Dataset 5

3.2 Annotation of Shape and Pose

The reason that we use multi-camera system to acquire image data is that multi-camera system provides much more information than a single-camera system.So the annotation of human shape and pose in 3D is more reliable.

After camera calibration and plane segmentation of human body in depthimages, we have a point cloud of human surface fused from three-view depthimage, and also noisy 3D pose by Kinect SDK at hand. The annotation of SMPLhuman shape and 3D joint position has three main steps as follows.

Initial guess of 3D pose As the 3D pose given by Kinect SDK is noisy, weuse the predicted 2D pose by OpenPose [3] as the criterion to decide whichjoint position given by Kinect SDK is correct. We select 14 aligned joints thatboth Openpose and Kinect have. For joint i in view j = {1, 2, 3}, 2D jointposition by OpenPose is denoted by (voij , u

oij), and 3D joint position by Kinect

by (xij , yij , zij) and its projected 2D joint position by (vkij , ukij). Since we cannot

figure out which joint is detected correctly by Kinect, we use the joint positionby OpenPose as the criterion to decide whether this joint is correctly estimatedby Kinect, that is

wij =

{1 if

√(vkij − voij)

2 + (ukij − uo

ij)2 < 50,

0 otherwise,(1)

where 50 means the pixel distance. Then, we get the initial guess of the 3D jointposition (xi, yi, zi) of joint i by averaging the valid positions given by three-viewKinects as

xi =

∑3j=1 wijxij∑3j=1 wij

, yi =

∑3j=1 wijyij∑3j=1 wij

, zi =

∑3j=1 wijzij∑3j=1 wij

. (2)

If none of the three-view joint positions by Kinect is correct, we consider it asa missing joint. We discard the frame with more than 2 joints missing (14 intotal).

Fitting shape to pose The next step is similar to SMPLify [4], but insteadof fitting to the 2D joints which have inherent depth ambiguity, we fit SMPLmodel to the initial guess of 3D pose.

Fine-tuning shape to the point cloud The final step is fine-tuning theshape to the point cloud of human surface so that the annotated SMPL shapeparameters are more accurate. We iteratively optimize SMPL parameters byminimizing the distance between vertices of SMPL shape to their nearest point.Finally, we have the annotated SMPL shape parameters and 3D pose.

Besides, we render the boundary of SMPL shape on the image to get themask of background, and calculate the target normal using three depth images

6 S. Zou et al.

based on [5]. Although the target normal is noisy, our experiment result showsour model can still learn to predict good and smooth normal maps.

The annotation process is shown in Fig. 3. Starting from the initial guessof 3D pose, we fit SMPL shape to the initial 3D pose and further fine-tune tothe point cloud of human surface. Finally, we get the annotated human shapeand pose for each frame. We can find from Fig. 3 that the third step is criticalto make the annotated shape align better to the subject in the image in thatthe pint cloud of human surface gives much more information than a skeleton-based pose. So the third step can adjust the shape to improve the alignment ofbody parts. Besides, we also show our annotated shape on multi-view images(one polarization image and three-view color image) and the human pose in 3Dcoordinate space in Fig 1.

3.3 Annotation of Actions

We select a subset of dataset PHSPD, and cut the long pose sequences intosmaller pieces which aligns with prescribed action types, named as HumanAct12.The statistics of our action annotations is given in Tab. 3, where there are 1061motion clips which are categorized into 12 action classes and 34 sub-classes.

Polarization Human Shape and Pose Dataset 7

initial guess of 3D pose

initial shape with point cloud

initial shape render on the image

final shape fitted to point cloud

final shape rendered on the image

final 3D pose projected on the image

Fig. 3. The figure shows our three-step annotation process. The first column shows theinitial guess of 3D pose, which is projected on the polarization image. After fitting theSMPL shape to the initial pose, we show the initial fitted shape with the point cloudof human surface (black points) in the second column and the rendered shape on theimage in the third column. The fourth and fifth columns show the annotated shapeafter fine-tuning the shape to the point cloud of human surface. The sixth columnshows the corresponding annotated 3D pose projected on the polarization image.

8 S. Zou et al.

Table 3. Statistics of dataset HumanAct12.

Coarse-grained Label Fine-grained Label # of Motions # Total

Warm up

Warm up wristankle 25

201

Warm up pectoral 45Warm up eblowback 39

Warm up bodylean right arm 25Warm up bodylean left arm 24

Warm up bow right 22Warm up bow left 21

Walk Walk 43 43

Run Run 44 44

JumpJump handsup 50

85Jump vertical 35

Drink

Drink bottle righthand 25

81Drink bottle lefthand 39Drink cup righthand 10Drink cup lefthand 3Drink both hands 4

Lift dumbbell

Lift dumbbell righthand 41

198Lift dumbbell lefthand 41

Lift dumbbell bothhands 43Lift dumbbell overhead 39

Lift dumbbell bothhands bend legs 34

Sit Sit 47 47

EatEat righthand 27

68Eat lefthand 18Eat pie/burger 23

Turn steering wheel Turn steering wheel 46 46

PhoneTake out phone, call and put back 19

52Call with left hand 33

Boxing

Boxing left right 21

116Boxing left upwards 34

Boxing right upwards 37Boxing right left 24

ThrowThrow right hand 47

80Throw both hand 33

Entire Dataset - - 1061

Polarization Human Shape and Pose Dataset 9

References

1. Ionescu, C., Papava, D., Olaru, V., Sminchisescu, C.: Human3.6m: Large scaledatasets and predictive methods for 3d human sensing in natural environments.IEEE Transactions on Pattern Analysis and Machine Intelligence 36(7) (2014) 1325–1339

2. Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: Smpl: A skinnedmulti-person linear model. ACM transactions on graphics (TOG) 34(6) (2015) 248

3. Cao, Z., Hidalgo Martinez, G., Simon, T., Wei, S., Sheikh, Y.A.: Openpose: Realtimemulti-person 2d pose estimation using part affinity fields. IEEE Transactions onPattern Analysis and Machine Intelligence (2019) 1–1

4. Bogo, F., Kanazawa, A., Lassner, C., Gehler, P., Romero, J., Black, M.J.: Keep itsmpl: Automatic estimation of 3d human pose and shape from a single image. In:European Conference on Computer Vision, Springer (2016) 561–578

5. Qi, X., Liao, R., Liu, Z., Urtasun, R., Jia, J.: Geonet: Geometric neural network forjoint depth and surface normal estimation. In: Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition. (2018) 283–291