1

Anticipating Human Activities using ObjectAffordances for Reactive Robotic Response

Hema S. Koppula and Ashutosh Saxena, Member, IEEE

Abstract—An important aspect of human perception is anticipation, which we use extensively in our day-to-day activities wheninteracting with other humans as well as with our surroundings. Anticipating which activities will a human do next (and how) canenable an assistive robot to plan ahead for reactive responses. Furthermore, anticipation can even improve the detection accuracyof past activities. The challenge, however, is two-fold: We need to capture the rich context for modeling the activities and objectaffordances, and we need to anticipate the distribution over a large space of future human activities. In this work, we represent eachpossible future using an anticipatory temporal conditional random field (ATCRF) that models the rich spatial-temporal relations throughobject affordances. We then consider each ATCRF as a particle and represent the distribution over the potential futures using a setof particles. In extensive evaluation on CAD-120 human activity RGB-D dataset, we first show that anticipation improves the state-of-the-art detection results. We then show that for new subjects (not seen in the training set), we obtain an activity anticipation accuracy(defined as whether one of top three predictions actually happened) of 84.1%, 74.4% and 62.2% for an anticipation time of 1, 3 and 10seconds respectively. Finally, we also show a robot using our algorithm for performing a few reactive responses.†

Index Terms—RGBD Data, 3D Activity Understanding, Human Activity Anticipation, Machine Learning, Robotics Perception.

F

1 INTRODUCTION

FOR many applications it is important to be able to detectwhat a human is currently doing as well as anticipate what

she is going to do next and how. The former ability is usefulfor applications such as monitoring and surveillance, but weneed the latter for applications that require reactive responses,for example, an assistive robot (see Figure 1). In this paper,our goal is to use anticipation for predicting future activitiesas well as improving detection (of past activities).

There has been a significant amount of work in detectinghuman activities from 2D RGB videos [1], [2], [3], frominertial/location sensors [4], and more recently from RGB-Dvideos [5], [6], [7]. The primary approach in these works isto first convert the input sensor stream into a spatio-temporalrepresentation, and then to infer labels over the inputs. Theseworks use different types of information, such as humanpose, interaction with objects, object shape and appearancefeatures. However, these methods can be used only to predictthe labeling of an observed activity and cannot be used toanticipate what can happen next and how.

Our goal is to predict the future activities as well as thedetails of how a human is going to perform them in short-term(e.g., 1-10 seconds). For example, if a robot has seen a personmove his hand to a coffee mug, it is possible he would movethe coffee mug to a few potential places such as his mouth,to a kitchen sink or just move it to a different location on thetable. If a robot can anticipate this, then it would rather notstart pouring milk into the coffee when the person is movinghis hand towards the mug, thus avoiding a spill. Such scenarioshappen in several other settings, for example, manufacturingscenarios in future co-robotic settings (e.g., [8], [9]).

• H. S. Koppula and A. Saxena are with the Computer Science Departmentat Cornell University, NY 14853. E-mail: {hema,asaxena}@cs.cornell.edu

†Parts of this work have been published at ICML 2013 and RSS 2013.

Activities often have a hierarchical structure where an ac-tivity is composed of a sequence of sub-activities and involveinteractions with certain objects. For example, a cup is usedin the drinking activity which is composed of a sequenceof reach, move and drink sub-activities. Therefore, we cananticipate the future by observing the sub-activities performedin the past and reasoning about the future based on thestructure of activities and the functionality of objects beingused (also referred to as object affordances [10]). For example,in Figure 1, on seeing a person carrying a bowl and walkingtowards the refrigerator, one of the most likely future actionsare to reach the refrigerator, open it and place the bowl inside.

For anticipating the future, we need to predict how thefuture sub-activities will be performed in terms of motiontrajectories of the objects and humans. In order to do this,we propose the use of object affordances and model them interms of the relative position of the object with respect tothe human and the environment1 and their temporal motionstrajectories during the activity, as described in Section 4.Modeling trajectories not only helps in discriminating theactivities,2 but is also useful for the robot to reactively planmotions in the workspace.

In our work, we use a conditional random field based on [5](see Figure 2) to model the spatio-temporal structure of activ-ities, as described in Section 5.1. For anticipation, we presentan anticipatory temporal conditional random field (ATCRF),where we model the past with the CRF described above butaugmented with the trajectories and with nodes/edges repre-senting the object affordances, sub-activities, and trajectoriesin the future. Since there are many possible futures, eachATCRF represents only one of them. In order to find the

1. For example, a drinkable object is found near the mouth of the personperforming the drinking activity and a placeable object is near a stable surfacein the environment where it is being placed.

2. For example, in stirring activity, the target position of the stirrer isimmaterial but the circular trajectory motion is.

2

(a) Robot’s RGB-D view. (b) Object affordance heatmaps.

(c) Trajectories heatmap. (d) Robot opening the door.Fig. 1: Reactive robot response through anticipation: Robotobserves a person holding an object and walking towards a fridge (a).It uses our ATCRF to anticipate the affordances (b), and trajectories(c). It then performs an anticipatory action of opening the door (d).

most likely ones, we consider each ATCRF as a particleand propagate them over time, using the set of particles torepresent the distribution over the future possible activities.One challenge is to use the discriminative power of the CRFs(where the observations are continuous and labels are discrete)for also producing the generative anticipation—labels oversub-activities, affordances, and spatial trajectories.

We evaluate our anticipation approach extensively onCAD-120 human activity dataset [5], which contains 120RGB-D videos of daily human activities, such as hav-ing meal, microwaving food, taking medicine, etc. Wefirst show that anticipation improves the detection ofpast activities: 85.0% with vs 82.3% without. Our algo-rithm obtains an activity anticipation accuracy (defined aswhether one of top three predictions actually happened) of(84.1%,74.4%,62.2%) for predicting (1,3,10) seconds into thefuture. Videos showing our experiments and code are availableat: http://pr.cs.cornell.edu/anticipation

The main contributions of this paper are:• While most previous works consider activity detection,

we consider anticipation.• We consider rich contextual relations based on object

affordances in RGB-D videos.• We propose ATCRFs, where each particle represents a

CRF.• We consider joint temporal segmentation and labeling

using our approach.• We test our method on a dataset containing long-term

activities, and also test in robotic experiments.The rest of the paper is organized as follows. We start

with a review of the related work in Section 2 and providean overview of our methodology in Section 3. We describethe affordances in Section 4 and model in Section 5. Section6 describes the features and Section 7 describes the learningand inference algorithms. We present the experimental resultsin Section 8 and finally conclude the paper in Section 9.

2 RELATED WORKActivity Detection. In recent years, much effort has been madeto detect human activities from still images as well as videos

!"#$%&'()*+,-'./01!2))13!"&'./%3*)4''5)

!"#$%&'()%,6".!)-)%,6".!)13!"&'./%3*)4%%5)

*$'/'()%,6".!-)%,6".!)13!"&'./%3*)4%%5)

*$'/'()%,6".!)7)*+,-'./01!2)13!"&'./%3*)4%'5)

Fig. 2: An example activity from the CAD-120 dataset (top row) andone possible graph structure (bottom row). Top row shows the RGBimage (left), depths (middle), and the extracted skeleton and objectinformation (right). (Graph in the bottom row shows the nodes at onlythe temporal segment level, the frame level nodes are not shown.)

(e.g., [11], [12]). Many methods have been proposed to modelthe temporal structure of low-level features extracted fromvideo, e.g., histograms of spatiotemporal filter responses. Thisincludes both discriminative [1], [13], [14], [15] and generativemodels [16], [17]. Another approach is to represent activitiesas collections of semantic attributes [18], [19], [20], [21].These methods use an intermediate level of representation suchas the presence or absence of semantic concepts (e.g., scenetypes, actions, objects, etc.) in order to generalize to unseeninstances. There are also a few recent works which addressthe task of early recognition [22], [23]. We refer the readerto [24] for a comprehensive survey of the field and discussworks that are closely related to ours.

Some works use human pose for action recognition bydetecting local pose features [25] and modeling the spatialconfiguration between human body parts and objects [5], [26],[27], [28], [29]. Recent availability of inexpensive RGB-Dsensors has enabled significant improvement in scene mod-eling [30], [31] and estimation of human poses [32], [33].This, together with depth information, has enabled somerecent works [6], [7], [34] to obtain good action recognitionperformance. However, these methods only address detectionover small time periods, where temporal segmentation (i.e.,knowledge of the spatio-temporal structure) is not an issue.

Koppula, Gupta and Saxena (KGS, [5]) proposed a modelto jointly predict sub-activities and object affordances bytaking into account spatio-temporal interactions between hu-man poses and objects over longer time periods. However,KGS found that not knowing the graph structure (i.e., thecorrect temporal segmentation) decreased the performancesignificantly. This is because the boundary between two sub-activities is often not very clear, as people often start per-forming the next sub-activity before finishing the current sub-activity. Moreover, all these work only detect the activitiesand affordance after the action is performed. None of thesemethods can anticipate what is going to happen next.Temporal Segmentation. In activity detection from 2Dvideos, much previous work has focussed on short video clips,assuming that temporal segmentation has been done apriori.It has been observed that temporal boundaries of actions arenot precisely defined in practice, whether they are obtained

3

automatically using weak-supervision [35] or by hand [36].These works represent the action clips by an orderless bag-of-features and try to improve classification of the action clips byrefining their temporal boundaries. However, they only modelthe temporal extent of actions, not their temporal structure.

Some recent effort in recognizing actions from longer videosequences take an event detection approach [15], [37], [38],[39], where they evaluate a classifier function at many dif-ferent segments of the video and then predict event presence.Similarly, change point detection methods [40], [41] performa sequence of change-point analysis in a sliding window alongthe time dimension. However, these methods only detect localboundaries and tend to over-segment complex actions whichoften contain many changes in local motion statistics.

Some previous works consider joint segmentation andrecognition by defining dynamical models based on kinematics[42], [43], but these works do not model the complex human-object interactions. [44] and [45] perform activity classifica-tion and clustering respectively but do not consider temporalcontext. In contrast, our application requires modeling of thetemporal context (as compared to just spatial).

In contemporary work, [46] performs joint segmentation andlabeling using a structural model which takes into account theactivity durations, motion features and context features. Theyuse a nonlinear dynamical model to obtain action segmentswhich are then merged into activities of interest. Wang etal. [47] predict sports moves and human activity in TVepisodes by solving a bilinear program to jointly estimatethe structure of an MRF graph and perform MAP inference.However, these works do not anticipate the future activities.Anticipation of Human Actions. Anticipation or forecastingfuture human actions has been the focus of few recent works.Maximum entropy inverse reinforcement learning was usedby [48], [49], [50] to obtain a distribution over possiblehuman navigation trajectories from visual data, and also usedto model the forthcoming interactions with pedestrians formobile robots [48], [50]. However, these works focus onlyon human actions which are limited to navigation trajectories.Wang et al. [51] propose a latent variable model for inferringunknown human intentions, such as the target ball position ina robot table tennis scenario, and Dragan et al. [52] use inversereinforcement learning to predict future goal for grasping anobject. In comparison, we address the problem of anticipationof human actions at a fine-grained level of how a humaninteracts with objects in more involved activities such asmicrowaving food or taking medicine compared to the genericnavigation activities or task-specific trajectories.Learning Algorithms. Our work uses probabilistic graphicalmodels to capture rich context. Such frameworks as HMMs[53], [54], DBNs [55], CRFs [5], [56], [57], semi-CRFs [58]have been previously used to model the temporal structure ofvideos and text. While these previous works maintain theirtemplate graph structure over time, in our work, new graphstructures are possible. Works on semi-Markov models [58],[59] are quite related as they address the problem of findingthe segmentation along with labeling. However, these methodsare limited since they are only efficient for feature maps thatare additive in nature. We build upon these ideas where we

use additive feature maps only as a close approximation tothe graph structure and then explore the space of likely graphstructure by designing moves. We show that this improvesperformance while being computationally efficient.

For anticipation, we use importance sampling for efficientestimation of the likelihood of the potential future activities.Particle filters have been applied with great success to a varietyof state estimation problems including object tracking [60],[61], mobile robot localization [62], [63], people tracking [64],etc. However, the worst-case complexity of these methodsgrows exponentially in the dimensions of the state space, it isnot clear how particle filters can be applied to arbitrary, high-dimensional estimation problems. Some approaches use factor-izations of the state space and apply different representationsfor the individual parts of the state space model. For example,Rao-Blackwellised particle filters sample only the discrete andnon-linear parts of a state estimation problem. The remainingparts of the states are solved analytically conditioned on theparticles by using Kalman filters [64], [65], [66], [67]. In ourwork, each of our particles is a CRF that models rich structureand lies in a high-dimensional space.

3 OVERVIEWOur goal is to anticipate what a human will do next giventhe current observation of his pose and the surrounding envi-ronment. These observations are from RGB-D videos recordedwith a Kinect sensor. From these videos, we obtain the humanpose using the Openni’s skeleton tracker [68] and extractthe tracked object point clouds using SIFT feature matchingas described in KGS [5].3 Note that we infer the objectaffordances based on its usage in the activity and do not requirethe object category labels. We discuss the effect of knowing theobject categories on the anticipation performance in Section5.3.

Since activities happen over a long time horizon, with eachactivity being composed of sub-activities involving differentnumber of objects, we perform segmentation in time, asdescribed in Section 7.1, such that each temporal segmentrepresents one sub-activity. We model the activity using aspatio-temporal graph (a CRF), as shown in Figure 3-left. Theextracted human pose and objects form the nodes in this graph,and the edges between them represent their interactions, asdescribed in Section 5.1. However, this graph can only modelthe present observations. In order to predict the future, weaugment the graph with an ‘anticipated’ temporal segment,with anticipated nodes for sub-activities, objects (their affor-dances), and the corresponding spatio-temporal trajectories.We call this augmented graph an anticipatory temporal CRF(ATCRF), formally defined in Section 5.2.

Our goal is to obtain a distribution over the future possibil-ities, i.e., a distribution over possible ATCRFs. Motivated byparticle filtering algorithm [69], we represent this distributionas a set of weighted particles, where each particle is a sampledATCRF. Partial observations become available as the sub-activity is being performed and we use these partial obser-vations to improve the estimation of the distribution. Section

3. The skeleton tracker identifies the human hand joints within an error of10 cm in 82.7% of frames. The SIFT based object tracker gives a boundingbox with atleast 10% overlap with the object in 77.8% of the frames.

4

temporal segment ‘k’! temporal segment ‘k+1’!

!past time ‘t’! !future time ‘t’!

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

A

O1

O2

O3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

A

O1

O2

O3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

A

O1

O2

O3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

A

O1

O2

O3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

A

O1

O2

O3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

A

O1

O2

O3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

A

O1

O2

O3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

A

O1

O2

O3

1

Fig. 3: An ATCRF that models the human poses H, object affordancelabels O, object locations L, and sub-activity labels A, over past time‘t’, and future time ‘d’. Two temporal segments are shown in thisfigure: k

th for the recent past, and (k + 1)

th for the future. Eachtemporal segment has three objects for illustration in the figure.

5.3 describes this approach. Since each of our ATCRF capturesstrong context over time (which sub-activity follows another)and space (spatial motion of humans and objects, and theirinteractions), each of our particles (i.e., possible future) is richin its modeling capacity. Later, our experiments in Section 8will show that this is essential for anticipating human actions.

Anticipated temporal segments are generated based on theavailable object affordances and the current configuration ofthe 3D scene. For example, if a person has picked up a coffeemug, one possible outcome could be drinking from it. There-fore, for each object, we sample possible locations at the endof the anticipated sub-activity and several trajectories based onthe selected affordance. The location and trajectory generationare described in Section 4.1 and Section 4.2 respectively.

The temporal segmentation determines the structure of theATCRF. It is quite challenging to estimate this structurebecause of two reasons. First, an activity comprises severalsub-activities of varying temporal length, with an ambiguity inthe temporal boundaries. Thus a single graph structure may notexplain the activity well. Second, there can be several possiblegraph structures when we are reasoning about activities in thefuture (i.e., when the goal is to anticipate future activities,different from just detecting the past activities). Multiplespatio-temporal graphs are possible in these cases and we needto reason over them in our learning algorithm.

Figure 4 shows two possible graph structures for an ac-tivity with two objects. We reason about the possible graphstructures for both past and future activities. The key ideais to first sample a few segmentations that are close to theground-truth using our CRF model instantiated with a subsetof features, and then explore the space of segmentation bymaking merge and split moves to create new segmentations.We do so by approximating the graph with only additivefeatures, which lends to efficient dynamic programming, asdescribed in Section 7.1.

4 OBJECT AFFORDANCESThe concept of affordances was proposed by Gibson [10] asall “action possibilities” provided by the environment. Manyrecent works in computer vision and robotics reason aboutobject functionality (e.g., sittable, drinkable, etc.) instead ofobject identities (e.g., chairs, mugs, etc.). These works take arecognition based approach to identify the semantic affordancelabels [70], [71], [72]. Few recent works explore the physical

future frames past frames

future frames past frames

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

A

O1

O2

O3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

A

O1

O2

O3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

A

O1

O2

O3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

A

O1

O2

O3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

A

O1

O2

O3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

A

O1

O2

O3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

A

O1

O2

O3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

A

O1

O2

O3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

A

O1

O2

O3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

A

O1

O2

O3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

A

O1

O2

O3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

A

O1

O2

O3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

A

O1

O2

O3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

A

O1

O2

O3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

A

O1

O2

O3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

A

O1

O2

O3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

A

O1

O2

O3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

A

O1

O2

O3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

1

gt,d(1)

gt,d(2)

gt,d(S)

H

L1

L2

L3

o1

o2

o3

1

Fig. 4: Figure illustrating two possible graph structures resulting fromtwo temporal segmentations (top and bottom), with six observedframes in the past and three anticipated frames in the future. Thisexample has one sub-activity node and two object nodes in eachtemporal segment.

aspects of affordances based on human interactions [5], [73],[74], [75], [76]. For example, Grabner et al. [73] detect thefunctionality of the object (specifically, chairs) with respectto possible human poses, Gupta et al. [74] predict stable andfeasible human poses given an approximate 3D geometry froman image, etc. In our work, we consider semantic affordanceswith spatio-temporal grounding which help in anticipating thefuture activities. Here, we describe how we model the spatio-temporal aspects of affordances.

4.1 Object Affordance HeatmapsTo represent object affordances we define a potential functionbased on how the object is being interacted with, when thecorresponding affordance is active. The kind of interactionwe consider depends on the affordance being considered. Forexample, when the active affordance of an object is drinkable,the object is found near the human’s mouth, the interactionconsidered is the relative position of the object with respect tothe human skeleton. In case of the affordance placeable, theinteraction is the relative position of the object with respect tothe environment, i.e., an object is placeable when it is abovea surface that provides stability to the object once placed. Thegeneral form of the potential function for object affordance o

given the observations at time t is:

o

=

Y

i

dist

i

Y

j

ori

j

(1)

where dist

i

is the i

th distance potential and ori

j

is the j

th

relative angular potential. We model each distance potentialwith a Gaussian distribution and each relative angular potentialwith a von Mises distribution. We find the parameters of theaffordance potential functions from the training data usingmaximum likelihood estimation. Since the potential functionis a product of the various components, the parameters ofeach distribution can be estimated separately. In detail, themean and variance of the Gaussian distribution have closedform solutions, and we numerically estimate the mean andconcentration parameter of the von Mises distribution.

5

Fig. 5: Affordance heatmaps. The learnt affordance heatmaps for placeability (top-left), reachability (top-right), pourability (bottom-left)and drinkability (bottom-right). The red signifies where the affordance is most likely, for example, the red signifies where the object isplaceable (top-left) and the most likely reachable locations on the object (top-right) (See Section 4.1).

Fig. 6: Heatmap of anticipated trajectories for moving sub-activity and how they evolve with time.

We categorize these functions into three groups dependingon the potentials used: (1) affordances drinkable and reachablehave one distance potential per skeleton joint and one angularpotential with respect to the head orientation, (2) affordancesdepending on the target object, such as pourable which de-pends on a pour-to object, have a distance potential and anangular potential with respect to the target object’s location, (3)the rest of the affordances which depend on the environment,such placeable and openable, have a distance potential withrespect to the closest surface and an angular potential withrespect to the head orientation.

We generate heatmaps for each affordance by scoring thepoints in the 3D space using the potential function, and thevalue represents the strength of the particular affordance atthat location. Figure 5 shows the heatmaps generated for theplaceable, reachable, pourable and drinkable affordances. Weobtain the future target locations of an object by weightedsampling of the scored 3D points.

4.2 Trajectory GenerationOnce a location is sampled from the affordance heatmap, wegenerate a set of possible trajectories in which the object canbe moved form its current location to the predicted targetlocation. We use parametrized cubic equations, in particularBezier curves, to generate human hand like motions [77].

B(x) = (1� x)

3L0 + 3(1� x)

2xL1

+ 3(1� x)x

2L2 + x

3L3 , x 2 [0, 1] (2)

We estimate the control points of the Bezier curves forthe proposal distribution component from the trajectories inthe training data. A cubic Bezier curve, as shown in Eq. 2,is parameterized by a set of four points: the start and endpoint of the trajectory (L0 and L3 respectively), and twocontrol points (L1 and L2) which define the shape of thecurve. We first transform and normalize the trajectories inthe training data so that all of them have the same start andend points. We then estimate the control points of the Beziercurve, one per sub-activity class, which best fit the normalizedtrajectories. In detail, L0 and L3 are the start and end points

of the normalized trajectories, respectively, and L1 and L2 areestimated using the least square fitting method to minimizethe distance between the fitted and the observed normalizedtrajectories. Figure 6 shows some of the anticipated trajectoriesfor moving sub-activity.

5 OUR APPROACHGiven the observations of a scene containing a human andobjects for time t in the past, and its goal is to anticipatefuture possibilities for time d.

However, for the future d frames, we do not even knowthe structure of the graph—there may be different number ofobjects being interacted with depending on which sub-activityis performed in the future. Our goal is to compute a distribu-tion over the possible future states (i.e., sub-activity, humanposes and object locations). We will do so by sampling severalpossible graph structures by augmenting the graph in time,each of which we will call an anticipatory temporal conditionalrandom field (ATCRF). We first describe an ATCRF below.

5.1 Modeling Past with a CRFMRFs/CRFs are a workhorse of machine learning and havebeen applied to a variety of applications. Recently, withRGB-D data they have been applied to scene labeling [78] andactivity detection [5]. Conditioned on a variety of features asinput, the CRFs model rich contextual relations. Learning andinference is tractable in these methods when the label spaceis discrete and small.

Following [5], we discretize time to the frames of thevideo4 and group the frames into temporal segments, whereeach temporal segment spans a set of contiguous framescorresponding to a single sub-activity. Therefore, at time ‘t’we have observed ‘t’ frames of the activity that are groupedinto ‘k’ temporal segments. For the past t frames, we knowthe nodes of the CRF but we do not know the temporalsegmentation, i.e., which frame level nodes are connected to

4. In the following, we will use the number of videos frames as a unit oftime, where 1 unit of time ⇡ 71ms (=1/14, for a frame-rate of about 14Hzin our experiments).

6

each of the segment level node. The node labels are alsounknown. For a given temporal segmentation, we represent thegraph until time t as: Gt

= (Vt

, Et

), where Et represents theedges, and Vt represents the nodes {Ht

,Ot

,Lt

,At}: humanpose nodes Ht, object affordance nodes Ot, object locationnodes Lt, and sub-activity nodes At. Figure 3-left part showsthe structure of this CRF for an activity with three objects.

Our goal is to model the P (Ht

,Ot

,Lt

,At|�t

H,�

t

L), where�

t

H and �t

L are the observations for the human poses and ob-ject locations until time t. Using the independencies expressedover the graph in Figure 3, for a graph Gt, we have:

PGt(Ht

,Ot

,Lt

,At|�t

H,�

t

L) =

P (Ot

,At|Ht

,Lt

)P (Ht

,Lt|�t

H,�

t

L) (3)

The second term P (Ht

,Lt|�t

H,�

t

L) models the distributionof true human pose and object locations (both are continuoustrajectories) given the observations from the RGB-D Kinectsensor. We model it using a Gaussian distribution. The firstterm P (Ot

,At|Ht

,Lt

) predicts the object affordances andthe sub-activities that are discrete labels—this term furtherfactorizes following the graph structure as:

P (Ot

,At|Ht

,Lt

)/

object affordancez }| {Y

o

i

2O

O(o

i

|`o

i

)

sub-activityz }| {Y

a

i

2A

A(a

i

|ha

i

)

Y

v

i

,v

j

2E

edge termsz }| { E(vi, vj |·)

(4)Given the continuous state space of H and L, we

rely on [5] for powerful modeling using a discriminativeframework for the above term. Each node potential func-tion in Eq. (4), O(oi|`o

i

) and A(ai|ha

i

), has the formPk2K

y

k

i

⇥w

k

n

· �n

(i)

⇤, where y

k

i

denotes a binary variablerepresenting the node i having label k, K is the set of labels,�

n

(i) is the node feature map and w

k

n

are the correspondingnode feature weights. Similarly, each edge potential function, E(vi, vj |·), has the form

P(l,k)2K⇥K

y

l

i

y

k

j

⇥w

lk

e

· �e

(i, j)

⇤,

where �

e

(i, j) is the edge feature map and w

lk

e

are thecorresponding edge feature weights. The feature maps, �

n

(i)

and �

e

(i, j), are described in detail in Section 6. We canrewrite Eq. (4) by taking logarithm on both sides and groupingthe node potentials as the following energy function expressedover a graph, G = (V, E) (as illustrated in Figure 2):

E(y|�(x);w) =

X

i2V

X

k2K

y

k

i

⇥w

k

n

· �n

(i)

⇤

+

X

(i,j)2E

X

(l,k)2K⇥K

y

l

i

y

k

j

⇥w

lk

e

· �e

(i, j)

⇤(5)

Depending on the nodes and edges in the graph G, theappropriate subset of features and class labels are used inenergy function given in Eq. (5). The learning and inferencealgorithms are described in Section 7.

5.2 Modeling one Possible Future with an aug-mented temporal CRF (ATCRF).We defined the anticipatory temporal conditional random fieldas an augmented graph Gt,d

= (Vt,d

, Et,d

), where t isobserved time and d is the future anticipation time. Vt,d

=

{Ht,d

,Ot,d

,Lt,d

,At,d} represents the set of nodes in the pasttime t as well as in the future time d. Et,d represents the setof all edges in the graph (see Figure 3). The observations (notshown in the figure) are represented as set of features, �t

H and

!" !#$"!%$"

!" !#$"!%$"

!" !#$"!%$"

CRF for k observed temporal segments

&'()*+",-.

)/"

01'%2*345+6

"",-

.)/"

!"!%$"

ATCRFs with anticipated temporal segment k+1

gt,d(1)

gt,d(2)

gt,d(S)

1

gt,d(1)

gt,d(2)

gt,d(S)

1

gt,d(1)

gt,d(2)

gt,d(S)

1

Fig. 7: Figure showing the process of augmenting the CRF structureto obtain multiple ATCRFs at time t for an activity with three objects.The frame level nodes are not shown for the sake of clarity.

�

t

O, extracted from the t observed video frames. Note that wedo not have observations for the future frames.

In the augmented graph Gt,d, we have:PGt,d

(Ht,d

,Ot,d

,Lt,d

,At,d|�t

H,�

t

L) =

P (Ot,d

,At,d|Ht,d

,Lt,d

)P (Ht,d

,Lt,d|�t

H,�

t

L) (6)

The first term is similar to Eq. (4), except over the augmentedgraph, and we can still rely on the discriminatively trainedCRF presented in [5]. We model the second term with aGaussian distribution.

5.3 Modeling the Distribution over Future Possibili-ties with ATCRFs.There can be several potential augmented graph structures Gt,d

because of different possibilities in human pose configurationsand object locations that determines the neighborhood graph.Even the number of nodes to be considered in the futurechanges depending on the sub-activity and the configurationof the environment.

Let gt,d represent a sample augmented graph structure withparticular values assigned to its node variables. I.e., one samplemay represent that a person and object move in a certain way,performing a sub-activity with certain object affordances, andanother sample may represent a person moving in a differentway performing a different sub-activity.

Figure 7 shows the process of augmenting CRF structurecorresponding to the seen frames with the sampled anticipa-tions of the future to produce multiple ATCRF particles at timet. The frame level nodes are not shown in the figure. The leftportion of the figure shows the nodes corresponding to the k

observed temporal segments. This graph is then augmentedwith a set of anticipated nodes for the temporal segmentk + 1, to generate the ATCRF particles at time t. The framelevel nodes of k + 1 temporal segment are instantiated withanticipated human poses and object locations.

The goal is now to compute the distribution over theseATCRFs g

t,d, i.e., given observations until time t, we wouldlike to estimate the posterior distribution p(g

t,d|�t

) fromEq. (6). However, this is extremely challenging because thespace of ATCRFs is infinite, and to even represent the distri-bution we need an exponential number of labels. We thereforerepresent the posterior using a set of weighted particles asshown in Eq. (7) and choose the weights using importancesampling as shown in Eq. (8).

7

p(g

t,d|�t

) ⇡SX

s=1

w

s

t

�

gt,d

(s)(gt,d

) (7)

w

s

t

/ p(g

t,d

(s)|�t

)

q(g

t,d

(s)|�t

)

(8)

Here, �x

(y) is the Kronecker delta function which takes thevalue 1 if x equals y and 0 otherwise, w

s

t

is the weightof the sample s after observing t frames, and q(g

t,d|�t

)

is the proposal distribution. We need to perform importancesampling because: (a) sampling directly from p(g

t,d|�t

) isnot possible because of the form of the distribution in adiscriminative framework, and (b) sampling uniformly wouldbe quite naive because of the large space of ATCRFs and mostof our samples would entirely miss the likely futures.

We now describe how we sample particles from the proposaldistribution q(g

t,d|�t

) and how to evaluate the posterior forthe generated samples.Sampling. In order to generate a particle ATCRF, we need togenerate possible human pose and object locations for the d

future frames. We write the desired distribution to sample as:q(g

t,d|�t

) = PGt,d

(Ht,d

,Ot,d

,Lt,d

,At,d|�t

H,�

t

L)

= PGt(Ht

,Ot

,Lt

,At|�t

H,�

t

L)

P (Hd

,Ld|Od

,Ad

,�

t

H,�

t

L, )P (Od

,Ad|Ot

,At

,�

t

H,�

t

L) (9)

We first sample the affordances, one per object in thescene, and the corresponding sub-activity from the distributionP (Od

,Ad|�t

H,�

t

L). This is a discrete distribution generatedfrom the training data based on the object type (e.g., cup,bowl, etc.) and object’s current position with respect to thehuman in the scene (i.e., in contact with the hand or not).For example, if a human is holding an object of type ‘cup’placed on a table, then the affordances drinkable and movablewith their corresponding sub-activities (drinking and movingrespectively) have equal probability, with all others being 0.5

Once we have the sampled affordances and sub-activity,we need to sample the corresponding object locations andhuman poses for the d anticipated frames from the distribu-tion P (Hd

,Ld|Od

,Ad

,�

t

H,�

t

L). In order to have meaningfulobject locations and human poses we take the followingapproach. We sample a set of target locations and motiontrajectory curves based on the sampled affordance, sub-activityand available observations. We then generate the correspond-ing object locations and human poses from the sampled endpoint and trajectory curve. The details of sampling the targetobject location and motion trajectory curves are described inSection 4.1 and Section 4.2 respectively.Scoring. Once we have the sampled ATCRF particles, weobtain the weight of each sample s by evaluating the posteriorfor the given sample, q(gt,d

(s)|�t

), as shown in Eq. (9) andnormalize the weights across the samples.

Note that the aforementioned methods for the affordanceand trajectory generation are only for the proposal distribution

5. If we know the object type as well, then it helps in generating informativesamples based on the discrete distribution learnt from the training data, whichcan save computation time as fewer samples are sufficient. In our experiments,when object type information is not provided, the anticipation performance(micro P/R) for sub-activities and affordances only drops by a maximum of0.3% for our full model (Table 2-row 6).

to sample. The estimated trajectories are finally scored usingour ATCRF model.

6 FEATURES: ADDITIVE AND NON-ADDITIVEIn this section we describe the feature maps used in theenergy function Eq. 5. In detail, we use the same featuresas described by KGS [5]. These features include the nodefeature maps �

o

(i) and �

a

(j) for object node i and sub-activity node j respectively, and edge feature maps �

e

(i, j)

capturing the relations between various nodes. The object nodefeature map, �

o

(i), includes the (x, y, z) coordinates of theobject’s centroid, the coordinates of the object’s boundingbox and transformation matrix w.r.t. to the previous framecomputed at the middle frame of the temporal segment, thetotal displacement and distance moved by the object’s centroidin the set of frames belonging to the temporal segment. Thesub-activity node feature map, �

a

(j), gives a vector of featurescomputed using the noisy human skeleton poses obtainedfrom running Openni’s skeleton tracker on the RGBD video.We compute the above described location (relative to thesubject’s head location) and distance features for each theupper-skeleton joints excluding the elbow joints (neck, torso,left shoulder, left palm, right shoulder and right palm).

The edge feature maps, �t

(i, j), include relative geometricfeatures such as the difference in (x, y, z) coordinates ofthe object centroids and skeleton joint locations and thedistance between them. In addition to computing these valuesat the first, middle and last frames of the temporal segment,we also consider the min and max of their values acrossall frames in the temporal segment to capture the relativemotion information. The temporal relational features capturethe change across temporal segments and we use the verticalchange in position and the distance between correspondingobject and joint locations. We perform cumulative binning ofall the feature values into 10 bins for each feature.

We categorize the features into two sets: additive features,�

A

(x), and non-additive features, �NA

(x). We compute theadditive features for a set of frames corresponding to atemporal segment by adding the feature values for the framesbelonging to the temporal segment. Examples of the additivefeatures include distance moved and vertical displacement ofan object within a temporal segment. The features that do notsatisfy this property are referred to as the non-additive features,for example, maximum and minimum distances between twoobjects. As we discuss in the next section, additive features al-low efficient joint segmentation and labeling by using dynamicprogramming, but may not be expressive enough.

Non-additive features sometimes provide very useful cuesfor discriminating the sub-activity and affordance classes. Forexample, consider discriminating cleaning sub-activity froma moving sub-activity: here the total distance moved couldbe similar (an additive feature), however, the minimum andmaximum distance moved being small may be strong indicatorof the activity being cleaning. In fact, when compared toour model learned using only the additive features, the modellearned with both additive and non-additive features improvesmacro precision and recall by 5% and 10.1% for labelingobject affordance respectively and by 3.7% and 6.2% forlabeling sub-activities respectively.

8

7 LEARNING AND INFERENCE7.1 InferenceThe inference problem is to find the best labeling of the pastnodes of the CRF for detecting the past sub-activities andaffordances and past as well as augmented future nodes ofthe ATCRF for anticipation. The prediction ˆ

y is computed asthe argmax of an energy function E(y|�(x);w).

ˆ

y = argmax

yE(y|�(x);w) (10)

For a given temporal segmentation, where the graph struc-ture is fully known, finding the argmax over labelings is aNP hard problem. However, its equivalent formulation as thefollowing mixed-integer program has a linear relaxation whichcan be solved efficiently as a quadratic pseudo-Boolean opti-mization problem using a graph-cut method [79] as describedin [5].

ˆy=argmax

ymax

z

X

i2V

X

k2K

yk

i

⇥wk

n

· �n

(i)⇤

+

X

(i,j)2E

X

(l,k)2K⇥K

zlkij

⇥wlk

e

· �e

(i, j)⇤

(11)

8i, j, l, k : zlkij

yl

i

, zlkij

yk

j

, yl

i

+ yk

j

zlkij

+ 1, zlkij

, yl

i

2 {0, 1}(12)

Note that the products yli

y

k

j

have been replaced by auxiliaryvariables z

lk

ij

. Relaxing the variables z

lk

ij

and y

l

i

to the interval[0, 1] results in a linear program that can be shown to alwayshave half-integral solutions (i.e. yl

i

only take values {0, 0.5, 1}at the solution) [80]. When we consider the additional con-straints that each node can take only one label, the problemcan no longer be solved via graph cuts. We compute the exactmixed integer solution including these additional constraintsusing a general-purpose MIP solver6 during inference.

However, during inference we only know the nodes in thegraph but not the temporal segmentation, i.e., the structureof the graph in terms of the edges connecting frame levelnodes to the segment level label nodes. We could search forthe best labeling over all possible segmentations, but this isvery intractable because our feature maps contain non-additivefeatures (that are important as described in Section 6).

Efficient Inference with Additive Features. We expressthe feature set, �(x), as the concatenation of the additive andnon-additive feature sets, �A

(x) and �NA

(x) respectively.Therefore, by rearranging the terms in Eq. (4), the energyfunction can be written as:

E(y|�(x);w) = E(y|�A

(x);w) + E(y|�NA

(x);w)

We perform efficient inference for the energy termE(y|�A

(x);w) by formulating it as a dynamic program (seeEq. (13)). In detail, let L denote the max length of a temporalsegment, i denote the frame index, s denote the temporalsegment spanning frames (i� l) to i, and (s� 1) denote theprevious segment. We write the energy function in a recursiveform as:

V (i, k) = max

k

0,l=1...L

V (i� l, k

0) +

X

k2K

y

k

s

⇥w

k

n

· �A

n

(s)

⇤

+

X

k2K

y

k

s

⇥w

lk

e

· �A

e

(s� 1, s)

⇤(13)

Here, �An

(s) and �

A

e

(s� 1, s) denote the additive featuremaps and can be efficiently computed by using the concept

6. http://www.tfinley.net/software/pyglpk/readme.html

of integral images.7 The best segmentation then correspondsto the path traced by max

a

V (t, a), where t is the number ofvideo frames. Using E(y|�A

(x);w), we find the top-k scoredsegmentations8 and then evaluate them using the full modelE(y|�(x);w) in order to obtain more accurate labelings.

Merge and Split Moves. The segmentations generated bythe approximate energy function, E(y|�A

(x);w), are oftenvery close to the given ground-truth segmentations. However,since the energy function used is only approximate, it some-times tends to over-segment or miss the boundary by a fewframes. In order to obtain a representative set of segmentationsamples, we also perform random merge and split moves overthese segmentations, and consider them for evaluating with thefull model as well. A merge move randomly selects a boundaryand removes it, and a split move randomly chooses a framein a segment and creates a boundary.

Heuristic Segmentations. There is a lot of informationpresent in the video which can be utilized for the purposeof temporal segmentation. For example, smooth movementof the skeleton joints usually represent a single sub-activityand the sudden changes in the direction or speed of motionindicate sub-activity boundaries. Therefore, we incorporatesuch information in performing temporal segmentation ofthe activities. In detail, we use the multiple segmentationhypotheses proposed by KGS. These include graph basedsegmentation method proposed by [81] adapted to temporallysegment the videos. The sum of the Euclidean distancesbetween the skeleton joints and the rate of change of theEuclidean distance are used as the edge weights for twoheuristic segmentations respectively. By varying the thresh-olds, different temporal segmentations of the given activitycan be obtained. In addition to the graph based segmentationmethods, we also use the uniform segmentation method whichconsiders a set of continuous frames of fixed size as thetemporal segment. There are two parameters for this method:the segment size and the offset (the size of the first segment).However, these methods often over-segment a sub-activity, andeach segmentation would result in a different graph structurefor our CRF modeling.

We generate multiple graph structures for various values ofthe parameters for the above mentioned methods and obtainthe predicted labels for each using Eq. (10). We obtain thefinal labeling over the segments by either using the second-steplearning method presented in KGS, or by performing votingand taking the label predicted by majority of the sampledgraph structures (our experiments in Section 8.2 follow thelatter). During anticipation, we only consider the best graphstructure obtained form the using the additive features andsplit-merge moves. Algorithm 1 gives the summary of theinference algorithm for anticipation.

7. The additive features for temporal segments starting at the first frame andending at frame l, for l = 1..t are precomputed, i.e., the segment features fora total of t temporal segments are computed. This needs (n⇥ t) summations,where n is the number of features. Now the segment features for a temporaltemporal segment starting and ending at any frame can be computed by nsubtractions. Therefore, the total feature computation cost is linear in thenumber of possible segmentations.

8. k is 2 in experiments.

9

Data: RGB-D video framesResult: Future sub-activity and affordance anticipationst = 0, P = {} ;while new frame f

t

observed doGenerate frame features for frame f

t

(Section 6);if temporal segmentation not given then

Find best segmentation using additive energyE(y|�A

(x);w) (solve Eq. 13);Sample segmentations by split and merge moves(Section 7.1);

endCompute segment features �

n

and �e

(Section 6);Compute ˆ

y, best labeling of the past-CRF (Eq. 11-12);for each object do

Sample possible future affordance and sub-activityfrom the discrete distribution P (Od

,Ad|�t

H,�

t

L);Sample future object location based on the affordanceheatmaps

o

;Generate corresponding object trajectory and humanposes for d future frames;Augment the past-CRF to generate an ATCRF particleg

t,d

(s) ;P = P [ {gt,d

(s)};endfor each particle g

t,d

(s) 2 P dofor each augmented frame do

Generate frame features (Section 6);endif temporal segmentation not given then

Find best segmentation using additive energyE(y|�A

(x);w) (solve Eq. 13);Sample segmentations by split and merge moves(Section 7.1);

endCompute segment features �

n

and �e

(Section 6);Compute ˆ

y, best labeling for the ATCRF particle(Eq. 11-12);Compute weight ws

t

(Eq. 8);endP = top-k scored particles in P ;A

t

= future sub-activity and affordance labels of top-3particles based on E(y|�(x);w);t = t+ 1;

endAlgorithm 1: Summary of our inference method.

7.2 LearningThe structure of the graph is fully known during learning.We obtain the parameters of the energy function in Eq. (5)by using the cutting plane method [82] as described in [5].Given M labeled training examples (x1,y1), .., (xM

,y

M

), weoptimize the regularized upper bound on the training error.

R(h) =

1M

PM

m=1�(y

m

,

ˆ

y

m

),

where ˆ

y

m

is the optimal solution of Eq. (10) and �(y,

ˆ

y) isthe loss function defined as

�(y,

ˆ

y) =

Pi2V

o

Pk2K

o

|yk

i

� y

k

i

|+P

i2Va

Pk2K

a

|yk

i

� y

k

i

|.

To simplify notation, note that Eq. (11) can be equivalentlywritten as w

T

�(x,y) by appropriately stacking the w

k

n

andw

lk

t

into w and the y

k

i

�

a

(i), y

k

i

�

o

(i) and z

lk

ij

�

t

(i, j) into�(x,y), where each z

lk

ij

is consistent with Eq. (12) given y.Training can then be formulated as the following convexquadratic program [82]:

min

w,⇠

1

2

w

T

w + C⇠ (14)

s.t. 8¯y1, ..., ¯yM

2 {0, 0.5, 1}N·K:

1

M

w

T

MX

m=1

[�(x

m

,y

m

)� �(xm

,

¯

y

m

)] � �(y

m

,

¯

y

m

)� ⇠,

where N is the number of temporal segments per exampleand K is the total number class labels. While the numberof constraints in this QP is exponential in M , N and K, itcan nevertheless be solved efficiently using the cutting-planealgorithm [82]. The algorithm needs access to an efficientmethod for computing

¯

y

m

= argmax

y2{0,0.5,1}N·K

⇥w

T

(x

m

,y) +�(y

m

,y)

⇤. (15)

Due to the structure of �(., .), this problem is identical tothe relaxed prediction problem in Eqs. (11)-(12) and can besolved efficiently using graph cuts.

8 EXPERIMENTSIn this section we describe the detailed evaluation of ourapproach on both offline data as well as realtime roboticexperiments. We first give the details of the dataset in Section8.1. We then present the detection results in Section 8.2,anticipation results in Section 8.3 and the robotic experimentsfor anticipation in Section 8.4.

8.1 DataWe use CAD-120 dataset [5], which has 120 RGB-D videosof four different subjects performing 10 high-level activities.The data is annotated with object affordance and sub-activitylabels and includes ground-truth object categories, trackedobject bounding boxes and human skeletons. The set of high-level activities are: {making cereal, taking medicine, stackingobjects, unstacking objects, microwaving food, picking objects,cleaning objects, taking food, arranging objects, having ameal}, the set of sub-activity labels are: {reaching, moving,pouring, eating, drinking, opening, placing, closing, scrub-bing, null} and the set of affordance labels are: {reachable,movable, pourable, pourto, containable, drinkable, openable,placeable, closable, scrubbable, scrubber, stationary}. We useall sub-activity classes for prediction of observed frames butdo not anticipate null sub-activity.

8.2 Detection ResultsFor comparison, we follow the same train-test split describedin KGS [5] and train our model on activities performed bythree subjects and test on activities of a new subject. We reportthe results obtained by 4-fold cross validation by averagingacross the folds. We consider the overall micro accuracy(P/R), macro precision and macro recall of the detected sub-activities, affordances and overall activity. Micro accuracy isthe percentage of correctly classified labels. Macro precisionand recall are the averages of precision and recall respectivelyfor all classes.

Table 1 shows the performance of our proposed approach onobject affordance, sub-activity and high-level activity labelingfor past activities. Rows 3-5 show the performance for thecase where ground-truth temporal segmentation is providedand rows 6-9 show the performance for the different methodswhen no temporal segmentation is provided. With known

10

TABLE 1: Results on CAD-120 dataset for detection, showing average micro precision/recall, and average macro precision and recall foraffordances, sub-activities and high-level activities. Computed from 4-fold cross validation with testing on a new human subject in each fold.Standard error is also reported. With ground-truth segmentation.

Object Affordance Sub-activity High-level Activitymicro macro micro macro micro macro

method P/R Prec. Recall P/R Prec. Recall P/R Prec. Recallchance 8.3 (0.0) 8.3 (0.0) 8.3 (0.0) 10.0 (0.0) 10.0 (0.0) 10.0 (0.0) 10.0 (0.0) 10.0 (0.0) 10.0 (0.0)max class 65.7 (1.0) 65.7 (1.0) 8.3 (0.0) 29.2 (0.2) 29.2 (0.2) 10.0 (0.0) 10.0 (0.0) 10.0 (0.0) 10.0 (0.0)KGS [5] 91.8 (0.4) 90.4 (2.5) 74.2 (3.1) 86.0 (0.9) 84.2 (1.3) 76.9 (2.6) 84.7 (2.4) 85.3 (2.0) 84.2 (2.5)Our model: all features 93.9 (0.4) 89.2 (1.3) 82.5 (2.0) 89.3 (0.9) 87.9 (1.8) 84.9 (1.5) 93.5 (3.0) 95.0 (2.3) 93.3 (3.1)Our model: only additive features 92.0 (0.5) 84.2 (2.2) 72.4 (1.2) 86.5 (0.6) 84.2 (1.3) 78.7 (1.9) 90.3 (3.8) 92.8 (2.7) 90.0 (3.9)

Without ground-truth segmentation.Our DP seg. 83.6 (1.1) 70.5 (2.3) 53.6 (4.0) 71.5 (1.4) 71.0 (3.2) 60.1 (3.7) 80.6 (4.1) 86.1 (2.5) 80.0 (4.2)Our DP seg. + moves 84.2 (0.9) 72.6 (2.3) 58.4 (5.3) 71.2 (1.1) 70.6 (3.7) 61.5 (4.5) 83.1 (5.2) 88.0 (3.4) 82.5 (5.4)heuristic seg. (KGS) 83.9 (1.5) 75.9 (4.6) 64.2 (4.0) 68.2 (0.3) 71.1 (1.9) 62.2 (4.1) 80.6 (1.1) 81.8 (2.2) 80.0 (1.2)Our DP seg. + moves + heuristic seg. 85.4 (0.7) 77.0 (2.9) 67.4 (3.3) 70.3 (0.6) 74.8 (1.6) 66.2 (3.4) 83.1 (3.0) 87.0 (3.6) 82.7 (3.1)

TABLE 2: Anticipation Results of Future Activities and Affordances, computed over 3 seconds in the future (similar trends hold for otheranticipation times).

modelAnticipated Sub-activity Anticipated Object Affordance

micro P/R macro F1-score robot anticipation metric micro P/R marco F1-score robot anticipation metricchance 10.0 ± 0.1 10.0 ± 0.1 30.0 ± 0.1 8.3 ± 0.1 8.3 ± 0.1 24.9 ± 0.1Nearest-neighbor 22.0 ± 0.9 10.6 ± 0.6 48.1 ± 0.5 48.3 ± 1.5 17.2 ± 1.0 60.9 ± 1.1KGS [5] + co-occurence 28.6 ± 1.8 11.1 ± 0.4 34.6 ± 2.8 55.9 ± 1.7 11.6 ± 0.4 62.0 ± 1.8ATCRF-discrete 34.3 ± 0.8 12.2 ± 0.2 44.8 ± 1.1 59.5 ± 1.5 12.4 ± 0.3 67.6 ± 1.3ATCRF 47.7 ± 1.6 37.9 ± 2.6 69.2 ± 2.1 66.1 ± 1.9 36.7 ± 2.3 71.3 ± 1.7Ours-full 49.6 ± 1.4 40.6 ± 1.6 74.4 ± 1.6 67.2 ± 1.1 41.4 ± 1.5 73.2 ± 1.0

graph structure, the model using the the full set of features(row 4) outperforms the model which uses only the additivefeatures (row 5): macro precision and recall improve by 5%and 10.1% for labeling object affordance respectively and by3.7% and 6.2% for labeling sub-activities respectively. Thisshows that additive features bring us close, but not quite, tothe optimal graph structure.

When the graph structure is not known, the performancedrops significantly. Our graph sampling approach based onthe additive energy function (row 6) achieves 83.6% and71.5% micro precision for labeling object affordance and sub-activities, respectively. This is improved by sampling addi-tional graph structures based on the Split and Merge moves(row 7). Finally, combining these segmentations with the otherheuristically generated segmentations presented by KGS, ourmethod obtains the best performance (row 9) and significantlyimproves the previous state-of-the-art (KGS, row 8).

Figure 8 shows the confusion matrix for labeling af-fordances, sub-activities and high-level activities using ourmethod (row 9). Note that there is a strong diagonal with a fewerrors such as pouring misclassified as moving, and pickingobjects misclassified as having a meal. Figure 9 shows thelabeling output of the different methods. The bottom-most rowshow the ground-truth segmentation, top-most row is the la-beling obtained when the graph structure is provided, followedby three heuristically generated segmentations. The fifth row isthe segmentation generated by our sampling approach and thesixth and seventh rows are the labeling obtained by combiningthe multiple segmentations using a simple max-voting and bythe multi-segmentation learning of KGS. Note that some sub-activity boundaries are more ambiguous (high variance amongdifferent methods) than the others. Our method has an end-to-end (including feature computation cost) frame rate of 4.3frames/sec compared to 16.0 frames/sec of KGS.

8.3 Anticipation ResultsBaseline Algorithms. We compare our method against thefollowing baselines: 1) Chance. The anticipated sub-activityand affordance labels are chosen at random.

Fig. 9: Illustration of the ambiguity in temporal segmentation.We compare the sub-activity labeling of various segmentations. Here,making cereal activity comprises the sub-activities: reaching, moving,pouring and placing as colored in red, green, blue and magentarespectively. The x-axis denotes the time axis numbered with framenumbers. It can be seen that the various individual segmentationmethods are not perfect.

2) Nearest Neighbor Exemplar. It first finds an example fromthe training data which is the most similar to the activityobserved in the last temporal segment. The sub-activity andobject affordance labels of the frames following the matchedframes from the exemplar are predicted as the anticipations.To find the exemplar, we perform a nearest neighbor search inthe feature space for the set of frames, using the node featuresdescribed in KGS [5].3) Co-occurrence Method. The transition probabilities for sub-activities and affordances are computed from the training data.The observed frames are first labelled using the MRF modelfrom KGS. The anticipated sub-activity and affordances for thefuture frames are predicted based on the transition probabilitiesgiven the inferred labeling of the last frame.4) ATCRF without {H,L} anticipation (ATCRF-discrete). OurATCRF model with only augmented nodes for discrete labels(sub-activities and object affordances).5) ATCRF. Our method that samples the future nodes (bothsegment and frame level) as described in Section 5.3, anduses a fixed temporal structure, which in this case is thesegmentation output of KGS.

Evaluation: We follow the same train-test split described inKGS and train our model on activities performed by threesubjects and test on activities of a new subject. We report the

11

.93 .02.01 .01.01.02

.01.96.01 .01.01

.09.31.51 .02.03.04 .01

.34 .66.38 .63.38.21 .13 .08.04.04.13

.22 .78

.10.23.02 .56.08.02

.16.21 .02.60.01 .01

.03.33 .03.61.25 .75

.08.08 .83

movablestationaryreachablepourablepourto

containabledrinkableopenableplaceableclosable

scrubbablescrubber

movablestationary

reachable

pourablepourto

containable

drinkableopenable

placeableclosable

scrubbable

scrubber

.65 .15 .04 .05 .04 .06 .01

.03 .87 .02 .01 .01 .03 .04

.41 .59

.05 .31 .03 .31 .03 .28

.25 .75

.04 .15 .65 .08 .02 .06

.05 .24 .02 .63 .02 .04

.03 .17 .61 .19

.08 .17 .08 .08 .58

.09 .14 .01 .01 .05 .01 .68

reaching

moving

pouring

eating

drinking

opening

placing

closing

scrubbing

nullreaching

movingpouring

eatingdrinking

openingplacing

closingscrubbing

null

1.0

.92 .08

.67 .33

.92 .08

.92 .08

.67 .33

.08 .08 .67 .08 .08

.25 .08 .67

.17 .08 .50 .25

1.0

Making Cereal

Taking Medicine

Microwaving Food

Stacking Objects

Unstacking Objects

Picking Objects

Cleaning Objects

Taking Food

Arranging Objects

Having MealMaking Cereal

Taking Medicine

Microwaving Food

Stacking Objects

Unstacking Objects

Picking Objects

Cleaning Objects

Taking Food

Arranging Objects

Having Meal

Fig. 8: Confusion matrix for affordance labeling (left), sub-activity labeling (middle) and high-level activity labeling (right) of the testRGB-D videos.results obtained by 4-fold cross validation by averaging acrossthe folds. We consider the following metrics:1) Labeling Metrics. For detecting and anticipating labels (forsub-activity and affordances), we compute the overall microaccuracy (P/R), macro precision, macro recall and macro F1score. Micro accuracy is the percentage of correctly classifiedlabels. Macro precision and recall are the averages of precisionand recall respectively for all classes.2) Robot Anticipation Metric. It is important for a robot toplan ahead for multiple future activity outcomes. Therefore,we measure the accuracy of the anticipation task for the topthree predictions of the future. If the actual activity matchesone of the top three predictions, then it counts towards positive.3) Trajectory Metric. For evaluating the quality of anticipatingtrajectories, we compute the modified Hausdorff distance(MHD) as a physical measure of the distance between theanticipated object motion trajectories and the true objecttrajectory from the test data.9

Table 2 shows the frame-level metrics for anticipating sub-activity and object affordance labels for 3 seconds in the futureon the CAD-120 dataset. We use the temporal segmentationalgorithm from KGS for obtaining the graph structure of theobserved past frames for all the baseline methods. ATCRF(row 5) outperforms all the baseline algorithms and achievesa significant increase across all metrics. Our full model (row6), which estimates the graph structure for both past and thefuture, improves the anticipation performance further. Figure10 shows the highest scored anticipations for the cleaningobjects activity. We will now study our results on anticipationin the form of the following questions:

How does the performance change with the durationof the future anticipation? Figure 11 shows how the macroF1 score and the robot anticipation metric changes with theanticipation time. The average duration of a sub-activity in theCAD-120 dataset is around 3.6 seconds, therefore, an anticipa-tion duration of 10 seconds is over two to three sub-activities.With the increase in anticipation duration, performance ofthe others approach that of a random chance baseline, theperformance of our ATCRF declines. It still outperforms otherbaselines for all anticipation times.

How does the performance change with the durationof the past observations? Figure 12 shows how the macroF1 score changes with the past observation time and futureanticipation time. The algorithm has lower performance when

9. The MHD allows for local time warping by finding the best local pointcorrespondence over a small temporal window. When the temporal windowis zero, the MHD is same as the Euclidean distance between the trajectories.We normalize the distance by the length of the trajectory in order to compareacross trajectories of different lengths. The units of the MHD are centimeters.

1 2 3 4 5 6 7 8 9

10

5 10 15 20 25 30

Antic

ipat

ion

time

(sec

)

Observed time (sec)

10

15

20

25

30

35

40

45

Fig. 12: Plot showing how macro F1 score depends on observed timeand anticipation time.

0

20

40

60

80

100

0 20 40 60 80 100

dista

nce

erro

r (cm

)

progress of sub-activity (%)

ATCRF MAP trajectoryATCRF MLE trajectory

No-affordance model trajectoryVelocity model trajectory

Fig. 13: Plot showing how the trajectory distance error (MHD)changes with the progress of the activity for our ATCRF (top particleand local mean) and other baselines (Kalman Filter velocity modelusing object affordance as target, and one without object affordanceinformation).

predicting longer into the future, but this improves as moreobservations become available. Therefore, context from thepast helps in anticipating longer into the future.

How good are the anticipated trajectories? Since tra-jectories are continuous variables, we perform two types ofestimation: MAP, where we take the highest scored particlegenerated by our model, and MLE where we take the weightedsum. Figure 13 shows how these distance errors, averagedover all the moving sub-activities in the dataset, change withthe progress of the sub-activity. Figure 6 shows the sampledtrajectories along with the heatmap corresponding to thedistribution of trajectories. At the beginning of the sub-activitythe anticipations correspond to moving the cup to other placeson the table and near the mouth to drink. As the sub-activityprogresses, depending on the current position of the cup, afew new target locations become probable, such as moving thecup on to the lap (such instances are observed in the trainingdata). These new possibilities tend to increase the distancemeasure as can be seen in the plot of Figure 13. However,on observing more frames, the intent of the human is inferredmore accurately resulting in better anticipated trajectories, forexample in Figure 6-last frame, anticipating only moving to

12

Fig. 10: Highest scored future anticipations for cleaning objects activity (top-row) and arranging objects activity (bottom-row).

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10

robo

t ant

icipa

tion

met

ric

future prediction time (sec)

Anticipated Sub-activity

Ours-fullATCRF

ATCRF-discrete[19] with co-occurance

nearest neighborchance

0

10

20

30

40

50

60

1 2 3 4 5 6 7 8 9 10

F1 s

core

future prediction time (sec)

Anticipated Sub-activity

Ours-fullATCRF

ATCRF-discreteKGS with co-occurance

nearest neighborchance

0

10

20

30

40

50

60

1 2 3 4 5 6 7 8 9 10

F1 s

core

future prediction time (sec)

Anticipated Object Affordance

Ours-fullATCRF

ATCRF-discreteKGS with co-occurance

nearest neighborchance

Fig. 11: Plots showing how robot anticipation metric and macro F1 score changes with the future anticipation time for all methods.

TABLE 3: Online Detection Results of Past Activities and Affor-dances.

modelPast Sub-activity Detection Past Object Affordance Detection

micro macro micro macroP/R Prec. Recall P/R Prec. Recall

chance 10.0 (0.1) 10.0 (0.1) 10.0 (0.1) 8.3 (0.1) 8.3 (0.1) 8.3 (0.1)KGS [5] - online 80.3 (1.5) 78.0 (1.3) 68.1 (2.6) 89.6 (0.8) 80.7 (2.8) 67.8 (1.4)ATCRF-discrete 84.0 (1.3) 72.2 (2.3) 60.7 (2.3) 87.7 (1.0) 67.9 (2.4) 48.9 (2.6)ATCRF 84.7 (1.4) 80.6 (1.0) 75.6 (2.4) 92.3 (0.7) 84.8 (2.3) 77.1 (1.1)

drink trajectories.Effect of anticipation on detection of past activities.

Table 3 shows the detection results of the sub-activities andobject affordances of the past temporal segments, computedin an online fashion. When we label each past segment,we observe that segment’s features but not the future. Theonline metrics are computed by aggregating performance onthe recent past of three segments. (KGS’s method was tolabel a segment given past, present, as well as the future.) Inthis experiment, we assumed ground-truth segmentation andobject tracks for consistent comparison across the methods.If we instead use an algorithm to segment [5], the overallperformance drops, however similar trends hold. We see thatboth the anticipation methods (rows 3-4) improve the detectionresults over the one that does not anticipate (row 2). Thisshows that anticipating the future can improve present andpast performance on detection.

Effect of unknown activities on anticipation perfor-mance. For activities not present in the training data, ourapproach generates most likely anticipations based on theaffordances of the objects present in the environment andthe detected past sub-activities. However, our approach cannotanticipate a sub-activity on which the anticipation model wasnot trained. In fact, around 14% of the segments in our datasetare sub-activities which do not belong to the nine sub-activitycategories used to train our anticipation model. These sub-activities include various background actions such as checkingtime, standing still, etc., which are not relevant to the high-level activity being performed. We label these segments asnull sub-activities and include them for learning the energy

function in Eq. 4. Therefore, even though our model is unableto anticipate such sub-activities in the future, they are correctlydetected as null sub-activities. This allows us to ignore suchirrelevant sub-activities and proceed to anticipate the mostlikely future. However, if null sub-activities are performedvery often, the performance of our anticipation model wouldgo down further.

8.4 Robotic ExperimentsIn this section we show how future activity predictions canhelp the robot perform appropriate actions in response to whatthe human is going to do next. By incorporating such reactiveresponses, the robot can better assist humans in tasks whichthey are unable to perform as well as work along side thehumans much more efficiently.

We use a PR2 robot to demonstrate the following anticipa-tory response scenarios: (i) Robot is instructed to refill waterglasses for people seated at a table, but when it sees a personreaching a glass to drink, it waits for him to finish drinkingbefore refilling, in order to avoid spilling, and (ii) Robot opensthe fridge door when a person approaches the fridge to placesomething inside the fridge. PR2 is mounted with a Kinectas its main input sensor to obtain the RGB-D video stream.We used the OpenRAVE libraries [83] for programing therobot to perform the pre-programmed tasks described in theaforementioned scenarios by incorporating the anticipationsgenerated with our ATCRFs. Figure 1 and Figure 14 show thesnapshots of the robot observing the human, the anticipatedactions and the response executed by the robot.

In our experiments, on the first scenario, we evaluate thesuccess rate which is defined as the percentage of times therobot identifies the correct response. We have a new subject(not seen in the training data) performing the interaction taskmultiple times in addition to other activities which should noteffect the robot’s response, such as reaching for a book, etc.We considered a total of 10 interaction tasks which involvefour objects including the cup, and 5 of these tasks were to

13

Fig. 14: Robot Anticipatory Response for refilling water task. See Figure 1 for opening fridge door task.reach for the cup and drink from it. The robot is given an in-struction to pour water in the cup at four random time instantsduring each interaction tasks (40 total pour instructions). Therobot makes a decision whether to execute the pouring task ornot, based on the anticipated activity and object affordance.The robot considers the three top scored anticipations fortaking the decision following the robot anticipation metric.

We obtain a success rate of 85%, which is the fraction oftimes the robot correctly identifies its response (‘pour’ or ‘notpour’). Out of the 6 failed instances, 3 instances are false-negatives, i.e., the robot anticipated an interaction with thecup when no interaction occurred in future. Videos showingthe results of our robotic experiments and code are availableat: http://pr.cs.cornell.edu/anticipation/.

9 CONCLUSIONIn this work, we considered the problem of detecting thepast human activities as well as anticipating the future us-ing object affordances. We showed how the anticipation offuture activities can be used by a robot to perform look-ahead planning of its reactive responses. We modeled thehuman activities and object affordances in the past usinga rich graphical model (CRF), and extended it to includefuture possible scenarios. Each possibility was representedas a potential graph structure and labeling over the graph(which includes discrete labels as well as human and objecttrajectories), which we called ATCRF. We used importancesampling techniques for estimating and evaluating the mostlikely future scenarios. The structure of the ATCRF wasobtained by first considering the potential graph structuresthat are close to the ground-truth ones by approximating thegraph with only additive features. We then designed movesto explore the space of likely graph structures. We showedthat anticipation can improve performance of detection of evenpast activities and affordances. We also extensively evaluatedour algorithm, against baselines, on the tasks of anticipatingactivity and affordance labels as well as the object trajectories.

In the recently growing field of RGB-D vision, our workthus shows a considerable advance by improving the state-of-the-art results on both the detection and anticipation tasks.We have focused on the algorithms for estimating the graphstructure for both past and future activities while using givennoisy skeleton and object tracks. Improvements to objectperception would further improve these results. In our ex-periments, we see that there is still a large gap between thedetection performance with and without ground-truth temporalsegmentation. Incorporating additional priors about the activi-ties in future work would improve the estimation of the graphstructure. Also, we see that the anticipation accuracies fallrapidly with future prediction time. We believe that modelinglarger temporal-range dependencies and hierarchical structureof activities is an interesting direction to explore for obtainingbetter anticipation.

ACKNOWLEDGMENTSThe authors would like to thank Rudhir Gupta and Jerry Yehfor useful discussions. This work was partially supported byARO award W911NF-12-1-0267, and by Microsoft FacultyFellowship, NSF Career Award and Google PhD Fellowship.REFERENCES[1] K. Tang, L. Fei-Fei, and D. Koller, “Learning latent temporal structure

for complex event detection,” in CVPR, 2012.[2] M. Rohrbach, S. Amin, M. Andriluka, and B. Schiele, “A database for

fine grained activity detection of cooking activities,” in CVPR, 2012.[3] H. Pirsiavash and D. Ramanan, “Detecting activities of daily living in

first-person camera views,” in CVPR, 2012.[4] J.-K. Min and S.-B. Cho, “Activity recognition based on wearable

sensors using selection/fusion hybrid ensemble,” in SMC, 2011.[5] H. S. Koppula, R. Gupta, and A. Saxena, “Learning human activities

and object affordances from rgb-d videos,” IJRR, 2013.[6] J. Sung, C. Ponce, B. Selman, and A. Saxena, “Unstructured human

activity detection from rgbd images,” in ICRA, 2012.[7] B. Ni, G. Wang, and P. Moulin, “Rgbd-hudaact: A color-depth video

database for human daily activity recognition,” in ICCV Workshop onCDC4CV, 2011.

[8] E. Guizzo and E. Ackerman, “The rise of the robot worker,” Spectrum,IEEE, vol. 49, no. 10, pp. 34 –41, October 2012.

[9] S. Nikolaidis and J. Shah, “Human-robot cross-training: computationalformulation, modeling and evaluation of a human team training strategy,”in HRI, 2013.

[10] J. Gibson, The Ecological Approach to Visual Perception. HoughtonMifflin, 1979.

[11] S. Maji, L. Bourdev, and J. Malik, “Action recognition from a distributedrepresentation of pose and appearance,” in CVPR, 2011.

[12] Z. Xing, J. Pei, G. Dong, and P. S. Yu, “Mining Sequence Classifiersfor Early Prediction,” in SIAM ICDM, 2008.

[13] I. Laptev, M. Marszalek, C. Schmid, and B. Rozenfeld, “Learningrealistic human actions from movies,” in CVPR, 2008.

[14] J. Niebles, C. Chen, and L. Fei-fei, “Modeling temporal structure ofdecomposable motion segments for activity classification,” in ECCV,2010.

[15] A. Gaidon, Z. Harchaoui, and C. Schmid, “Actom sequence models forefficient action detection,” in CVPR, 2011.

[16] B. Laxton, L. Jongwoo, and D. Kriegman, “Leveraging temporal,contextual and ordering constraints for recognizing complex activitiesin video,” in CVPR, 2007.

[17] R. Chaudhry, A. Ravichandran, G. Hager, and R. Vidal, “Histograms oforiented optical flow and binet-cauchy kernels on nonlinear dynamicalsystems for the recognition of human actions,” in CVPR, 2009.

[18] J. Liu, B. Kuipers, and S. Savarese, “Recognizing human actions byattributes,” in CVPR, 2011.

[19] S. Sadanand and J. Corso, “Action bank: A high-level representation ofactivity in video,” in CVPR, 2012.

[20] M. Rohrbach, M. Regneri, M. A., S. Amin, M. Pinkal, and B. Schiele,“Script data for attribute-based recognition of composite activities,” inECCV, 2012.

[21] Y. Fu, T. M. Hospedales, T. Xiang, and S. Gong, “Attribute learning forunderstanding unstructured social activity,” in ECCV, 2012.

[22] M. Ryoo, “Human activity prediction: Early recognition of ongoingactivities from streaming videos,” in ICCV, 2011.

[23] M. Hoai and F. De la Torre, “Max-margin early event detectors,” inCVPR, 2012.

[24] J. K. Aggarwal and M. S. Ryoo, “Human activity analysis: A review,”ACM Comp Surveys (CSUR), 2011.

[25] W. Yang, Y. Wang, and G. Mori, “Recognizing human actions from stillimages with latent poses,” in CVPR, 2010.

[26] A. Gupta, A. Kembhavi, and L. S. Davis, “Observing human-objectinteractions: Using spatial and functional compatibility for recognition,”IEEE PAMI, vol. 31, no. 10, pp. 1775–1789, 2009.

[27] B. Yao and L. Fei-Fei, “Modeling mutual context of object and humanpose in human-object interaction activities,” in CVPR, 2010.

14

[28] Y. Jiang, M. Lim, and A. Saxena, “Learning object arrangements in 3dscenes using human context,” in ICML, 2012.

[29] S. Tran and L. S. Davis, “Event modeling and recognition using markovlogic networks,” in ECCV, 2008.

[30] H. S. Koppula, A. Anand, T. Joachims, and A. Saxena, “Semanticlabeling of 3d point clouds for indoor scenes,” in NIPS, 2011.

[31] Z. Jia, A. Gallagher, A. Saxena, and T. Chen, “3d-based reasoning withblocks, support, and stability,” in CVPR, 2013.

[32] J. Shotton, R. Girshick, A. Fitzgibbon, T. Sharp, M. Cook, M. Finocchio,R. Moore, P. Kohli, A. Criminisi, A. Kipman, and A. Blake, “Efficienthuman pose estimation from single depth images,” IEEE TPAMI, 2012.

[33] D. L. Ly, A. Saxena, and H. Lipson, “Co-evolutionary predictors forkinematic pose inference from rgbd images,” in GECCO, 2012.

[34] H. Zhang and L. E. Parker, “4-dimensional local spatio-temporal featuresfor human activity recognition,” in IROS, 2011.

[35] O. Duchenne, I. Laptev, J. Sivic, F. Bach, and J. Ponce, “Automaticannotation of human actions in video,” in ICCV, 2009.

[36] S. Satkin and M. Hebert, “Modeling the temporal extent of actions,” inECCV, 2010.

[37] Y. Ke, R. Sukthankar, and M. Hebert, “Event detection in crowdedvideos,” in ICCV, October 2007.

[38] M. H. Nguyen, L. Torresani, F. De la Torre, and C. Rother, “Weaklysupervised discriminative localization and classification: a joint learningprocess,” in ICCV, 2009.

[39] T. Simon, M. H. Nguyen, F. De la Torre, and J. F. Cohn, “Action unitdetection with segment-based svms,” in CVPR, 2010.

[40] X. Xuan and K. Murphy, “Modeling changing dependency structure inmultivariate time series,” in ICML, 2007.

[41] Z. Harchaoui, F. Bach, and E. Moulines, “Kernel change-point analysis,”in NIPS, 2008.

[42] S. Oh, J. Rehg, T. Balch, and F. Dellaert, “Learning and inferring motionpatterns using parametric segmental switching linear dynamic systems,”IJCV, 2008.

[43] E. B. Fox, E. B. Sudderth, M. I. Jordan, and A. S. Willsky, “Nonparamet-ric Bayesian learning of switching linear dynamical systems,” in NIPS21, 2009.

[44] M. Hoai, Z. Lan, and F. De la Torre, “Joint segmentation and classifi-cation of human actions in video,” in CVPR, 2011.

[45] M. Hoai and F. De la Torre, “Maximum margin temporal clustering,” inAISTATS, 2012.

[46] Y. Zhu, N. M. Nayak, and A. K. Roy-Chowdhury, “Context-awaremodeling and recognition of activities in video,” in CVPR, 2013.

[47] Z. Wang, Q. Shi, C. Shen, and A. van den Hengel, “Bilinear pro-gramming for human activity recognition with unknown mrf graphs,”in CVPR, 2013.

[48] B. D. Ziebart, N. Ratliff, G. Gallagher, C. Mertz, K. Peterson, J. A.Bagnell, M. Hebert, A. Dey, and S. Srinivasa, “Planning-based predictionfor pedestrians,” in IROS, 2009.

[49] K. Kitani, B. D. Ziebart, J. A. Bagnell, and M. Hebert, “Activityforecasting,” in ECCV, 2012.

[50] M. Kuderer, H. Kretzschmar, C. Sprunk, and W. Burgard, “Feature-based prediction of trajectories for socially compliant navigation,” inRSS, 2012.

[51] Z. Wang, M. Deisenroth, H. B. Amor, D. Vogt, B. Scholkopf, andJ. Peters, “Probabilistic modeling of human movements for intentioninference,” in RSS, 2012.

[52] A. Dragan and S. Srinivasa, “Formalizing assistive teleoperation,” inRSS, 2012.

[53] S. Hongeng and R. Nevatia, “Large-scale event detection using semi-hidden markov models,” in ICCV, 2003.

[54] P. Natarajan and R. Nevatia, “Coupled hidden semi markov models foractivity recognition,” in WMVC, 2007.

[55] S. Gong and T. Xiang, “Recognition of group activities using dynamicprobabilistic networks,” in ICCV, 2003.

[56] A. Quattoni, S. Wang, L.-P. Morency, M. Collins, and T. Darrell,“Hidden-state conditional random fields,” IEEE PAMI, 2007.

[57] C. Sminchisescu, A. Kanaujia, Z. Li, and D. Metaxas, “Conditionalmodels for contextual human motion recognition,” in ICCV, 2005.

[58] S. Sarawagi and W. W. Cohen, “Semi-markov conditional random fieldsfor information extraction,” in NIPS, 2004.

[59] Q. Shi, L. Wang, L. Cheng, and A. Smola, “Human action segmentationand recognition using discriminative semi-markov models,” IJCV, 2011.

[60] Z. Khan, T. Balch, and F. Dellaert, “Mcmc-based particle filtering fortracking a variable number of interacting targets,” IEEE PAMI, 2005.

[61] R. Hess and A. Fern, “Discriminatively trained particle filters forcomplex multi-object tracking,” in CVPR, 2009.

[62] D. Fox, “Kld-sampling: Adaptive particle filters,” in NIPS, 2001.

[63] P. Jensfelt, D. Austin, O. Wijk, and M. Andersson, “Feature basedcondensation for mobile robot localization,” in ICRA, 2000.

[64] D. Schulz, D. Fox, and J. Hightower, “People tracking with anonymousand id-sensors using rao-blackwellised particle filters,” in IJCAI, 2003.

[65] A. Doucet, N. d. Freitas, K. Murphy, and S. Russell, “Rao-blackwellisedparticle filtering for dynamic bayesian networks,” in UAI, 2000.

[66] F. Gustafsson, F. Gunnarsson, N. Bergman, U. Forssell, J. Jansson,R. Karlsson, and P.-J. Nordlund, “Particle filters for positioning, nav-igation, and tracking,” IEEE Trans. SP, 2002.

[67] M. Montemerlo, S. Thrun, D. Koller, and B. Wegbreit, “Fastslam: Afactored solution to the simultaneous localization and mapping problem,”in AAAI, 2002.

[68] “Openni,” https://github.com/OpenNI/OpenNI.[69] M. Montemerlo, S. Thrun, and W. Whittaker, “Conditional particle

filters for simultaneous mobile robot localization and people-tracking,”in ICRA, 2002.

[70] H. Kjellstrom, J. Romero, and D. Kragic, “Visual object-action recog-nition: Inferring object affordances from human demonstration,” CVIU,vol. 115, no. 1, pp. 81–90, 2011.

[71] J. Sun, J. L. Moore, A. Bobick, and J. M. Rehg, “Learning visual objectcategories for robot affordance prediction,” IJRR, 2009.

[72] T. Hermans, J. M. Rehg, and A. Bobick, “Affordance prediction vialearned object attributes,” in ICRA: Workshop on Semantic Perception,Mapping, and Exploration, 2011.

[73] H. Grabner, J. Gall, and L. Van Gool, “What makes a chair a chair?”in CVPR, 2011.

[74] A. Gupta, S. Satkin, A. Efros, and M. Hebert, “From 3d scene geometryto human workspace,” in CVPR, 2011.

[75] V. Delaitre, D. Fouhey, I. Laptev, J. Sivic, A. Gupta, and A. Efros, “Scenesemantics from long-term observation of people,” in ECCV, 2012.

[76] Y. Jiang, H. S. Koppula, and A. Saxena, “Hallucinated humans as thehidden context for labeling 3d scenes,” in CVPR, 2013.

[77] J. J. Faraway, M. P. Reed, and J. Wang, “Modelling three-dimensionaltrajectories by using bezier curves with application to hand motion,”JRSS Series C, vol. 56, pp. 571–585, 2007.

[78] H. S. Koppula, A. Anand, T. Joachims, and A. Saxena, “Semanticlabeling of 3d point clouds for indoor scenes,” in NIPS, 2011.

[79] C. Rother, V. Kolmogorov, V. Lempitsky, and M. Szummer, “Optimizingbinary mrfs via extended roof duality,” in CVPR, 2007.

[80] P. Hammer, P. Hansen, and B. Simeone, “Roof duality, complementationand persistency in quadratic 0–1 optimization,” Mathematical Prog.,vol. 28, no. 2, pp. 121–155, 1984.

[81] P. F. Felzenszwalb and D. Huttenlocher, “Efficient graph-based imagesegmentation,” IJCV, vol. 59, no. 2, 2004.

[82] T. Joachims, T. Finley, and C. Yu, “Cutting-plane training of structuralSVMs,” Mach. Learn., vol. 77, no. 1, 2009.

[83] R. Diankov, “Automated construction of robotic manipulation pro-grams,” Ph.D. dissertation, CMU, Robotics Institute, August 2010.

Ashutosh Saxena is in the Faculty of the Com-puter Science department at Cornell University.His research interests include machine learning,robotics and computer vision. He received hisPh.D. in 2009 from Stanford University, and hisB.Tech. in 2004 from IIT Kanpur, India. He haswon best paper awards in 3DRR, RSS and IEEEACE. He was named a co-chair of IEEE tech-nical committee on robot learning. He has alsoreceived Sloan Fellowship in 2012, NSF Careeraward in 2013, and RSS Early Career Award in

2014. He has developed robots that perform household chores suchas unload items from a dishwasher, arrange a disorganized house,checkout groceries, etc. Previously, he has developed Make3D, analgorithm that converts a single photograph into a 3D model.

Hema S. Koppula is a PhD student in the Com-puter Science department at Cornell University.Her research lies at the intersection of machinelearning, computer vision and robotics: she isinterested in understanding people from visualdata to build smart assistive devices. She hasdeveloped machine learning algorithms for per-ceiving environments from RGB-D sensors suchas scene understanding, activity detection andanticipation. She has won the best student paperaward at RSS and is a Google PhD Fellow.