Temporal Anticipation and Adaptation Methods for FluentHuman-Robot Teaming

Tariq Iqbal1 and Laurel D. Riek2

Abstract— As robots work with human teams, they willbe expected to fluently coordinate with them. While peopleare adept at coordination and real-time adaptation, robotsstill lack this skill. In this paper, we introduce TANDEM:Temporal Anticipation and Adaptation for Machines, a seriesof neurobiologically-inspired algorithms that enable robots tofluently coordinate with people. TANDEM leverages a human-like understanding of external and internal temporal changesto facilitate coordination. We experimentally validated theapproach via a human-robot collaborative drumming taskacross tempo-changing rhythmic conditions. We found that anadaptation process alone enables a robot to achieve human-levelperformance. Moreover, by combining anticipatory knowledgealong with an adaptation process, robots can potentially per-form such tasks better than people. We hope this work willenable researchers to create robots more sensitive to changesin team dynamics.

I. INTRODUCTION

Robots are entering daily life in a variety of roles, andare working proximately with people [1–3]. For example,robots are being used to provide physical and behavioralhealthcare, both in clinical and home care settings [4–7], tohelp children with learning [8–10], to guide people in socialspaces [11–13], and to support work in factories [14–16].

The successes of many of these mixed-initiative teamsdepend on how fluently and efficiently robots can interact andcoordinate with people [17, 18]. There are several mechanismsrobots can employ to achieve interactional fluency, definedby Hoffman and Breazeal as the quality of achieving a highlevel of mutual coordination and adaptation among agents[19]. One way for robots to accomplish this is to observe howpeople achieve fluency in groups, and use that knowledge toanticipate the other team members’ future actions and takenecessary adaptive steps to interact effectively (see Fig. 1).

In groups, members create a state of interdependence,where everyone’s outcomes are determined in part by actionsof other members [20]. One important aspect of successfulcoordination is precise motor timing, and people employsensorimotor synchronization to achieve it [21]. This is thetemporal coordination of an action with events in a predictableexternal rhythm, and is considered to be a fundamentalhuman skill [22]. People encounter many group interactionscenarios during everyday activities, and are skilled atperforming sensorimotor synchronization even with sequencesthat contain tempo changes, such as walking in groups invaried speeds, moving objects with the help of other people,dancing in a group, or rowing a boat with others.

1School of Engineering and Applied Science, University of Virginia, USA.(tiqbal@virginia.edu). 2Computer Science and Engineering, Uni-versity of California San Diego, USA. (lriek@eng.ucsd.edu).

Fig. 1: Coordination phenomena observed in human-robot teams [30–32].

Many researchers in neuroscience and psychology haveinvestigated how people effectively time their coordinatedactions with others, especially in constantly changing environ-ments [23–28]. Researchers have found two main conceptsthat people employ during coordination with other externalrhythms: temporal adaptation and temporal anticipation[23, 29]. People use temporal adaptation mechanisms tocompensate for temporal errors, which may occur whilesynchronizing with an external rhythm due to biologicalnoises (i.e., the motor variance associated with central vs.peripheral processing). Additionally, people employ temporalanticipation to start their task early enough to coordinatewith the external event by extracting structural regularities inrhythmic patterns [22].

While people can adaptively synchronize with others,robots still do not know how to take advantage of theseconcepts. If robots can achieve those skills, then it will bepossible for them to adapt rapidly to achieve fluent interactionin teams. It will enable the creation of adaptable robots acrossa range of fields, from socially assistive robots that help peoplewith social and cognitive disabilities, to physically assistiverobots that can enable older adults to perform their dailytasks independently.

In this paper, we introduce a series of neurobiologically-inspired algorithms, TANDEM: Temporal Anticipation andAdaptation for Machines, which enable robots to fluentlycoordinate with people in tempo-changing environments.TANDEM is inspired by recent advancements in neuroscience(Sect. III), specially designed to work on robots and supporthuman-robot teaming. We validated TANDEM by conductingan experiment where two people drummed together syn-chronously, and a robot joined in by synchronously drummingwith them across three tempo-changing rhythmic conditions(Sects. IV and V). The results (Sect. VI) suggest that a robotcan adapt to people in tempo-changing rhythmic conditionsat least as well as a person, and in some cases better, byleveraging the approaches that people employ to interactwith others. TANDEM is computationally inexpensive andsensitive to rhythmic changes in interaction patterns, and thussuitable for real-time implementation on robots. It is both

sensor agnostic and agnostic to group sizes, thus flexible toextend to group interaction scenarios (Sect. VII).

II. RELATED WORK

How people interact amongst themselves has inspiredresearchers to build models for robots to interact fluently ingroups [33–41]. To make policies for robots, many approachestake advantage of human demonstrations. Researchers usethese demonstrations as training data to develop models forrobots to perform action planning and generation in teams.

For example, to play music with people, Hoffman and Wein-berg [42, 43] developed an autonomous, jazz-improvisingrobot, which used a human-inspired anticipatory action planto play in real-time with people. By analyzing human-humanhandovers, Kshirsagar et al. [44] identified four types of gazebehaviors of a receiver and implemented the behaviors on arobot to perform handover with a person. Their experimentalresults suggest that when the robot exhibited human-likebehavior, participants considered the handover more likable,anthropomorphic, and communicative of timing. Fitter etal. [45, 46] developed a hand-clapping robot where userscan synchronously clap with it across various clappingconditions. Their results suggest that when a robot madehuman-like errors, they were more widely accepted thanmechanical errors.

To generate a robot’s collaborative policy, Nikolaidis et al.[47] proposed a framework to cluster activities from humandemonstrations and to learn a reward function for a MOMDPpolicy using inverse reinforcement learning. Inspired byhuman-human coordination in groups, Iqbal et al. [48, 49]developed a method to measure the degree of coordination ina human-robot team and incorporated that into robots’ actiongeneration while working with people in a team.

Others have explored mechanisms of fluent coordinationby drawing inspiration from fields such as cognitive science,neuroscience, psychology, and language [3, 50–59]. Many ofthese investigations included human-human and human-agentjoint action scenarios, e.g., finger tapping, dancing, drumming.For example, drawing inspiration from neuropsychologicalprinciples of anticipation and perceptual simulation, Hoffmanet al. [19] developed a cognitive architecture for robots,which they implemented on a mechanical robotic lamp,enabling a high degree of fluent teaming. In the computationalneuroscience space, Van Der Steen et al. [22] introducedADAM, a computational model to explain how peoplesynchronize to an external, tempo-changing rhythm. Usingsimulations and fitting models to human behavioral data, theauthors concluded that both adaptation and anticipation playa significant role in human sensorimotor synchronization.

While many of the aforementioned approaches can charac-terize adaptation processes in human-human teams, they areoften developed to work on a consistent interaction patternand are not adaptable to scenarios where the interactionpattern varies over time. Moreover, many of these methodsare intended to model human-human dyadic interaction andare not suitable to extend the methods to model a human-robotgroup interaction. This is a significant gap in the literature

A(n+1)

TimeS(n-1) Sn S(n+1)

A(n-1)

Robot Events

Perceived Events A’(n+1)R(n)

async(n)Predicted Events

An

Fig. 2: An overview of TANDEM’s prediction methods. Solid orange andblue circles present the perceived external event’s timing (Sn) and performedrobot event (An) at time step n, respectively. Rn presents the rhythm-keeperat time step n. The dashed blue and orange circle present the predicted timingof the event from the adaptation (A(n+1)) and the anticipation algorithm(S(n+1)), respectively. The dashed green circle shows the predicted timingfrom the dyadic adaptation and anticipation algorithm (A

′(n+1)

).

of building fluent human-robot teams. Our work addressesthis gap by proposing a series of algorithms, TANDEM.

III. TEMPORAL ANTICIPATION AND ADAPTATION FORMACHINES (TANDEM)

Inspired by the processes that people employ duringcoordination [21–23, 27, 29], this work aims to build adaptivealgorithms for robots to enable them to coordinate fluentlywith people, both in dyads and in groups. Thus, we introduceTANDEM, which contains four parts particularly designedfor robots: (1) an adaptation algorithm, by which a robotadjusts its movement timings while synchronizing with anexternal rhythm, (2) an anticipation algorithm, by which arobot predicts the next external event timing, (3) a dyadicadaptation and anticipation algorithm, which combines theadaptation and anticipation algorithms to predict an actiontiming, and (4) a group adaptation and anticipation algorithm,which enables a robot to predict its action timing in a groupsituation. We present the algorithms in Algos. 1-4 (see Fig. 2).

A. Adaptation Algorithm

People use temporal adaptation mechanisms to compensatefor their temporal errors that occur due to variability inmovement timings while synchronizing with an externalrhythm [27]. By employing these mechanisms, people adjusttheir movements to better coordinate with the rhythm.

TANDEM’s anticipation algorithm draws inspiration fromhuman information processing [22, 29]. In this paradigm,a robot requires a linear rhythm-keeper to model the errorcorrection process. First, the robot observes the activitiesperformed by other agents to understand the dynamics(rhythmic tempo of actions) of the team. Based on thatobservation, the robot can set the initial value of the rhythm-keeper variable.

The timing of the next robot action (A(n+1)) is measuredbased on the current robot action time (An) and the rhythm-keeper (Rn). The next robot action is computed by adding thecurrent action time and the current rhythm-keeper values, andby subtracting the product of the most recent asynchrony(async(n)) measure and the phase correction parameter(α). The asynchrony (async(n)) is measured by taking thedifference between the timings of the recent action and the

Algorithm 1 AdaptationInput: Action time (An), Rhythm-keeper (Rn), Phase correction (α), Period

correction (β), Asynchrony (asyncn)Output: Next action time (A(n+1)), Rhythm-keeper (R(n+1))

1: A(n+1) ← A(n) + R(n) − α ∗ async(n) . next action time2: R(n+1) ← R(n) − β ∗ async(n) . updated rhythm-keeper3: return A(n+1), R(n+1)

Algorithm 2 AnticipationInput: External events (S(n−k−1), . . . , S(n)), Prediction-tracking ratio (φ)Output: Next external event (S(n+1))

1: Pred(n+1) = a + b ∗ (n + 1) . predicted time calculated from(S(n−k−1), . . . , S(n)) by using linear-extrapolation; a and b are the y-intercepts and the slope of the line best fitted on the inter-event intervals

2: Track(n+1) ← S(n) − S(n−1) . tracking behavior3: S(n+1) ← S(n) + (φ ∗ Pred(n+1) + (1− φ) ∗ Track(n+1))4: return S(n+1) . anticipated time of an external event

perceived rhythmic signal (Algo. 1, Line 1). The rhythm-keeper value is also updated (using a period correctionparameter (β)) to compensate for the asynchrony (async(n))(Algo. 1, Line 2). The adaptation algorithm reflects the robot’sinternal model and only corrects the timing of the next actionbased on its perceived asynchrony from the rhythmic signal.

B. Anticipation Algorithm

When synchronizing with an external rhythm, peoplepredict the timing of a future event so that they can starttheir task early enough to facilitate coordination. During thisprocess, the brain extracts structural regularities in rhythmicpatterns and utilizes those patterns for prediction [22, 60].

TANDEM’s anticipation algorithm draws inspiration fromhuman event prediction processes [22, 29]. Through thisprocess, a robot generates a prediction about the next externalevent timing so that it can generate an action to coincide withthat rhythmic signal. Thus, the robot considers the predictionsmade by the anticipation algorithm as an external model.

First, a predicted timing of a future event (Pred(n+1))is calculated by using a linear-extrapolation. This linear-extrapolation operation is performed on the k previous inter-event intervals of perceived events (Algo. 2, Line 1).

Further, the interval between the current and previous eventsis computed as a tracked timing (Track(n+1)) (Algo. 2, Line2). Now, combining the tracked and the predicted timings, arobot can anticipate the future timing of an event (Algo. 2,Line 3). Here, φ represents the prediction-tracking ratio. Ifφ = 0, then the anticipation is based only on the trackedtiming. On the other hand, if φ = 1, then the anticipatedevent timing is entirely based on the predicted timing.

C. Dyadic Adaptation and Anticipation Algorithm

Using anticipation and adaptation processes, the humanbrain generates two predicted timings for a future event [29].The person then combines these to produce a single predictionabout the next event’s timing. Similarly, TANDEM’s dyadicadaptation and anticipation algorithm combines the predic-tions from Algos. 1 and 2 to predict a future event timing.

Through this process, a robot first measures the asynchronybetween the predictions from the adaptation (internal model)and the anticipation (external model) algorithms (Algo. 3,Lines 1-3). The robot then computes the next action time

Algorithm 3 Dyadic Adaptation and Anticipation

Input: Action time (An), Rhythm-keeper (R), External events (S), Phasecorrection (α), Period correction (β), Prediction-tracking ratio (φ), Anticipatorycorrection gain (γ)

Output: Next action time (A(n+1)), Rhythm-keeper (R(n+1))

1: async(n) ← A(n) − S(n) . perceived asynchrony2: A(n+1), R(n+1) ← ADAPTATION(An, Rn, asyncn, α, β)3: S(n+1) ← ANTICIPATION((S(n−k−1), . . . , S(n)), φ)4: async′(n+1) ← A(n+1) − S(n+1) . aynchrony in predictions5: A(n+1) ← A(n+1) − γ ∗ async′(n+1) . predicted next action time6: return A(n+1), R(n+1)

Algorithm 4 Group Adaptation and AnticipationInput: Action time (An), Rhythm-keeper (R[1], . . . , R[H]), External events

(S[1], . . . , s[H]), Phase correction (α), Period correction (β), Prediction-trackingratio (φ), Anticipatory correction gain (γ), Number of external rhythms (H)

Output: Next action time (A(n+1)), Rhythm-keeper (R[1], . . . , R[H])

1: TA(n+1) ← 02: async′(n+1) ← 0

3: for i← 1 : H do4: async[i](n) ← A(n) − S[i](n) . perceived asynchrony5: A[i]n+1, R[i](n+1) ← ADAPTATION(An, R[i]n, async[i](n), α, β)6: S[i](n+1) ← ANTICIPATION((S[i](n−k−1), . . . , S[i](n)), φ)7: async′(n+1) ← async′(n+1) + A[i]n+1 − S[i](n+1)

8: TA(n+1) ← TA(n+1) + A[i](n+1)

9: TA(n+1) ← TA(n+1)/H10: async′(n+1) ← async′(n+1)/H

11: A(n+1) ← TA(n+1) − γ ∗ async′(n+1) . predicted next action time12: return A(n+1), (R[1], . . . , R[H])

(A(n+1)) by adding this asynchrony with the error correctedtime (adaptation) of the event (Algo. 3, Line 4), whichcombines the robot’s internal and external models. Here,γ represents the anticipatory correction gain.

D. Group Adaptation and Anticipation Algorithm

Beyond human-robot dyads, a robot may encounter multiplerhythmic signals in a group setting. For example, if a robot isperforming a rhythmic coordination task along with people,such as working with multiple people in a factory whoare working at slightly different speeds, it will perceivemixed rhythmic signals based on how each person performshis/her actions. TANDEM’s group adaptation and anticipationalgorithm enables a robot to adapt to multiple rhythmic signalsand maintain coordinatation with the rest of the group.

Suppose that a robot perceives H separate rhythmic signals,where each of these signals are the timing of the actionsperformed by each human. The robot keeps a separate rhythm-keeper for each rhythmic signal. The robot updates its internalmodel for each rhythmic signal (Algo. 4, Line 5). Then, bycombining the action times of all H external rhythms, thenext adapted action time is calculated (Algo. 4, Line 9).

Additionally, a robot generates an external model for eachperceived signal. Thus, a robot needs a total of H externalmodels, each of which can anticipate the next event timingusing the process described in Sect. III-B (Algo. 4, Line 6).

We can measure the asynchrony between the action timereceived from the adaptation algorithm and the times receivedfrom the anticipation algorithm for H external rhythms, bytaking a similar approach described in Sect. III-C (Algo. 4,Line 7). Thus, we can combine the timings from the robot’sinternal and external models together to compute the nextrobot action time (A(n+1)) (Algo. 4, Line 11).

Another approach to combine multiple rhythmic signals isto take a weighted average of the asynchronies (the internalmodel) or of the anticipated timings of each signal (theexternal model) based on how synchronous they were inprevious events. Their stability score can represent theseweights, measured by their standard deviation of inter-eventintervals. A robot may choose to put more weight on thetimings of the person who has exhibited more stable rhythmthan a person who has shown abrupt actions.

IV. IMPLEMENTATION

As our work employs neurobiologically-inspired mecha-nisms to support human robot teaming, it is important tocompare the performance of TANDEM to the performanceof people in similar conditions. Moreover, it is essential totest whether a robot can achieve human-level performanceby employing TANDEM, because if a robot can accomplishthat, then the human team members need to put less effort toadapt to that robot to attain interactional fluency. Furthermore,although there are other approaches to generate beat sequencesfor robots, they are not suitable for comparison as they donot handle temporal variations, which is a crucial aspect ofachieving fluent interaction in teams. To explore this, weimplemented the first three parts of TANDEM (Alogs. 1-3)to enable a robot to perform synchronous actions with people.We plan to validate the group adaptation and anticipationalgorithm (Algo. 4) in a future experiment.

A. Experimental TestbedAs an experimental testbed, we designed a human-robot

synchronous drumming scenario. In this scenario, two peopledrum together synchronously, and a robot observes theparticipants and joins by drumming with them synchronously.During an experimental trial, two people stood near two tables,and a barrier visually blocked them (see Fig. 3-Left). e.g.,participant could not see one another or the robot drumming.

We designed our experiment such that all members of theteam had to use similar sensing modalities while coordinatingwith others. Thus, in this experiment, all the participants onlyrelied on an audible signal to perceive the rhythm, and therobot solely relied on a piezoelectric sensor as a proxy ofthe acoustic signal. All participants wore sound-blockingheadphones, which reduced the outside noise and helpedparticipants concentrate on their primary rhythm.

During an experiment, Person 1 (P1) listened to an externalrhythm through his/her headphones (a metronomic signal). P1was instructed to follow that rhythm and drum on the beats.We placed a microphone underneath P1’s drum, which wasdirectly connected to Person 2’s (P2) headphones. P2 wasinstructed to drum synchronously with P1, only by listeningto the drumming of P1 through their headphones. At thesame time, a robot detected the drumming events of P1 fromsensors attached to P1’s drum, and drummed synchronouslyby employing the algorithms described in Sect. III.

B. Drumming Robot and Drumming Activity DetectionWe built a robot capable of drumming with a human group

(see Fig. 3-Right), which can move a drumstick at a rate of

Fig. 3: Left) The experimental setup, where the participants and robotwere situated separately. Participants wore sound-blocking headphones.Person 1 (P1) listened to an external tempo, Person 2 (P2) listened toaudio of P1’s drumming, and the robot “listened” to P1’s drum hits via apiezoelectric sensor. Right) A prototype of the drumming robot and schematicdiagram of the different ROS nodes of the system. The arrows denote theconnectivity and the message passing directions.

4-5 Hz to hit a drum. This rate is very close to how fast anaverage person (non-drummer) generally can drum (at 5-7 Hz[61]). Thus, this robot was well-suited to our experiment. Therobotic arm (the drumstick) was attached to a servo, whichwas connected to an Arduino Mega and a Renbotic ServoShield. All the devices were controlled using ROS.

To track the human and robot drumming events precisely,we attached a piezoelectric sensor under each drum head,which were attached to an Arduino microcontroller. Wheneverthe participants or the robot hit a drum head, the correspond-ing Arduino ROS node generated a hit event instantaneously.

C. Robot Action Adaptation

After receiving the events from the drums, the AlgoNodegenerated the next action time for the robot by utilizing theprediction algorithms. However, the robot needed to startearly to perform the hit on the drum head exactly on time.We define this early start time as earlyStartTime (et).

For example, suppose a robot needs to hit the drum at timeA(n+1) predicted by our algorithms (in the AlgoNode). TheArduinoNode starts the drumming action at time A(n+1)−et.Due to physical noise, the robotic arm may hit the drum alittle earlier or later than A(n+1). Suppose that the roboticarm hits the drum at time A(hit). We update et as a firstorder autoregressive variable based on the difference betweenA(n+1) and A(hit), and a movement alignment factor f . Thus,et(i+1) = eti + f ∗ (A(n+1) −Ahit).

V. EXPERIMENTS

We performed a 3 x 3, within-subjects study to exploreour research goals. We used three different algorithms forthe robot to drum with people in three tempo-changingrhythmic signals.

A. Rhythms

We used three rhythmic tempos for this experiment. Eachrhythm had 76 beats in total. Each metronome beat lastedfor 5 ms. The first rhythm was a uniform tempo of 80 beatsper minute (BPM), and lasted for 76 beats (Fig. 4).

The second rhythm started with a tempo of 80 BPM forthe first 20 beats. It slowed down gradually (ritardando)starting from the 21st beat for next 24 beats to reach 60 BPM.After reaching 60 BPM, the tempo sped up (accelerando)from the 45th beat for 24 beats to reach 80 BPM again. Therhythm then ended at 80 BPM for the last 8 beats.

Fig. 4: Three tempo conditions. A) Uniform tempo condition, B) Singletempo change condition, and C) Double tempo change condition.

The third rhythm also had a similar tempo change likerhythm 2. It started with a tempo of 80 BPM. However, thisrhythm had two ritardando and accelerando phases. Eachphase lasted for 12 beats and the tempo changed from 80BPM to 60 BPM and returned to 80 BPM again.

B. Procedure and Participants

After giving informed consent, both participants wereinstructed about the experimental procedures. The participantsthen had time to practice the drumming session as a group.During the practice session, the robot did not participate.

The main experiment was performed in two sessions. Inthe first session, one person acted as P1, and the other personacted as P2 for all nine experimental conditions. In the secondsession, these experimental conditions were repeated oncemore, with each of the participants swapping roles. P1 wasinstructed to listen to the first four beats of the rhythm toperceive the tempo and to start drumming from the fifth beat.The experimental conditions were counter-balanced.

We recruited a total of eight groups (16 participantsin total, four women and twelve men) via word-of-mouth.Their average age was 24 years (s.d. = 2.38 years). Onlythree participants had prior drumming experience of twoyears; thirteen did not have any drumming experience. Theexperiments per group took about 40-50 minutes, and eachparticipant was compensated with a $5 gift card.

C. Experimental Conditions

During the experiments, we used a phase correctionparameter (α) = 0.8, a period correction parameter (β) = 0.5,an anticipation parameter (k) = 3, a prediction-tracking ratio(φ) = 0.8, and an anticipatory correction gain (γ) = 0.4. Theparameter values were selected based on the recommendationby Van Der Steen el al. [29] and from pilot experiments. Frompilot experiments, we found f = 0.2 as the optimal valuefor our setup, and the robot needed approximately 70 ms toperform a drumming action. Thus, we initialized the et = 70.

VI. DATA ANALYSIS AND RESULTS

To compare the performance of the algorithms, we firstmeasured how closely the robot (R) drummed to P1. We alsomeasured how closely P2 drummed to P1. Then we performeda non-inferiority trial on these two measures (See Sect. VI-A), which enabled us to investigate whether the robot’sperformance was at least as good as the human performance.

A. Analysis Method

Non-inferiority trials are a widely used analysis technique,frequently used in the health sciences. Non-inferiority trialsare usually used to test whether a new treatment is not

TABLE I: Presents differences in effects for all nine experimental conditions.We subtracted the effect of the control condition (dt(P1, P2)) from theexperimental condition (dt(P1, R)). Thus, negative mean values indicatethat the robot drummed closer to P1’s drumming events compared to thehuman participant (P2). These values are represented in ms.

dt(P1 , R) - dt(P1 , P2)

Robot Algorithms Rhythms Mean s.d.95% CI

Lower UpperUniform Tempo -4.82 12.92 -11.70 2.07

Anticipation Single Tempo -9.74 25.59 -23.38 3.89

Double Tempo 80.62 316.78 -88.17 249.43

Uniform Tempo -8.09 6.86 -11.75 -4.43

Adaptation Single Tempo -14.47 28.53 -29.68 0.73

Double Tempo -58.04 213.78 -171.96 55.87

Uniform Tempo -10.26 10.79 -16.01 -4.51

Dyadic Single Tempo -18.47 25.18 -31.89 -5.06

Double Tempo 111.26 464.64 -136.33 358.85

unacceptably worse than an active treatment that is already inuse [62–65]. In our work, we performed a non-inferiority trialto investigate whether the robot’s performance is not inferiorto a person. A non-inferiority trial allows a limit on theamount an experimental condition is permitted to be inferiorto a control condition while guaranteeing a similar benefitand efficacy from the control condition. This limit is calleda non-inferiority margin (δ). If an experimental condition iswithin the non-inferiority margin of the control condition, thenthe experimental condition is considered as non-inferior (asgood) to the control condition and as beneficial as the controlcondition [62, 64]. First, the difference between the effectof the experimental and the control condition is calculated.Then from the distribution of these values (i.e., 2-sided 95%confidence intervals), the non-inferiority is tested. The resultsfrom non-inferiority trials indicate whether the experimentalcondition is superior (better), non-inferior (as good), inferior(worse), or inconclusive, compared to the control condition.

B. Data Processing

We only considered the events that occurred from the18th to the 76th beat of the metronome signal (a total of59 beats), as the robot began performing using differentalgorithms starting from the 18th beat based on experimentalconditions. We then calculated the timing of all drummingevents performed by P1, P2 and the robot (R). For example,for a single drumming event e, we calculated the timingof the event performed by P1 and P2, and the robotrespectively as t(P1, e), t(P2, e), and t(R, e). Then, wemeasured the absolute difference between the timing ofeach event performed by P1 and P2, and P1 and R, andaveraged it over each session. For example, if a sessionhas a total of a E number of drumming events, wheree ∈ E, then we measured the mean difference of timings(dt(P1, P2) =

∑Ee=1 abs(t(P1,e)−t(P2,e))

E and dt(P1, R) =∑Ee=1 abs(t(P1,e)−t(R,e))

E ), and performed a non-inferioritytrial over those values.

C. Results

We considered the time difference between P1 and P2as the control condition, and the time difference between

P1 and R as the experimental condition. Thus, a negativevalue of dt(P1, R) − dt(P1, P2) represents that the robotperformed that drumming event closer to P1 in time thanP2. On the other hand, a positive value represents that P2performed it closer to P1 in time than the robot.

To perform a non-inferiority trial, we choose a non-inferiority margin δ = 70 ms. We choose this value forδ, as this is an approximate time that our robot takes to hitthe drum head after receiving a command. If we choose avalue larger than this, then a reactive robot (which can reactand can start drumming after listening to a person hit) wouldbe better than our algorithms. On the other hand, being closerthan this margin indicates that the algorithms are better thanby chance (reactive controls).

We show the results on the non-inferiority test in Table I.We present dt(P1, R)− dt(P1, P2) for all our experimentalconditions. We also present the 95% confidence interval ofdt(P1, R)− dt(P1, P2) in last two columns of Table I.

Tables I and II indicates that for the uniform tempo-changing condition, the anticipation algorithm was non-inferior (as good) (95 % CI, −11.70 to 2.07), the adaptationalgorithm was superior (better) (95% CI, −11.75 to −4.43)and the dyadic algorithm was superior (better) to the controlcondition (95% CI, −16.01 to −4.51). This means that therobot achieved human-level performance using the adaptationalgorithm, and was better than people when the adaptationand the dyadic algorithms were used in the uniform tempo-changing condition.

For the single tempo-changing condition, the anticipationalgorithm was non-inferior (as good) (95% CI, −23.38 to3.89), the adaptation algorithm was non-inferior (as good) (95% CI, −29.68 to 0.73) and the dyadic algorithm was superior(better) to the control condition (95% CI, −31.89 to −5.06).This means that the robot achieved human-level performanceusing the adaptation and the anticipation algorithms, and wasbetter than people when using the dyadic algorithm.

For the double tempo-changing condition, the performanceof the anticipation algorithm and the DA3 were inconclusivecompared to the control condition (Anticipation: 95% CI,−88.17 to 249.43, and DA3: 95% CI, −136.33 to 358.85).However, the adaptation algorithm was non-inferior (as good)to the control condition (95% CI, −171.96 to 55.87). Thismeans that the robot achieved human-level performanceusing the adaptation algorithm in the double tempo-changingcondition, but we cannot draw any conclusions about theanticipation algorithm or the dyadic algorithm.

VII. DISCUSSION

The main contribution of this work is the introduction ofTANDEM, a suite of neurobiologically-inspired algorithmsthat specifically designed for robots to support fluent human-robot coordination in groups. The results of our human-robotvalidation study indicate that TANDEM enables a robot toadapt to tempo-changing rhythmic conditions at least as wellas an average person and in some cases to perform even better.

Overall, our experimental results suggest that the adaptationalgorithm alone enables a robot to achieve human-level

TABLE II: Summary of findings from the experiments. Here, each valuerepresents whether the robot performed better (marked in blue), as good(marked in green), or worse than the human. The inconclusive value indicatesneither the robot nor the human performed better than one another. Thenon-inferiority margin (δ) was 70 ms.

Robot

Algorithms

Rhythms

Uniform TempoSingle Tempo

Change

Double Tempo

Change

Anticipation As good As good Inconclusive

Adaptation Better As good As goodDyadic Better Better Inconclusive

performance in the task of synchronously drumming withanother person only through the use of auditory cues. Further-more, by combining anticipatory knowledge (the anticipationalgorithm), along with an adaptation process (the combineddyadic algorithm), a robot can perform even better than peoplein uniform and single tempo-changing conditions.

In this paper, we focused on synchronous teaming con-ditions; however, this approach is flexible in applying tonon-synchronous situations. For example, many human-robotteams working in factory settings have specific activitysequences to perform, which might be independent andmight not have to happen synchronously. In such time-variedcollaborative tasks, these proposed methods are applicable.

Although the anticipation algorithm was non-inferior andthe combined dyadic algorithm was superior to humanperformers during the uniform and single tempo-changingconditions, we cannot draw the same conclusion for thedouble tempo-changing conditions. This may be because theanticipation algorithm uses linear extrapolation to calculatefuture event timings, which may be insufficient to performcorrect prediction when the tempo changes very quickly, asit did in the double tempo-changing conditions. For example,the ritardando and the accelerando phases occurred twiceas quickly in the double tempo-changing conditions (lastingonly for 12 beats) than the single tempo-changing conditions(lasted for 24 beats).

In the future, we plan to explore the effects of the combinedgroup algorithm (described in Sect. III-D) in a group context.This exploration will provide an opportunity to investigatemany dimensions of the problem, such as how a robot cananticipate and adapt to multiple external rhythms and howother group members can adapt to a robot’s actions in teams.

We hope this work will enable the next generation ofassistive robots that help people with cognitive and physicaldisabilities. For example, a assistive robot, which acts as askilled practice partner for people with cognitive impairments,can predict its interactional partners’ verbal and non-verbalbehaviors and adjust its actions to ensure fluent coordination.

Humans have a unique ability to coordinate with others.How people coordinate with others can yield significant in-sights for developing algorithms for robots to work in teams.The methods described in this paper are a step towardbuilding fluent human-robot teams and may help the roboticscommunity explore new human-team-inspired approaches inthe future.

REFERENCES

[1] G. Hoffman, “Evaluating fluency in human–robot collaboration,” IEEETransactions on Human-Machine Systems, 2019.

[2] S. Tellex, N. Gopalan, H. Kress-Gazit, and C. Matuszek, “Robots thatuse language,” Annual Review of Control, Robotics, and AutonomousSystems, 2020.

[3] A. Thomaz, G. Hoffman, and M. Cakmak, “Computational Human-Robot Interaction,” Foundations and Trends in Robotics, 2016.

[4] O. Zuckerman, D. Walker, A. Grishko, T. Moran, C. Levy, B. Lisak,I. Y. Wald, and H. Erel, “Companionship is not a function: The effect ofa novel robotic object on healthy older adults’ feelings of” being-seen”,”in CHI, 2020.

[5] A. Kubota, E. I. Peterson, V. Rajendren, H. Kress-Gazit, and L. D.Riek, “Jessie: Synthesizing social robot behaviors for personalizedneurorehabilitation and beyond,” in ACM/IEEE HRI, 2020.

[6] L. Riek, “Healthcare robotics,” Communications of the ACM, 2017.[7] A. Taylor, H. Lee, A. Kubota, , and L. D. Riek, “Coordinating Clinical

Teams: Using Robots to Empower Nurses to Stop the Line,” in CSCW,2019.

[8] M. Suguitan and G. Hoffman, “Blossom: A handcrafted open-sourcerobot,” ACM Trans. on THRI, 2019.

[9] I. Leite, M. McCoy, M. Lohani, D. Ullman, N. Salomons, C. Stokes,S. Rivers, and B. Scassellati, “Emotional storytelling in the classroom:Individual versus group interaction between children and robots,” inACM/IEEE HRI, 2015.

[10] N. T. Fitter, Y. Chowdhury, E. Cha, L. Takayama, and M. J. Mataric,“Evaluating the effects of personalized appearance on telepresencerobots for education,” in ACM/IEEE HRI, 2018.

[11] M. Swofford, J. Peruzzi, N. Tsoi, S. Thompson, R. Martın-Martın,S. Savarese, and M. Vazquez, “Improving social awareness throughdante: Deep affinity network for clustering conversational interactants,”Proceedings of the ACM on Human-Computer Interaction, 2020.

[12] S. Strohkorb Sebo, M. Traeger, M. Jung, and B. Scassellati, “The rippleeffects of vulnerability: The effects of a robot’s vulnerable behavioron trust in human-robot teams,” in ACM/IEEE HRI, 2018.

[13] A. Taylor, D. M. Chan, and L. D. Riek, “Robot-centric perception ofhuman groups,” THRI, 2020.

[14] T. Iqbal, S. Li, C. Fourie, B. Hayes, and J. A. Shah, “Fast Online Seg-mentation of Activities from Partial Trajectories,” in IEEE InternationalConference on Robotics and Automation (ICRA), 2019.

[15] A. Kubota, T. Iqbal, J. A. Shah, and L. D. Riek, “Activity recognitionin manufacturing: The roles of motion capture and semg+ inertial wear-ables in detecting fine vs. gross motion,” in International Conferenceon Robotics and Automation (ICRA), 2019.

[16] B. Hayes and J. A. Shah, “Interpretable models for fast activityrecognition and anomaly explanation during collaborative roboticstasks,” in IEEE International Conference on Robotics and Automation(ICRA), pp. 6586–6593, 2017.

[17] T. Iqbal and L. Riek, “Human robot teaming: Approaches from jointaction and dynamical systems,” Springer Handbook of HumanoidRobotics, 2017.

[18] D. W. Vinson, L. Takayama, J. Forlizzi, W. Ju, M. Cakmak, andH. Kuzuoka, “Human-robot teaming,” in Extended Abstracts of the 2018CHI Conference on Human Factors in Computing Systems, p. panel07,ACM, 2018.

[19] G. Hoffman and C. Breazeal, “Anticipatory Perceptual Simulation forHuman-Robot Joint Practice: Theory and Application Study.,” AAAI,2008.

[20] D. R. Forsyth, Group dynamics. T. Wadsworth, fourth ed., 2009.[21] B. H. Repp and P. E. Keller, “Sensorimotor synchronization with

adaptively timed sequences,” Human movement science, vol. 27, no. 3,pp. 423–456, 2008.

[22] M. C. M. van der Steen and P. E. Keller, “The ADaptation andAnticipation Model (ADAM) of sensorimotor synchronization.,” Front.Hum. Neurosci., 2013.

[23] B. H. Repp, “Sensorimotor synchronization: A review of the tappingliterature,” Psychonomic bulletin & review, 2005.

[24] E. W. Large, “Resonating to musical rhythm: theory and experiment,”Psychology of time, pp. 189–232, 2008.

[25] S. Grondin, “Timing and time perception: a review of recent behav-ioral and neuroscience findings and theoretical directions,” Attention,Perception, & Psychophysics, 2010.

[26] K. Overy and I. Molnar-Szakacs, “Being together in time: Musicalexperience and the mirror neuron system,” Music Perception: AnInterdisciplinary Journal, 2009.

[27] B. H. Repp and Y.-H. Su, “Sensorimotor synchronization: a review ofrecent research (2006–2012),” Psychonomic bulletin & review, 2013.

[28] H. Merchant, D. L. Harrington, and W. H. Meck, “Neural basis ofthe perception and estimation of time,” Annual review of neuroscience,2013.

[29] M. M. van der Steen, N. Jacoby, M. T. Fairhurst, and P. E. Keller,“Sensorimotor synchronization with tempo-changing auditory sequences:Modeling temporal adaptation and anticipation,” Brain research, 2015.

[30] R. M. Aronson, T. Santini, T. C. Kubler, E. Kasneci, S. Srinivasa, andH. Admoni, “Eye-hand behavior in human-robot shared manipulation,”in ACM/IEEE HRI, 2018.

[31] V. V. Unhelkar, S. Li, and J. A. Shah, “Semi-supervised learning ofdecision-making models for human-robot collaboration,” in Conferenceon Robot Learning, 2020.

[32] T. Iqbal and L. D. Riek, “Coordination dynamics in multi-humanmulti-robot teams,” IEEE Robotics and Automation Letters (RA-L),2017.

[33] M. J. Gielniak and A. L. Thomaz, “Enhancing interaction throughexaggerated motion synthesis,” in HRI, 2012.

[34] M. M. Islam and T. Iqbal, “Multi-gat: A graphical attention-basedhierarchical multimodal representation learning approach for humanactivity recognition,” IEEE Robotics and Automation Letters, 2021.

[35] S. Matsumoto and L. D. Riek, “Fluent Coordination in ProximateHuman Robot Teaming,” in Robotics, Science, and Systems (RSS)Workshop on AI and Its Alternatives for Shared Autonomy in Assistiveand Collaborative Robotics, 2019.

[36] M. Vazquez, E. J. Carter, B. McDorman, J. Forlizzi, A. Steinfeld,and S. E. Hudson, “Towards robot autonomy in group conversations:Understanding the effects of body orientation and gaze,” in ACM/IEEEHRI, 2017.

[37] M. Yasar and T. Iqbal, “A scalable approach to predict multi-agentmotion for human-robot collaboration,” IEEE Robotics and AutomationLetters, 2021.

[38] M. M. Islam and T. Iqbal, “Hamlet: A hierarchical multimodal attention-based human activity recognition algorithm,” in IEEE/RSJ InternationalConference on Intelligent Robots and Systems (IROS), pp. 10285–10292,2020.

[39] M. Yasar and T. Iqbal, “Transferring representation for long-termlearning in human-robot interaction,” in ACM/IEEE InternationalConference on Human-Robot Interaction (HRI), Lifelong Learningand Personalization in Long-Term Human-Robot Interaction (LEAP-HRI), 2021.

[40] M. Doering, P. Liu, D. F. Glas, T. Kanda, D. Kulic, and H. Ishiguro,“Curiosity did not kill the robot: A curiosity-based learning system fora shopkeeper robot,” ACM Transactions on Human-Robot Interaction(THRI), vol. 8, no. 3, pp. 1–24, 2019.

[41] D. M. Chan and L. D. Riek, “Unseen salient object discovery formonocular robot vision,” IEEE Robotics and Automation Letters, vol. 5,no. 2, pp. 1484–1491, 2020.

[42] G. Hoffman and G. Weinberg, “Interactive improvisation with a roboticmarimba player,” Auton. Robots, 2011.

[43] G. Hoffman and G. Weinberg, “Synchronization in human-robotMusicianship,” 19th Int. Symp. Robot Hum. Interact. Commun., pp. 718–724, sep 2010.

[44] A. Kshirsagar, M. Lim, S. Christian, and G. Hoffman, “Robotgaze behaviors in human-to-robot handovers,” IEEE Robotics andAutomation Letters, 2020.

[45] N. T. Fitter and K. J. Kuchenbecker, “How does it feel to clap handswith a robot?,” International Journal of Social Robotics, pp. 1–15,2019.

[46] N. T. Fitter and K. J. Kuchenbecker, “Synchronicity trumps mischiefin rhythmic human-robot social-physical interaction,” in RoboticsResearch, pp. 269–284, Springer, 2020.

[47] S. Nikolaidis, R. Ramakrishnan, K. Gu, and J. Shah, “Efficient modellearning from joint-action demonstrations for human-robot collaborativetasks,” in ACM/IEEE HRI, ACM, 2015.

[48] T. Iqbal and L. D. Riek, “A Method for Automatic Detection ofPsychomotor Entrainment,” IEEE Transactions on Affective Computing,2015.

[49] T. Iqbal, S. Rack, and L. D. Riek, “Movement coordination in human-robot teams: A dynamical systems approach,” IEEE Transactions onRobotics, vol. 32, no. 4, pp. 909–919, 2016.

[50] P. Vanzella, J. B. Balardin, R. A. Furucho, G. A. Zimeo Morais,T. Braun Janzen, D. Sammler, and J. R. Sato, “fNIRS responses inprofessional violinists while playing duets: evidence for distinct leader

and follower roles at the brain level,” Frontiers in psychology, vol. 10,p. 164, 2019.

[51] G. Pezzulo, F. Donnarumma, H. Dindo, A. D’Ausilio, I. Konvalinka,and C. Castelfranchi, “The body talks: Sensorimotor communicationand its brain and kinematic signatures,” Physics of life reviews, vol. 28,pp. 1–21, 2019.

[52] I. D. Colley, P. E. Keller, and A. R. Halpern, “Working memory andauditory imagery predict sensorimotor synchronisation with expres-sively timed music,” Quarterly Journal of Experimental Psychology,vol. 71, no. 8, pp. 1781–1796, 2018.

[53] S. Nozaradan, I. Peretz, and P. E. Keller, “Individual differences inrhythmic cortical entrainment correlate with predictive behavior insensorimotor synchronization,” Scientific Reports, vol. 6, p. 20612,2016.

[54] I. Leite, C. Martinho, and A. Paiva, “Social robots for long-terminteraction: a survey,” International Journal of Social Robotics, vol. 5,no. 2, pp. 291–308, 2013.

[55] M. Bretan and G. Weinberg, “A survey of robotic musicianship,”Communications of the ACM, vol. 59, no. 5, pp. 100–109, 2016.

[56] H. Admoni, A. Dragan, S. S. Srinivasa, and B. Scassellati, “Deliberatedelays during robot-to-human handovers improve compliance with gazecommunication,” in ACM/IEEE HRI, 2014.

[57] A. Washburn, R. W. Kallen, C. A. Coey, K. Shockley, and M. J.Richardson, “Harmony from chaos? perceptual-motor delays enhancebehavioral anticipation in social interaction.,” Journal of experimentalpsychology: human perception and performance, 2015.

[58] G. Milliez, R. Lallement, M. Fiore, and R. Alami, “Using human knowl-edge awareness to adapt collaborative plan generation, explanation andmonitoring,” in HRI, 2016.

[59] L. Noy, E. Dekel, and U. Alon, “The mirror game as a paradigm forstudying the dynamics of two people improvising motion together,”Proceedings of the National Academy of Sciences, vol. 108, no. 52,pp. 20947–20952, 2011.

[60] R. I. Schubotz, “Prediction of external events with our motor system:towards a new framework,” Trends in cognitive sciences, vol. 11, no. 5,pp. 211–218, 2007.

[61] S. Fujii, K. Kudo, T. Ohtsuki, and S. Oda, “Tapping performance andunderlying wrist muscle activity of non-drummers, drummers, and theworld’s fastest drummer,” Neurosci. Lett., vol. 459, no. 2, pp. 69–73,2009.

[62] M. D. Rothmann, B. L. Wiens, and I. S. F. Chan, Design and Analysisof Non-Inferiority Trials. CRC Press, fourth ed., 2012.

[63] G. Piaggio, D. R. Elbourne, D. G. Altman, S. J. Pocock, S. J. W.Evans, and for the CONSORT Group, “Reporting of Noninferiorityand Equivalence Randomized Trials,” Jama, vol. 295, no. 10, p. 1152,2006.

[64] S. Hahn, “Understanding noninferiority trials.,” Korean J. Pediatr.,vol. 55, no. 11, pp. 403–7, 2012.

[65] J. Schumi and J. T. Wittes, “Through the looking glass: understandingnon-inferiority,” Trials, vol. 12, no. 1, p. 106, 2011.