Towards Seamless Autonomous Function Handoversin Human-Robot Teams

Andrey Kiselev1, Andre Potenza1, Barbara Bruno2, and Amy Loutfi1

Abstract— Various human-robot collaboration scenarios mayimpose different requirements on the robot’s autonomy, rangingfrom fully autonomous to fully manual operation. The paradigmof sliding autonomy has been introduced to allow adaptingrobots’ autonomy in real time, thus improving flexibility ofa human-robot team. In sliding autonomy, functions can behanded over between the human and the robot to addressenvironment changes and optimize performance and workload.This paper examines the process of handing over functionsbetween humans by looking at a particular experiment scenarioin which the same function has to be handed over multiple timesduring the experiment session.

We hypothesize that the process of function handover issimilar to already well-studied human-robot handovers whichdeal with physical objects. In the experiment, we attempt todiscover natural similarities and differences between these twotypes of handovers and suggest further directions of work thatare necessary to give the robot the ability to perform thefunction handover autonomously, without explicit instructionfrom the human counterpart.

Index Terms— Function Handovers, Teleoperation, RobotControl

I. INTRODUCTION

In Mobile Robotic Telepresence (MRP) [8] and, morebroadly, in robot teleoperation [7], a human operator iscontrolling a robot from a distance. If the robot possessessome degree of autonomy, then the operator is sharing controlfunctions with the robot itself. In many settings it is naturalthat functionality should be distributed in some way oranother between the human and the robot, thereby requiringthe implementation of various control paradigms. Master-slave control systems rely on the human operator in thecontrol loop - for instance, the Soviet Lunokhod-1 missionrover was directly operated from Earth, with a 2.5s roundtrip delay. Other systems, such as that of NASA’s MarsExploration Rover (MER) mission, implemented supervisorycontrol: Here the operators issued the daily missions whichwere then performed by the rovers autonomously [5]. Variousmodes of teleoperation are discussed in detail by [11].

An ideal telepresence system is a transparent communica-tion channel that provides natural human-human interaction.However, real current systems are limited by today’s tech-nology, and therefore other types of interactions have to be

1A. Kiselev and A. Loutfi are with Center for AppliedAutonomous Sensor Systems, Orebro University, Fakultetsgatan1, 70182, Orebro, Sweden andrey.kiselev@oru.se,andre.potenza@oru.se, amy.loutfi@oru.se

2B. Bruno is with the Department DIBRIS, University of Genova, viaOpera Pia 13, 16145, Genova, Italy barbara.bruno@unige.it

*This work has received funding from the European Union’s Horizon2020 research and innovation program under the Marie Skłodowska-Curiegrant agreement No 721619 for the SOCRATES project.

taken into account when attempting to establish telepresence.These are:

• human-robot interaction between the local user and therobot;

• human-robot interaction between the remote user andthe robot (through the communication channel);

• human-computer interaction between the remote userand a robot control point (being that a PC computer, ora tablet device, or anything else);

• human-computer interaction between the remote userand the robot control interface.

Equipping the robot with more features to improve ex-pressiveness also puts an additional burden on the usercontrolling it, unless the robot can handle certain functionsautonomously. For example, standard first person shooter(FPS) controls used frequently in computer games allow2DOF movements of the character using arrow or WASDkeys and 2DOF of viewpoint change via mouse. This issufficient for typical tasks like pointing and shooting, how-ever it is inadequate when used as a means of non-verbalcommunication between characters. At the same time, addingmode degrees of freedom to a virtual character or a robotrequires the introduction of corresponding user controls and,consequently, results in an extra burden on users.

This study looks at the interaction between the remote userand the robot in the scenario, specifically when the robotis able to act autonomously or semi-autonomously. In suchcases, the robot has to adjust its functionality based on theuser’s needs. Sliding autonomy (, or adjustable autonomy peroriginal definition by [10],) addresses precisely this issue,allowing the robot to ”slide” according to user requirementsand environmental changes. Thus, sliding autonomy aims toreduce the burden on the user who otherwise has to carry outall actions and adjustments fully manually. For this purpose,[4] proposes an ”explicit theory of delegation (and trust)” toachieve the dynamic collaboration in human-robot teams.

Function redistribution in sliding autonomy means that oneor multiple functions are transferred between the agents (thehuman and the robots). Sheridan emphasizes the need foran allocation authority that handles the process of functionredistribution [11]. With this authority established and thecommand issued, however, it is still necessary to hand thefunction over between the agents. In the manual operationscenario this is accomplished by means of a user interface(UI), so the robot is clearly instructed on what functionis handed over at what moment in time. In some cases,heuristics are used to automate the transition. For instance, an

autopilot system in certain aircrafts may be disengaged whenthe pilot puts deliberate force on the yoke. In such cases, thepilot’s actions may lead to a function transition regardlessof whether they are actually prepared to take over. In fact,this unintended function transition is cited as a main causein some aircraft accidents [3], [13].

While sliding autonomy is concerned with how to redis-tribute functionality between the human and the robot, thispaper looks at the actual process of switching a function inteleoperation, and, more broadly, in human-robot collabora-tion scenarios. In particular, our goal is to investigate whatthe process of handing over a function between humans androbots looks like, and whether there are any behavioral cuesthat can be utilized to help make this process as smooth andseamless as possible, similar to physical handovers [12].

Contrary to existing works that aim to improve the qualityof the switching process by providing a better feedback to theuser, we, in this work, look at the process of switching func-tions as a handover. In this function handover both agents,the user and the robot, play equal roles in determining detailsof the handover. Function handover denotes the transfer of atask or control from one agent to another. In the context ofthis paper, the terms function, duty and task may be usedinterchangeably, since the sole type of function handoverfound here essentially constitutes an exchange of duties.

Since teleoperation scenarios allow for diverse constella-tions including several human operators or groups of robots,function handovers may also occur between a human and allor a subset of the team’s robots. It is worth investigating ifthere are also insights to be gained from analyzing handoversbetween more than two humans.

In this first step, however, we present a scenario that allowsdata collection from a function handover in dyadic human-human and human-robot teams. The proposed scenario isimplemented in a state-of-the-art robotics simulator andtested in a preliminary experiment.

The contribution of this work is in developing an ex-perimental platform that allows data collection in order tostudy function handovers in human-human teams. Below, wedescribe the experimental platform as well as an introductorystudy that has been performed using this platform.

The paper is organized as follows: The general approachis described in Section II. Section III show the design andthe results of the initial experiment, conducted to verify thescenario and the general approach. The paper is concludedby Section IV.

II. METHOD

The ultimate goal of this study (as well as follow-up en-deavors) is to enable a robot to perform a function handoverwith a human partner both autonomously and seamlessly. Inpractice, this means that the robot has first to identify theneed to adjust its autonomy by handing over functionalityto and from the human – the problem addressed by slidingautonomy. But it also has to identify the precise momentwhen the handover is required to take place, as well as the

exact way of signaling and acknowledging if there are severaloptions.

There are multiple ways to facilitate autonomous functionhandovers. First of all, observing and understanding theenvironment (and the own status within this environment)allows the robot to take on a ”human’s view” and identifywhen the pilot’s actions are inadequate. A circumstance ofthis kind can qualify as a sign for the robot to proactivelyinitiate a handover.

A second option is to observe the behavior of the humanoperator and to search for cues that might indicate the needto perform a function handover – ideally before seriousmistakes have occurred. For instance, it has been shownbefore that high workload can be a good predictor of loss ofefficiency. Furthermore, we hypothesize that predictors canbe found to indicate not only the need, but also the timingof the handover.

To this end, recording and incorporating data from boththe environment and the operator is expected to provide thenecessary means to make informed decisions on when it isadvisable to perform a handover.

At this early point of the study our goal is to establishan experimental framework that would allow us to observehuman-human and human-robot function handovers in aplausible scenario. As this study is based on empiricalhypothesis testing, a scenario has been developed in orderto facilitate data collection from function handovers.

A number of baseline criteria have been set for the overallscenario design. These are as follows:

• a two-party (human-human or human-robot) team actingin the same environment;

• the two parties collaborate towards a common goal;• a score is assigned based on the overall team perfor-

mance;• achieving the goal implicitly requires that one (and

always the same) function be handed over between teammembers in both directions;

• the function handover occurs multiple times during oneexperiment session;

• the entire experiment session is observable by the ex-perimenter;

• the environment is fully controlled.

A. Game Overview

The game scenario has been developed in order to fulfillthe aforementioned criteria (Fig. 1a) and the game envi-ronment consists of two robotic arms (6 and 7) and threeconveyor belts (1, 2, and 3). Both robotic arms are controlledby the subjects. The arms differ in that arm (6) operatesautomatically once activated, while the arm (7) (locatedbetween the belts 1 and 2) is controlled manually using ajoystick. Subjects move the gripper and pick up the boxesby pressing the trigger button (hold the trigger until you wantthe gripper to release the box). Inverse kinematics is used toderive joint angles based on a tentative gripper pose. Thesecond arm (6, on the left) is automatic. Subjects only need

to turn it on and off by pressing another button. An indicatoron the screen shows the status of the automatic arm.

The task is to take boxes (4 and 5) from the straight belts(1 and 2) and put them on the circular conveyor belt (3).The belts carry two kinds of boxes, namely green and redones. Green packages are always on the left belt (1). Botharms can reach this belt, however, if the automated arm (6)is enabled, the manual arm (7) is not permitted to enter thebelt area and pick up a box. The right belt (2) carries redboxes. These can only be reached by the manually controlledarm (7). The frequency of spawning green boxes is adjustedbased on subjects’ success rate, thereby making it impossibleto pick up all boxes.

For each box that subjects put on the circular belt (3) withthe manually controlled arm they receive one point. If theautomated arm successfully places the box on the target belt,then it is awarded the one point. If a green box is lost (i.e.,it falls from the belt), then subjects lose a point. Moreover,losing a red box costs the subjects ten points, as it is solelytheir responsibility to pick it up. The idea behind the scoringis that subjects need to enable the automated arm to take careof those boxes they cannot take themselves. At the sametime, it is neither a good idea to let the automated arm pickup all the green boxes, because subjects will not earn anypoints for that.

The role of allocation authority in the current experimentis to switch the function (and thereby the responsibility) topick up boxes from belt (1) between the two agents. Thus,if the role of allocation authority is assigned to the subjectcontrolling the manual arm (7), then the scoring systemfavors a strategy in which handovers are being performedirregularly, depending on the current situation.

B. Implementation

The experimental environment is developed using the V-Rep simulator [9]1. Subjects control the robotic arm usinga joystick. The autonomous arm is activated and deactivatedvia a push button. The button is detached from the joystickand can be pressed independently. X and Y axes of thejoystick move the gripper within the horizontal plane (ofthe environment coordinate system). The gripping action iscontrolled by using the index-finger button. The fingers onthe gripper on the subject-controlled arm are arranged in a120o configuration, so it is tolerant towards imprecise gripperpositioning over boxes. Subjects observe the environmentfrom the top view to simplify mapping of the joystickmovements to arm actions (see Fig. 1b).

III. HUMAN-HUMAN FUNCTION HANDOVERS

The main goal of the experiment is to perform an initialdata collection and analysis and make some first observationsof the function handover. When observing the functionhandover between two humans, we postulate the followinghypotheses based on observations from physical handovers:

1The physics engine used in the experiment is Bullet v.0.78.

(a) General overview of the experiment environment.

(b) Operator’s view of the experiment environment.

Fig. 1: The experiment environment. Boxes (4 and 5) arecarried by the conveyor belts (1 and 2) and must be placedon the circular belt (3). (7) is a manually controlled roboticarm and (6) is an automated arm which has to be engagedand disengaged.

H1 As in the case of physical handovers, function han-dovers involve a signaling phase, in which the giverand the receiver agree on performing the handover.

H2 As in the case of physical handovers, the signaling phaseof function handovers mostly relies on non-verbal cues.

H3 There is a causal relationship between the current stateof the environment, i.e., the positions of the robots andboxes on the belts, and the gaze of the giver, which thereceiver can learn to predict and anticipate the handoverrequest.

Thus, rejecting the null hypotheses would require us toobserve signaling before performing handovers (to reject nullhypothesis of [H1]), the signaling is non-verbal ([H2]), andto observe a correlation between the current situation on theconveyor belts (for example, two concurrent boxes) and thehandover event ([H3]).

A. Experiment Design

The experiment was structured in three conditions de-scribed below and utilized a complete within-subject design.

Condition I: The purpose of Condition I was to analyzethe modalities with which the participant acting as theoperator requested the help of the participant acting as theassistant, to assess the validity of hypotheses [H1] and [H2].In this condition, the operator and the assistant were ableto communicate however they preferred to, as long as theyremained seated in their respective positions.

Before the experiment, the subjects were instructed on thepurpose of the game and the commands to issue to control thevirtual robots; they were told that they could communicatethe need for assistance anytime during the game in whateverway they wanted. No comment was made about how orwhen the operator was supposed to ask for the assistant’sintervention. Importantly, the assistant was unable to see thevirtual workspace and was instructed to press the push buttonwhenever the operator requested assistance. Only a visualindicator of the automated arm was visible to the assistant,to allow them to confirm the status of the arm (on or off)and to avoid undesired confusion in the operator-assistantcommunication.

In this and the following studies, we saw the functionhandover as an event of switching the automated arm onand off. In this handover, the function which is handed overbetween the operator and the assistant was represented as theduty to take the green boxes from the left and place themon the circular conveyor belt.

In accordance with our hypotheses, we predicted that:i at least in the initial stages of the experiment session,

there is a correlation between the subjects’ non-verbalcues and the verbal assistance requests;

ii the majority of all assistance requests is verbal;iii at least in the initial stages of the experiment session,

eye contact precedes the verbal assistance requests;iv the assistant correctly addresses the majority of the

assistance requests.If the predictions were correct, Hypothesis 1 would be

supported and Hypothesis 2 partially supported.

Condition II: Condition II aimed at assessing the validityof [H1]. The experiment setup was identical with that inCondition I, with the exception that this time around subjectswere not allowed to talk or use any verbal cues. Constrainingthe subjects to only non-verbal communication was supposedto force the subjects to explore other ways to communicate.

As in Condition I, the assistant could not see theworkspace, but only the visual automated arm status indi-cator.

In accordance with our hypotheses, we predicted that:i the user and the assistant use eye contact to communi-

cate the assistance request (and acknowledgment);ii the assistant will correctly address most of the user’s

requests (differences to scenario (1) will be minimum);iii there is a correlation between the scene in the virtual

environment and the timing of the assistance requests.If the predictions were correct, Hypothesis 1 would be

supported and so would Hypothesis 2.

Fig. 2: Overview of the experiment environment.

Condition III: Condition III aimed at testing [H2]. In thiscondition, the experimental setup was exactly the same as inCondition II, only in this case the assistant was also able tosee the workspace. The predictions were as follows:

i the operator and the assistant use eye contact to com-municate the assistance request and acknowledgment;

ii the assistant will correctly address most of the user’srequests (differences to scenarios (1) and (2) will beminimum);

iii there is a correlation between the scene in the virtualenvironment and the timing of the assistance requests;

iv there is a correlation between the scene in the virtualenvironment and the assistant’s gaze (i.e., they try topredict an assistance request).

If the predictions were correct, Hypothesis 1 would besupported and Hypothesis 2 would be supported as well.Moreover, if prediction 4 was correct, then Hypothesis 3would also be supported.

B. Experimental Settings

In this experiment, subjects were placed in a vis-a-visposition (see Fig. 2). Two identical monitors were placed sothat both subjects could see the simulated workspace and atthe same time each other’s faces clearly. During Conditions1 and 2, the assistant’s monitor was covered with the maskwhich only revealed the automated arm indicator to confirmthat it had been switched on and off.

Throughout the experiment, data was continuously col-lected from user controls (all joystick axes and buttons,assistant’s button) and the simulated environment (spawned,collected, lost boxes, current scores, etc.). The non-verbal be-haviors of the subjects were analyzed using video recordingscollected with monitor-mounted cameras. The NASA TLXquestionnaire [6] was employed to collect subjects’ perceivedworkload after each condition, with the weighting sessionbeing performed after the experiment on a per-subject basis.

The experiment was designed in the within-subject form.Every subject was exposed to all experimental conditionsand participated in two blocks - as an operator and as anassistant. The order of blocks was counter-balanced acrossthe subjects. The protocol of the complete experiment sessionwas as follows:

• Introduction and signing of an informed consent form.

• Background questionnaire and example videos of op-eration sessions: (bad) assistant is always off, (bad)assistant is always on, (good) assistant is switched onand off, based on the current situation.

• Block I:– Practice session - 6 min.– Condition I - 6 min, TLX questionnaire.– Condition II - 6 min, TLX questionnaire.– Condition III - 6 min, TLX questionnaire.

• Interim results questionnaire.• Subjects switch their places and corresponding roles.• Block II:

– Practice session - 6 min.– Condition I - 6 min, TLX questionnaire.– Condition II - 6 min, TLX questionnaire.– Condition III - 6 min, TLX questionnaire.

• Final results questionnaire.• TLX weighting procedure.• Debriefing.

C. Results

In total, six subjects participated in the pilot experiment;three male and three female, all with a background intechnology, ages ranging from 23 to 46 (µ = 33.2, σ =8.52). Each subject participated in both experimental blocks(as the operator and the assistant); the order of blocks wascounterbalanced. No effect of order on other performancemetrics was found. No further statistical analysis of perfor-mance data was conducted due to the small sample size.

Despite the small sample size, the observations derivedfrom individual teams and subjects revealed valuable infor-mation for the study and experiment design. The followingparagraphs summarize the observations along with the anal-ysis of the questionnaire data regarding subjects’ feedbackand workload assessment.

The workload was measured after each condition, usingthe NASA TLX questionnaire. The weighting procedurewas administered after the experiment and the weights werecalculated per each subject. Five out of six subjects reported aconsiderably higher workload in the operator role than in theassistant role. At the same time, despite the higher workload,all subjects reflected during the debriefing that they enjoyedthe operator role more. In the case of the assistant role, foursubjects reported an increased workload in the third condition(when they were able to see the environment). This, however,had no visible effect on the workload of the operator inthe corresponding condition. Five subjects reported an onaverage 3.5 times higher workload in operator mode relativeto assistant mode. One subject had a higher workload in theassistant mode.

When asked what was the main impediment in the task,the assistants’ responses were: blindness, focus, signaling,anticipation, conflict (with the operator), physics (w.r.t. non-trivial box handling), in that order. To the same questions,the operators’ responses were: control, physics, signaling,dropping (boxes), grasping, physics. All subjects surmised

that they would perform better if they trained more. Asthe main challenges in Condition 2, subjects regarded thedifficulty to remember to give a signal, response time anddifficulty to handle concurrent tasks (gesturing). When askedwhat had changed in Condition 3 (when the assistant wasable to see the screen), the subjects listed the ability toanticipate, misunderstandings and action by the assistantswhen they were not expected or instructed to act.

On average, the subjects performed µ = 27.2 (σ =4.3) handovers in both directions per session (each sessionbeing 6:00 minutes), with a total of 499 handovers observedacross all conditions and all subjects. Explicit eye contactbetween both subjects was not observed in every handover,but was rather common in situations in which conflicts ormisunderstandings occurred. Thus, the predictions regardingthe stable eye contact preceding each handover occurrencewere not supported. Prediction (i) of Condition I was notconfirmed either, since no eye contact or gestures werepresent during this condition (i.e., all subjects communicatedverbally). Predictions on the success of handovers werein fact confirmed, with an overall success rate of 99.6%.Prediction (iii) in Conditions II and III were supported, asthe operators requested assistance in all cases in which therewere concurrent boxes present on the belts. Prediction (iv)for Condition III could not be assessed due to a too smallsample size, however preliminary data shows that assistantswere more inclined to look at the screen when they wereable to see the environment, compared to Conditions I andII in which they were not.

The important qualitative observation from all sessions isthat all subjects across all conditions agreed on their sig-naling prior to the session start (subjects were not restrictedto communicate between the experimental conditions). Theagreement ranged from brief negotiation of keywords todeveloping deliberate strategies for collaborating over thecentral conveyor belt. The signaling in verbal mode in-cluded simple commands such as ”engage”, ”disengage”,”on”, ”off”. For the signaling in non-verbal conditions, foursubjects opted for hand gestures (finger or arm up/down) andtwo chose to use nods of the head.

In this experiment, one of the two subjects (the operator)has been explicitly taking on the role of allocation authority.This was due to the fact that in two experiment conditionsthe assistant did not have information about the experimentalenvironment. The order of conditions in each block wasnot randomized due to the small number of subjects. Thatmay explain the fact that in the third condition the role ofallocation authority always remained with the operator.

Another important observation in this initial data collec-tion is that all subjects negotiated signaling before eachsession. This contradicts the prediction that signaling hasto precede each handover, and moreover, this eliminates theneed for eye contact during handover. It also reveals the morecomplex nature of handing over functions than was expectedin the beginning of the study.

IV. CONCLUSIONS AND FUTURE WORKS

This paper was written specifically with regard to a robotteleoperation scenario, in which the robot is able to adjust itsown level of autonomy. In such a case, there are two actingentities, controlling the robot simultaneously. Adjustableautonomy allows to switch functions between actors, as away to optimize for changing environments. However, it alsorequires that the functions be switched seamlessly, with theoperator always being prepared to take over or yield control.This latter detail on which this paper is focusing is that ofhow exactly functions are handed over between actors.

Multiple options to switch functions are used in currenthuman-machine systems. Depending on who is responsiblefor function allocation, the operator can have full control overthe switching process (e.g. when using traditional dashboardcontrols), or can be notified about the need to switch. Theultimate goal of this study is to allow the robot to synchronizethe handover with the operator, thereby eliminating any needfor explicit action from the user.

In this paper, we discussed function handovers as a keyissue for HRI and human-robot collaboration. We furtherpresented an experimental framework that allows us to studythese handovers between humans and - in future applications- humans and virtual agents. Finally, a number of preliminaryexperiments were conducted with the presented experimentalplatform as a first investigation into the parameters whichpromote function handovers.

The collected initial observations do not support any ofthe experiment predictions. Moreover, the human-humanfunction handover experiment demonstrated the negotiationbetween the subjects on signaling and error handling prior tothe entire experiment session. A possible explanation of thismight be the fact that subjects knew that they would haveto perform a series of handovers rather than just one, as isthe case with different scenarios. The main direction of ourfuture work is to take a closer look at 1) individual functionhandovers between humans and 2) a series of handovers,similar to the scenario in the presented study, but dealingwith physical objects. For instance, it can be observed indifferent daily situations that two people handing over objectsbetween each other multiple times do not establish eyecontact during each handover event. A formal experimentcould be established to observe this situation. If, in suchan experiment, the negotiation phase between subjects willprecede an experiment session, this will indicate supportfor Hypothesis 1 in the human-human function handoverexperiment.

Another direction of future work includes extending thedata collection and analysis to obtain more quantitative per-formance data such as reaction and response time, and alsoextend the interaction analysis between subjects to includebrief and peripheral vision, and possibly motion detection.Collecting physiological data can also be beneficial as it hasbeen shown to be a strong predictor of workload or stress [2],[1] , both of which can affect the assistance requests.

In the human-human experiment, the subjects’ gaze behav-

ior analysis is currently implemented using video analysis.The resulting robust and reliable data collection is necessaryto observe the operator-assistant gaze interaction fully.

Possible explanations of why no strong behavioral cueswere observed in the experiments could lie in the natureof handovers. The skill of handing over an object is highlydeveloped in humans and is a conscious action to indicateintention. There is nothing suggesting that functional han-dover is similar enough to handing over of physical objectsthat it would be trained simultaneously. A second directionof future work thus is to elicit behavioral cues intentionally,by modifying the experiment in such a way that subjects areasked to give a ”cue” prior to the explicit request.

Yet another planned setup involves two or more assistants(in the human-human scenario) or several robots with over-lapping radii of action. In the former case, the experiment isagain comprised of three conditions in which the assistantsmay or may not see the environment and they may or maynot be allowed to use verbal cues. In all three conditionsonly one assistant’s robot can be active at a time and it isthe operator’s responsibility to signal who is supposed to takeover in a given situation. It will be interesting to see how thenature of the cues change compared to the single-assistantexperiments and whether the dynamics change drastically inthe third condition.

REFERENCES

[1] H. Ayaz, P. A. Shewokis, S. Bunce, K. Izzetoglu, B. Willems, andB. Onaral. Optical brain monitoring for operator training and mentalworkload assessment. Neuroimage, 59(1):36–47, 2012.

[2] G. Borghini, L. Astolfi, G. Vecchiato, D. Mattia, and F. Babiloni.Measuring neurophysiological signals in aircraft pilots and car driversfor the assessment of mental workload, fatigue and drowsiness. Neu-roscience & Biobehavioral Reviews, 44:58–75, 2014.

[3] B. d. et d’Analyses et al. Final report on the accident on 1st june2009 to the airbus a330-203 registered f-gzcp operated by air franceflight af 447 rio de janeiro–paris. Paris: BEA, 2012.

[4] R. Falcone and C. Castelfranchi. The human in the loop of a delegatedagent: The theory of adjustable social autonomy. IEEE Transactionson Systems, Man, and Cybernetics Part A:Systems and Humans.,31(5):406–418, 2001.

[5] T. Fong and C. Thorpe. Vehicle teleoperation interfaces. Autonomousrobots, pages 9–18, 2001.

[6] S. Hart & Staveland, L. Development of the NASA-TLX: Results ofempirical and theoretical research. In Human mental workload, pages139–183. 1988.

[7] P. F. Hokayem and M. W. Spong. Bilateral teleoperation: An historicalsurvey. Automatica, 42(12):2035–2057, 2006.

[8] A. Kristoffersson, S. Coradeschi, and A. Loutfi. A Review of MobileRobotic Telepresence. Advances in Human-Computer Interaction,2013:1–17, 2013.

[9] E. Rohmer, S. P. N. Singh, and M. Freese. V-REP: A versatile andscalable robot simulation framework. IEEE International Conferenceon Intelligent Robots and Systems, pages 1321–1326, 2013.

[10] P. Scerri, D. V. Pynadath, and M. Tambe. Towards adjustable auton-omy for the real-world. Journal of Artificial Intelligence Research,17:171–228, 2002.

[11] T. B. Sheridan. Adaptive automation, level of automation, allocationauthority, supervisory control, and adaptive control: Distinctions andmodes of adaptation. IEEE Transactions on Systems, Man, andCybernetics Part A:Systems and Humans, 41(4):662–667, 2011.

[12] K. Strabala, M. K. Lee, A. Dragan, J. Forlizzi, S. S. Srinavasa, M. Cak-mak, and V. Micelli. Towards Seamless Human-Robot Handovers.Journal of Human-Robot Interaction (2013), 1(1):112–132, 2012.

[13] T. S. Times. Tape Reveals Kids Got Flying Lesson Before Crash,1994.