An On-Going Evaluation of Domestic Robots

Position paper

Gabriella CortellessaInstitute for Cognitive Science and Technology

Italian National Research CouncilVia S. Martino della Battaglia 44, 00185 Rome,

Italygabriella.cortellessa@istc.cnr.it

Amy Loutfi and Federico PecoraAASS Mobile Robotics Lab

Dpt of Technology, Örebro UniversityS-70182 Örebro, Sweden

fpa@aass.oru.se,amy.loutfi@tech.oru.se

ABSTRACTIn this position paper we describe an on-going effort to pro-vide an in-depth and cross-cultural evaluation of how el-derly users perceive robotic systems for domestic cognitivesupport. Our work is grounded on two implemented smart-home prototypes, namely the RoboCare Smart Home de-veloped in Italy, and the PEIS Home developed in Swe-den. The former project has provided a testbed for an a-posteriori evaluation of smart home technology with Italianuser groups. The presence in Sweden of the PEIS Home,a system which shares numerous commonalities with theRoboCare Smart Home, gives us the opportunity to extendthese results by (1) providing a cross-cultural perspectiveon the perception of smart home technology, and (2) lay thefoundations for a live, Wizard of Oz based evaluation withinthe PEIS Home.

1. INTRODUCTIONIn this brief paper, we report our current strategy to push

the envelope in the development of acceptable and effectiveHRI schemes for social assistive robots. Specifically, we sum-marize an on-going evaluation effort which started last yearwithin a project called RoboCare [4] developed at ISTC-CNR with Italian user groups. The availability of a proto-typical smart home (which was the primary outcome of theRoboCare project — see http://robocare.istc.cnr.it),made it possible to study aspects related to the interactionbetween users and a set of intelligent artifacts in the smarthome. These artifacts include a mobile robot, which is theprincipal mediator of all human-system interactions. Afterproviding some points of discussion on the results obtainedso far in the RoboCare evaluation, we describe our cur-rent effort in extending this evaluation with a cross-culturalassessment performed on Swedish user groups. This fur-ther evaluation is currently in set-up phase, and is groundedon another“robotically-rich”environment, namely the PEISHome [11], which focuses on the development of an ecologyof Physically Embedded Intelligent Systems (PEIS).

Both RoboCare and PEIS were initially conceived tofocus primarily on technology development. The use of sim-ilar AI and robotics based technology for supporting elderlypeople at home has been addressed in various other researchprojects in the last years [8, 9]. More recent research hasbeen increasingly focusing on Cognitive Systems to produceaids that enhance human cognition capabilities [7]. Thestate-of-the-art in robotics today is bringing about an in-

creasing focus on human-robot interaction in general and onsocial assistive robotics in particular. Indeed, as the robotic,sensory and software infrastructure began to enter a morestable stage of development in RoboCare and PEIS, thequestion of evaluation naturally came to our attention.

In RoboCare, a team of psychologists was employed todrive technology development since its early stages, althoughits recommendations and the technological possibilities ap-peared to be misaligned in numerous occasions. In laterstages of development, the availability of a complete pro-totypical system was employed to produce an a-posteriorievaluation with Italian user groups. This evaluation under-scored several interesting indications concerning the percep-tion of an assistive robot and the smart home’s capabilities.At the same time, the issue of evaluation emerged in thePEIS Home, and an investigation into possible evaluationstrategies and methodology began. This lead to the idea ofperforming the same a-posteriori evaluation that was car-ried out on the Italian user groups with Swedish users, inorder to assess the extent to which the findings could begeneralized across different European cultures.

In the following paragraphs, we briefly summarize theprincipal findings of the RoboCare evaluation, and putforth some hypotheses on the results of the latter evaluation.We conclude with a brief discussion on further possibilitiesfor continuing the evaluation effort in both the PEIS andthe RoboCare contexts.

2. EVALUATING USER PERCEPTIONS OFSOCIAL ASSISTIVE ROBOTS

The emphasis in social assistive robotics is to support hu-man users through social rather than physical interaction [5].A key aspect in the development of social assistive robotsis the evaluation of social interaction between human usersand robotic agents. A number of guiding principles exist inthe literature. For example, [10] highlights how observationand behavioral analysis of human-robot social interaction inreal environments is necessary in order to take into consid-eration all the divergent factors pertaining to the design ofsocial robots. The design of social robots also raises a num-ber of ethical issues that need to be discussed within the re-search community to provide guidance to system designers.[14] considers some of the ethical implications of human-robot interaction, mainly related to the kind of authenticitywe require in our technology as well as to the choice of themost appropriate relationship between children/elders and

relational artifacts.

2.1 Previous Results: the RoboCare Evalua-tion

The RoboCare project examines some of the relevantfactors for the design of assistive robots. RoboCare has in-volved research groups with different backgrounds with thegoal of investigating how state of the art AI and roboticstechniques can be combined to create new domestic servicesfor elderly people. The project has produced a prototype ofintegrated home environment (the RoboCare Smart Home)consisting in a robotic interactive agent, some vision-basedsensors for continuous monitoring, and additional intelligentsystems that store and reason upon knowledge about the as-sisted elder’s scheduled activities. A multi-agent coordina-tion algorithm guarantees the coherence of the behavior ofthe whole environment. This provides a functional cohesivewhich invokes the smart home’s services so as to preservethe person’s safety and to provide suggestions. The detailsof how these different interactive functionalities are obtainedand integrated are described in [4].

The RoboCare Smart Home includes a mobile roboticplatform with interaction capabilities1. The multi-agent en-vironment interacts with the assisted person through thisrobot, which we have called a robotic mediator (see Fig-ure 1). The base on top of which the robotic mediator

Figure 1: The RoboCare Smart Home and the robotic

mediator.

is built consists in a Pioneer platform. The mobile plat-form is equipped with additional sensors, namely a laserrange finder, a stereo camera and an omni-directional cam-era, as well as additional computational resources consistingin two laptops, one for on-board sensor processing and nav-igation and one for human-robot interaction. The robot isendowed with verbal user interaction skills: speech recog-nition is achieved with the Sonic speech recognition system(University of Colorado)2, while speech synthesis is drivenby a simple text-to-speech system.

Since its beginning, RoboCare has raised a number ofuser acceptability challenges. In particular one, also re-

1In the spirit described in [5] the Smart Home is an ex-ample of Social Assistive Robot, a concept which can bedistinguished from Social Interactive Robot [6] because itsmain task is to monitor and assist the person rather thansimply interacting with him/her.2For details, see cslr.colorado.edu/beginweb/speech_recognition/sonic.html

ported in [13] has been paramount in our work: “what arethe circumstances in which people accept an assistive robotin their environment?”. Other important questions we havestrived to answer (or at least investigated) are “how shouldan elder user communicate with a robot?”, “should the robotlook like a human being?”, and, last but not least,“are robotsuseful in the domestic environment?”.

RoboCare set out to provide a first investigation into theabove questions. The full results of this investigation as wellas the details of the methodology are reported in [2, 3]. Theevaluation was performed through eight short films (rang-ing from about 30 seconds to little more than one minute)depicting potential interactions occurring in a real domesticenvironment between an elderly person and the robotic me-diator. Various experimental manipulations of the featuresof the robotic agent were employed (e.g., two variants of therobot, one in which it had a 3D representation of a face, andone with no anthropomorphic traits). In the eight scenar-ios we presented common everyday life situations in whichthe robot provides cognitive support to the elderly person.Scenarios referred to critical areas of every day living, in-cluding (a) management of personal/environmental safety,(b) healthcare, (c) reminding events/deadlines, (d) supportto activity planning, (e) suggestions. A sample movie isavailable at http://robocare.istc.cnr.it.

After viewing the videos, respondents were asked to an-swer a questionnaire on the eight scenarios regarding (a) thelikelihood of the eight situations, (b) the physical aspect ofthe robot, (c) an emotional scale for evaluating the pos-sible presence of the robot in their own home, and (d) theperceived health of the interviewee. The evaluation was per-formed on forty male and female users, aged between 56 and88.

Among the most interesting results, we highlight the fol-lowing.

Perceived usefulness. The proposed scenarios were eval-uated as significant in the everyday life of respondents,and the Smart Home’s support was evaluated as bothuseful and appreciated. In particular, we found a higherlevel of perceived utility for personal safety scenarios,and a lower level for instances in which the system sug-gests non-critical activities, such as going for a walk orreminding someone’s birthday.

General evaluation of the robot. Mean scores referringto key features of the robot were calculated. On thewhole, the robot emerged to be positively evaluatedwith respect to physical aspect, interactive behaviorand communication modalities, level of integration withthe domestic environment, absence of perceived in-trusion/disturbance in everyday life and routines, andpersonal advantages of having such a device at home.

Similarity to human beings. A comparison between ex-perimental conditions showed a stronger preference forthe least anthropomorphic version of the robot. Inparticular, a significant difference in favour of this ver-sion of the robot emerged with respect to physical as-pect, integration with the home environment, and ad-vantages provided by the robot at home. No signifi-cant difference between the different versions emergedwith respect to interactive behavior and communica-tion modalities and level of perceived intrusion/distur-bance.

Influence of psychological variables. Our analyses out-lined a clear influence of both perceived health andworry about loss of cognitive competence on evalua-tions expressed by elderly people. Elderly people per-ceiving better health conditions expressed more posi-tive evaluations about the integration of the robot inthe domestic environment, and perceived the robot asless intrusive in everyday life than did those who com-plained about worse health conditions. No significantdifference emerged with respect to physical aspect, in-teractive behavior and communication modalities, nordid poorer health affect the perception of the advan-tages provided by having the robot at home.

Overall, although the perceived utility of and the expressedpreference towards a proactive robot intervening in case ofan emergency is higher than in other scenarios, a positivereaction to different interactive situations was undoubtedlyfound. This is in line with current research in aging, inwhich the key role of compensation strategies to managethe loss of personal resources is underscored. In addition,even though elderly people do not think that a robotic agentliving in the domestic environment can be as useful for un-compelling activities as it is perceived to be in safety-relatedcontexts, they do not perceive it as completely out of placewhen supporting the former. This result shows an opennessof elderly people towards a variety of functionalities, at leastto some extent: they may be rather unfamiliar with someof the smart home’s technology, but certainly not against ita-priori.

Lastly, we should mention that a further evaluation withItalian users is currently in progress, and is aimed at as-sessing the perception user have of the RoboCare SmartHome with no robotic mediator. Specifically, the eight sce-narios are presented in a smart home without an embodiedrelational agent, the interaction between the assisted per-son and the system occurring through environmental mi-crophones and loud speakers. Given that the RoboCarerobot is not endowed with manipulatory capabilities, we areinterested in understanding the value of embodiment in theinteraction between users and the intelligent cognitive sup-port system. Although the evaluation is currently in its finalstages, preliminary results show that the embodied versionof the RoboCare Smart Home maintains overall preference.Specifically, the overall preferred configuration among theproposed scenarios is the one in which the robot is presentand has less anthropomorphic traits.

2.2 Evaluation in PEIS

The concept of PEIS, originally proposed in [11], com-bines insights from the fields of ambient intelligence, au-tonomous robotics and sensor networks (see Figure 2) togenerate a new approach to the inclusion of robotic technol-ogy into smart environments. In this approach, advancedrobotic functionalities are achieved through the cooperationof many simple robotic, sensory and software components.PEIS builds upon the principle that any device incorporat-ing some computational and communication resources, andpossibly able to interact with the environment via sensorsand/or actuators, is included as a “building block” within aunifying framework. A PEIS can be as simple as a toasterand as complex as a humanoid robot. In general, we define aPEIS-Ecology to be a set of inter-connected software com-ponents, called PEIS-components, residing on distributed

Figure 2: The PEIS approach combines insights from

several fields.

physical entities. Each component may include links to sen-sors and actuators, as well as input and output ports thatconnect it to other components in the same or another PEIS-Ecology. The framework connecting all PEIS is a uniformcommunication and cooperation model. This allows the ex-change of information among PEIS, and can cope with themjoining and leaving the ecology dynamically. Also, a com-mon cooperation schema allows each participating PEIS touse functionalities from other PEIS in the ecology in orderto compensate or to complement its own.

Figure 3: Two views of the PEIS environment.

The approach underlying user interaction implemented inthe PEIS Home is based on the idea that the human is justanother PEIS in the PEIS-Ecology. As any other PEIS, thehuman can be made aware of which functionalities can beprovided by the environment, and decide to use any one ofthose. In the reverse direction, the PEIS-Ecology may relyon the human user to provide certain functionalities, likechoosing between alternative courses of action.

In practice, this intuitively appealing viewpoint is compli-cated by a number of issues. First, the set of tasks that thewhole PEIS-Ecology could perform is, in general, very large.These tasks can be at very different levels of abstraction andcomplexity, ranging from basic tasks which are performedby a single PEIS, like dimming the lights in the bedroom;to tasks which involve the coordinated activity of severalPEIS, like collecting all the expired items from the fridgeand putting them in the garbage. Since the set of possibletasks is so large, the human user cannot be presented withthe whole set at all times. Instead, an effective interfaceshould filter the tasks to be presented to the user accordingto their relevance to the current context.

In addition to the possibility of starting a large numberof tasks, the human user may also be interactively involvedin the execution of a task: the ecology may ask the user to

provide some functionalities which are needed for the perfor-mance of the task. Typically, these will be cognitive decisionmaking functionalities, like deciding which ones of the ex-pired items in the fridge should be reordered. A preliminaryuser interface design which attempts to overcome these is-sues in the PEIS Home is reported in [1].

As in RoboCare, the elementary functionalities providedby the PEIS Home can be combined to obtain high-levelcognitive support services. While there exists a strong dif-ference in infrastructure, the two systems are both conceivedfor providing support services in safety and healthcare man-agement, memory-aides, activity planning support and pro-viding suggestions on daily activities. For this reason, anevaluation of the scenarios originally scripted on the Robo-Care system would provide results that can be used to drivefurther high-level service implementation in the PEIS Home.In addition to this, such an evaluation provides a rare oppor-tunity to compare the bias of different cultural backgroundson the perceived benefit of the various features which canbe implemented in the smart home.

The evaluation will focus on a user group with the samecharacteristics as the RoboCare sample, most importantlywith respect to parameters such as number of participants,age, educational and social background. Also, the adminis-tration and evaluation methodology will be kept consistent.Specifically, all material has been reproduced exactly to beas coherent as possible between the two evaluations: thequestionnaires were translated and the order of questionspreserved; the videos were subtitled3 in Swedish, and theorder of presentation of the eight scenarios will be preserved.

3. DISCUSSIONThe anticipated outcome of the cross-cultural evaluation is

a set of indications that disprove or confirm the general ideathat the perceived advantage of employing current state-of-the-art smart home technology does not vary significantlyamong Italian and Swedish societies. There are, indeed,reasons to suspect either result. On one hand, the uptake ofrelatively new technologies is very different in the two coun-tries. In Italy, just 2.8% of households of people aged over65 has access to the Internet, while 45% of them own a mo-bile phone; in Sweden, the percentage of households havingaccess to the Internet at home was 73% in 2005, and thepercentage of individuals in the 55-74 age range using theInternet at least once a week is 51%4. This may suggest avery different level of confidence in smart home technology,and possibly different expectations with respect to the po-tential benefit of living in a smart home. However, there aretwo factors that add complexity to the comparative analy-sis. First, the fact that the analysis is concerned with veryspecific aspects of acceptability — eight different scenarios,in which the system supports the elderly person in a varietyof specific tasks. The level of appreciation of the different

3A short pre-evaluation on one elderly user was performed toascertain whether subtitling or dubbing would be a preferredmethod for conveying the spoken language in the videos. Asdubbing is virtually absent in Sweden, the user reportedthat reading subtitles seemed positively more natural thanwatching a dubbed scene, thus subtitling was chosen as themethod of translation.4Data taken from the“Supporting Policy Development for e-Inclusion” project — see the the EU country reports sectionon http://www.ipolicy.eu/.

features most probably depends on different aspects of theuser’s background which are not necessarily related to theiracquaintance with ICT. Secondly, the user base in both ex-periments represents a small sample of a very specific partof society, namely middle class elderly people with an ad-equate level of schooling and previous jobs in the servicesector (e.g., teaching).

Lastly, another factor which may distinguish Swedish users’perception of smart home technology lies in different na-tional elderly care culture and policy [12]. In Sweden, 98%of elderly people lived either alone or with their spouse in2002, while 35% of the Italian elderly lived with other familymembers and/or within communities in the year 2000. Evenmore interestingly, only 17% of the elderly in Nordic coun-tries (Sweden, Denmark, Finalnd and Norway) received carefrom family members within their own household in 1992,while in 67% of the elderly residing in Southern Europeancountries (Italy, Portugal, Spain, Greece, Ireland and Aus-tria) found themselves in the same situation. It is reasonableto assume that these and many other social differences prob-ably affect Swedish and Italian users’ perception of smarthome technology in different ways.

Video-based vs. live evaluation On the basis of the re-sults from the cross-cultural evaluation, we are investigatingthe possibility to conduct user evaluation within the phys-ically instantiated environment of the PEIS Home. Specif-ically, a possibility is to employ the Wizard of Oz (WoOz)technique, an experimental condition in which subjects in-teract with a system that is perceived as autonomous, butwhich is actually being operated or partially operated by anunseen human being. This study can contribute to overcom-ing the inevitable limitations of video-based evaluation: toget an accurate feel of being in the same room of a robotrequires all senses (e.g., the robot makes noise, the height ofthe robot may be intimidating, etc.) Most importantly, com-paring results of a WoOz evaluation can provide a criticalassessment of video-based vs. live evaluation methods [15].

Wizard of Oz

Evaluation Evaluation

Video−Based

Video sessions as

preparatory material

Figure 4: Using video-based evaluation to prepare users

for live evaluation in the PEIS.

Finally, a third possible form of evaluation we are in-vestigating involves the possibility to include some of theusers employed for the video-based evaluation also in thelive WoOz sessions (see Figure 4). Specifically, this wouldadd the possibility to assess whether the video material andthe fact that the user has thought about the situations byanswering the questions can serve as a“preparatory”stage tocurb the potentially disturbing presence of live components.

AcknowledgementThe work referenced in this paper on evaluation in Robo-Care was performed jointly with Maria Vittoria Giuliani,Massimiliano Scopelliti and Lorenza Tiberio of ISTC-CNR.The Authors are grateful to Alessandro Saffiotti and GionKoch Svedberg for their support.

4. REFERENCES[1] M. Broxvall, A. Loutfi, and A. Saffiotti. Interacting

with a robot ecology using task templates. InProceedings of the 16th IEEE Int Symp on Robot andHuman Interactive Communication (RoMan), 2007.

[2] A. Cesta, G. Cortellessa, M. Giuliani, F. Pecora,R. Rasconi, M. Scopelliti, and T. Tiberio. ProactiveAssistive Technology: An Empirical Study. InProceedings of the Eleventh International Conferenceon Human-Computer Interaction (INTERACT’07),2007.

[3] A. Cesta, G. Cortellessa, M. Giuliani, F. Pecora,M. Scopelliti, and L. Tiberio. Psychologicalimplications of domestic assistive technology for theelderly. PsychNology Journal, 5(3):229–252, 2007.

[4] A. Cesta, G. Cortellessa, F. Pecora, and R. Rasconi.Supporting Interaction in the RoboCare IntelligentAssistive Environment. In Proccedings of AAAI SpringSymposium on Interaction Challenges for IntelligentAssistants, 2007.

[5] D. Feil-Seifer and M. Mataric. Defining SociallyAssistive Robotics. In Proceedings of IEEEInternational Conference on Rehabilitation Robotics(ICORR-05), Chicago, IL, pages 465–468, 2005.

[6] T. Fong, I. Nourbakhsh, and K. Dautenhahn. Asurvey of socially interactive robots. Robotics andAutonomous Systems, 42(3–4):143–166, 2003.

[7] K. Myers. Calo: Building an intelligent personalassistant. Invited Talk. The Twenty-First NationalConference on Artificial Intelligence and theEighteenth Innovative Applications of ArtificialIntelligence Conference (AAAI-06), 2006.

[8] J. Pineau, M. Montemerlo, M. Pollack, N. Roy, andS. Thrun. Towards Robotic Assistants in NursingHomes: Challenges and Results. Robotics andAutonomous Systems, 42(3–4):271–281, 2003.

[9] M. Pollack. Intelligent Technology for an AgingPopulation: The Use of AI to Assist Elders withCognitive Impairment. AI Magazine, 26(2):9–24, 2005.

[10] S. Sabanovic, M. P. Michalowski, and R. Simmons.Robots in the wild: observing human-robot socialinteraction outside the lab. In Proceedings of the 9thInternational Workshop on Advanced Motion Control(AMC 2006), pages 596 – 601. IEEE, March 2006.

[11] A. Saffiotti and M. Broxvall. Peis ecologies: ambientintelligence meets autonomous robotics. InsOc-EUSAI ’05: Proceedings of the 2005 jointconference on Smart objects and ambient intelligence,pages 277–281, New York, NY, USA, 2005. ACM.

[12] G. Sundstrom and L. Johansson. The changingbalance of government and family in care for theelderly in Sweden and other European countries.Australasian Journal on Ageing, 24(s1):S5–S11, 2005.

[13] A. Tapus, M. Mataric, and B. Scassellati. The grand

challenges in socially assistive robotics. IEEE Roboticsand Automation Magazine, 14(1):35–42, 2007.

[14] S. Turkle, W. Taggart, C. Kidd, and O. Daste.Relational artifacts with children and elders: Thecomplexities of cybercompanionship. ConnectionScience, 18(4):347–361, 2006.

[15] S. Woods, M. Walters, K. Koay, and K. Dautenhahn.Comparing human robot interaction scenarios usinglive and video based methods: Towards a novelmethodological approach. In Proceedings of the NinthInternational Workshop on Advanced Motion Control(ACM 2006), Istanbul, March 27-29, pages 750–755,2006.