An Alternative Approach for Developing Socially Assistive Robots

Chandimal Jayawardena1 and Abdolhossein Sarrafzadeh

Abstract— This paper presents the design of the socially as-sistive companion robotic wheelchair named RoboChair. Unlikein most current companion robotics projects, the approach ofRoboChair is not to build a completely new robotic device.Instead, the focus of the RoboChair project is to convert analready useful device (i.e. wheelchair) to a socially assistivecompanion robot. The authors argue that there are number ofadvantages in this approach.

The proposed robotic chair is a mobile robot that can carrya person. It is equipped with several measuring devices formeasuring vital signs. The robot chair is capable of engagingusers with interactive dialogs through a touch screen and byusing human-robot interaction techniques. It has a scalablemodular software architecture so that adding new hardwareand software modules is straightforward. The software frame-work is based on Robot Operating System (ROS) open sourcerobotic middleware.

I. INTRODUCTION

Developing socially assistive robots is an emerging inter-disciplinary research area, which requires collaboration be-tween a wide range of disciplines, including robotics, healthsciences, psychology, gerontology, and human-computer in-teraction [1] [2]. In recent years, there is an increased interestin this area due to multiple reasons. Increasing capabilities ofmobile service robots, increasing needs of people for assis-tance, and increasing numbers of older population around theworld are some of the important reasons [3]. On the otherhand, there is an increasing shortfall in numbers of healthprofessionals and caregivers [4].

Among recent research projects, there have been attemptsto develop assistive robotic solutions to solve various healthand social issues. Mobility aids [5], manipulation aids [6],therapeutic aids [7], surgical robots [8], physical and mentalrehabilitation robots [9] [10], medication reminding robots[11] and elder-care robots [12] [13] are some examples .Among these solutions, ‘socially assistive robots’ belong toa distinct category.

Socially assistive robots are different from social robotsand entertainment robots, which provide relatively simplehuman-robot interactions. In contrast, socially assitive robotsare expected to provide a broad range of services to supportdaily activities of users.

In most current research attempts to design socially as-sistive robots, the focus is on designing new robotic agentsthat can interact with people by various means [14], [15],[16]. Since people do not have much experience with robots,usually extensive field trials are conducted in order to assessthe usability of these robots. However, determining the

1C. Jayawardena is with the Department of Computing, Unitec Institute ofTechnology, New Zealand. cjayawardena at unitec.ac.nz

usability of these robotic agents is a difficult task, since theresults of field trials are not always conclusive [15], [17].The difficulty of assessing usability directly affects productdevelopment and commercialization.

A. An alternative approach for developing useful sociallyassistive robots

In this paper, an attempt to overcome the difficulty ofevaluating usability of assistive robots is presented. Theauthors suggest a novel approach to develop socially assistiverobots and the current status of a research in that directionis presented.

The authors approach is to convert an already useful assis-tive device to a socially assistive robot by embedding variousrobotic features. The objective of the research presented onthis paper is to convert a normal powered wheel chair to asocially assistive companion robot. Functionally, this wheelchair will act as a socially assistive companion robot, whilefulfilling its function as a wheel chair.

II. SYSTEM OVERVIEW

This paper presents the design of the first version of a com-panion robot called RoboChair. RoboChair is a project con-ducted at the Unitec Institute of Technology, New Zealand.Fig. 1 shows RoboChair when it is being tested in a corridor.

Fig. 1. RoboChair being tested in a corridor

The motivation for designing this robotic chair comesfrom the service robot domain, in particular from health

2014 5th IEEE RAS & EMBS International Conference onBiomedical Robotics and Biomechatronics (BioRob)August 12-15, 2014. São Paulo, Brazil

978-1-4799-3127-9/6/14/$31.00 ©2014 IEEE 573

care robot domain. Currently, there many robots are beingdeveloped for health monitoring and assessment. HealthBotrobot can do medication reminding, vital signs measurement,falls detection among many other things [17].

RoboChair is in the form of a wheel chair. But, function-ally it is a socially assistive companion robot. The featuresincluded in the first version are as follows:

• Voice controlled navigation using fuzzy voice com-mands

• Vital signs (blood pressure, blood oxygen saturation,body weight, and bio impedance

• Cardiovascular risk prediction• Communicating with friends, family, and health profes-

sionals• Provide companionship through multi-modal interac-

tionsThese features are very similar to the features commonly

associated with personal robots. However, the design philos-ophy of RoboChair is quite different. The authors argue thatit is more beneficial to embed features of socially assistivecompanion robots into already useful devices rather thanbuilding completely new robots. Since the core device isalready in use, it is expected that the resistance to useracceptance would be minimal. However, this hypothesisshould be tested through user trials, which will be the nextstep of this research.

The following key requirements were considered in thedesign of RoboChair:

• Low cost: Since this robot is intended to be a personalrobot, cost is a primary consideration. Therefore, sim-pler and cheaper technologies were used.

• Simple user interface: Since the target user group wasolder or disabled people simple user interfaces wereused.

• Simple operation: All services provided by the robotcould be accessed via a touch screen. Speech synthesiswas used, but speech recognition was not used.

• Voice controlled navigation: For the convenience ofthe users with physical limitations, voice controllednavigation was included.

• Robustness: The working environment of the servicerobot is a domestic setting, where technical support isnot readily available. Therefore, special attention waspaid to the robustness.

• Open-source software: For software development, open-source software was used. In particular, Robot Operat-ing System (ROS) was used as the middleware.

Another important design consideration of RoboChair isthe ability to measure body weight. This is a limitation ofalmost all existing socially assistive robots designed for thehealth care domain. Most healthcare robots help users invarious areas related to health. Medication management, helpquit smoking are some examples. However, weight is animportant measurement in health and inability to measurethe weight of a user is a major limitation.

The first version of RoboChair is shown in Fig. 3(a).

III. ROBOT HARDWARE

The robot presented in this paper is a result of an on-going project at the Unitec Institute of Technology, NewZealand. The robot hardware consists of a differential drivemobile platform, a note book computer (main controller), atablet computer (GUI), sonar sensors, microphone, speakers,a kinect sensor, a laser range finder, a load cell and vitalsigns measuring devices. RoboChair is powered by a 24vLi-Polymer battery.

The tablet mounted on a rotatable mount is the main modeof interaction with the users. It displays instructional text,graphics, and video as well as takes user inputs by means oftouch events,

The kinect sensor is used for people detection using cob-people-detection ROS package [18]. The robots syntheticspeech was generated through Festival open-source speechsynthesis system [19].

IV. SOFTWARE ARCHITECTURE

The software architecture is based on the ROS open-sourceframework. An overview of the architecture is shown inFig. 3(b).

The architecture comprised of a main controller, GUI gen-erator, several distributed components and robot hardware.

A. Main controller

The main controller shown in Fig. 3(b) controls the corebehaviour of the robot. It is a Finite State Machine (FSM)created using SMACH package of ROS [18]. The maincontrol thread of the robot is in the FSM and it controlsand coordinates all the other distributed components.

Each state is associated with actions, GUI, and events. Thisis illustrated in Fig. 2. This representation has similarities

Statei

Events

Actions

(Physical actions, service requests)

Hello! Jack

CONTINUE

Visual output

Statei+1Statei-1

Events

(user inputs +

external events)

Speech

Fig. 2. State behavior

with the finite-state-machine based behaviour presented in[17]. However, there are major difference and improve-ments in RoboChair software architecture, when comparedto HealthBots Version 2 architecture presented in [17]

In HealthBots Version 2 architecture, robot behaviour andscreen layout definitions are in a single FSM and behaviourgeneration and screen generation are done by a singlemonolithic software. In RoboChair architecture, the robot

574

behaviour description and screen layout generation are sepa-rated. Still both robot behaviour and screen layout generationare controlled by a single FSM due to reasons describedin [17], but the actual screen generation and all front-endactivities are done by a separate software component (GUIgenerator), coordinating with the main FSM.

It was decided to combine the definitions of robot actionsand screen generation due to following reasons.

Usually, in robotic applications, the GUI is not includedin the robot behaviour design. Instead, the focus is onrobot behaviour such as path planning, navigation, and otheractions. However, in service robot applications, the GUI is adominant part of the robot behavior, since the user experiencehighly depends on the audio visual output and the interactionwith the robot. Therefore, to have effective human-robotinteraction, robot actions, user inputs, as well as changesin GUI, should be synchronized [17].

B. GUI generator

GUI generator is a ROS component implemented usingQT-ROS package [18]. IT uses the widely used QT frame-work for GUI generation.

In RobotChair GUI there are no fixed components. Themain controller determines what should be displayed on theGUI depending on the current state of the main controller andsends a message to GUI generator. This is an XML formattedmessage that completely describes the screen layout andcomponents. Upon receiving this, the GUI generator dynami-cally generates the screen layout required by the current stateof the robot.

C. Distributed components

There are several distributed ROS components to imple-ment various robot functions. Following are the componentsin the current implementation:

• Navigation: This components receives fuzzy voice con-trolled commands described below and converts thesecommands into motor control commands. Then it sendsmessages to the micro controller responsible for sendingPWM signals to motor drives.

• Weight measurement: This components receives com-mands for acquiring weight. IT communicates with themicro controller responsible for reading the load-celloutput. It returns the weight as a ROS message.

• Blood pressure and pulse rate measurement: This com-ponents received commands for acquiring blood pres-sure of a user and then send control signals to a bloodpressure meter. Blood pressure measurement consistsof several steps and this component is responsible formonitoring and controlling all steps and finally takingthe blood pressure measurement. It returns the systolicblood pressure, diastolic blood pressure, and pulse rateas a single ROS message.

• Blood oxygen saturation measurement: Similar to bloodpressure measurement, this module controls a bloodoxygen meter and returns the reading.

• People detection: This component detects the numberof people in front of the robot as a ROS message. Thisinformation is used by the main controller to generateengaging dialogs.

• Text-to-speech: This components receives text to bespoken and then generates voice using Festival.

Fig. 7 is a run-time view of main software componentsof the system, which was generated using RxGraph ROSutility..

D. Fuzzy voice controlled navigation

Fuzzy voice controlled navigation is a key feature ofRoboChair. For voice controlled navigation, a fuzzy coach-player approach is used [20], [21], [22], [23], [24] is used. Inthis approach, in a voice command there are two parts; actionand action modification. For the whee chair, go forward, gobackward, turn right, turn left and stop are used as actionsand very slow, slow, fast, and very fast are used as actionmodifications.

E. GUI Generator

GUI generator is a ROS component implemented us-ing QT-ROS package [18]. IT uses the widely used QTframework for GUI generation. More details about the GUIgenerator is given in Section V.

V. DYNAMIC USER INTERFACE GENERATION

In RobotChair GUI there are no fixed components. Themain controller determines what should be displayed on theGUI depending on the current state of the main controller andsends a message to GUI generator. This is an XML formattedmessage that completely describes the screen layout andcomponents. Upon receiving this, the GUI generator dynami-cally generates the screen layout required by the current stateof the robot.

As shown in Fig. 3(b), all software components are con-nected by ROS middleware. The complete robot behaviour iscontrolled by the main controller, by sending and receivingmessages. GUI is yet another component controlled by themain controller, by sending messages.

As explained in Section IV-A, the main controller is anFSM developed using SMACH. Some states of the statemachine are associated with GUI. In such states, the maincontroller sends messages to GUI and depending on themessages received, GUI can dynamically generate interfaces,which consists of text boxes, buttons, images, video clips,and sound clips.

Figure 5 shows a sample message sent to GUI generator.This message defines the type, size, location, and proper-ties of components as well as event associated with somecomponents such as buttons.

The complete FSM of the robot consists of a large numberof states. Figure 4 shows an example work flow. This showsstate transitions after receiving a scheduled event. Dottedlines show screens associated with some states.

575

Kinect sensor

Touch screen

Vital signs

measuring

devices

Main controller

Body weight

sensor

devices

Laser range

finder

(a) RoboChair

Controller (SMACH FSM)

Interface layout

definition

• Components

• Layout

Back-end Actions

• Speech

• Motion

• Measurements

TTS BP

Navigation

GUI Generator (QT FSM)

Interface layout

templates

Front-end Actions

• Photo viewer

• Movie viewer

• Skype

• Facebook

RO

S

ROS

Distributed components

SPO2

People

detection

Weight

Speakers BP

Motor

controller

Robot Hardware

SPO2

Kinect

Load cell

(b) Software architecture

Fig. 3. RoboChair and software architecture

Wait

Alert

Lunch 1

Lunch 2

Wait

clickYes

clickNo

clickOk

Scheduled event received

(evt:lunchTime)

clickNo

It is you lunch time

enter

menu

exit

menu

enter

navigate

exit

navigate

select

menu

send

menu

sendMenu(menu)

sendText(msg, caregiver)

clickNo

Fig. 4. A sample work flow - receiving a scheduled event

VI. DISCUSSION

The first version of the RoboChair is currently being testedin the laboratory. A typical user interaction session is shown

in Fig. VI. Although the development of RoboChair has beencompleted, so far a user trial has not been done. RoboChairresearch group is currently working on this. Some of theother current works are as follows:

576

<components>

<textfield x="50" y="50" width="924" textsize="50" background="1" border="1">

Please click on PLAY if you want to watch the following movie clip.

</textfield>

<movie>

...

</movied>

<button label="PLAY" width="80" height="30" x="920" y="10" textsize="14" icon="play.jpg">

<event name="clicked">

<action>

…

</action>

</event>

</button>

</components>

Fig. 5. A sample message sent to GUI generator

Fig. 7. Main software components (ROS nodes)

Events:

1.Face detected

2.Screen touched

Manual

authentication

FR success

Face

recognition

Frequent

user

N

Y

User profile

DB

Main menu

Navigation CallingBody

weightBP SPO2

Bio

impedance

Use

r se

lect

ed

task

sIn

itia

l in

tera

ctio

n

Wa

itin

g f

or

a u

ser

Self

introduction

Fuzzy voice

commands

Key board

controlVoice controlled

navigation

Event:

Session end

FR not success

Crisp voice

commands

Fig. 6. Experimental scenario

• Integration with social networks: Being a companionrobot, RoboChair can help users with disabilities tobe engaged with the society through social networks.Although the users may not be able to use socialnetworks directly due to physical limitations, RoboChaircan help them to connect to social networks and usetheir features.

• Video conferencing: This is an essential feature, spe-cially for lonely people in rest homes. The video confer-encing feature can be used to communicate with familymembers, friends and with health care professionals.

• Health monitoring and risk prediction: This is one of thekey application areas of RoboChair, once completed. Bymeasuring some vital clinical parameters, it is possibleto predict things such cardiovascular risk. Being acompanion a robot, RoboChair can be used to providerisk information to users, educate them, and help themto reduce the risk through education.

• Health educational content: As a companion robot,providing educational information such as medicationinformation will enhance the usefulness of the robot.[11]

• Integration with sensor networks (intelligent environ-ment): By integrating with sensor networks, it is possi-ble to extend the usefulness of RoboChair by providingservices such as falls monitoring [15].

• HL7 compliance: HL7 is an application layer protocolused by health information systems. HL7 compliance isan essential feature of RoboChair since it is necessary tocommunicate with existing health information systems.

The above features have been given priority in the currentdevelopment work of RoboChair.

REFERENCES

[1] L. Boccanfuso and J. M. O’Kane, “Charlie : An adaptive robot designwith hand and face tracking for use in autism therapy,” InternationalJournal of Social Robotics, vol. 3, pp. 337–347, 2011.

[2] Y. Yamaji, T. Miyake, Y. Yoshiike, P. R. S. Silva, and M. Okada, “Stb:Child-dependent sociable trash box,” International Journal of SocialRobotics, vol. 3, pp. 359–370, 2011.

577

(a) Initial interaction 1. (b) Initial interaction 1.

(c) Initial interaction 1. (d) Initial interaction 1.

(e) Initial interaction 1. (f) Initial interaction 1.

Fig. 8. HealthBot robot and test scenarios.

[3] W. Lutz, W. Sanderson, and S. Scherbov, “The coming acceleration ofglobal population ageing,” Nature, vol. 451, no. 7179, pp. 716–719,2008, 10.1038/nature06516.

[4] Establishing and Monitoring Benchmarks for Human Resources forHealth: the Workforce Density Approach, ser. Spotlight on Statistics,no. 6, World Health Organization, Department of Human Resourcesfor Health, WHO, 2008.

[5] Y. Hasegawa, J. Jang, and Y. Sankai, “Cooperative walk control ofparaplegia patient and assistive system,” in Proc. IEEE/RSJ Interna-tional Conference on Intelligent Robots and Systems, Oct. 2009, pp.4481–4486.

[6] J. Kofinan, X. Wu, T. Luu, and S. Verma, “Teleoperation of arobot manipulator using a vision-based human-robot interface,” IEEE

Transactions on Industrial Electronics, vol. 52, no. 5, pp. 1206–1219,Oct. 2005.

[7] K. Kiguchi, T. Tanaka, and T. Fukuda, “Neuro-fuzzy control of arobotic exoskeleton with emg signals,” IEEE Transactions on FuzzySystems, vol. 12, no. 4, pp. 481–490, Aug. 2004.

[8] W. Shin and D. Kwon, “Surgical robot system for single-port surgerywith novel joint mechanism,” IEEE Transactions on Biomedical En-gineering, vol. —, no. —, pp. —.

[9] H. Krebs, B. Volpe, D. Williams, J. Celestino, S. Charles, D. Lynch,and N. Hogan, “Robot-aided neurorehabilitation: A robot for wrist re-habilitation,” IEEE Transactions on Neural Systems and RehabilitationEngineering, vol. 15, no. 3, pp. 327–335, 2007.

[10] A. Di Nuovo, D. Marocco, A. Cangelosi, V. De La Cruz, andS. Di Nuovo, “Mental practice and verbal instructions execution:A cognitive robotics study,” in Proc. The 2012 International JointConference on Neural Networks (IJCNN), 2012, pp. 1–6.

[11] P. Tiwari, J. Warren, K. Day, B. MacDonald, C. Jayawardena, I. H.Kuo, A. Igic, and D. C., “Feasibility study of a robotic medicationassistant for the elderly,” in Proc. Australasian Conf. User Interface(AUIC), 2011, pp. –.

[12] I. Kuo, C. Jayawardena, P. Tiwari, E. Broadbent, and B. MacDonald,“User identification for healthcare service robots: Multidisciplinarydesign for implementation of interactive services,” in Social Robotics,ser. Lecture Notes in Computer Science, S. Ge, H. Li, J.-J. Cabibihan,and Y. Tan, Eds. Springer Berlin Heidelberg, 2010, vol. 6414, pp.20–29.

[13] I.-H. Kuo, C. Jayawardena, E. Broadbent, and B. MacDonald,“Multidisciplinary design approach for implementation of interactiveservices,” International Journal of Social Robotics, vol. 3, no. 4, pp.443–456, 2011. [Online]. Available: http://dx.doi.org/10.1007/s12369-011-0115-x

[14] A. Tapus, M. Mataric, and B. Scasselati, “Socially assistive robotics[grand challenges of robotics],” IEEE Robotics Automation Magazine,vol. 14, no. 1, pp. 35–42, 2007.

[15] C. Jayawardena, I. H. Kuo, U. Unger, A. Igic, R. Wong, C. Watson,R. Stafford, E. Broadbent, P. Tiwari, J. Warren, J. Sohn, and B. Mac-Donald, “Deployment of a service robot to help older people,” in Proc.IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS), Oct. 2010, pp. 5990–5995.

[16] R. Stafford, E. Broadbent, C. Jayawardena, U. Unger, I. H. Kuo,A. Igic, R. Wong, N. Kerse, C. Watson, and B. MacDonald, “Improvedrobot attitudes and emotions at a retirement home after meeting arobot,” in Proc. IEEE RO-MAN, 2010, pp. 82–87.

[17] C. Jayawardena, I. Kuo, C. Datta, R. Stafford, E. Broadbent, andB. MacDonald, “Design, implementation and field tests of a sociallyassistive robot for the elderly: Healthbot version 2,” in Proc. 4thIEEE RAS EMBS International Conference on Biomedical Roboticsand Biomechatronics (BioRob), 2012, pp. 1837–1842.

[18] M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs,R. Wheeler, and A. Y. Ng, “Ros: an open-source robot operatingsystem,” ICRA workshop on open source software, vol. 3, no. 3.2,2009.

[19] A. W. Black, P. Taylor, and R. Caley. (2013, Jul.)The festival speech synthesis system. [Online]. Available:http://www.cstr.ed.ac.uk/projects/festival

[20] C. Jayawardena, K. Watanabe, and K. Izumi, “Posture control of robotmanipulators with fuzzy voice commands using a fuzzy coachplayersystem,” Advanced Robotics, vol. 21, pp. 293–328, 2007.

[21] K. Watanabe, C. Jayawardena, and K. Izumi, “Intelligent interfaceusing natural voice and vision for supporting the acquisition of robotbehaviors,” in Proc. 5th IEEE Conference on Sensors, Oct. 2006, pp.374–377.

[22] C. Jayawardena, K. Watanabe, and K. Izumi, “Controlling a robotmanipulator with fuzzy voice commands using a probabilistic neuralnetwork,” Neural Computing Applications, vol. 16, no. 2, pp. 155–166,2007.

[23] ——, “Probabilistic neural network based learning from fuzzy voicecommands for controlling a robot,” in Proc. of International Confer-ence on Control, Automation, and Systems (ICCAS), 2004, pp. 2011–2016.

[24] ——, “Knowledge acquisition by a sub-coach in a coach player systemfor controlling a robot,” in Proc. of the 4th International Conferenceon Advanced Mechatronics, Hokkaido, Japan, 2004, pp. 601–606.

578