Heuristic usability evaluation of user interfaces for a semi-autonomous vineyard robot sprayerGeorge AdamidesOpen University of Cyprus

What is and why do we care about usability in a human-robot interaction system?

• A user interface that supports human-robot interaction (HRI) needs to meet specific non-functional requirements, such as reliability, efficiency and usability.

• Usability refers to whether a system can be used with effectiveness, efficiency, and satisfaction with which users achieve specified goals in a particular context of use.

• A usability issue is anything that can affect the user experience in a negative way.

• What makes a robotic interface effective is no different than what makes anything else usable, be it a door handle or a piece of software.

• “Cut” and “Paste” or “Undo” does not always work!

HRI usability evaluation• Heuristic evaluation is a ‘discount usability engineering’

method for evaluating user interfaces to find their usability problems. • “discount” because a small number of evaluators, usually

3 to 7, suffice to reliably evaluate the usability of a user interface against a list of heuristics (the usability principles).

Which usability principles we used? Adamides et al (2015)

1. Platform architecture and scalability: “Provide the flexibility to iterate robotic and computing technological developments in the UI of the HRI system.”

2. Error prevention and recovery: “Provide information and alerts to avoid and recover from user errors.”

3. Visual design: “Provide an aesthetic, clear, and simple design of the UI with the relevant information necessary.”

4. Information presentation: “Provide the necessary information, in the right context, moment, and modality.”

5. Robot state awareness: “The knowledge that the robot has about its own systems’ situation and the information it gives to the operator about its health status and mode of operation.”

6. Interaction effectiveness and efficiency: “Provide efficient and effective interactions between human and robot.”

7. Robot environment/surroundings awareness: “Provide spatial information about the robot’s surroundings and the environment where it is operating.”

8. Cognitive factors: “Use mental models and metaphors to lower the cognitive load.”

Source: G. Adamides, G. Christou, C. Katsanos, M. Xenos, and T. Hadzilacos, "Usability Guidelines for the Design of Robot Teleoperation: A Taxonomy," IEEE Transactions on Human-Machine Systems, vol. 45, pp. 256-262, 2015.

Which usability principles we used? Adamides et al (2015)

Semi-Autonomous Vineyard Robot Sprayer

Sprayer nozzle

End-effector cameraPeripheral camera

Sprayer tank

Main pan-tilt zoom camera

Summit XL robot platform

3 systems under evaluation

SAARS V0

• On-screen controls for robot movement and camera movement• Presentation of camera views• Elements for displaying sensor information (visual and auditory feedback) for

distance from the robot sides and battery level. • Can use the entire screen and support interaction though either the keyboard

or the mouse.

3 systems under evaluation

SAARS V1• Similar to V0• Plus functionality for target pointing.• SAARSv1 supports both manual (user points to targets) and automated target

specification through a pattern recognition algorithm.

3 systems under evaluation

SAARS V2

• Similar to V1• Added feedback from a laser scanner.

HRI heuristic evaluation• Four usability experts conducted a heuristic usability

evaluation on three user interfaces.• The evaluators were situated at the Hellenic Open

University Software Quality Assessment laboratory at Patra, Greece and remotely controlled (over HTTP) the robot, which was located in Cyprus at the Open University of Cyprus premises. An appropriate simulation environment was created, including various paths and targets.

HRI heuristic evaluation results – SAARS V0• 13 usability issues were identified. • Most (77%) of these usability issues were related to

violations of the following four heuristics: a) 23% were violations of heuristic 4 (Information presentation), b) 23% were violations of heuristic 5 (Robot state awareness), c) 15% were violations of heuristic 6 (Interaction effectiveness and efficiency) and d) 15% were violations of heuristic 8 (Cognitive factors). • All in all, the system is at a satisfactory level of usability.

HRI heuristic evaluation results – SAARS V1• 10 usability issues were identified. • Most (80%) of these usability issues were related to

violations of the following four heuristics: a) 20% were violations of heuristic 4 (Information presentation), b) 20% were violations of heuristic 5 (Robot state awareness), c) 20% were violations of heuristic 6 (Interaction effectiveness and efficiency) and d) 20% were violations of heuristic 8 (Cognitive factors). • These findings tend to provide support that the system is

at a good level of usability.

HRI heuristic evaluation results – SAARS V2• 3 usability issues were identified. • These issues were related to violations of the following

three heuristics: a) one violation of heuristic 3 (Visual design), b) one violation of heuristic 6 (Interaction effectiveness and efficiency), and c) one violation of heuristic 7 (Robot environment/surroundings awareness). • All in all, the system is at a very good level of usability.

Conclusion – Recommendations • These findings provide evidence that the final version of the system provides

satisfactory services to its typical users. • This can be attributed to the iterative design, development and evaluation

process followed. • These advantages, combined with the increased usability of the SAARv2 (final)

system, may result in high adoption from its end users.• However, there is always room for improvement. The expert evaluators argued

that a next version of the system could benefit from: • a) an embedded representation of the robot’s body in the user interface

displaying sensors’ information and robot’s direction in relation to the active camera views (heuristic 7),

• b) embedded help explaining functionality and controls (heuristic 8), e.g. simplify and explain algorithmic settings for automated target identification, embed tooltips and/or labels on the buttons related to user-defined targets,

Conclusion – Recommendations • c) mechanisms for error prevention in target identification and

spraying (heuristic 2), e.g. confirmation message for the “erase-all-targets” action,

• d) additional information that is important for the task (heuristic 4) e.g. remaining level of spraying liquid,

• e) improvements in the visual design of the user interface (heuristic 3), e.g. visual clarification for currently active control, larger text labels to increase readability.

Latest news: A new robot with added functionality is currently being built with a robotic arm. New user interface is also designed based on the above recommendations.

THANK YOU FOR YOUR ATTENTION!

George AdamidesOpen University of Cyprus – http://www.ouc.ac.cy Email: george.adamides@st.ouc.ac.cyAgricultural Research Institute – http://www.ari.gov.cy Email: gadamides@ari.gov.cy