Agricultural Robot Sprayer and Evaluation of User Interfaces in Field

Experiments

George AdamidesSenior Agricultural Research OfficerAgricultural Research Institute

Presentation overview

Technical characteristics of the Agricultural Robot Sprayer

User Interfaces for HRI Design & implementation

Field Experiment design & implementation

Findings

Conclusion

Summit XL by Robotnik

Technical characteristics of the Agricultural Robot Sprayer

http://www.robotnik.es/en/products/mobile-robots/summit-xl

Summit XL technical characteristics

• The SUMMIT XL is a medium-sized high mobility all-terrain robot with extreme performance.

• The Summit XL has skid-steering kinematics based on 4 high power motor wheels

– Dimensions 693x626x417 mm– Weight 30 Kg– Load capacity 20 Kg– Speed 3 m/s– Traction system 4 wheels– Batteries 8x3.2V LiFePO4 (~5 hours autonomy)

• 45 minutes for full charge– Temperature range 0oC +50oC – Max climbing angle 45o

– Controller ROS embedded PC with Linux Real Time– Communications WiFi 802.11n– Connectivity Internal: USB, RS232, GPIO y RJ45– External: USB and power supply 12 VDC

Technical characteristics of the Agricultural Robot Sprayer

Main parts of Summit XL robot

Building the Robot Sprayer

Technical characteristics of the Agricultural Robot Sprayer

September 2012

The Sprayer• Serena electric sprayer

Weight (with full tank) 13.6 kgMeasurements: 315x145x400Water flow rate with a fan nozzle 26 liters/hBattery life 11.30 hFull cycle charging 10hCapacity 10 liters

Technical characteristics of the Agricultural Robot Sprayer

Sprayer installation• Modbus IO• Sprayer case• ROS module programming

<?xml version="1.0"?> <launch> <!-- start modbus_io node --> <param name="modbus_io_node/ip_address" value="192.168.2.185" /> <param name="digital_outputs" value="8"/> <param name="digital_inputs" value="8"/> <param name="analog_outputs" value="2"/> <param name="analog_inputs" value="2"/> <node pkg="modbus_io" type="modbus_io_node" name="modbus_io_node" output="scr een"/> </launch>

ssh summit@192.168.0.200 summit> ping 192.168.0.185 3. Launch the modbus_io module roslaunch modbus_io test_io.launch 4. Test the digital outputs rosservice call /modbus_io/write_digital_output 0 false rosservice call /modbus_io/write_digital_output 0 true rosservice call /modbus_io/write_digital_output 1 false rosservice call /modbus_io/write_digital_output 1 true ... Outputs 5,6,7 and 8 are RELAYS

5. Once it is working, modify the summit_xl_complete launch file in order to launch the IO module every time you start the robot and roscd summit_xl_complete edit the launch/summit_xl_complete.launch file and add the following line: <!-- start modbus_io node --> <include file="$(find modbus_io)/launch/test_io.launch"/>

Sprayer case and installation by AgroWise

Building the Robot Sprayer – modified version

December 2012

Technical characteristics of the Agricultural Robot Sprayer

Initial trial-out findings

• Issues with the camera – Viewing angle– Drops on dome cover

• Issues with PC screen– Lighting/shading/reflection

• Issues with wireless connection

– Distance

• Issues with Bluetooth connection

– PS3 (distance)

Technical characteristics of the Agricultural Robot Sprayer

Improving the Agrirobot

• Hardware – Installation of two USB cameras to improve

Peripheral vision and End-effector vision– Bigger wifi antenna– Moved higher the spraying nozzle

• Software – Installation and configuration of the

mjpeg_server ROS module and the Apache webserver

– Installation and programming the pr2_keyboard_teleop ROS module for the Summit XL navigation and the sprayer on/off state

Technical characteristics of the Agricultural Robot Sprayer

Peripheral Visual aid

Technical characteristics of the Agricultural Robot Sprayer

End-effector visual aid

Technical characteristics of the Agricultural Robot Sprayer

WiFi Antenna

Building the Robot Sprayer – current version

Technical characteristics of the Agricultural Robot Sprayer

May 2013

Technical issues and troubleshooting

Short-movie

Technical characteristics of the Agricultural Robot Sprayer

Presentation overview

Technical characteristics of the Agricultural Robot Sprayer

User Interfaces for HRI - Design and Implementation

Field Experiments Design & Implementation

Findings

Conclusion

Reality Based Interaction (RBI) styles [1]

New interaction styles that draw strength by building on users’ pre-existing knowledge of the everyday, non-digital world to a much greater extent than before.Examples of RBI: VR, AR, TUI, ubiquitous and pervasive computing, handheld or mobile interaction…

[1] Jacob, Robert JK, et al. "Reality-based interaction: a framework for post-WIMP interfaces." Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 2008.

User Interfaces for HRI – Design and Implementation

User Interface for Robot Teleoperation – Development Phases

User Interfaces for HRI – Design and Implementation

Spraying

Designing for HRI Awareness

HRI Awareness [2]

Given one human and one robot working on a task together, HRI awareness is the understanding that the human has of the location, activities, status, and surroundings of the robot; and the knowledge that the robot has of the human’s commands necessary to direct its activities and the constraints under which it must operate.

LASSO technique [3]

• Location Awareness• Activity Awareness• Status Awareness• Surroundings

Awareness• Overall Mission

Awareness[3] Jill L. Drury, Holly A. Yanco & Keyes, B 2007, 'LASSOing HRI: analyzing situation awareness in map-centric and video-centric interfaces', Proceedings of the ACM/IEEE international conference on Human-robot interaction.

[2] Scholtz, J.; Young, J.; Drury, J.L.; Yanco, H.A., "Evaluation of human-robot interaction awareness in search and rescue," Robotics and Automation, 2004. Proceedings. ICRA '04. 2004 IEEE International Conference on, vol.3, no., pp.2327,2332 Vol.3, 26 April-1 May 2004

Robot teleoperation through Human-Robot user interfaces

Mental model

User Interfaces for HRI – Design and Implementation

Phase 1. Using ROS command line

~> roslaunch usb_cam low_res.launch

User Interfaces for HRI – Design and Implementation

~> export ROS_MASTER_URI=http://V3:11311~> rostopic list~> rosrun image_view image_view image:=/logitech_usb_webcam/image_raw

Robot PC Remote PCssh summit@V3

• Using ROS command line to display Peripheral and End-Effector cameras

• Web interface of Axis Ethernet camera

ROS environment with three cameras

User Interfaces for HRI – Design and Implementation

• Installed the mjpeg_server module• Installed the Apache web server

The mjpeg_server is a streaming server that subscribes to requested image topics in ROS and publishes those topics as MJPEG streams via HTTP

Preparing for phase 2

User Interfaces for HRI – Design and Implementation

Phase 2. First attempt for Web UI in HTML

User Interfaces for HRI – Design and Implementation

Improved version of the Web UIin PHP

UI Design and Implementation by Istognosis

UI for drivingMain camera & Peripheral camera

UI for spraying (rejected)Main camera & spraying camera

UI for spraying Main camera & ssupport cameras

Phase 4. Using a patriot wireless tracker & digital glasses

User Interface Design and Implementation by Istognosis

Presentation overview

Technical characteristics of the Agricultural Robot Sprayer

User Interfaces for HRI – Design & Implementation

Field Experiments Design & implementation

Findings

Conclusion

Field experiments Design & Implementation

Setting up the stage

Field experiments Design & Implementation

Preparing the robot

Field experiments Design & Implementation

Path and obstacles

Field experiments Design & Implementation

Grape clusters (targets)

Field experiments Design & Implementation

Experiment design & implementation

1 PC Screen + PS3 + Main Camera Only

2 PC Screen + PS3 + Main & Support Cameras

3 PC Screen + Keyboard + Main Camera Only

4 PC Screen + Keyboard +Main & Support Camera

5 AR Glasses + PS3 + Main Camera Only

6 AR Glasses + PS3 + Main & Support Cameras

7 AR Glasses + Keyboard + Main Camera Only

8 AR Glasses + Keyboard + Main & Support Cameras

Field experiments Design & Implementation

Tasks were randomized and were conducted in two day visits. Four tasks were carried out on day 1 and the remaining tasks on day 2 (not consecutive, period between experiments varied 2 to 10 days)

25 participants:AgronomistsAgricultural TechniciansAgricultural Laborers

Experiment procedures• Consent form• Pre-Questionnaire

– Demographics, Immersion Tendency Questionnaire, General Self-Efficacy Scale, Santa Barbara Sense of Direction Scale, CEW Fluency Scale

• Briefing and getting familiar with the UIs• Post-Questionnaire after each run

– SUS, Presence, NASA TLX– Metrics (collisions, path divergence, targets sprayed,

targets missed, percent completed, duration)

• Experiment duration ~3 hours per participant to complete 8 tasks

Field experiments Design & Implementation

The Agrirobot in the field

Photo Album

Field experiments Design & Implementation

Spraying – extended nozzle antenna

The Agrirobot UI findings

Screenshots

FindingsMain camera VS Main Camera and Support Cameras

Identifying obstacles

Identifying obstacles

Peripheral visionIdentifying obstacles

Shading Issues (left wheel)

Spraying

Stopped Spraying

Shading Issues (left wheel)

Shading issuesIdentifying obstacles

Shading Issues (wheels)

Identifying red grape clusters

Identifying green grape clusters

The Agrirobot VS obstacles in the field

Image album

Keep going…

Presentation overview

Technical characteristics of the Agricultural Robot Sprayer

User Interfaces for HRI – Design & Implementation

Field Experiments Design & Implementation

Findings

Conclusion

Preliminary results

NASA Task Load Index per UI

Findings

User Interfaceswith PS3 controller

User Interfacesusing keyboard controller

Preliminary results

Effectiveness: Number of grape clusters sprayed per UI

Findings

User Interfaceswith main andSupport (3) cameras

User Interfaceswith main (1) camera

Preliminary findingsMean number of collisions per UI

UI # Cameras N Minimum Maximum Mean Std. Deviation

UI1 - 1 Camera

25 0 3 ,76 ,779

UI2 - 3 Cameras

25 0 4 ,60 ,913

UI3 - 1 Camera

25 0 7 1,36 1,777

UI4 - 3 Cameras

25 0 2 ,56 ,861

UI5 - 1 Camera

25 0 5 1,28 1,242

UI6 - 3 Cameras

25 0 4 ,84 ,987

UI7 - 1 Camera

25 0 4 ,80 1,118

UI8 - 3 Cameras

25 0 3 ,28 ,678

Presentation overview

Technical characteristics of the Agricultural Robot Sprayer

User Interface Design

Field Experiment design & implementation

Findings

Conclusion

In Summary – Problems faced and overcome

• Transformation of a “off-the-shelf” robot into a robotic sprayer

• Pilot trials revealed issues with WiFi, Bluetooth, camera view points

• A lot of –smaller or bigger- practical, “non-research” problems, turned into valuable experience for the future

In Summary - what we did• Designed and implemented several

user interfaces• Used WIMP and RBI interaction styles• Field experiments

Conclusions• Yes, it is feasible! (Agri Robot tele-operation )• The user interface design does make a

difference• There are many small, practical issues to

resolve– Agricultural task are demanding and take place in a

difficult environment– Many issues to overcome

• PS3 Bluetooth, WiFi, monitor shading/light, web cameras• Robot wheels, sprayer hose• ROS module programming

• Promising findings

Future work• Incorporation of sensor information in the

UI to include ultrasound sensor information, battery-life (under development)

• Robot improvements – Servos for extending sprayer antenna and

USB/Ethernet cameras control/rotation – Sprayer antenna with multiple nozzles

• Learnability issues need further study• Long hours? • Cost-benefit analysis

Spraying short movie

Thank you for your attention!

Discussion / Coffee time