Exploring Alternative

Control Modalities for

Unmanned Aerial VehiclesThesis Presentation

David Qorashi

Grand Valley State University

April 2015

Committee Members

Dr. Engelsma, Dr. Alsabbagh, Dr. Dulimarta

Unmanned Aerial Vehicles

(UAVs)• Primary Usage

• Defense & Security Sectors

• Potential Usage

• Private Security

• News Gathering (CNN)

• Agricultural Practices

• Product Transport

• Aerial Advertising

• … Source: New York Daily News (2014)

–Business Insider (2014)

“We predict that 12% of an estimated $98 billion

in cumulative global spending on aerial drones

over the next decade will be for commercial

purposes.”

ChallengeHard to fly for inexperienced users

Problems with current

approaches

• Steep Learning Curve

• Failing to pilot the drone in initial attempts

• Damage to drone/environment

• Comprise safety of pilot and/or bystanders

Drone flies

away

Endangers People’s Safety

Roman Pirozek was killed while flying a

remote-controlled model helicopter on

September 2013 (Source: Wall Street

Journal).

Purpose of this study

Alternative Control

Mechanisms

Existing Literature

Human robot interaction: A

new field in HCI• Goodtich & Olsen defined a general Interface on designing effective UIs

• Dury et.al defined a set of HRI taxonomies

• Quigley et al. used a numeric parameter-based interface for controlling

• Quigley et al. used voice control interaction

• Natural User Interfaces

• Gesture-based Interaction for collocated ground robots (Rogolla et al.

2002)

• Gesture-based Interaction for collocated Drones (Ng et al. 2011)

Most gesture-based

approaches use drone’s built-in

camera for image recognition

Source: Collocated Interaction with Flying Robots (Ng et al. 2011)

Microsoft

Kinect

Source: Microsoft Corp. (2014)

New Approach: Using motion sensing input devices

A Kinect-Base Natural Interface for Quadrotor Control (Sanna et al. 2012)

Challenges using Body Parts

Movements

• Not very comforting

• Not very receptive by the pilots

• Very hard to do in a long run

Proposed Solution?

Hand Gestures

The purpose of study

Develop an improved & more intuitive method of

piloting

Thesis Hypothesis

• Increasing efficiency in piloting

• Adding overall enjoyable experience in piloting

Quadcopters

• Also known as quadrotor

• 4 rotors

• Fixed pitch

• Controlling happens via Wi-Fi

Bebop

AR-DroneAn affordable Quadcopter

• Rich Programming API

• Affordable Price

Leap Motion ControllerSource: Forbes Magazine (2014)

Core Experiment

Implemented gesture-based interface vs. conventional

multi-touch methods

Comparative Analysis

• Sample: A subset of students

• First, they were asked to complete a very simple mission

in an specific route using multi-touch approach

• Second, we ask them to pilot the drone in the same

route using the implemented approach

• Finally, we ask them to fill a questionnaire to rate each

approach.

• Also, the trainer examined the accuracy of landings after

each flight

Defined Mission

Implementation

What needs to be done?

• Controller for sending the commands to the drone

• Gesture Recognition System

• Interfacing Gesture system with Drone Controller

Drone Controller

• Bebop Drone released in December 2014

• Documentations for the new API: Extremely weak

• A high-level wrapper was created around C API

• https://github.com/gvsucis/c-bebop-drone

Implemented Functions• ARDrone3SendSettingsAllSettings

• ARDrone3SendCommonAllStates

• ARDrone3SendTakeoffCommand

• ARDrone3SendLandCommenad

• ARDrone3SendPCMD

• ARDrone3SendSendSpeedSettingsHullProtection

• ARDrone3SendPilotingFlatTrim

• ARDrone3SendYawRightCommand

• ARDrone3SendYawLeftCommand

• ARDrone3SendAscendCommand

• ARDrone3SendDescendCommand

• ARDrone3SendHoverCommand

• ARDrone3SendMoveForwardCommand

• ARDrone3SendMoveBackwardCommand

Using the implemented

wrapper we were able to send

commands to the Bebop

Second component of

implementation …

Gesture Recognition System

• Q: What method of implementation would be better?

(better here, means general and accurate)

• A: Machine Learning

ML Applications

• Classification

• Grouping

• Ranking

Live Preview Demo

Gesture Recognizer

• Method in use: Supervised Machine Learning

• Steps:

• Sample Collection And Tagging

• Machine Leaning Recognizer Code

Usual Development

VS.

ML Development

Usual DevelopmentWorkflow

Recognizer DevelopmentWorkflow

Every ML algorithm needs

some features

• Features are extracted from the data

• Features in use for this application:

• Tips’ positions

• Center of Palm position

• Velocities

Leap Motion API provided

us with positions of tips,

palm and velocities

Training Algorithms

With huge set of data all these methods converge

Domingos, Pedro. "A few useful things to know about machine

learning." Communications of the ACM 55.10 (2012): 78-87.

For this App, we chose $P

• Why?

• We didn’t have lots of training data

• $P is optimized for gesture recognition

Cloud Recognizer: a 2-D gesture recognizer designed for rapid prototyping of gesture

As the last step we connected

gesture recognizer to drone

controller

Voila!

Experiment

Asked subjects to try out

both approaches

Mission

Data collected

Results

Surprise? Does it mean

multi-touch method is

better?

Why?

• Gesture controller involves nondeterministic factors

• Quality of the sensor

• Quality of data gathered during training phase

• Lighting conditions of the environment during

the test

• Suitableness of algorithm used for training

As expected, majority of

people thought that GESTURE

method is more intuitive

Other challenging

questions

Raised Questions During

Research??

• How should we handle combinatorial commands?

• How can we make GR 100% accurate and more

robust?

• How can we define more intuitive gestures?

Future work?

• Use another kind of sensor for gathering data

• Run image processing algorithms instead of relying

on the API provided by Leap

• Make a huge data set instead of using just two

people’s hand gestures

• Considering multimodal controls

Thank you for your time

Questions?