Design of Hands-Free Interface for Object...
Transcript of Design of Hands-Free Interface for Object...
![Page 1: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/1.jpg)
Design of Hands-Free Interface for Object Relocation
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
GDMS Sr Engineer Mike DeMichele
User Robotic Aid Proposed design eliminates need for operator to perform
physical motion
![Page 2: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/2.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 2
AGENDA
• Motivation
• Problem Statement and Stakeholders
• Technology Introduction
• Gap Analysis
• CONOPS
• Simulation
• Trade-off Analysis
• Business Case
2
![Page 3: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/3.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 3
PHYSICALLY DISABLED STATISTICS
3
• 5,596,000 people in the US, reported
some form of paralysis [1]
• 2,900,000 severely paralyzed persons in
America [1]
• Physical movement required by control
interfaces.
• Limiting factor in the system usability
and user workload.
[1] 2012 study performed by the Christopher and Donna Reeve Foundation
![Page 4: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/4.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 4
COMPLEXITY OF HUMAN MACHINE INTERACTION
4
Challenges of using human machine
interaction in a practical setting
• Systems requiring a huge amount
of user input and focus
• Time sensitive systems
• Systems requiring user to perform
small, precise movements
• Systems requiring frequent
operator turnover
[3] Robot FRIEND, Institute of Automation (IAT)
of University of Bremen
• Process driven by input from human user
• Relies on interface to continue data exchange
![Page 5: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/5.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 5
AGENDA
• Motivation
• Problem Statement and Stakeholders
• Technology Introduction
• Gap Analysis
• CONOPS
• Simulation
• Trade-off Analysis
• Business Case
5
![Page 6: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/6.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 6
PROBLEM STATEMENT
6
The 2.9 million severely paralyzed individuals in the US require a technology
enable them to relocate objects without requiring motion. This design has the
potential to assist users in other daily activities that ordinarily rely on physical
input, such as navigating a robot to perform fetch and deliver tasks.
![Page 7: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/7.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 7
STAKEHOLDER ANALYSIS
Stakeholder Significance Tensions
Control Technology
Companies
- Lead on current development work for
alternatives
- Potential to modify control technologies to
best fit need
- Competition between companies
- Tension with government approvers
Severely Paralyzed - Benefit from Hands-Free Control System - Learning time
- May initially distrust system
Insurance Companies - Create new policies to include HFCS - Cost of insurance
- Additional costs to insurance
companies
HHS and FDA - Under Medical Device Act, aid would require
approval
- Tension if design is not accepted
In-home Care
Organizations
- Leading care for severely paralyzed
- Potential to integrate new robotic care device
- Concerns of being replaced/ losing
market dominance
![Page 8: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/8.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 8
STAKEHOLDER INTERACTIONS
8
![Page 9: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/9.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 9
AGENDA
• Motivation
• Problem Statement and Stakeholders
• Technology Introduction
• Gap Analysis
• CONOPS
• Simulation
• Trade-off Analysis
• Business Case
9
![Page 10: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/10.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 10
ALTERNATIVES
• Eye Tracking Software
• Head Movement (Gyroscope)
• Voice Control
• Muscle Contraction Detection
• Brain-Computer Interface
10
TRADE SPACE
[4]RoboNurse, Computers Making Decisions- Standford
[5] Accompany Care-O-Bot, Fraunhofer Institute
Methods to Perform Hands-Free
Control
![Page 11: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/11.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 11
SELECTING ALTERNATIVES
• EEG will be evaluated as the BCI Alternative as it is the most
researched and cost effective BCI
• Muscle Contraction Detection and Gyroscope will not be evaluated as
these alternatives require physical input or movement
Alternatives to Evaluate:
- EEG
- Eye Tracking
- Voice Commands
![Page 12: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/12.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 12
VOICE COMMAND SUMMARY
• Unlimited Number of Commands
• Ready to Use Technology
• Study by School of Information Technology in China set up
a wheelchair to be driven via voice command • 92% average success rate
• 2.2s average delay
![Page 13: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/13.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 13
ELECTROENCEPHALOGRAPHY (EEG) SUMMARY
• Non-invasive BCI for detecting brain activity
• Uses electrodes to measure ERP pulses along the scalp
• Event related potentials (ERP) - Brain’s electrical response to a sensory, cognitive, or
motor event
• Key to utilizing EEG as a control method is to detect desired ERP from other ERPs (noise)
and map to a command
• Non-invasive BCI Interface for Device Operation
• 96-channel EEG system
• Cursor guidance commands
• User “mastery” within 5 sessions
• Patient initial success rate of 50-70% to 80-100% accuracy
13
https://drive.google.com/file/d/0BzLzei3_VBoAeFVCSFg0U2ljdDQ/view
*J. del R. Millán is with the IDIAP Research Institute, CH-1920 Martigny, Switzerland, and also with the Laboratory of Computational Neuroscience
![Page 14: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/14.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 14
EYE TRACKING SUMMARY
• Subject to false positives
• Calibration is key to success
• 2014 study published by Max Planck Institute for Informatics
in Germany • 40ms slower than using a mouse
![Page 15: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/15.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 15
AGENDA
• Motivation
• Problem Statement and Stakeholders
• Technology Introduction
• Gap Analysis
• CONOPS
• Simulation
• Trade-off Analysis
• Business Case
15
![Page 16: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/16.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 16
GAP ANALYSIS
• The necessity of physical movement of the control interface is a highly limiting factor
is the system usability and user workload.
• For the 2,900,000 severely paralyzed persons in America, the necessity of physical
motion to operate a device interface renders the system unusable.
Win-Win - Create a hands-free control system interface which can match the
quality and cost of a traditional interface.
16
![Page 17: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/17.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 17
SCOPE
Before a new control interface can be selected for the robotic aid, we must examine
the maneuvers which the robot will need to perform.
This project will perform a simulation to collect and analyze the data crucial to
creating a hands-free control interface.
![Page 18: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/18.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 18
AGENDA
• Motivation
• Problem Statement and Stakeholders
• Technology Introduction
• Gap Analysis
• CONOPS
• Simulation
• Trade-off Analysis
• Business Case
18
![Page 19: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/19.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 19
CONOPS
A paralyzed person cannot perform basic motor tasks without
assistance. A robot, controlled by the person, can cross the room and
retrieve an item (prepackaged meal, medicine, cell phone).
19
User with HFCS YouBot by Kuka [9] Youbot, Vertically-Integrated Projects, GT
![Page 20: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/20.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 20
Functional Diagram
![Page 21: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/21.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 21
USE CASE
21
![Page 22: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/22.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 22 22
MISSION REQUIREMENTS
R.0 The HFCS shall be operable without tactile user input.
R.1 The HFCS shall not harm the user in any way.
R.2 The HFCS shall provide flexibility for use for a variety of functions.
R.3 The HFCS shall operate in real-time.
R.4 *The HFCS shall be capable of differentiating between [X] commands.
R.5 *The HFCS shall be capable of executing [Y] commands.
* where [X] and [Y] to come from simulation
![Page 23: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/23.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 23
COMMANDS FOR CONOPS
23
Loop 1: Correct direct
1. Stop motion
2. Pivot to correct direction
3. Resume forward motion
Assumption 1:
Hand contacting object will result in grasp.
(No slipping, etc)
![Page 24: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/24.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 24
VIRTUAL ROBOTIC SIMULATOR
• V-REP as a virtual robot simulator • Physics based collisions
• Large number of ready made models
• Free for educational use
• Matlab
• TRS and RTB - binds VREP and
Matlab
• YouBot will pick up an object and
move it to another location • Limited preprogrammed movement
YouBot, a popular multi-function robot
![Page 25: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/25.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 25
YouBot VIRTUAL ROBOT[9]
• Platform
• Omnidirectional wheels
• Zero turn radius pivoting
• Surface for placing multiple objects
• Arm
• 5 joints for 5 DoF movement (Arm is not capable
of full pitch motion)
• Each joint has a limited range of movement
• Gripper
• Open and close
• To simplify the YouBot movement, all the joint movement can be automated according to
the x-y-z position of the gripper in order to reduce user complexity
![Page 26: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/26.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 26
YouBot Dimensions
Platform
• Raise distance from ground .7m
• L x W x D: .58m x .38m x 14m
• Payload 20 kg
• Max Velocity .8m/s
Arm (Scale factor x1.4)
• Total reach .917m
• Payload .7kg
Gripper (Scale factor x1.4)
• Range .007m
Work Envelope .7182 m^2
![Page 27: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/27.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 27
AGENDA
• Motivation
• Problem Statement and Stakeholders
• Technology Introduction
• Gap Analysis
• CONOPS
• Simulation
• Trade-off Analysis
• Business Case
27
![Page 28: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/28.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 28
Simulation Objectives
• For object relocation be domestic robot: • How many & what turn commands are needed
• How many & what travel commands are needed
• How many & what arm commands are needed
• How many & what gripper commands are needed
• What tasks are infeasible for robot?
![Page 29: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/29.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 29
Simulation Requirements
• The simulation shall model the path of a robot fetching objects in a domestic setting.
• The simulation shall model the arm movement of a robot fetching objects in a domestic
setting.
• The simulation shall model the gripper of a robot fetching objects in a domestic setting.
• The simulation shall collect platform travel statistics.
• The simulation shall collect platform turn statistics.
• The simulation shall collect arm movement statistics.
• The simulation shall collect gripper statistics.
• The simulation shall define range of robot.
![Page 30: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/30.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 30
SIMULATION OVERVIEW Fetch item - an item the virtual robot is instructed to pick up
Fetch point - the location of the fetch item, random variable
Task - a randomly generated trip from a start point, to the fetch point, picking up the fetch item and carrying it to an assigned finish point
Start location - initial (x,y) coordinates of the center of the robot’s platform, random variable
Finish location - final location of the center of the robot’s platform, random variable
Start Points (Green), End Point (Red) Fetch Points (Blue)
![Page 31: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/31.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 31
SIMULATION STATISTICS
Statistics to Record per Task
1. Start location
2. Fetch item location
3. End location
4. Total time on task
5. Time spent on fetch trip
6. Time spent on return trip
7. Time spent picking up object (frame
between fetch trip and return trip)
8. Total distance traveled
9. # of platform rotations
10. degrees rotated for each rotation
11. # lateral arm movements
12. Arm displacement per movement
(cm)
![Page 32: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/32.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 32
2D- Simulation Inputs
Room Map - Red marks obstacles
Start and fetch location
coordinates
![Page 33: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/33.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 33
2D- Simulation Modules
Name Description Output
GeneratePath.m Select start and fetch coordinates
then uses D* navigation algorithm
from Robotics Toolbox
Array of coordinates stepping through
shortest path to fetch location.
StatisticsScript.m Locates turns along shortest path.
Calculates angle of turn and
distance traveled on straight-aways
Matrix
Column 1,2 - (x,y)
Column 3 - angle of turn @ pivot point
Column 4 - distance traveled in previous
straight away
GraphicalData Displays frequency of maneuvers Bar graphs for frequency of travel distance
and angle
![Page 34: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/34.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 34
2D- Simulation Output for Single Run
![Page 35: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/35.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 35
2D- Simulation Output for 50 runs
![Page 36: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/36.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 36
3D Simulation
Depending on the position of the YouBot base, the arm joint positions are different even for moving
an object on the same height.
![Page 37: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/37.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 37
Alternative to Manual Control of Arm/Gripper
• Vision sensor to track objects to be picked up to simplify the
pick up action
• Option for user would be displayed an array of detected distinct
objects and be prompted to choose the distinct object to be
picked up (via voice command)
• Used as an aid to guide the arm movement
![Page 38: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/38.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 38
Vision Sensor for HFCS Display
VREP Vision sensor has an adjustable resolution, in the ranges that an object is detected, it will
return a depth value, color and intensity values for every point. From the angle of the vision
sensor, the relative coordinates can be calculated.
![Page 39: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/39.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 39
Simulation Results
-6 platform commands - turn left 18 degrees, turn left 2
degrees, turn right 18 degrees, turn right 2 degrees, start, stop
- Vision Sensor and active select algorithm
- Automated collision avoidance
- Return to user function
![Page 40: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/40.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 40
Simulation Impact
Completed Mission Requirements R.4 *The HFCS shall be capable of differentiating between 9 commands.
R.5 *The HFCS shall be capable of executing 9 commands.
Additional Requirement 1. Implement a vision sensor instead of manual control once the object is in “sight”
of the robot.
![Page 41: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/41.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 41
AGENDA
• Motivation
• Problem Statement and Stakeholders
• Technology Introduction
• Gap Analysis
• CONOPS
• Simulation
• Trade-off Analysis
• Business Case
41
![Page 42: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/42.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 42
Decision Criteria
![Page 43: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/43.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 43
Weighting the Attributes
Goal
Category
Weight Measures Measure Weight Total Weight
Capabilities 0.5 Number of
Commands 0.7 0.35
Time Sensitivity 0.3 0.15
TRL 0.1 TRL 1 0.1
Usability 0.2 User Population 0.75 0.15
New User Time 0.25 0.05
Performance 0.2 Accuracy 0.4 0.08
Maintainability 0.2 0.04
Reliability 0.4 0.08
![Page 44: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/44.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 44
Utility Calculation
![Page 45: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/45.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 45
SENSITIVITY ANALYSIS
![Page 46: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/46.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 46
Cost vs Utility for 1 Year
![Page 47: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/47.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 47
AGENDA
• Motivation
• Problem Statement and Stakeholders
• Technology Introduction
• Gap Analysis
• CONOPS
• Simulation
• Trade-off Analysis
• Business Case
47
![Page 48: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/48.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 48
● Eliminates need for 24/7 in-home caregiver
● Fill in-home caregiver shortage [4]
● Returns level of independence to physically disabled persons
VALUE
● Unit is defined as a Robotic Aid + Hands-Free Control System
● Sell units to Home Health Care Organizations
○ Open channel for improvement in future releases
○ Allow professionals to monitor ratio of robotic care vs human interaction
● Potential to form partnership
○ Leverage knowledge of in-home care agencies to further design
● Cost per Unit = $4,500
![Page 49: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/49.jpg)
COST
49
● TOTAL 5 years $699k
● Nonrecurring $195K
● Recurring $504k
REVENUE ● Break even at 300 sales
Sell 100 per year Sell 200 for year 1 and 2, sell 100 after that
![Page 50: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/50.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 50
QUESTIONS?
Hand-held controller
Robotic Aid
Proposed design eliminates need for hand-held controller User
![Page 51: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/51.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 51
BACKUP SLIDES
51
![Page 52: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/52.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 52
References
[1] "Prevalence of Paralysis." Christopher & Dana Reeve Foundation. The Reeve Foundation Paralysis Resource Center, n.d. Web. 19 Nov. 2015.
[2] "Human Factors and Medical Devices." U.S. Food and Drug Administration. U.S. Food and Drug Administration, n.d. Web. 19 Nov. 2015.
[3] "Care-Providing Robot FRIEND." Institute of Automation. University of Bremen, n.d. Web. 19 Nov. 2015. [4] "Computers and Robots: Decision-Makers in an Automated World." Computers and Robots DecisionMakers in an Automated World. University of Stanford, n.d. Web. 19 Nov. 2015.
[5] "Care-O-bot 3." Fraunhofer Institute for Manufacturing Engineering and Automation. Fraunhofer Institute, n.d. Web. 19 Nov. 2015.
[6] "Brain Cells Chat, Even Without a Synapse." Science Magazine. AAAS, n.d. Web. 19 Nov. 2015.
[7]"Parts of Central Nervous System." , Control and Coordination, Science Help. Tutorvista, n.d. Web. 19 Nov. 2015.
[8] "A Closer Look at EEG." Epilepsy Society. Epilepsy Society, n.d. Web. 19 Nov. 2015. [9] "YouBot." YouBot Store. Kuka, n.d. Web. 19 Nov. 2015. [10] "Emotiv EPOC / EPOC+." Emotiv Epoc. Emotiv, n.d. Web. 19 Nov. 2015.
[11] "Quantitative EEG and Event-Related Potentials." Neuronetrix. COGNISION , n.d. Web. 19 Nov. 2015.
[12] T. Pierce, T. Watson, J. King, S. Kelly and K. Pribram, 'Age Differences in Factor Analysis of EEG', Brain Topography, vol. 16, no. 1, pp. 19-27, 2003.
52
![Page 53: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/53.jpg)
MARKET VALUE
According to a 2014 New York
Times article [4], ● 1.3 million new paid caregivers will
be needed to meet demand over
the next decade.
● By 2020 the direct care workforce
(5 million people) will become the
largest occupation in the United
States, surpassing the number of
retail salespeople.
● About 75 percent of services
provided by home care agencies
are paid by Medicaid and
Medicare.
53
![Page 54: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/54.jpg)
Alternative Scores
![Page 55: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/55.jpg)
EEG CONTEXT
![Page 56: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/56.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 56
EEG TERMINOLOGY
Signals - The electrical activity that travels through a user’s
brain
Synapses - Spaces between neurons that conduct the electrical
activity in the brain
Noise - General term for unwanted and, in general, unknown
modifications that a signal may suffer during capture,
storage, transmission, processing, or conversion
Event-Related Potential - electrophysiological response to a
stimulus
56
[6] AAAS, Brain and Behavior
User command - The instruction a user gives to perform an action (user signal to EEG)
Input Command - A collection of steps the end-device will perform in order to execute a
single user command (digital signals to end-device)
![Page 57: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/57.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 57
BRAIN-COMPUTER INTERFACE (BCI)
Direct communication pathway between the brain and a computing device
57
BCI Type Signal
Quality
Sensor Location Cost
Commitment
Other
1. Invasive Best In Brain Matter High Risk of scar tissue
2. Partially-
Invasive
Medium Under skull High
3. Non-invasive Poor On Scalp Low
EEG is the most studied non-invasive BCI method due to its low cost and ease of use. The data
quality is improving quickly.
![Page 58: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/58.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 58
Status of EEG as a Control Method Projects that follow a similar approach using EEG technology:
• Non-invasive BCI Interface for Device Operation
• 96-channel EEG system
• Cursor guidance commands
• User “mastery” within 5 sessions
• Patient initial success rate of 50-70% to 80-100% accuracy
• Brain Actuated Control of a Mobile Robot
• Combined with machine-learning techniques
• Six commands combined with “mental states”
• Worse than manual control by a slim factor of 1.5
https://drive.google.com/file/d/0BzLzei3_VBoAeFVCSFg0U2ljdDQ/view
*J. del R. Millán is with the IDIAP Research Institute, CH-1920 Martigny, Switzerland, and also with the Laboratory of Computational Neuroscience
![Page 59: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/59.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 59
PHYSIOLOGY OF THE BRAIN
Human Brain
- Comprised of
100,000,000,000 neurons
- Neurons connect via
synapses
- When thought occurs,
neurons generate spikes
of electrical activity
59
[7] Canadian Institutes of Health Research, Institutes of Neurological, Mental Health and Addiction
![Page 60: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/60.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 60
ELECTROENCEPHALOGRAHY (EEG)
• Offers a non-invasive method to detect electrical activity by using electrodes to
measure the electrical pulses along the scalp to ultimately graph these impulses
• Predominantly used in medical field for sleeping/brain abnormalities and
detection, stress evaluation, and prosthetics
• Event related potentials (ERP)
• Correspond to a brain response related to a sensory, cognitive or motor event
• The key to using EEG as a control method is to detect the desired ERP from
other ERPs (noise) and map it to a command
• EEG is not widely used as a control method due to poor signal quality from
being outside of the skull. Technological advances are improving quality.
60
![Page 61: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/61.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 61
How the EEG Hands-Free System Works
1) When a user thinks, electrical pulses fire within synapses in lobes of the brain specific to
that “thought” - user command
2) The EEG headset reads these signals as voltage and generates a graph indicating the
intensity and pattern of the electrical signals (The same stimuli or thought should produce
roughly the same EEG patterns when repeated)
3) These graphs will then be analyzed and chosen patterns will be stored in the database with
it’s corresponding thought or stimulus
4) These selected patterns will then be mapped to varying device actions through a system
that this project will create - input command
![Page 62: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/62.jpg)
62 DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
User concentrates
on a trained user
command
System filters out
noise and converts
signals to digital
format
System sends digital
signals to library to be
matched to a command
System outputs
corresponding
command to
simulator
14
Sensors
EEG raw data stream
EEG receives
signals and captures
them in the form of
graphs Raw signals are
stored in database and
inputted into the
system
[8] A Closer Look at EEG, Epilepsy Society
![Page 63: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/63.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 63
PATTERN LIBRARY BLOCK DIAGRAM
63
![Page 64: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/64.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 64
OPERATIONAL BLOCK DIAGRAM
64
![Page 65: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/65.jpg)
65 DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
![Page 66: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/66.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 66
EEG SIGNAL PROCESSING
• The EEG signal will be composed of 14 continuous and numerical data streams, one
from each of the EEG.
• General steps to preprocess the signal:
1. High pass filter to remove low frequency noise
2. Use a window function to group a variable length interval of data such as the Hann or
Hamming window function and eliminate edge artifacts
3. Fast Fourier Transform
4. Convert the imaginary part of any complex numbers into amplitude values
• Use a machine learning approach such as neural networks to build a learning model,
then continuously test in real time to measure accuracy
66
![Page 67: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/67.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 67
OPTIMIZING EEG Event-Related Potentials
Goal: Determine largest limiting factor in EEG Control System reliability
• Use simulation to examining the effects of different variables on
detection rate and reliability of ERPs such as: • Optimal training procedures or increasing training time
• Use of different machine learning approaches to improve detection
• Tentative input commands to be detected: • 6 directional commands for positive and negative displacement along each
of the x, y, z axes
• 1 toggle command for switching platform control vs arm control
• May need to simplify the number of different commands to reduce user
complexity
67
![Page 68: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/68.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 68
SIMULATION REQUIREMENTS
S.0.1 Virtual Robot shall accept input from HFCS.
S.0.1.1 Virtual Robot model shall contain a robot with 2D movement capability
S.0.1.2 Virtual Robot model shall contain a robot capable of grasping and
relocating
S.0.1.3 Virtual Robot model shall contain an object to be relocated
S.0.2 Simulation shall accept EEG ERPs from an input device
S.0.2.1 Simulation shall respond to a detected ERP in time < 2 seconds
S.0.2.2 Simulation shall map ERPs to input commands and interface with VREP
S.0.2.3 Simulation should detect and respond to 2+ simultaneous ERP
S.0.3 Virtual robot shall employ automated collision avoidance.
68
![Page 69: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/69.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 69
EPOC Cognitive Suite
1)Create user account
2)Train neurological signal to a basic motion
3)Repeat thought to perform signal
4)Map signal to another vehicle motion
69 Source: Emotiv EPOC
![Page 70: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/70.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 70
EEG Headset Alternatives
Standard EPOC
• 14 EEG channels and two references for accurate spatial resolution
• High performance wireless device
• iOS and Android compatible
EPOC+
• 9 axis-inertial motion sensors, Bluetooth capabilities, additional applications are
enabled
Raw Data Add-on
• Includes EEG firmware that allows the raw EEG data stream and marker
events in TestBench software
70
![Page 71: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/71.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 71
Brainwaves
Neural oscillations are rhythmic or repetitive neural activities in the central nervous system. Neural tissue generate oscillations through
neuron interactions.
A large number of neurons activating for a particular neural computation is called a neural ensemble.
Neural oscillations can be categorized into 5 main frequencies
• Beta (14-40Hz) - Waking consciousness
• Alpha (7.5-14Hz) - Conscious relaxation
• Theta (4-7.5Hz) - Sleeping and light meditation
• Delta (0.5-4Hz) – Deep sleep
• Gamma (above 40Hz) – Insight
• Only recently discovered
71
![Page 72: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/72.jpg)
72 DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 72
http://neuronetrix.com/technology-i-36.html
![Page 73: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/73.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 73
Age
• A study conducted by Thomas W. Pierce et. al suggested that
age has an effect on EEG
• Results showed that electrode groupings were higher in older
adults than younger adults
• Older adults had more electrode locations that did not load
than the younger adults
73
![Page 74: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/74.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 74 74
[1]T. Pierce, T. Watson, J. King, S. Kelly and K. Pribram, 'Age Differences in Factor
Analysis of EEG', Brain Topography, vol. 16, no. 1, pp. 19-27, 2003.
![Page 75: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/75.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 75
Gender
• A study by Corsi-Cabrera et. al suggested differences in the
brain waves between male and female
• Men showed higher beta power, women showed higher alpha
power
• The alpha waves of men decreased during analytic,spatial,
and mixed processing, while women decreased significantly
only in analytic and mixed processing
75
![Page 76: Design of Hands-Free Interface for Object Relocationcatsr.ite.gmu.edu/SYST490/490_2015_HandsFreeControl/Final... · Design of Hands-Free Interface for Object Relocation Kassidy Kenney,](https://reader031.fdocuments.in/reader031/viewer/2022022509/5ad5bf487f8b9a5c638d741f/html5/thumbnails/76.jpg)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 76
CURRENT TECHNOLOGIES UTILIZED BY PHYSICALLY
DISABLED PERSONS
76
• Prosthesis
• Wheelchair
• Voice Recognition [13] Johns Hopkins- Modular Prosthetic Limb