Mind Control to Major Tom: Is It Time to Put Your EEG Headset On?
-
Upload
steve-poole -
Category
Software
-
view
164 -
download
1
Transcript of Mind Control to Major Tom: Is It Time to Put Your EEG Headset On?
@spoole167 @Luc_At_IBM
Mind Control to Major Tom:
Is It Time to Put Your EEG Headset On?
CON2338
@spoole167 @Luc_At_IBM
Who we areSteve Poole aka Obiwan
Java guru, Developer extraordinaireDelivery lead
Luc Desrosiers aka Lobot (Think Star Wars Cloud city)
Canadian expat living in the UKGadget & IoT enthousiast… and Cloud architect
@spoole167 @Luc_At_IBMhttps://www.flickr.com/photos/stawarz/
@spoole167 @Luc_At_IBM
Who we really are
Terminators from the future…
@spoole167 @Luc_At_IBM
Here to promote our book series
“Skynet for Dummies”
Today we’ll be talking about the basics for creating your own terminator
@spoole167 @Luc_At_IBM
Skynet will be built with the greatest and best tools
We picked Java because, even in your day it’s on many, many devices
We will be using Java 8 - because even in the future Java 9 hasn’t shipped yet
@spoole167 @Luc_At_IBM
Outline
Ways to interact with your human
understanding your human
controlling your human
@spoole167 @Luc_At_IBM
https
://w
ww
.flic
kr.c
om/p
hoto
s/gy
duxa
/
Ways to interact with your human
@spoole167 @Luc_At_IBM
Vision
AuditoryTouch
GustatoryOlfactory
Proprioception
Key input/output components of ahuman being
@spoole167 @Luc_At_IBM
Human being interaction map
Virtual Smell
Electronic noses
Wearables
Haptic sensorsMotion sensors
Molecular analysis
Virtual Reality
Computer VisionText-to-Speech
Speech-to-Text
Touch and proprioception
Gustatory Olfactory
Auditory Vision
Author: Allan-Hermann Pool
@spoole167 @Luc_At_IBM
Convincing your human its in a new world
Teaching your terminatorto understand the real world
Virtual RealityAugmented Reality
World Domination
@spoole167 @Luc_At_IBM
Goal is full immersive sensory replacement.
Environment Awareness
Object and Facial detection
IBM WatsonVisual Recognition
@spoole167 @Luc_At_IBM
Virtual Reality, or the many challenges of fooling the human beings
Images by Pdenbrook - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=31920588
From user interface
You want to reduce motion sickness?Add a nose!
Pointman
@spoole167 @Luc_At_IBM
Wait, where did my hands go?frame.hands().forEach(hand->{
Arm arm = hand.arm(); if (arm.isValid()) { renderArm(arm);}
hand.fingers().forEach(finger->{if (finger.isValid()) {
for(int i = 3; i >= 0; i--) {Bone bone = finger.bone(Type.swigToEnum(i));renderCylinder(bone.width(), bone.length(),
bone.prevJoint(), bone.direction()); }
} });});
@spoole167 @Luc_At_IBM
Making the humans feel the virtual world
Sense of touch has been attempted using:• Vibration• Small pockets of air balloons• Electrical impulse• Inertial sensors
Many startup and currently many failures…
@spoole167 @Luc_At_IBM
Recognizing objects in a scene
Completely possible… However it requires a lot of training!
Complexity does not lie in the coding but in getting a good set of positive and negative images:- Positive images have the object you want to identify- Negative images do not
Group positive images together and we call them class.Group related class together and we call them classifier.
@spoole167 @Luc_At_IBM
Demo: Visual Recognition with Watson
VisualRecognition service = new VisualRecognition(VisualRecognition.VERSION_DATE_2016_05_19);
service.setApiKey("{api-key}");
System.out.println("Classify an image"); ClassifyImagesOptions options = new ClassifyImagesOptions.Builder().images(
new File("src/test/resources/visual_recognition/car.png")).build();
VisualClassification result = service.classify(options).execute();
System.out.println(result);
@spoole167 @Luc_At_IBM
Hasta la vista Baby:
it’s not just about vision
@spoole167 @Luc_At_IBM
Understanding what your human is saying
Getting your human to understand who’s the boss
Speech-to-Text Text-to-Speech(aka)
World Domination
@spoole167 @Luc_At_IBM
Demo: Text to Speech Furbynator speaks
TextToSpeech service = new TextToSpeech();
String text = "Hello world.";
InputStream stream = service.synthesize (text, Voice.EN_ALLISON, "audio/wav");
OutputStream out = new FileOutputStream("hello_world.wav");
@spoole167 @Luc_At_IBM
Demo: Speech to Text
SpeechToText service = new SpeechToText();
RecognizeOptions options = new RecognizeOptions().contentType("audio/flac”).timestamps(true).wordAlternativesThreshold(0.9).continuous(true);
File file = new File("audio-file1.flac”);
SpeechResults results = service.recognize(file, options);
System.out.println(results);
@spoole167 @Luc_At_IBM
Today VirtualReality
ComputerVision
HandTracking
Speech-to- Text
Text-to-Speech
Current Applicability 6/10 7/10 8/10 7/10 8/10Java Coverage 4/10 8/10 9/10 9/10 9/10Ease of use 4/10 6/10 9/10 8/10 8/10Reliability 6/10 7/10 8/10 8/10 9/10
FuturePotential for improvement High High Medium Medium Medium
Scorecard
@spoole167 @Luc_At_IBM
Understanding the emotional state of your Human
https
://w
ww
.flic
kr.c
om/p
hoto
s/pi
xiet
art/
@spoole167 @Luc_At_IBM
Understanding emotion requires combining detection and analysis of facial features, prosody and lexical content in speech
This is hard!
@spoole167 @Luc_At_IBM
Recognizing facial features
Two approaches:
Take many pictures of humans in different moods and teach a neural net to do pattern matching
Detect interesting face points (nose tips, mouth corners, eyes etc and determine relationship between them.
@spoole167 @Luc_At_IBM
OpenCV has good Face Detection
VideoCapture camera = new VideoCapture();
CascadeClassifier cc = new CascadeClassifier( "haarcascade_frontalface_alt.xml");
MatOfRect faces = new MatOfRect();
Mat frame = new Mat();
camera.read(frame);cc.detectMultiScale(frame, faces);
Rect[] hits=faces.toArray();
for(int i=0;i<hits.length;i++) {Mat face = new Mat(frame,hits[i]);Imgcodecs.imwrite("face"+i+".jpg", face);
}
@spoole167 @Luc_At_IBM
Simple Face Recognition / Emotion detection is much harder
OpenCV has image training capability but it’s not available as a Java API
It’s computationally very expensive
You need a database of images that contain the items you want to detectYou need a database of images that do not contain the items you want to detect
OpenCV training requires creating images from the second set with items from the first set added in at various scales and rotational angles
http://coding-robin.de/2013/07/22/train-your-own-opencv-haar-classifier.html
@spoole167 @Luc_At_IBM
Detection alternatives:
Get humans to score a series of pictures of other humans suffering emotions
http://www.ipsp.ucl.ac.be/recherche/projets/FaceTales/en/Home.htm
Apply various point analysis techniques to extract key parts of a face
http://www.codeproject.com/Articles/110805/Human-Emotion-Detection-from-Image
https://github.com/mpillar/java-emotion-recognizer
@spoole167 @Luc_At_IBM
Creating test images of the various emotions
Challenge:
By Crosa (Flickr: Scream) [CC BY 2.0 (http://creativecommons.org/licenses/by/2.0)], via Wikimedia Commons
https://www.flickr.com/photos/huphtur/
https://www.flickr.com/photos/auroredelsoir/
For some reason humans only ever scream?
https://www.flickr.com/photos/tenaciousme/https://www.flickr.com/photos/morton/
@spoole167 @Luc_At_IBM
Analysing emotion from text and speech
This is still hard. Terminators are not good with speech
I’ll be back
asta la vista....baby
Come with me if you want to liveYou’ve been terminated
"Get out."
I need your clothes, your boots and your motorcycle.
Stay here. I'll be back
@spoole167 @Luc_At_IBM
ToneAnalyzer service = new ToneAnalyzer(ToneAnalyzer.VERSION_DATE_2016_05_19);
ToneAnalysis tone = service.getTone(text, null).execute();
System.out.println(tone);
Watson Tone Analysis sample
@spoole167 @Luc_At_IBM
Analysing tone with IBM Watson:
I need your clothes, your boots and your motorcycle.I’ll be back.Get out.asta la vista....baby.Stay here.I'll be back.You’ve been terminated.Come with me if you want to live
Conscientiousness
Extraversion
Analytical
@spoole167 @Luc_At_IBM
Analysing tone: the more words the better:
Using your mind to interact with computers is a long-standing desire. Advances in technology have made it more practical, but is it ready for prime time? This session presents practical examples and a walkthough of how to build a Java-based end-to-end system to drive a remote-controlled droid with nothing but the power of thought. Combining off-the-shelf EEG headsets with cloud technology and IoT, the presenters showcase what capabilities exist today. Beyond mind control (if there is such a concept), the session shows other ways to communicate with your computer besides the keyboard. It will help you understand the art of the possible and decide if it's time to leave the capsule to communicate with your computer.
@spoole167 @Luc_At_IBM
Analysing tone: the more words the better:
@spoole167 @Luc_At_IBM
But, but, Terminators
can have a personality too!
@spoole167 @Luc_At_IBM
Maybe, but Personality of Terminators tends to be too predictable… Exterminate??
So lets look at the personality of the father of Robotics
https://en.wikipedia.org/wiki/I,_Robot_(film)
Isaac Asimov
@spoole167 @Luc_At_IBM
This analysis was based from an extract of the text of iRobot
We could not contact Isaacnext of kin to validate Watson’s findings…
but Wikipedia seems to agree:https://en.wikipedia.org/wiki/Isaac_Asimov
@spoole167 @Luc_At_IBM
Watson Personality Insight sample
PersonalityInsights service = new PersonalityInsights();service.setUsernameAndPassword("{username}", "{password}");try {
JsonReader jReader = new JsonReader(new FileReader("profile.json"));Content content = GsonSingleton.getGson().fromJson(jReader, Content.class);
ProfileOptions options = new ProfileOptions().contentItems(content.getContentItems());
Profile profile = service.getProfile(options);System.out.println(profile);
} catch (FileNotFoundException e) {e.printStackTrace();
}
@spoole167 @Luc_At_IBM
Today ToneAnalysis
PersonalityAnalysis
Current Applicability 8/10 8/10Java Coverage 9/10 9/10Ease of use 8/10 7/10Reliability 8/10 7/10
FuturePotential Improvement Medium Medium
Scorecard
@spoole167 @Luc_At_IBM
Ways to control with your human
https://www.flickr.com/photos/adforce1/
@spoole167 @Luc_At_IBM
We tried inserting machines into humans.
Too messy and somewhat obvious.
https
://w
ww
.flic
kr.c
om/p
hoto
s/au
spic
es/
@spoole167 @Luc_At_IBM
We tried adding machines outside humans.
Mixed success
By C
cmsh
arm
a2 -
Ow
n w
ork,
CC
BY-S
A 4.
0, h
ttps
://c
omm
ons.w
ikim
edia
.org
/w/in
dex.
php?
curid
=449
2070
7
43IBM _
In fact it is much simpler!!! Just hack the brain!
@spoole167 @Luc_At_IBM
Capturing brain activity (when applicable) of a human
IOT Command• Push• Pull• Stop
BluetoothBB8.roll
IOT Event•Push•Pull•Neutral
@spoole167 @Luc_At_IBM
Today MindControl
Current Applicability 3/10Java Coverage 7/10Ease of use 4/10Reliability 2/10
FuturePotential Improvement High
Scorecard
@spoole167 @Luc_At_IBM
So its clear that we’re not quite ready to take over the world…
@spoole167 @Luc_At_IBM
“we’ll be back”