@spoole167 @Luc_At_IBM

Mind Control to Major Tom:

Is It Time to Put Your EEG Headset On?

CON2338 

@spoole167 @Luc_At_IBM

Who we areSteve Poole aka Obiwan

Java guru, Developer extraordinaireDelivery lead

Luc Desrosiers aka Lobot (Think Star Wars Cloud city)

Canadian expat living in the UKGadget & IoT enthousiast… and Cloud architect

@spoole167 @Luc_At_IBMhttps://www.flickr.com/photos/stawarz/

@spoole167 @Luc_At_IBM

Who we really are

Terminators from the future…

@spoole167 @Luc_At_IBM

Here to promote our book series

“Skynet for Dummies”

Today we’ll be talking about the basics for creating your own terminator

@spoole167 @Luc_At_IBM

Skynet will be built with the greatest and best tools

We picked Java because, even in your day it’s on many, many devices

We will be using Java 8 - because even in the future Java 9 hasn’t shipped yet

@spoole167 @Luc_At_IBM

Outline

Ways to interact with your human

understanding your human

controlling your human

@spoole167 @Luc_At_IBM

https

://w

ww

.flic

kr.c

om/p

hoto

s/gy

duxa

/

Ways to interact with your human

@spoole167 @Luc_At_IBM

Vision

AuditoryTouch

GustatoryOlfactory

Proprioception

Key input/output components of ahuman being

@spoole167 @Luc_At_IBM

Human being interaction map

Virtual Smell

Electronic noses

Wearables

Haptic sensorsMotion sensors

Molecular analysis

Virtual Reality

Computer VisionText-to-Speech

Speech-to-Text

Touch and proprioception

Gustatory Olfactory

Auditory Vision

Author: Allan-Hermann Pool

@spoole167 @Luc_At_IBM

Convincing your human its in a new world

Teaching your terminatorto understand the real world

Virtual RealityAugmented Reality

World Domination

@spoole167 @Luc_At_IBM

Goal is full immersive sensory replacement.

Environment Awareness

Object and Facial detection

IBM WatsonVisual Recognition

@spoole167 @Luc_At_IBM

Virtual Reality, or the many challenges of fooling the human beings

Images by Pdenbrook - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=31920588

From user interface

You want to reduce motion sickness?Add a nose!

Pointman

@spoole167 @Luc_At_IBM

Wait, where did my hands go?frame.hands().forEach(hand->{

Arm arm = hand.arm(); if (arm.isValid()) { renderArm(arm);}

hand.fingers().forEach(finger->{if (finger.isValid()) {

for(int i = 3; i >= 0; i--) {Bone bone = finger.bone(Type.swigToEnum(i));renderCylinder(bone.width(), bone.length(),

bone.prevJoint(), bone.direction()); }

} });});

@spoole167 @Luc_At_IBM

Making the humans feel the virtual world

Sense of touch has been attempted using:• Vibration• Small pockets of air balloons• Electrical impulse• Inertial sensors

Many startup and currently many failures…

@spoole167 @Luc_At_IBM

Recognizing objects in a scene

Completely possible… However it requires a lot of training!

Complexity does not lie in the coding but in getting a good set of positive and negative images:- Positive images have the object you want to identify- Negative images do not

Group positive images together and we call them class.Group related class together and we call them classifier.

@spoole167 @Luc_At_IBM

Demo: Visual Recognition with Watson

VisualRecognition service = new VisualRecognition(VisualRecognition.VERSION_DATE_2016_05_19);

service.setApiKey("{api-key}");

System.out.println("Classify an image"); ClassifyImagesOptions options = new ClassifyImagesOptions.Builder().images(

new File("src/test/resources/visual_recognition/car.png")).build();

VisualClassification result = service.classify(options).execute();

System.out.println(result);

@spoole167 @Luc_At_IBM

Hasta la vista Baby:

it’s not just about vision

@spoole167 @Luc_At_IBM

Understanding what your human is saying

Getting your human to understand who’s the boss

Speech-to-Text Text-to-Speech(aka)

World Domination

@spoole167 @Luc_At_IBM

Demo: Text to Speech Furbynator speaks

TextToSpeech service = new TextToSpeech();

String text = "Hello world.";

InputStream stream = service.synthesize (text, Voice.EN_ALLISON, "audio/wav");

OutputStream out = new FileOutputStream("hello_world.wav");

@spoole167 @Luc_At_IBM

Demo: Speech to Text

SpeechToText service = new SpeechToText();

RecognizeOptions options = new RecognizeOptions().contentType("audio/flac”).timestamps(true).wordAlternativesThreshold(0.9).continuous(true);

File file = new File("audio-file1.flac”);

SpeechResults results = service.recognize(file, options);

System.out.println(results);

@spoole167 @Luc_At_IBM

Today VirtualReality

ComputerVision

HandTracking

Speech-to- Text

Text-to-Speech

Current Applicability 6/10 7/10 8/10 7/10 8/10Java Coverage 4/10 8/10 9/10 9/10 9/10Ease of use 4/10 6/10 9/10 8/10 8/10Reliability 6/10 7/10 8/10 8/10 9/10

FuturePotential for improvement High High Medium Medium Medium

Scorecard

@spoole167 @Luc_At_IBM

Understanding the emotional state of your Human

https

://w

ww

.flic

kr.c

om/p

hoto

s/pi

xiet

art/

@spoole167 @Luc_At_IBM

Understanding emotion requires combining detection and analysis of facial features, prosody and lexical content in speech

This is hard!

@spoole167 @Luc_At_IBM

Recognizing facial features

Two approaches:

Take many pictures of humans in different moods and teach a neural net to do pattern matching

Detect interesting face points (nose tips, mouth corners, eyes etc and determine relationship between them.

@spoole167 @Luc_At_IBM

OpenCV has good Face Detection

VideoCapture camera = new VideoCapture();

CascadeClassifier cc = new CascadeClassifier( "haarcascade_frontalface_alt.xml");

MatOfRect faces = new MatOfRect();

Mat frame = new Mat();

camera.read(frame);cc.detectMultiScale(frame, faces);

Rect[] hits=faces.toArray();

for(int i=0;i<hits.length;i++) {Mat face = new Mat(frame,hits[i]);Imgcodecs.imwrite("face"+i+".jpg", face);

}

@spoole167 @Luc_At_IBM

Simple Face Recognition / Emotion detection is much harder

OpenCV has image training capability but it’s not available as a Java API

It’s computationally very expensive

You need a database of images that contain the items you want to detectYou need a database of images that do not contain the items you want to detect

OpenCV training requires creating images from the second set with items from the first set added in at various scales and rotational angles

http://coding-robin.de/2013/07/22/train-your-own-opencv-haar-classifier.html

@spoole167 @Luc_At_IBM

Detection alternatives:

Get humans to score a series of pictures of other humans suffering emotions

http://www.ipsp.ucl.ac.be/recherche/projets/FaceTales/en/Home.htm

Apply various point analysis techniques to extract key parts of a face

http://www.codeproject.com/Articles/110805/Human-Emotion-Detection-from-Image

https://github.com/mpillar/java-emotion-recognizer

@spoole167 @Luc_At_IBM

Creating test images of the various emotions

Challenge:

By Crosa (Flickr: Scream) [CC BY 2.0 (http://creativecommons.org/licenses/by/2.0)], via Wikimedia Commons

https://www.flickr.com/photos/huphtur/

https://www.flickr.com/photos/auroredelsoir/

For some reason humans only ever scream?

https://www.flickr.com/photos/tenaciousme/https://www.flickr.com/photos/morton/

@spoole167 @Luc_At_IBM

Analysing emotion from text and speech

This is still hard. Terminators are not good with speech

I’ll be back

asta la vista....baby

Come with me if you want to liveYou’ve been terminated

"Get out."

I need your clothes, your boots and your motorcycle.

Stay here. I'll be back

@spoole167 @Luc_At_IBM

ToneAnalyzer service = new ToneAnalyzer(ToneAnalyzer.VERSION_DATE_2016_05_19);

ToneAnalysis tone = service.getTone(text, null).execute();

System.out.println(tone);

Watson Tone Analysis sample

@spoole167 @Luc_At_IBM

Analysing tone with IBM Watson:

I need your clothes, your boots and your motorcycle.I’ll be back.Get out.asta la vista....baby.Stay here.I'll be back.You’ve been terminated.Come with me if you want to live

Conscientiousness

Extraversion

Analytical

@spoole167 @Luc_At_IBM

Analysing tone: the more words the better:

Using your mind to interact with computers is a long-standing desire. Advances in technology have made it more practical, but is it ready for prime time? This session presents practical examples and a walkthough of how to build a Java-based end-to-end system to drive a remote-controlled droid with nothing but the power of thought. Combining off-the-shelf EEG headsets with cloud technology and IoT, the presenters showcase what capabilities exist today. Beyond mind control (if there is such a concept), the session shows other ways to communicate with your computer besides the keyboard. It will help you understand the art of the possible and decide if it's time to leave the capsule to communicate with your computer.

@spoole167 @Luc_At_IBM

Analysing tone: the more words the better:

@spoole167 @Luc_At_IBM

But, but, Terminators

can have a personality too!

@spoole167 @Luc_At_IBM

Maybe, but Personality of Terminators tends to be too predictable… Exterminate??

So lets look at the personality of the father of Robotics

https://en.wikipedia.org/wiki/I,_Robot_(film)

Isaac Asimov

@spoole167 @Luc_At_IBM

This analysis was based from an extract of the text of iRobot

We could not contact Isaacnext of kin to validate Watson’s findings…

but Wikipedia seems to agree:https://en.wikipedia.org/wiki/Isaac_Asimov

@spoole167 @Luc_At_IBM

Watson Personality Insight sample

PersonalityInsights service = new PersonalityInsights();service.setUsernameAndPassword("{username}", "{password}");try {

JsonReader jReader = new JsonReader(new FileReader("profile.json"));Content content = GsonSingleton.getGson().fromJson(jReader, Content.class);

ProfileOptions options = new ProfileOptions().contentItems(content.getContentItems());

Profile profile = service.getProfile(options);System.out.println(profile);

} catch (FileNotFoundException e) {e.printStackTrace();

}

@spoole167 @Luc_At_IBM

Today ToneAnalysis

PersonalityAnalysis

Current Applicability 8/10 8/10Java Coverage 9/10 9/10Ease of use 8/10 7/10Reliability 8/10 7/10

FuturePotential Improvement Medium Medium

Scorecard

@spoole167 @Luc_At_IBM

Ways to control with your human

https://www.flickr.com/photos/adforce1/

@spoole167 @Luc_At_IBM

We tried inserting machines into humans.

Too messy and somewhat obvious.

https

://w

ww

.flic

kr.c

om/p

hoto

s/au

spic

es/

@spoole167 @Luc_At_IBM

We tried adding machines outside humans.

Mixed success

By C

cmsh

arm

a2 -

Ow

n w

ork,

CC

BY-S

A 4.

0, h

ttps

://c

omm

ons.w

ikim

edia

.org

/w/in

dex.

php?

curid

=449

2070

7

43IBM _

In fact it is much simpler!!! Just hack the brain!

@spoole167 @Luc_At_IBM

Capturing brain activity (when applicable) of a human

IOT Command• Push• Pull• Stop

BluetoothBB8.roll

IOT Event•Push•Pull•Neutral

@spoole167 @Luc_At_IBM

Today MindControl

Current Applicability 3/10Java Coverage 7/10Ease of use 4/10Reliability 2/10

FuturePotential Improvement High

Scorecard

@spoole167 @Luc_At_IBM

So its clear that we’re not quite ready to take over the world…

@spoole167 @Luc_At_IBM

“we’ll be back”