TouchNet preview

by Jason Toy

TouchNet

• A touch simulator and dataset of touchable objects to teach AIs how to interact with their environments via motor-sensory neurons

history

• idea around 2015

• started coding September 2017

• initial release 2017 Q4

why TouchNet?

why TouchNet?

• On Intelligence

• future predicting machines

• neuroscience based principles

why TouchNet?

• Helen Keller

• intelligence through fingers

• consciousness “started” from I

Helen Keller• “Every object is associated in my mind with tactual

qualities which, combined in countless ways, give me a sense of power, of beauty.”

• “..with the fingers I see only a very small portion of a surface, and that I must pass my hand continually over it before my touch grasps the whole. It is still more important, however, to remember that my imagination is not tethered to certain points, locations, and distances. It puts all the parts together simultaneously as if it saw or knew instead of feeling them.”

where is AI now?• images, CNNs, LSTMs, GANs

• most business focus on static models; some work in RL

• where is time?

• motor sensory neurons?

• where is the agent compared to the world?

- John Locke, philosophy of mind

“An unknown, but potentially large, fraction of animal and human intelligence is a direct

consequence of the perceptual and physical richness of our environment, and is unlikely to

arise without it.”

TouchNet consists of 4 parts

• dataset of 3D objects to touch and interact with

• an evolving simulation

• a simple extendable API

• benchmarks to measure and track progress

Dataset

Dataset

• procedurally generated 3D objects

• common objects: cups, plates, bowls, etc

• starting with tool objects

• cifar100 …. imagenet

Simulator

Simulators and physics

• need a real physics engine

• needs to be fully programmable & modifiable

• tested all of them: MuJoCo, DeepMind Lab, Roboschool, project Malmo, VizDoom,Gazebo, bullet , Processing

Simulators and physics• bullet and pybullet are a secret weapon!

• robust physics engine used for many applications

• super easy to install

• movies: Shrek, Hancock,Sherlock Holmes….

• games: Toy Story 3, Grand Theft Auto 4,Hot Wheels…

pybullet• rigid and soft body simulations

• collision detection

• works with unity, blender,CUDA,OpenCL,…..

• developed by Erwin Coumans & community since ~2003

• now at Google and used for RL & Robotics

• http://pybullet.org

What we want to simulate

What we have

simulation demo

coevolving simulator & agent

• index finger with merkel touch sensors

• all sensor types in finger

• all fingers

• full arm

• full body?

API

• simple to get started and use

• easy to install

• modifiable and customizable

reinforcement learning api

• based off OpenAI’s gym and universe API

• can plug in almost any code for reinforcement learning

OpenAI’s Universe API

Same API

Benchmarks

“Blind” objection classification

• goal: classify the object

• CNN and actor critic DNN models trained concurrently

• pytorch and tensorflow versions

• 4% accuracy, please beat mine!

“blind” classification code walk through

Benchmarks

• HTM version?

• more benchmarks in the future e.g:

• moving objects, differentiating textures,…..

Help!

• very ambitious project, too big for me

• 100% open source

• I do a little of everything ( suck at everything? )

• looking for contributors and experts in physics,math, neuroscience, ml

Jason Toy jtoy@jtoy.net

twitter.com/jtoy

Thank you!

Questions and Feedback

http://github.com/jtoy/touchnet