© 2001-2002 Sony Computer Science Laboratories, Inc.

Jun RekimotoInteraction Laboratory

SONY Computer Science Laboratories, Inc.

SmartSkin ：An Infrastructure for Freehand Manipulati

on on Interactive Surfaces

© 2001-2002 Sony Computer Science Laboratories, Inc.

Background• Various types of “realworld” surfaces become int

eractive media– Ex: 　 DigitalDesk 　 [Wellner]– Support collaborative work– Fusion of virtual & real

• Camera-based input technologies– Sensitive to lighting conditions, occlusion– Camera cannot be integrated in the surface

• Possibility of other principle ？

© 2001-2002 Sony Computer Science Laboratories, Inc.

© 2001-2002 Sony Computer Science Laboratories, Inc.

Lev Theremin1896-1993

© 2001-2002 Sony Computer Science Laboratories, Inc.

Lev Theremin1896-1993

© 2001-2002 Sony Computer Science Laboratories, Inc.

SmartSkin: sensing principle

transmitter

receiver

transmitter receiver

2πfCV

© 2001-2002 Sony Computer Science Laboratories, Inc.

SmartSkin: sensing principle

transmitter

receiver

Conductive & Grounded Object(Human Body) transmitter receiver

© 2001-2002 Sony Computer Science Laboratories, Inc.

wave signalTo host

PC

low-passfilter

Analog-Digitalconverter

receiver

receiver

receiver

receiver

receiver

receiver

reference signal

ampamp

reference signal analogswitch

signalfrom electrode

to ADC

© 2001-2002 Sony Computer Science Laboratories, Inc.

Prototype I: 　 Interactive Table

• An wooden table covered by 8×9 electrodes.

• 30fps

© 2001-2002 Sony Computer Science Laboratories, Inc.

Prototype I: 　 Interactive Table

• An wooden table covered by 8×9 electrodes.

• 30fps

© 2001-2002 Sony Computer Science Laboratories, Inc.

© 2001-2002 Sony Computer Science Laboratories, Inc.

© 2001-2002 Sony Computer Science Laboratories, Inc.

Multiuser, Multihand interactions

© 2001-2002 Sony Computer Science Laboratories, Inc.

Distance measurement

© 2001-2002 Sony Computer Science Laboratories, Inc.

Input by hand(arm) shape

© 2001-2002 Sony Computer Science Laboratories, Inc.

Peak position estimation

a potential fieldcreated by bicubic interpolation

peak positionsensor values

© 2001-2002 Sony Computer Science Laboratories, Inc.

Potential Driven Movement

© 2001-2002 Sony Computer Science Laboratories, Inc.

PrototypeII: GesturePad24 (transmitters) 　 × 32 (receivers) 　　　　

(1cm pitch)

© 2001-2002 Sony Computer Science Laboratories, Inc.

© 2001-2002 Sony Computer Science Laboratories, Inc.

© 2001-2002 Sony Computer Science Laboratories, Inc.

Multiple finger detection

Various image processing algorithm can be applied.

• Connected region labeling

• Pattern matching• Object tracking• Etc.

© 2001-2002 Sony Computer Science Laboratories, Inc.

© 2001-2002 Sony Computer Science Laboratories, Inc.

Interactions by Multi-fingers

© 2001-2002 Sony Computer Science Laboratories, Inc.

Interaction by physical objects

electrodes

As a platform of tangible interfaces.

© 2001-2002 Sony Computer Science Laboratories, Inc.

SmartSkin

• Capacitive sensing (c.f., Thermin)• Mesh-shaped electrodes• Multiple object tracking• 2.5D (2D position, distance) ＋ shape• Fingertip detection by interpolating• Can be embedded in various objects• Low-cost, scalable

© 2001-2002 Sony Computer Science Laboratories, Inc.

Discussions ： analogy between biological “Skin” and artificial surfaces• form• Sensors on the surface• Input （ outside→skin ）

– Touch, pressure, temperature

• Output （ skin→outside ）– image (e.g., Chameleon)– Tactile

• communication• Artificial surface -> Skin

© 2001-2002 Sony Computer Science Laboratories, Inc.

Future Directions

• Interaction techniques featuring multi hands/fingers, shapes

• Non planar surface• Integration with a flat-panel display• Data communication

– Ex: table becomes a communication medium

• Special Thanks to …– Takaaki Ishizawa 、 Asako Toda （ Keio University ）– Kentaro Fukuchi (Tokyo Institute of Technology)

© 2001-2002 Sony Computer Science Laboratories, Inc.

Smart Keyboard

• Two hand input→ 　 Text Input

• Single Hand→ 　 Command

© 2001-2002 Sony Computer Science Laboratories, Inc.

Interactive Clothing

transmitterreceiver

body

shield layerfabric

© 2001-2002 Sony Computer Science Laboratories, Inc.

© 2001-2002 Sony Computer Science Laboratories, Inc.

© 2001-2002 Sony Computer Science Laboratories, Inc.

DiamondTouch [MERL2001]

• Identifies individual users• Requires external electrodes

(chairs)

© 2001-2002 Sony Computer Science Laboratories, Inc.

Near-Field data transmission

© 2001-2002 Sony Computer Science Laboratories, Inc.

Hand tracking without camera

© 2001-2002 Sony Computer Science Laboratories, Inc.

動作原理（ 2 ）

Transmitter(AVR)

I-Vamp

ampA/D

ConverterLPF

100-500KHz (5Vp-p)

1-10μA

analogswitch

© 2001-2002 Sony Computer Science Laboratories, Inc.

Wireless.11, BT etc.ID

© 2001-2002 Sony Computer Science Laboratories, Inc.

物体を介した操作

電極

© 2001-2002 Sony Computer Science Laboratories, Inc.

Application Domains

• Where to install?– Furniture, Floor, Wall– Wearables -- Clothing, Bag, Shoes– Tablet, Keyboard, Mouse, Gamepad– Mobile Devices– Robot

• How to use– as a primary input device– to augment existing interaction

© 2001-2002 Sony Computer Science Laboratories, Inc.

Multi-Hand/Finger Input

© 2001-2002 Sony Computer Science Laboratories, Inc.

複数の指による入力

© 2001-2002 Sony Computer Science Laboratories, Inc.

距離の判定

© 2001-2002 Sony Computer Science Laboratories, Inc.

© 2001-2002 Sony Computer Science Laboratories, Inc.

The Concept

• Beyond The GUI dogma (Mouse+Display)– Inspired by analogies between biological skin &

artificial surfaces

• As An Input Device– Touch, Proximity, Digital Data (Electrostatic/magnetic

coupling), energy

• As An Output Device– Pixel Spatial Pattern, Pixel Temporal Pattern (beacon),

Digital Data, energy, tactile, sound?

• As A Communication Device– Information, Motion, Gesture, Emotion, Intimacy

© 2001-2002 Sony Computer Science Laboratories, Inc.

なぜ Skin?

• 生物の「皮膚」の機能– 保護、形態保持– 入力（触覚）– 出力（表示）