Leveraging Conductive Inkjet Technology to Build a Scalable and Versatile Surface

for Ubiquitous Sensing

Nan-Wei Gong1,2, Steve Hodges2, Joseph A. Paradiso1,2

1MIT Media Lab, Responsive Environments Group, Cambridge, USA2Microsoft Research Cambridge, Sensors and Devices Group, Cambridge, UK

13th ACM International Conference on Ubiquitous Computing September 17-21, 2011

Sensate Surface - Electronic skins as Dense Sensor Networks

Sensate Media - Multimodal Electronic Skins as Dense Sensor Networks, Paradiso, J.A., Lifton. J., and Broxton, M., BT Technology Journal, Vol. 22, No. 4, October 2004, pp. 32-44.

ChainMail – A Configurable Multimodal Lining to Enable Sensate Surfaces and Interactive Objects, Mistree, B.F.T., and Paradiso, J.A., in Proc. of TEI 2010, Cambridge MA, January 25-27, 2010, pp. 65-72.

S.N.A.K.E.: A Dynamically Reconfigurable Artificial Sensate Skin (MS) August 2006

J. Lifton et al. “Experiences and Directions in Pushpin Computing” Symposium on Information Processing in Sensor Networks (IPSN05).

Tribble

ChainMail

Pushpin

SNAKE

Works from the Responsive Environments Group at MIT Media Lab

Flexile and stretchable electronicsRoll-to-roll process, widely used on LCD/OLED connector circuit for providing extra elasticity to the connection and embeds/holds the back panel ICs

JainK,Klosner M,Zemel M,Raghunandan S “FlexibleElectronics and Displays: High- Resolution, Roll-to-Roll, Projection ‐Lithography and Photoablation Processing Technologies for High- Throughput Production “(2005) Proc IEEE93:1500–1510‐

Low-cost Flexible Electronics

(a) Copper-on-Kapton substrate (allflexinc.com)(b) Conductive inkjet flex technology (conductiveinkjet.com/)(c) Inkjet printed electronics using metallic nanoparticle ink (T-ink.com)

~＄ 100 USD (30 cm2) ~＄ 10 USD (30 cm2)

Depending on the Material used

Using the body for signal transmissionYour noise is my command: sensing gestures using the body as an antenna. Gabe Cohn, Daniel Morris, Shwetak N. Patel, and Desney S. Tan. CHI 2011. EMS Synthi AKS, 1971

(picking up electric hum coupled in human body)

Zimmerman, T. G., Personal Area Networks: Near-field intrabody communication in IBM Systems Jour-nal, vol. 35, nos. 3&4, 1996, pp. 609-617.

DiamondTouch: a multi-user touch technology. Paul Dietz and Darren Leigh. UIST 2001.

Passive signal detection

Active signal transmit and receive

Sensate Floor Systems

Lee Middleton et al., A Floor Sensor System for GaitRecognition, AUTOID '05. Washington, DC, USA, 171-176.

a prototype floor sensor as a gait recognition system:1536 individual sensors arranged in a 3 x 0.5 m strip with an individual sensor area of 3 cm2.

6 x 10 foot mat surface atop a matrix of 64 pressure-sensitive piezoelectric (PVDF) wires, measures the position and intensity of footsteps, turning them into MIDI note events.

Paradiso, J., Abler, C., et al., The Magic Carpet: Physical Sensing for Immersive Environments. CHI 1997. ACM Press, pp.277-278.

Z-tiles: an array of force-sensitive resistors on each node to detect pressure, and that pressure information is output by way of a self-organized network formed by the floor nodes

Richardson, et.al., “Z-Tiles: building blocks for modular, pressure-sensing floor spaces. Human factors and computing systems, Vienna, Austria, pp. 1529–1532 (2004)

Our Hybrid Approach

30 cm

30 c

m

Traditional rigid PCB + flexible PCB

solder points

Overall Architecture and Construction

-GND-SDA-GND-SCL-GND-HEART BEAT-GND

Physical topology

Sensing Modalities

PCB module circuitry

Basic operation of each slave sensor unit

• The simplest and the most power-efficient mode for tracking people moving across the surface.

• Signals can distinguish walking direction, heal strikes and mid-swing -useful information for gait analysis.

•The four different colors in the right-hand figures represent the signals from the four different electrodes in one sensing tile.

Capacitive Sensing - Passive Mode

Gait Signatures from Passive Mode

Different signatures typically detected with the passive capacitive sensing method. (a) Forefoot strike, (b) heel strike pattern (left feet), (c) and (d) mid-swing between steps(right feet), detected by adjacent electrodes. The decay time is from the RC response of the envelope detector.

Capacitive Sensing – Active Mode

• Two possible scenarios when a user’s body comes into the electric field between transmit and receive electrodes – transmit mode and shunt mode.

Sample and average the charging and discharging amplitude change (voltage) for 32 cycles

Joseph A. Paradiso and Neil Gershenfeld, Musical Applications of Electric Field Sensing, Computer Music Journal 21(2), Summer 1997, pp. 69-89.

Capacitive Sensing – Active Transmit Mode

• Transmit mode dominates when direct contact with the transmit electrode.

(a) The user was touching the transmit electrode and moved from towards electrode (1). The strength of the signal pick-up is plotted as a function of distance.

(b) The electrode pattern of a single tile, where the electrode marked by the red dot served as the transmitter.

(c) Signal pickup on all the receive electrodes as a function of time

Capacitive Sensing – Active Shunt Mode

• The testing environment was set up on the floor, with ~4cm of high-dielectric constant foam on top of the sensors to avoid transmit mode.

• This effect is less marked than the passive sensing results (around 4 bits of resolution).

• During each step, the user effectively blocks the electromagnetic field flux, hence the signal drop: (a) heel strikes and (b) mid-swing.

• The red dots mark the transmit electrodes.

Piezoelectric Pickup

• Piezoelectric sensors was integrated for low power, passive detection of pressure and vibration.

• The signal can be used to trigger wake up of the microcontroller from a low power sleep node.

•Also infer the weight of a person and provide insight into gait dynamics.

•Vibration from adjacent units is perceptible.

Images from Digikey.com

Cellular signals versus localization and identification

13.56MHz NFC square loop antenna

900/1800MHz ¼ wavelength GSM antenna

Cutouts on the electrode eliminate Eddy currents that would decrease performance.

• The pattern and signal strength of NFC are consistent and can easily be used to determine range by measuring peak thresholds. • GSM signals have stronger signal response that can infer longer distance tracking by integrating and averaging the signal patterns.

(a) Signal response versus sensing unit location when a mobile device is held 1m from the surface. (b) Illustration of the experimental setup. (c) Close up of the antenna. (d) signal strength versus distance

Cellular signals versus localization and identification

NFC signal GSM signal

Example Applications

• Localization– Capacitive Sensing (~0.3 m), GSM (~0.8 m), NFC (~2m), Piezoelectric sensor

(~ 0.8m away)

• Identification– Gait signature

– Signals from the cellular network

• Gesture recognition• On-body information transmission

/exchange– Active tags on the shoe– Active signal transmission through

capacitive sensing

Conclusions and Future Works• Conclusions

– Our work presented a low-cost scalable and versatile distributed sensate surface based on a new conductive inkjet printing technology.

– We demonstrated the design and implementation of passive and active capacitive sensing, coupled with GSM and NFC RF signal pickup – all based on copper electrodes and antennas printed on the substrate.

– Pilot studies showed promising results which could change the way we think about covering large areas with sensors and associated circuitry.

• Future work

– designing minimum circuitry for direct surface mount.

– experimenting manufacturing processes for fast assembly.

– Develop a sheet of modular printed sensors and circuitry which can be scalable and adaptive for various applications.