SUNGJAE HWANG | EXP LAB

PseudoSensor: Emulation of Input Modality by

Repurposing Sensors on Mobile Devices

Keywords: sensor emulation; sensor repurposing; pressure; tactile; mobile device

PUBLICATIONS (RELATED TO THE THESIS)

S. Hwang, K.W. Wohn, “Designing Magnetically Driven Tangible Interfaces - Journal of Human Computer Studies”, Journal of Human Computer

Studies (SCI), 2015 – (Submitted)

S. Hwang, K.W. Wohn, "PseudoSensor: Emulation of Input Modality by Repurposing Sensors on Mobile Devices", Journal of Intelligence and Smart

Environments (SCIE), 2015 - (Accepted)

S. Hwang, D., Kim, S., Leigh, and K., Wohn, “NailSense: Fingertip force as a new input modality," ACM Symposium on User Interface Software and

Technology (UIST Poster), 2013

S. Hwang, M. Ahn, K.W. Wohn, "MagGetz: User Configurable Tangible Controllers On and Around Mobile Devices", ACM Symposium on User

Interface Software and Technology (UIST), 2013 - 19%

S. Hwang, A. Bianchi, K.W. Wohn, "VibPress: Enabling Pressure-Sensitive Interaction using Vibration Ab-sorption on Mobile Device", International

Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI), 2013 - 21%

S. Hwang, A. Bianchi, M. Ahn, K.W. Wohn, "MagPen: Magnetically Driven Pen Interactions On and Around Mobile Device", International Conference

on Human-Computer Interaction with Mobile Devices and Services (MobileHCI), 2013 - 21% (Best paper award)

S. Hwang, K.W. Wohn, "VibroTactor: Low-cost Placement-Aware Technique using Vibration Echoes on Mobile Devices", ACM International

Conference on Intelligent User Interfaces (IUI Poster), 2013

S. Hwang, Bianchi, A., K. W. Wohn, "MicPen: Pressure-Sensitive Pen Interaction Using Microphone with Standard Touchscreen", The ACM SIGCHI

Conference on Human Factors in Computing Systems (CHI EA), 2012

S. Hwang, K. W. Wohn, "Pseudo Button: Emulation of Touch Sensors by using Microphone on Mobile De-vice", The ACM SIGCHI Conference on

Human Factors in Computing Systems (CHI EA), 2012

CURRICULUM VITAE (Continued)

CHAPTER 1

INTRODUCTION

• Current mobile devices are equipped with a variety of sensors, offering numerous

input channels for expressive interaction techniques.

• Researchers have leveraged on these capabilities for creating new input

interaction modalities to enhance the user experience and explored the design

space to utilize them.

BACKGROUND

(http://www.noahlab.com.hk/labvision/research_hci)(http://www.qualcomm.com/news/snapdragon/2014/04/24/behind-

sixth-sense-smartphones-snapdragon-processor-sensor-engine)

Issue 1. Some sensors are impractical for small devices (e.g., wristwatches, glasses,

rings) due to the limited computing power and small interaction area available.

Issue 2. Some sensors (e.g., pressure, touchscreens, humidity) are seldom available

on mobile devices today; instead they have to be attached to the device as bulky

accessories, which ordinary users have seldom available (Maragos, Potamianos, & Gros, 2008).

Miyaki et al., 2009, Essl et al., 2009, Heo et al., 2011, Wilson et al., 2013

BACKGROUND : ISSUES

Images from Google, Fin ring, Pebble, and Samsung website (Nov. 20. 2014)

(a) (b)

Issue 3. Augmentation of input hardware causes additional production

costs for device vendors and additional maintenance cost for users.

http://www.ifixit.com (Nov. 20. 2014)

BACKGROUND : ISSUES

“What if we emulate unavailable sensors through available resources on mobile device?”

“What if we generate unavailable sensors through a software approach?”

This study starts from the question,

BACKGROUND : RESEARCH QUESTION

Schenkman, B. N., and Nilsson, M. E. (2010). "Human echolocation: blind and sighted persons’ ability to detect

sounds recorded in the presence of a reflecting object," Perception (39:4), p 483.

Hint from Human : Human echolocation

A blind person recognizes surrounding objects by detecting

sound recorded in the presence of reflecting objects.

BACKGROUND : RESEARCH QUESTION

GOAL 1.

To present a concept that repurposes input resources of the mobile device.

GOAL 2.

To empirically prove this method through various instance applications.

GOAL 3.

To build a unified set of guidelines that a broad range of HCI could utilize.

RESEARCH GOAL

RESEARCH STRUCTURE

CHAPTER 2

LITERATURE REVIEW

PRESSURE-BASED INPUT METHOD FOR MOBILE DEVICES

1. It adds a degree of freedom to the touch locations (Boring, Ledo, Chen, Marquardt, Tang, &

Greenberg, 2012) and so it can free the user from spatial restrictions and repetitive

movement (Miyaki, & Rekimoto, 2009).

2. Pressure input allows relatively stable and accurate interactions when user is in

mobile context (Wilson, Brewster, Halvey, Crossan, & Stewart, 2011).

3. Pressure input supports in-pocket operation. For instance, user can interact with

the device when it is in the pocket or bag.

4. It can be used to alleviate occlusion problems and can also provide ways for rich

contextual selections (Ramos, Boulos, & Balakrishnan, 2004).

Several advantages of pressure input for mobile device

Previous work on pressure input techniques for mobile devices either

introduced specialized sensing hardware or relied on software to estimate

the pressure from sensors commonly available on mobile devices

HARDWARE-AUGMENTATION APPROACH TO ENABLING PRESSURE INPUT

GraspZoom

(Miyaki et al., 2009)

Pressure-based Text Entry

(Brewster et al., 2009)

Squeezing the Sandwich

(Essl, 2009)

Pressure-based Menu

Selection

(Wilson et al., 2010)

ForceGestures

(Heo et al., 2011)

Indirect Shear Force

(Heo et al, 2013)

Multi-digit Pressure Input

(Wilson et al., 2012)

ForceDrag

(Heo et al., 2012)

HARDWARE-AUGMENTATION APPROACH TO ENABLING PRESSURE INPUT

SOFTWARE APPROACH TO ENABLING PRESSURE INPUT

A tangible controller

(Kato et al., 2009)

Vision-based Force Sensor

(Sato et al., 2012)

GripSense

(Goel et al., 2012)

The Fat Thumb

(Boring et al., 2012)

c c t g

Muscle Tremors

(Strachan et al., 2004)

Use the Force or something

(Essl et al., 2010)

ForceTap

(Heo et al., 2011)

Expressive typing

(Iwasaki et al., 2009)

c t a gcamera touchscreen accelerometer gyroscope

As a response to the limitation of hardware augmentation approach, some authors have attempted to

estimate input pressure with software, using readings from sensors available on most mobile devices.

a a a a

Sonicstrument

(Lee et al., 2011)

VibroTactor

(Hwang et al., 2012)

SoundWave

(Gupta et al., 2012)

Biomolecule detection

(Won et al., 2012)

WiSee

(Pu et al., 2013)

Medical Mirror

(Poh et al., 2011)

LIghtWave

(Gupata et al., 2011)

uTouch

(Chen et al., 2014)

m m m e

t w c e

m e t cmicrophone EMI touchscreen camera

OTHER SOFTWARE APPROACH TO EMULATING INPUT MODALITIES

LIMITATION OF PREVIOUS WORKS

1. Limitation of Hardware augmentation approach

- They have to be attached to the device in the form of bulky accessories, which

are seldom available to ordinary users (Maragos et al., 2008).

- The addition of hardware can mean additional production and maintenance

costs for both device vendors and users.

2. Limitation of Software approach

- Some do not measure pressure applied by the user continuously (Essl et al., 2010;

Heo et al., 2011b; Iwasaki et al., 2009) while others are fixed and limited with regard

to the location of cameras (Kato et al., 2009; Sato et al.).

- They only focus on local problems. The general method for repurposing sensors

have not yet investigated in HCI field.

CHAPTER 3

PROPOSED CONCEPT : PSEUDOSENSOR

PROPOSED CONCEPT : PSEUDOSENSOR

PseudoSensor is a sensor emulated

- using a (built-in) sensor or combination of (built-in) sensors

- for a different purpose from their original one (emulation of other functionality)

- without additional cost (sensors added)

PseudoSensorSensor Sensor

PROPOSED CONCEPT : PSEUDOSENSOR

Our approach is to overcome the device constraints (e.g., absence

or lack of sensors) by bypassing the effect the user create (e.g.,

pressure) to the emulated sensor (PseudoSensor).

Motor skillTouchscreen

HumanComputer

Touch Effect

device

constraints

PseudoSensorPressure

Pressure Effect

Modalities, constraints, and effects (Obrenovic, Abascal, & Starcevic, 2007)

PROPOSED CONCEPT : PSEUDOSENSOR

Active vs. Passive

Event-based vs. Streaming-based vs. Recognition-based modality

Unimodal vs. Multimodal

Followed by the Simplified model of computing modalities

(Obrenovic, Abascal, & Starcevic, 2007)

PseudoSensorSensor Sensor

THE HIERACHY OF PSEUDOSENSOR

CHAPTER 4

INSTANCES OF PSEUDOSENSOR

The Right Number of Pressure Levels

- ≤ 6 distinct levels (Ramos, Boulos, & Balakrishnan, 2004)

Selection Technique

- Dwell, Quick Release, Stroke, and Roll.

Feedback Design

- Continuous visual feedback is needed for a pressure widget (Ramos, Boulos, & Balakrishnan, 2004)

Perceptual Characteristics of Human Kinesthetic System

- The differential threshold of force is 7% - 10% (over a force range of 0.5-200 N), while that of

stiffness is 17% (Jones, 2000).

CONSIDERATIONS FOR DESIGNING PRESSURE INPUT TECHNIQUES

INSTANCES OF PSEUDOSENSOR

Eight applications to demonstrate how our approach supports

pressure interaction for mobile devices.

a camera a microphone an accelerometer a magnetometer

NailSense and CamPress MicPen and PseudoButton ForceTouch and VibPress MagGetz and MagPen

Pressure

Input

PRESSURE ESTIMATION BY REPURPOSING A CAMERA

Two applications that repurpose a camera to emulate a pressure sensor

NailSense is a novel interaction technique that repurposes a camera sensor to emulate a pressure

sensor. This technique allows users to control a mobile device by hovering and slightly

bending/extending fingers behind the device. It determines the pressure applied with a user’s

fingertip by tracking changes in coloration of a user’s fingernail with a built-in camera.

CamPress is another example that detects pressure asserted on a mobile phone by utilizing an

inertial camera on the mobile device. Our technique infers the amount of pressure applied on the

inertial camera of the mobile device by measuring the luminance of reflected light.

NailSense CamPress

MicPen is a pressure-sensitive pen interface that infers pressure applied on the pen by analyzing

the scratch sound. When the rubber tip of the pen is dragged across the glossy display, friction

produces a sound easily captured by a microphone and uses it to estimate the amount of pressure

ap-plied with the pen.

PseudoButton is another application that senses pressure applied on a pin-hole by repurposing a

built-in microphone on mobile devices. To emulate a pressure sensor, the system emits inaudible

sounds through the built-in speaker and analyzes the feedback through the built-in microphone on

the device.

MicPen PseudoButton

PRESSURE ESTIMATION BY REPURPOSING A MICROPHONE

PseudoButton (Active)MicPen (Passive)

The software of MicPen receives the sound captured by the microphone and computes a FFT using a

non-overlapping rectangular window sampled at 44 KHz. The software computes the sum of the

amplitude values in the range 15-32 KHz and uses it as a rough estimation for pressure.

In case of PseudoButton, the software generates ultrasonic sound (16.7K Hz) and simultaneously

captures the sound signal to emulate a pressure sensor. The software performs FFT algorithm on

incoming sounds using a non-overlapping rectangular window.

PRESSURE ESTIMATION BY REPURPOSING A MICROPHONE

VibPressForceTouch

ForceTouch is a software technique that enables pressure-like input interaction with a mobile device

by measuring the physical movement from accelerometer readings to estimate the force with which

the screen of a mobile device is tapped.

VibPress is another software technique that enables pressure input interaction on mobile devices by

measuring the level of vibration absorption with a built-in accelerometer when the device is in

contact with a damping surface (e.g., user’s hands).

PRESSURE ESTIMATION BY REPURPOSING AN ACCELEROMETER

PRESSURE ESTIMATION BY REPURPOSING AN ACCELEROMETER

Amount of pressure on a mobile device can be approximated by using an

accelerometer to measure the spatial displacement generated when the device is

touched (ForceTouch) or when the internal vibration motor vibrations (vibPress).

Light press (impact)

Hard press (impact)

VibPress (Active)ForceTouch (Passive)

touch

ed

touch

ed

touch

ed

touch

ed

MagPenMagGetz

MagGetz is an input technique that enables a pressure-sensitive interaction on and around mobile

devices without requiring power or wireless connections. This is achieved by tracking and analyzing

the magnetic field generated by magnetic controllers attached on and around the device through a

single magnetometer, which is commonly integrated into smartphones today.

MagPen is a magnetically driven pen that enables pressure sensitive interaction on the touchscreen

of the mobile devices. This technique is achieved using commonly available smartphones that

detect touch position with a touchscreen and analyze the magnetic field produced by a permanent

magnet embedded in a standard capacitive stylus with a magnetometer.

PRESSURE ESTIMATION BY REPURPOSING A MAGNETOMETER

Magnetic traces for pressure-sensitive control widgets (MagGetz)

PRESSURE ESTIMATION BY REPURPOSING A MAGNETOMETER

where p1 is a last reference point (e.g., a point when the

button is fully pressed), p0 is a first reference point (e.g.,

a point when the button is not pressed), and x’ is a vector

that new point x projected onto p0 p1.

where x is a new magnetic point according to the user’s

input, pa is a reference point of button a with maximum

pressure, pb is a maximum point of button b, and p0 is a

reference point of button a and b with minimum pressure.

Button a Button bButton a

We used a relative position between the two curves as a coarse proxy of pressure

(The farther the distance is, the denser the pressure levels).

110 uT

70mm

PRESSURE ESTIMATION BY REPURPOSING A MAGNETOMETER

MagPen (Passive & Multimodality)

EVALUATION

Goal

The goals of this experiment were to verify how fast and accurate our input techniques are for

different input levels and to show our approach can be successfully integrated for

conventional mobile devices.

Participants

- 10 volunteers, 6 males and 4 females between 27 and 34 (average 30.2, SD 2.30).

- A mix of researchers, students and professionals.

- Compensated for their time with a small gift (about $10).

Procedure

- Given a minute for familiarizing

- Calibration (to establish the minimum and maximum pressure that user could exert with it)

- Counterbalanced order

- A post-experiment interview.

- The entire experiment lasted approximately 100 minutes per user.

EVALUATION

We tested our applications for three input levels (2, 4, and 6 input levels, referred to as L2, L4, L6)

excluding two applications that use binary inputs (NailSense and CamPress). To report usability

metrics (completion time and error rate), the device keeps several data logs including participant’s

name, type of interface, input level presented, number of trial, time to reach a target box, time to

complete trial, target box presented, and target box selected while conducting the experiment.

The user selects a specific target using one of three selection techniques: selecting when

maintaining the cursor within the target box for one second (Dwell); selecting the target box when

quickly lifting (Quick release); or selecting the target box when touch event occurs (Touch).

RESULT : ERROR RATE (%)

The figure shows the error rate for all different input techniques of our applications. As can be seen,

the participants made few se-lection errors and errors increases as input level increases. The

overall error rate for levels L2, L4, L6 are, respectively, 3.85% (SD=0.45%), 10% (SD=0.65%), and

19.16% (SD=1.42%). Overall error rates for different interfaces in L2 are 0% for VibPress (SD=0%),

0.8% for MicPen (SD=0.26%), 1.7% for MagPen (SD=0.53%), 1.7% for ForceTouch (SD=0.35%),

5% for PseudoButton (SD=0.58%), 7.5% for CamPress (SD=0.73%), and 13.3% for NailSense

(SD=0.9%) in an ascending sequence. A two-way ANOVA analysis revealed significant differences

for errors across input levels (F(2,180)=29.29, p<0.01) and interaction techniques (F(7,180)=10.38,

p<0.01), and an interaction was found (F(10,180)=3.56, p<0.01).

RESULT : COMPLETION TIME (SEC)

The completion time increases as input level increases. are, respectively, 1.4 (SD=0.34), 1.73 (SD=0.14),

and 2.30 (SD=0.13) seconds. There were also differences in completion time in terms of different input

techniques. The average completion time for different interactions in L2 was: 0.91 seconds for MicPen

(SD=0.13); 1.24 seconds for VibPress (SD=0.04); 1.31 seconds for MagGetz Button (SD=0.07); 1.38

seconds for MagPen (SD=0.17); 1.45 seconds for PseudoButton (SD=0.32); 1.5 seconds for CamPress

(SD=0.26); and 2.03 seconds for NailSense (SD=0.7) in an ascending sequence. A two-way ANOVA

revealed significant differences for completion time across input levels (F(2,153)=112.4, p<0.01),

interaction techniques (F(6,153)=29.5, p<0.01), and an interaction was found (F(8,153)=5.98), p<0.01).

SUMMARY OF OUR APPLICATIONS

FINDINGS FROM THE EVALUTATION

The result is compelling when compared to previous works.

- We showed evidence that, with continuous visual feedback, users can reliably and quickly input using two to six

pressure levels with accuracy ranging from 3.85% to 19.16% and from 1.4 to 2.3 seconds interaction time,

depending on the input levels (L2~L6).

- All techniques showed selection times similar to those obtained with specialized hardware (Cechanowicz et al.,

2007; Ramos et al., 2004; Wilson et al., 2010) and MicPen, ForceTouch, VibPress, MagGetz Button, and MagPen

showed fewer errors than software pressure techniques (Goel et al., 2012; Heo et al., 2011b)

The errors and completion time varied depending on the number of

input levels.

- Any subsequent increment took at least 1.2 times longer with considerably more errors.

The errors and completion time varied depending on the type of

interactions.

- MagGetz Button (1.1%) vs. ForceTouch (20.3%)

- Repurposing a camera is cumbersome since it is easily affected by ambient light.

CHAPTER 5

GUIDELINE FOR SENSOR REPURPOSING

GRAMMAR OF PSEUDOSENSOR

HOW TO FORMALIZE THIS PSEUDOSENSOR?

Human.Motor<Pressure> → Computer.Microphone<Volume>

AttributeOperatorEntity Modality Entity Modality

Event ::= Entity[‘.’Modality]‘<’AttributeList’>’ (Operator Entity[‘.’Modality]‘<’AttributeList’>’)*

Entity ::= ‘Human’ | ‘Computer’ | ‘Environment’

Modality ::= ‘Microphone’ | ‘Accelerometer’ | ‘Camera’ | ‘Vibrator’ | ‘Motor’ | …

AttributeList ::= Attribute (‘,’ Attribute)*

Attribute ::= ‘Gesture’ | ‘Pressure’ | ‘Color’ | ‘Light’ | ‘Movement’ | ‘Intensity’ | ‘Vibration’ | …

Operator ::= ‘→’ | ‘+data’ | ‘+feature’ | ‘+decision’

FORMALIZATION OF PSEUDOSENSOR

Extended Backus Naur Form (EBNF) grammar to define the syntax of PseudoSensor.

Human.motor<Touch, Pressure> → ( Computer.touchscreen<Touch> + Computer.accelerometer<Movement> )

Human touch and pressure affects touch attribute of a touchscreen

and movement attribute of an accelerometer (ForceTouch)

Human.motor<Pressure> → Computer.accelerometer<Physical movement>

Human pressure affects the movement of accelerometer (repurposing an accelerometer).

Human.motor<Pressure> → ( Computer.speaker<Sound> → Computer.microphone<Spectrum, Volume> )

Human pressure affects the relationship in the way an inaudible sound from a speaker affects the

spectrum and volume attributes of microphone (repurposing a microphone).

Human.motor<Touch, Pressure> → ( Computer.touchscreen<Contact position> +feature

( Magnet<Intensity> → Computer.magnetometer<Intensity> ) )

The human touch and pressure affects the contact position of touchscreen and the relationship in the

way a purse of magnet affects the intensity of magnetometer

(repurposing a touchscreen and a magnetometer).

FORMALIZATION OF PSEUDOSENSOR

An overview of all applications we developed AN OVERVIEW OF ALL APPLICATIONS WE DEVELOPED

An Overview of Software Approach to Enable Pressure InteractionAN OVERVIEW OF PREVIOUS WORKS THAT ENBALE PRESSURE INPUT

AN OVERVIEW OF PREVIOUS WORKS FOR REPURPOSING SENSORS

SUMMARY OF DESIGN GUIDELINE FOR REPURPOSING SENSORS

1. PseudoSensor can be empowered by adding additional sensors

MicPen can be improved by compensating for sound energy variance according to the touch position

and the speed of dragging measured by additional inertial sensor, a touchscreen

2. PseudoSensor, in certain aspects, sometimes demonstrate a

better performance than a hardware-augmentation approach.

VibPress technique expands its interaction area beyond the touchscreen and enables in-pocket

interaction, and the NailSense technique enables pressure-sensitive interaction in the air.

3. Active PseudoSensor using public output channels may

influence the performance of other systems that use the same

technique.

PseudoButton, MagGetz, and MagPen use public output channels (e.g., sound from a speaker and

magnetism from a magnet) to build systems. Thus, we should consider the interference between

systems for active Pseudosensor using public channels.

CHAPTER 6

CONCLUSION

CONCLUSION : SUMMARY OF STUDY

1. We presented a novel method that overcomes the limitations of

hardware by emulating unavailable sensors through available input

resources. This concept is unique in that it has not yet been

introduced and summarized in the HCI field.

2. We presented a concrete set of applications and offered empirical

evidence through a set of evaluations. The results of the evaluation

are an important aspect of our work’s contribution to the field.

3. Through an exploration of both our own examples and the corpus of

related work, we established a set of design guidelines. These

unique guidelines can provide designers with flexible and reusable

solutions when faced with insufficient sensor resources or suboptimal

conditions.

1. PseudoSensor is limited in terms of hardware settings.

Since PseudoSensor relies on inertial sensors of the device, a limitation exists in terms of hardware

settings: VibPress technique (compatibility & battery consumption issue), and PseudoButton

technique (spatial constraints).

2. PseudoSensor affects the performance of other systems.

When PseudoSensor is active and using public output channels, it may influence the performance of

other systems: PseudoButton, MagGetz, and MagPen (interference issue).

3. PseudoSensor may cause negative effects on user experience.

Depending on a form of PseudoSensor, it may cause negative effect on user experience: VibPress

technique (fatigue and noisy sound), and CamPress technique (privacy concerns).

4. There is no comparison with the previous hardware-augmentations.

We have empirically proved our method through various applications, but there is no comparison with

the previous hardware-augmentation approaches. The comparison is needed.

CONCLUSION : LIMITATIONS

Expanding our vision by exploring a wider range of applications

using our technique.

- Various modalities repurposed and sensor emulated (e.g., EMG, EEG, humidity)

- Different devices and domains (e.g., wearable, robot, and Internet of Things).

Conducting a longitudinal study and comparing them with

corresponding real sensors.

More detailed investigation on the effect of repurposing sensors.

CONCLUSION : FUTURE WORK

Images from Google, Fin ring, Pebble, Samsung, and Amazon website (Nov. 26. 2014)

THANK YOU

Q & A