MIT General Exams. Grounding interfaces: shifting the body boundaries, 2008.

GROUNDING INTERFACESShifting the body boundaries

Oral Exam by Cati Vaucelle14th of November 2008

Committee:

Professor Hiroshi IshiiProfessor Edith AckermannProfessor Joe Paradiso

GROUNDING INTERFACES: Shifting the body boundaries �

INTRODUCTION


Oral ExamCati Vaucelle Tangible MediaMIT Media Lab14-11-2008

GroundingTaking something that’s ephemeral and pulling it down into reality and foundation.

Grounding InterfacesPhysical interfaces that actively construct a relationship between the digital and the physical body of an individual.

Part I - Motivation[] Inspiration[] Beyond Picture This![] Lifecycle[] Learning[] Communication[] Connection[] Mindful Learning[] Motivating Feedback

Part II - Oppportunities[] Percepts Vision . Smell . Hearing . Touch[] ConceptsMemory . Intelligence . Identity[] ConductsFeedback . Perspective . Friendship

Part III - Vision[] Where do I stand?[] Dimensions in my work[] Picture This![] Hap-bit[] Synthesis

Part IV - Sensing Technologies[] Exchange information[] Gesture communication[] Piezo vibration sensor[] Harware vs software[] Generalizing Picture This![] Other methods[] Mapping

Introduction Conclusion

Definitions

By Hannah HoneypieLiving (2008).



The dolls become the body, and then becomes the video. The doll is a gesture object.

Grounding via body and visual perspectives.

Physical body Digital body

Video of the dollThe body holds the dolls

Picture This! (2007).






ExamplePicture This! a grounding object



The child’s anthropomorphism makes the objects become part of the child’s physical body.

Combining the gestures with the objects grounds the child in his physical world while playing with a digital body. The physical body is then directly connected to the digital body via the doll (a gesture object).






ExamplePicture This! a gesture object








Semiotic squareGrounding Interfaces



Tangible interaction techniquesMain area with Dr. Hiroshi Ishii

Psychology, cultural artifacts and semioticsContextual area with Dr. Edith Ackermann

Technologies for body sensing and expression Technical area with Dr. Joe Paradiso






Committee



Interaction techniques are key to pushing the way people engage with digital information.

Psychology is key to an interaction that is defined by the nature of human attention, cognition, and perceptual abilities.

Sensing technology is the key to translating a user’s environment as part of new applications.






Committee


MOTIVATION

GROUNDING INTERFACES: Shifting the body boundaries 10


Is it really so sad and dangerous to be fed up with seeing with your eyes, breathing with your lungs, swallowing with your mouth, talking with your tongue, thinking with your brain…

Why not walk on your head, sing with your sinuses, see through your skin….

Where psychoanalysis says, stop, find your self again, we should say instead, let’s go further still! Find your body without organs. Find out how to make it. It’s a question of life and death!

–Gilles Deleuze and Felix Guattari, A Thousand Plateaus: Capi-talism and Schizophrenia.






Inspiration

GROUNDING INTERFACES: Shifting the body boundaries 11


Imagine when one can say:

“the technology delivered him from all his automatic reactions and restored him to his true freedom.”

Is it really so sad and dangerous to be fed up with seeing with your eyes, breathing with your lungs, swallowing with your mouth, talking with your tongue, thinking with your brain…

Why not walk on your head, sing with your sinuses, see through your skin….

Where psychoanalysis says, stop, find your self again, we should say instead, let’s go further still! Find your body without organs. Find out how to make it. It’s a question of life and death!

–Gilles Deleuze and Felix Guattari, A Thousand Plateaus: Capitalism and Schizophrenia.






Inspiration

GROUNDING INTERFACES: Shifting the body boundaries 1�


We have already partially discovered a body without organs by creating:

[] A digital copy of oneself.[] A copy that will coexist online starting from birth.[] A copy that will perpetuate its existence after death, living in a digital reality.

This copy will be our digital body, a body without organs.

The Human Speechome Project by Deb Roy. Record every minute of a baby’s life, 2006.

Socioscope by Alex Pentland, 2006.Track location and conversation patterns.

MyLifeBits by Gordon Bell, 2003.Capture everything and going paperless.

The goal of Lifelogging: to record and archive all information in one’s life.






Digital BodyBeyond Picture This!



[] Google indexes our name and we can be found at any time!

[] Even after our death, we remain on the Google search engine!

[] If Google sentences us to an electronic death, we cease to exist online and we cease to be found!






Digital BodyLifecycle



50 years agoProgressive learning under the family’s supervision.

NowVery fast “learning” curve. Children believe that one can be a champion of Karate without moving a finger. However there is almost no common point between physical and digital Karate.

Despite a learning curve, there is no limitation from the physical body.

ReferenceGauthier, J. M. and Moukalou, R. (2007) De la guerre des boutons a Harry Potter: Un siecle d’evolution de l’espace-temps des adolescents, pp 5-16 and pp153-173.

A champion of Karate!






Digital BodyLearning



50 years agoProgressive learning under the family’s supervision.

NowVery fast “learning” curve. Children believe that one can be a champion of Karate without moving a finger. However there is almost no common point between physical and digital Karate.

Despite a learning curve, there is no limitation from the physical body.

You will never be denied the opportunity of being a digital Karate champion because you are physically weak!






Digital BodyLearning

ReferenceGauthier, J. M. and Moukalou, R. (2007) De la guerre des boutons a Harry Potter: Un siecle d’evolution de l’espace-temps des adolescents, pp 5-16 and pp153-173.

A champion of Karate!



[] No need to interact through body language!

[] It might avoid miscommunication!

[] Now we coommunicate via our digital body, on social networks controlling the level of engagement and disengagement.

[] Construct your digital body *free of risk*!

With the computer, the relationship between dis-tance and proximity, direct communication, corporeal versus indirect and me-diation is transformed. (Gauthier and Moukalou, 2007).

Twins in tree by Rodney Smith, 1999.In face to face communication, distance between people can be an obstacle.

ReferenceHall, E. T (1973) The Silent Language. A Doubleday Anchor Book.






Digital BodyCommunication



[] Children with atypical interests find other children with the same hobbies.

[] By connecting to someone else, one can relate and create bonds, can project onto someone else’s life and empathize.

[] Connecting with a digital body might sharpen our perception through sharing perspectives.

Moving Pictures, 2004.Children capture media to share with children who live in a different city.

ReferenceHarel, I. and Papert, S. (1991) Constructionism, Ablex Publishing Corporation.






Digital BodyConnection



Walking through the Internet requires:

[] being open to novelty[] being alert to distinction[] being sensitive to different contexts[] being aware of multiple perspectives

Scratch, Resnick et al. (2008).

ReferenceLanger, E. J. (1998) The Power of Mindful Learning. Published by Da Capo Press, 1998

The Internet teaches children to adapt themselves to new situations rather than providing them with conditional learning -- fitting themselves to an external norm (Langer, 1998).






Digital BodyMindful Learning



[] A digital body motivates us in exercising!

[] Joysticks that vibrate are ironic: they remind us of our body! As if the joystick said: Remember! You had a body!

[] This is not entirely grounding: this is the difference between a reminder that your hands exist and giving someone tools, asking them to craft a piece of furniture.

My virtual model!Free weight-loss calculator on a virtual model, Glamour magazine, 2007.

Wii Fit, 2008.






Digital BodyMotivating Feedback

GROUNDING INTERFACES: Shifting the body boundaries �0


Delicate Boundaries by Chris Sugrue.Using human touch to dissolve the barrier of the computer screen.






Things get lost with de-materialization.

Where are the senses?

The digital world gives us power at the same time that it disconnects us from our physical body.

We connect to the world using our senses.

To have the benefits of technology without loosing our connections to the body, technology needs to bridge to our senses, grounding us in our surroundings.

Only being grounded in experience can give birth to new ways of living and new dreams.


OPPORTUNITIES

GROUNDING INTERFACES: Shifting the body boundaries ��







OpportunitiesPercepts - Concepts - Conducts



Observation [] Current displays restrict vision to one depth of field, supporting visual fatigue and myopia.[] Stereoscopic vision rehabilitates the effects of looking at one depth of field.

MechanismTrain stereoscopy as new means of environment interaction.

OpportunitySee all surfaces in 3D with your own eyes!

Tri-Vision, a stereo camera, (1946-49).






PerceptsVision



ObservationSmell is key to past experience and is unexplored.

Mechanism [] Collect senses of smell to create and recall stories. [] Associate memories of smells, places and objects.

OpportunitySmell the grass by reading a book and remember an afternoon tea party!






PerceptsSmell



ObservationThe spectrum of high frequencies that one can hear drops off with age.

MechanismEncode information at frequencies discernible by some populations and not others.

OpportunityHear secret messages that no one else can hear!






PerceptsHearing



ObservationTouch interfaces between the physical space and the nervous system.

Mechanism [] Injectable electrodes at the peripheral limb where the leg has been amputated. [] Electrodes stimulate the nervous system, actuated by a prosthetic

OpportunityFeel the sand from a foot that doesn’t exist!

ReferenceAaron R., Herr H., Ciombor D., Hochberg L., Donoghue J., Briant C., Morgan J., Ehrlich M.(2006). Horizons in Prosthesis Development for the Restoration of Limb Function, Journal of the American Academy of Orthopaedic Surgeons. 14 (10): 198-204.






PerceptsTouch



ObservationMemory aids could both allow us to focus on more important matters, or destroy our ability to recall.

MechanismDevelop a taxonomy of cognitive tasks to determine when cognitive aids liberate higher cognitive functions.

OpportunityThe computer does low level tasks while you create bridges between disciplines!

Externalizing body functions with technology might be liberating (Serres, M.).

People use pen and paper to simplify cognitive tasks! You could just lean on the structure of the environment to remember!

Putting brain, body and world together in epistemic action! (Andy Clark).






ConceptsMemory



Observation [] Humans often outperform technology in pattern recognition. [] Technology outperforms humans in raw calculation.

MechanismAssist a player in becoming a better calculator, and defer to the person what she is uniquely expert in.

OpportunityBecome a superhuman computer!

A Go competition. MoGo (2008). MoGo beat Myungwan Kim who is an 8-dan master on a 19 x 19 board (most previous wins by computers were done on smaller amateur level 9 x 9 boards). Kin gave the computer a nine-stone handicap.






ConceptsIntelligence as calculation



Observation [] A majority of westerners think that as we get old, we don’t remember as well. [] Langer showed that the degree of memory loss is dependent on how we value aging!

MechanismStructure societal change in the conception of value in aging.

OpportunityRevisit every stereotype you can think of!






ConceptsIntelligence as adaptation



Observation [] Some role playing games have more than 10.9 million monthly subscribers! [] In RPG we are equipped with sonic hearing, super vision, abnormal strength!

MechanismUse virtual environments to ask people to rediscover what they can do with their senses in the real world.

OpportunityBe an everyday super hero!

Borderland by Lucien Alma and Laurent Hart, 2001.

“Borderland” is based on video game duels such as “Tekken” or “Mortal Kombat”. They use ordinary people as the characters: old ladies, tramps, workmen to fight each other against the backdrop of the suburbs of Paris that have become a desolated no-man’s land.






ConceptsIdentity



ObservationUngrounded, “magical” interaction, romanticized via video games and TV, deny the power of real-life cause and effect.

MechanismCombine children’s physical limits with digital action and reward to reinstate the relationship between cause and effect.

OpportunityFeel your own body limits when you play video games!

50 years ago children had a direct understanding and relationship to cause an effect of a physical action.

In Harry Potter they use a “magic wand” to confront others!






ConductsFeedback



Observation50 years ago, kids with ADHD were given means to catalyze their energy (Golse). Now quick fix: pharmaceutical treatment.

MechanismKids with ADHD can focus if they vary the target of their attention in their own minds (Langer).

OpportunityPlay hopscotch in a museum and recall all the details of a Botticelli painting!

You are getting better day after day by Broadhong Design, 2006.

Designers envisioned the physical representation of the healing process as part of a pharmaceutical treatment.






ConductsPerspective



ObservationWith social networks, we build “instant” friends. We are perpetually connected but out of touch!

MechanismSocial networks could better support realistic relationships by monitoring and reflecting activity between users.

OpportunityLive on Mars and have millions of real earth friends!






ConductsFriendship



The link between the dematerialized digital body and sensory opportunities is:

Grounding

Creating a shift in perspectives, from a new discovery of the environment to playing with our senses.

Don’t just passively deliver information or stimulate the senses, actively construct a relationship between digital and physical body in the individual!

Squeeze Me! (2006) This vest simulates therapeutic holding to “ground” patients with sensory issues.

Grounding the body: the Me ++






Things get lost with de-materialization.

Where are the senses?

The digital world gives us power at the same time that it disconnects us from our physical body.

We connect to the world using our senses.

To have the benefits of technology without loosing our connections to the body, technology needs to bridge to our senses, grounding us in our surroundings.

Only being grounded in experience can give birth to new ways of living and new dreams.


VISION



[] Between being nostalgic of the past and being a technophile.

[] The ones who innovate create new bridges between the past and the future.

[] Each time one transcends, one builds a hybrid with a nostalgic vision from the past/present and from the real/virtual.

Wallmaster & Home transporter by Antal Lakner, 1999.

Everyday activities are transformed into indoor sport activi-ties!






Where do I stand?



There are four consistent dimensions that comprise my work:

Stance in the world.Perspective.Time.Space and proxemics.

All these dimensions merge to ground the body in the physical environment while interacting with a digital body

STANCE IN THE WORLDI design mechanisms of interaction to enable users to position themselves in their environment.

PERSPECTIVEI design sensing technology to translate the undiscovered environment, and create a shift in perspective.

TIMEI aim to synchronize users with their internal motion and with life. Users take time and go through an iterative process of discovering / understanding / reflecting.

SPACE AND PROXEMICSI design systems for users to understand spatially their environment, where they are situated in relation to someone else and where they are located in regards to objects






Dimensions in my work



A child projects his personal experiences while playing with toys. His stories reflect his stance in the world.

He wants to make a movie from the point of view of his Naruto action figure!


Lego people can even embed a scene drawn by Edward Hopper, Nighthawks, from 1942!






Picture This!The Secret Lives of Objects



The dolls become the body - a point of new perpective.

What if the toy had a visual perspective?

What if any objects could share their perspectives with a child?

Wouldn’t this open a new world to the child?







Picture This!The dolls become the body



Children methodically progress …

[] iterate back and forth to discover new visual angles

[] slowly appropriate the “new eye”

[] expand their visual sense

[] synchronize their senses







Picture This!Iterative process via visual feedback



Children discover spatially their environment: [] where their body is situated in relation to someone else [] where they are located in regards to objects.

Picture This! is a gesture objects interface at the extension of natural play.







Picture This!Spatial understanding








Semiotic squareGesture Object Interfaces



Parkinson’s disease, a neurodegenerative disorder with primary symptoms occurring due to a loss of motor timing mechanisms in the brain.

What if patient with Parkinson’s disease could communicate to objects their need to be synchronized?

Would they be able to walk with accuratetiming and not fall?


Sir William Richard Gowers, 1886.






Hap-bitTalking to Objects



It has been shown that the delivery of an external auditory metronome, in home-based rehabilitation programs, significantly increases gait rehabilitation of patients.

Would it work with a haptic stimulus? Could it be customized?

Patients could alternate amplitude of pulse stimuli to alternate their body perspectives.







Hap-bitAlternating body perspectives



Patients can customize features of the stimulus presentation such as frequency and amplitude by pulse compressing the garment.

The customized haptic rhythm might allow the patient to resynchronize her body, providing her with a temporal grounding reference.

Hap-bit is a membrane knitted into the fabric to deliver a rhythmic stimulus.







Hap-bitTime: a grounding reference



Our goal is to provide an interface for patients to modulate their neurological function as it relates to Parkinson’s motor timing.

We would like to design garments that support customizable intervention outside of the physical therapy laboratory.


Neurologists investigated intrasubjective relationships among organs, the nervous system, and sensory experience through neural prosthetics (Cartwright & Goldfarb, 2006).






Hap-bitThe haptic becomes part of the skin



The dolls become the body, and then becomes the video. The doll is a gesture object.

Grounding via body and visual perspectives. The child’s anthropomorphism makes the objects become part of the child’s body.

The body is then directly connected to the digital body (video of the dolls).

Physical body Digital body

Video of the dollThe body holds the dolls






SynthesisGrounding with gesture objects



Physical bodyDigital body

Digital actuated clothing The body will wear theenveloping fabric

Hap-bit skecth, 2008

This interface grounds the patient in her physical world while being held by a digital body, the digital actuation of youself.






SynthesisGrounding with a temporal reference



Connect your body to an Avatar and feel your body limits as part of that connection!

My World of Warcraft Avatar, now level 70!






SynthesisMore applications to come!


SENSING TECHNOLOGIES



How to exchange information between the digital and the physical body?

Let’s look at the gesture communication in Picture This!

ReferenceVaucelle, C. and Ishii, H. 2008. Picture this!: film assembly using toy gestures. In Proceedings of the 10th international Conference on Ubiquitous Computing. UbiComp ‘08, vol. 344. ACM, New York, NY, 350-359.






Exchange information



Rehearsing mode: The live video feed comes from the camera attached to the doll and is continuously displayed on a screen in front of the child

Recording mode: If a doll wants to be in the video, the doll needs to move. If it wants to be recorded it has to move three times quickly and the doll’s conversation partner will start the recording.

Playback mode: To play back the movie, the two dolls have to be moved in synchrony, in essence, jumping horizontally together.






Gesture communicationModes to control Picture This!



[] In Picture This! we use a Piezo vibration sensor (PVDF). We distinguish between vertical and horizontal motions while our sensor is a single axis accelerometer --detects mechanical stress.

[] We detect small variations of the off-axis motion with the on-axis accelerometer and are able to categorize strong motion in one axis and weak motion in the orthogonal axis. Piezo Sensor

MiniSense 100 is a vibration sensor loaded by a mass to offer high sensitivity at low frequencies.

Impacts containing high frequency components will excite the resonance frequency (1oohz).






Piezo vibration sensorTo distinguish between motions

In the piezo vibration sensor, crystal structures are stressed which generate voltage that can be converted to acceleration.



Why use a Piezo film?A Piezo vibration sensor is low-cost, easy to use and it offers a nice signal, but it is sensitive to temperature variations.

Why not use optical flow calculation? Because the doll wears a camera, we could take advantages of the camera as input to detect when it is moving ...

The optical flow vector of a moving object in a video sequence.

The camera is on the doll.To be recorded the doll hasto shake! “film me!”






Hardware versus softwarePiezo sensor vs optical flow calculation



Why not use optical flow calculation?

[] we avoid the problem of line of sight from the optical flow calculation.

[] with a Piezo sensor, we can select the strong variations in a short period of time (not the medium variations that occur during play).

There is a problem with the optical flow method: the child can easily occlude the camera because she shakes the doll that will not record, thus does not have a direct feedback for motion detection occlusion.






Hardware versus softwareAvoiding the problem of line of sight



We can generalize Picture This! technology to new platforms

Using a software-based Picture This! can democratize film making with toys.

ExampleMum and Dad’s cherished objects become the platforms of a child’s perspective taking, i.e. imagining Picture This! on cell phones.

A Piezo vibration sensor can be a constraint, requiring the purchase of a peripheral for the toys.

We can be hopeful, lots of camera phones now have an integrated accelerometer!

Children already use stuffed animals cell phone holder!

Picture This! could easily be integrated within these character looking holders!






Generalizing Picture This!Picture This! on new platforms



Let’s look at other hardware methods to detect gestures with toys and wearable:

[] Tilt mechanical switch: ON/OFF.

[] Capacitive sensing: measuring the distance between an object and its reference point. Change in capacitance related to acceleration.

[] Piezo resistive effect: Resistance changes with acceleration.

[] Hall effect: Motion converted to an electrical signal detecting change in magnetic fields.

[] Heat transfer: Location of heated mass tracked during acceleration by sensing temperature.

[] Magnetoresistive: Material resistivity changes in presence of magnetic field






Hardware methodsTo detect gestures



How to exchange information between the digital and physical body?

[] Sensing technology initiated with the physical body and output to the digital body:

e.g. gesture, touch, pressure.

[] Sensing technology from the digital body mapped onto the physical body:

e.g. actuation could be identified as “tiring”, “annoying” or “relaxing”. Actuation can be based on vibration; temperature: Peltier junction; pressure: pneumatics.






Mapping dataWhat kind of actuation to select?



How to find the right mapping between the digital and physical body?

[] User study: testing the mechanism of actuation.

[] Measuring the relationship between stimulus and sensation (psychophysics).

[] Allowing user customization of the technology: give users control and study it!






Mapping informationMethods for mapping data


CONCLUSION



I design grounding interfaces that translate the undiscovered environment:

[] to create a shift in perspective[] to ground the user in her physical body

I actively construct a relationship between the digital and physical body of the individual.

I do not deliver information passively or stimulate the senses.

I create a shift in body boundaries from the digital to the physical body.






Conclusion

MIT General Exams. Grounding interfaces: shifting the body boundaries, 2008.

Technology

Transcript of MIT General Exams. Grounding interfaces: shifting the body boundaries, 2008.