Audio & Audio & HapticsHapticsD. Dayan, J. Ellis, T. Ono & N. ShahD. Dayan, J. Ellis, T. Ono & N. Shah

AudioAudioAudioAudio

Why Do We Care About Audio?Why Do We Care About Audio?Why Do We Care About Audio?Why Do We Care About Audio?

Most prior research in virtual environment system has Most prior research in virtual environment system has focused on focused on visualization.visualization.

More More accurate acoustic modelling provides a user with a accurate acoustic modelling provides a user with a stronger sense of presence in VE.stronger sense of presence in VE.

Localizing objects which are not viewable. vehicle blind spots Localizing objects which are not viewable. vehicle blind spots for examplefor example..

Basic Basic AuralizationAuralization PipelinePipelineBasic Basic AuralizationAuralization PipelinePipeline

AcoustingAcousting ModellingModelling

I tI t Vi t l i t ( t f l ) l ti Vi t l i t ( t f l ) l ti •• Input:Input: Virtual environment (set of polygons), source location Virtual environment (set of polygons), source location (speakers), receiver location ((speakers), receiver location (micsmics).).

O t tO t t Filt ( d d l d )Filt ( d d l d )•• Output:Output: Filters (encode delays and errors).Filters (encode delays and errors).

Auditory DisplayAuditory Display

•• Input:Input: Filters audio inputFilters audio input•• Input:Input: Filters, audio input.Filters, audio input.

•• Output:Output: SpatializedSpatialized sound signal.sound signal.

Process of Sound PresentationProcess of Sound PresentationProcess of Sound PresentationProcess of Sound Presentation

Sound Simulate Simulate i l i l

Decoding for speaker array

Output from speaker

Sound source

environmental environmental acoustic effectsacoustic effects

Output from Output from headphones

HRTF

Light and SoundLight and SoundSimilarities and DifferencesSimilarities and Differences

WavelengthWavelength

SpeedSpeed

CoherenceCoherence

Dynamic RangeDynamic Range

Latency and Update RateLatency and Update Rate

Problems of Problems of ModellingModelling Sound PropagationSound PropagationProblems of Problems of ModellingModelling Sound PropagationSound Propagation

Computational Challenge:Computational Challenge: ModellingModelling the scattering of the scattering of sound waves in a 3D environment:sound waves in a 3D environment:

•• Sound waves travel from a source to receiver.Sound waves travel from a source to receiver.

•• Multitude of propagation paths.Multitude of propagation paths.

•• Influenced by reflections, diffractions and refractions.Influenced by reflections, diffractions and refractions.Influenced by reflections, diffractions and refractions.Influenced by reflections, diffractions and refractions.

Major Major ApproachesApproachesMajor Major ApproachesApproaches

Numerical solution to the wave equationNumerical solution to the wave equation•• Finite and boundary element Finite and boundary element methods.methods.Finite and boundary element Finite and boundary element methods.methods.

High frequency approximations based on geometric High frequency approximations based on geometric propagation pathspropagation pathspropagation pathspropagation paths•• Image source methods, ray tracing, and beam Image source methods, ray tracing, and beam tracing.tracing.

P llP ll b d b d i i l i i l d ld lPerceptuallyPerceptually--based sbased statistical tatistical modelsmodels•• Feedback/unitary Feedback/unitary feedback delay feedback delay networks.networks.

Finite & Boundary Element MethodsFinite & Boundary Element MethodsFinite & Boundary Element MethodsFinite & Boundary Element Methods

Subdivides time (and perhaps space) into elements.Subdivides time (and perhaps space) into elements.

Wave equation expressed as discrete set of linear equations Wave equation expressed as discrete set of linear equations for each of the elements.for each of the elements.

Can get complex in time and memory, so mainly used at low Can get complex in time and memory, so mainly used at low frequency.frequency.

Can also be used to model energy transfer between surfaces Can also be used to model energy transfer between surfaces (like (like radiosityradiosity)… but it’s tricky!)… but it’s tricky!

Geometric ModelsGeometric ModelsGeometric ModelsGeometric Models

Based on ray theoryBased on ray theory

Impulse ResponseImpulse ResponseImpulse ResponseImpulse Response

Direct Sound:Direct Sound: Earliest arriving & (usually) strongest wave.Earliest arriving & (usually) strongest wave.

Early Reflections:Early Reflections: Arrive within first few milliseconds of Arrive within first few milliseconds of impulse response (e.g., between 20ms and 80ms)impulse response (e.g., between 20ms and 80ms)

Late Reverberation:Late Reverberation: Sound has reflected off many surfaces.Sound has reflected off many surfaces.

ChallengesChallengesChallengesChallenges

Enumerating propagation paths.Enumerating propagation paths.

Modeling attenuation, reflection and scattering.Modeling attenuation, reflection and scattering.

Signal processing for geometric acoustics.Signal processing for geometric acoustics.

Artificial Reverberation AlgorithmsArtificial Reverberation AlgorithmsArtificial Reverberation AlgorithmsArtificial Reverberation Algorithms

Uses an artificial reverberator circuit for synthetic Uses an artificial reverberator circuit for synthetic reverberation.reverberation.

Does not provide an accurate model of a specific acoustic Does not provide an accurate model of a specific acoustic environment… but, has plausible model for late environment… but, has plausible model for late environment… but, has plausible model for late environment… but, has plausible model for late reverberation.reverberation.

Commonly used for late reverberations in video gamesCommonly used for late reverberations in video gamesCommonly used for late reverberations in video games.Commonly used for late reverberations in video games.

Process of Sound PresentationProcess of Sound PresentationProcess of Sound PresentationProcess of Sound Presentation

Sound Simulate Simulate i l i l

Decoding for speaker array

Output from speaker

Sound source

environmental environmental acoustic effectsacoustic effects

Output from Output from headphones

HRTF

HRTFHRTFHRTFHRTF

Head Related Transfer Function.Head Related Transfer Function.

Models Models the the effects on effects on headhead, ear and torso , ear and torso on propagation.on propagation.

How to measure HRTF?How to measure HRTF?•• Use simple sphere as a listener and compute diffraction Use simple sphere as a listener and compute diffraction filters.filters.

•• Use dummyUse dummy--heads (heads (pplacing lacing small microphones directly at the small microphones directly at the l)l)ear ear canal).canal).

•• Use 3D model of the head and run boundary element Use 3D model of the head and run boundary element simulationssimulations..

Problems with HRTFProblems with HRTFProblems with HRTFProblems with HRTF

HRTF varies from person to person.HRTF varies from person to person.•• Influenced by head or ear Influenced by head or ear shapes.shapes.Influenced by head or ear Influenced by head or ear shapes.shapes.

Headphones Headphones and and eardrum.eardrum.•• Acoustic cavity forms between Acoustic cavity forms between themthem•• Acoustic cavity forms between Acoustic cavity forms between them.them.

•• This cavity leads significant impact of the quality of the This cavity leads significant impact of the quality of the reproduction.reproduction.

and some solutionsand some solutions…and some solutions…and some solutions

Efficient Efficient implementation of binaural implementation of binaural filtering.filtering.•• For efficient rendering of multiple sound For efficient rendering of multiple sound paths.paths.For efficient rendering of multiple sound For efficient rendering of multiple sound paths.paths.

•• Based on geometrical Based on geometrical acoustics.acoustics.

CrossCross talk cancellation and talk cancellation and transauraltransaural stereostereoCrossCross--talk cancellation and talk cancellation and transauraltransaural stereo.stereo.

Output Output device is device is loudspeakerloudspeaker, the sound from left speaker , the sound from left speaker h ld b ll d h i h i h h ld b ll d h i h i h should be cancelled when it reaches to right should be cancelled when it reaches to right ear.ear.

HapticsHapticsHapticsHaptics

Why use Why use HapticsHaptics??Why use Why use HapticsHaptics??

Lack of Lack of hapticshaptics reminds the user the world is not reminds the user the world is not realreal, , detracts from their sense of detracts from their sense of presencepresence..pp

Presence Presence -- ““The extent to which human participants in a The extent to which human participants in a virtual environment virtual environment allow themselves to be convinced […] allow themselves to be convinced […] virtual environment virtual environment allow themselves to be convinced […] allow themselves to be convinced […] that that they are somewhere other than where the physically are they are somewhere other than where the physically are --that ‘somewhere’ being determined by the image, sounds, that ‘somewhere’ being determined by the image, sounds,

d d h i l ti h i l ti id d b h id d b h and and physical sensations physical sensations provided by the computerprovided by the computer--synthesized virtual environment to their sensessynthesized virtual environment to their senses.” (Slater and .” (Slater and UsohUsoh, 1993)., 1993)., ), )

Requirement of VE (steering mechanisms, dials, buttons).Requirement of VE (steering mechanisms, dials, buttons).

Other strategies inappropriate (e g video games just stop the Other strategies inappropriate (e g video games just stop the Other strategies inappropriate (e.g. video games just stop the Other strategies inappropriate (e.g. video games just stop the character, but try this in a CAVE!)character, but try this in a CAVE!)

All i t i l l dAll i t i l l dAlleviate visual load.Alleviate visual load.

Examples of applicationsExamples of applicationsExamples of applicationsExamples of applications

ForceForce--feedback GUIsfeedback GUIs

GamesGames

Multimedia publishing (communication)Multimedia publishing (communication)

Scientific discoveryScientific discovery

Arts and creationArts and creation

Editing existing media (sound & images)Editing existing media (sound & images)g g ( g )g g ( g )

Vehicle operation and control roomsVehicle operation and control rooms

Examples of applications (cont )Examples of applications (cont )Examples of applications (cont.)Examples of applications (cont.)

ManufacturingManufacturing

TeleroboticsTelerobotics & & telemedicinetelemedicine

Education and trainingEducation and training

Rehabilitation (use by the disabled)Rehabilitation (use by the disabled)

Research Research

Types of Types of HapticsHapticsypyp ppActive & PassiveActive & Passive

Active Active HapticsHapticsActive Active HapticsHaptics

Passive Passive HapticsHapticsPassive Passive HapticsHaptics

A technique that incorporates passive physical objects into A technique that incorporates passive physical objects into virtual environments to physically simulate the virtual objects.virtual environments to physically simulate the virtual objects.p y y jp y y j

Virtual Environment Real World

ExperimentsExperimentsppPassive Passive HapticsHaptics Significantly Enhances Virtual Significantly Enhances Virtual EnvironmentsEnvironments ((InskoInsko, 2001), 2001)

Visual Pit RevisitedVisual Pit RevisitedEffects of passive Effects of passive hapticshaptics on presence.on presence.

Remember the Visual Pit?Remember the Visual Pit?Remember the Visual Pit?Remember the Visual Pit?

Experiment conditionsExperiment conditionsExperiment conditionsExperiment conditions

Physical ledge No physical ledge

Data CaptureData CaptureData CaptureData Capture

UCL Presence Questionnaire:UCL Presence Questionnaire:

•• Reported presenceReported presence•• Reported presenceReported presence

•• Reported behavioral presence Reported behavioral presence

•• Ease of locomotionEase of locomotion

ProCompProComp+ tethered telemetry system+ tethered telemetry system

Fi d Fi d •• Finger mounted sensorsFinger mounted sensors

•• ThreeThree--electrode electrocardiograph on participant’s chestelectrode electrocardiograph on participant’s chest

Results from questionnaireResults from questionnaireResults from questionnaireResults from questionnaire

Results from biometricsResults from biometricsResults from biometricsResults from biometrics

“R l f hi d “R l f hi d “Results of this study support “Results of this study support the hypothesis that augmenting the hypothesis that augmenting the hypothesis that augmenting the hypothesis that augmenting

virtual environments with virtual environments with passive passive hapticshaptics increases increases

presence.”presence.”

Training RoomTraining RoomEffects of passive Effects of passive hapticshaptics on cognitive mapping and spatial knowledge transfer.on cognitive mapping and spatial knowledge transfer.

DefinitionsDefinitionsDefinitionsDefinitions

Cognitive mappingCognitive mapping –– Forming a mental model of an Forming a mental model of an environment.environment.

Spatial knowledgeSpatial knowledge –– Knowledge about the dimensions of Knowledge about the dimensions of objects in the environment and the distances between them.objects in the environment and the distances between them.

A priori A priori hypotheseshypothesesA priori A priori hypotheseshypotheses

Spatial knowledge training in a virtual environment Spatial knowledge training in a virtual environment augmented with passive augmented with passive hapticshaptics will produce increased will produce increased g pg p pp ppspatial knowledge transfer when compared to training in a spatial knowledge transfer when compared to training in a nonnon--augmented VE.augmented VE.

Participants will form better cognitive maps while Participants will form better cognitive maps while Participants will form better cognitive maps while Participants will form better cognitive maps while experiencing a virtual environment with passive experiencing a virtual environment with passive hapticshaptics than than when experiencing a VE without passive when experiencing a VE without passive hapticshaptics..

Room LayoutRoom LayoutRoom LayoutRoom Layout

Map of Training Room Aerial view of virtualTraining Room

Room Layout Room Layout cotdcotdRoom Layout Room Layout –– cotdcotd..

Virtual view of Actual view ofVirtual view oftraining environment

Actual view oftraining environment

The ExplorationThe ExplorationThe ExplorationThe Exploration

With Passive With Passive HapticsHaptics

In room with In room with styrofoamstyrofoam

Without Passive Without Passive HapticsHaptics

In a In a phsysicallyphsysically empty empty In room with In room with styrofoamstyrofoammodel for passive model for passive hapticshaptics..

N di i l N di i l

In a In a phsysicallyphsysically empty empty room.room.

T hi bj T hi bj No audiovisual cues.No audiovisual cues. Touching objects causes Touching objects causes audiovisual cues (hand audiovisual cues (hand turns red, turns red, turns red, turns red, buzzing/clicking).buzzing/clicking).

The RecallThe RecallThe RecallThe Recall

Participants required to sketch a map of the VE on a blank Participants required to sketch a map of the VE on a blank 8.5 x 118.5 x 11--inch sheet of paper (no instructions given on how to inch sheet of paper (no instructions given on how to p p ( gp p ( gdo this).do this).

Participants marked the heights of the four Participants marked the heights of the four colouredcoloured objects objects Participants marked the heights of the four Participants marked the heights of the four colouredcoloured objects objects on a sixon a six--foot strip of paper attached to the wall.foot strip of paper attached to the wall.

Participants marked the distance between the four Participants marked the distance between the four colouredcolouredParticipants marked the distance between the four Participants marked the distance between the four colouredcolouredobjects on a sixobjects on a six--foot strip of paper spread out on a table.foot strip of paper spread out on a table.

P ti i t th h d t l t l th h th P ti i t th h d t l t l th h th Participants then had to complete a lap through the Participants then had to complete a lap through the environment while blindfolded without touching any of the environment while blindfolded without touching any of the objects.objects.jj

ResultsResultsResultsResults

Other ObservationsOther ObservationsOther ObservationsOther Observations

Wrong turnsWrong turns

PresencePresence

Simulator sicknessSimulator sickness

Summary of ResultsSummary of ResultsSummary of ResultsSummary of Results

Navigation time and number of collisions support the Navigation time and number of collisions support the hypothesis that spatial knowledge transfer was greater in a hypothesis that spatial knowledge transfer was greater in a yp p g gyp p g gVE system augmented by passive VE system augmented by passive hapticshaptics than a VE without.than a VE without.

Sketch maps and height estimates did Sketch maps and height estimates did notnot support the support the Sketch maps and height estimates did Sketch maps and height estimates did notnot support the support the hypothesis that participants trained with passive hypothesis that participants trained with passive hapticshaptics form form better cognitive maps than those without.better cognitive maps than those without.

PseudoPseudo--Haptic FeedbackHaptic FeedbackCan Isometric Input Devices Simulate Force Feedback?Can Isometric Input Devices Simulate Force Feedback?

Isotonic Isotonic vsvs Isometric 3D Input DevicesIsometric 3D Input DevicesIsotonic Isotonic vsvs Isometric 3D Input DevicesIsometric 3D Input Devices

Isometric devicesIsometric devices -- offer resistance and stay put while you offer resistance and stay put while you exert force on exert force on themthem..

Isotonic devicesIsotonic devices -- offer no significant resistance and are used offer no significant resistance and are used to track users as they move around the virtual to track users as they move around the virtual world.world.

Isometric DeviceIsometric DeviceIsometric DeviceIsometric Device

Spaceball

ObjectiveObjectiveObjectiveObjective

Whether a passive isometric input device, such as a Whether a passive isometric input device, such as a spaceballspaceball, , used together with visual feedback, could provide the operator used together with visual feedback, could provide the operator g p pg p pwith a pseudowith a pseudo--haptic feedback.haptic feedback.

•• Simulating Haptic Simulating Haptic FeedbackFeedback

•• No No Active Haptic Feedback Active Haptic Feedback

PremisePremisePremisePremise

To use To use a virtual spring along side a real a virtual spring along side a real spring to see if the subject can perceive the spring to see if the subject can perceive the spring to see if the subject can perceive the spring to see if the subject can perceive the discrimination of stiffness between the two discrimination of stiffness between the two

iisprings.springs.

Real SpringReal SpringReal SpringReal Spring

Virtual SpringVirtual SpringVirtual SpringVirtual Spring

Virtual Spring InputVirtual Spring InputVirtual Spring InputVirtual Spring Input

ExperimentExperimentExperimentExperiment

Subject had to choose between two virtual springs displayed Subject had to choose between two virtual springs displayed on the same computer on the same computer screen.screen.pp

Three values of virtual reference stiffness were Three values of virtual reference stiffness were used.used.

Each spring was compared with five possible stiffer springs Each spring was compared with five possible stiffer springs whose stiffness varied from the reference whose stiffness varied from the reference stiffness.stiffness.

For each subject each pair appeared 25 times in random For each subject each pair appeared 25 times in random order. The total amount of trials was then 100 a pair and the order. The total amount of trials was then 100 a pair and the t t l t t l b b f t i l f t i l 1500 1500 total total number number of trials of trials was 1500was 1500. .

ResultsResultsResultsResults

InterpretationInterpretationInterpretationInterpretation

Configuration of this system can simulate the model of a Configuration of this system can simulate the model of a virtual spring realistically.virtual spring realistically.p g yp g y

Compliance discrimination in a VE can be performed with Compliance discrimination in a VE can be performed with most of other input devices, without force feedback, but by most of other input devices, without force feedback, but by most of other input devices, without force feedback, but by most of other input devices, without force feedback, but by using sound or visual substitutions as force cues.using sound or visual substitutions as force cues.

Necessary to study the concept with a task which could not Necessary to study the concept with a task which could not Necessary to study the concept with a task which could not Necessary to study the concept with a task which could not be performed, without using a force feedback devicebe performed, without using a force feedback device..

Compliance Discrimination Compliance Discrimination BBetween etween RReal and Virtual Springseal and Virtual Springs

During each trial the subject first had to test the stiffness of a During each trial the subject first had to test the stiffness of a real spring (the reference stimulus), and then, to test the real spring (the reference stimulus), and then, to test the p g ( )p g ( )stiffness of a virtual spring (the comparison stimulus).stiffness of a virtual spring (the comparison stimulus).

Subject had to tell which spring was Subject had to tell which spring was stiffer.stiffer.Subject had to tell which spring was Subject had to tell which spring was stiffer.stiffer.

Each real spring was compared with Each real spring was compared with 12 12 possible virtual possible virtual springsspringssprings.springs.

The total amount of trials was then 27 a pair and the total The total amount of trials was then 27 a pair and the total t f t i l t f t i l 972972amount of trials was amount of trials was 972.972.

ResultsResultsResultsResults

DiscussionDiscussionDiscussionDiscussion

Transition between Transition between real world real world and and VE may VE may have had a have had a negative negative effect.effect.gg

Subjects have a tendency to underestimate the virtual spring. Subjects have a tendency to underestimate the virtual spring. The less stiff the reference spring is, the more underestimated The less stiff the reference spring is, the more underestimated The less stiff the reference spring is, the more underestimated The less stiff the reference spring is, the more underestimated the comparison virtual spring is.the comparison virtual spring is.

May have been due to the visual perception of the spring May have been due to the visual perception of the spring May have been due to the visual perception of the spring May have been due to the visual perception of the spring moving a lot even though the thumb barely moves with the moving a lot even though the thumb barely moves with the SSpaceballpaceball..

ConclusionConclusionConclusionConclusion

Subjects Subjects were able to discriminate successfully betweenwere able to discriminate successfully betweena virtual spring and a real a virtual spring and a real one.one.p gp g

Possibility Possibility to feed back haptic information to the user to feed back haptic information to the user without using without using a force feedback device but simply by a force feedback device but simply by without using without using a force feedback device but simply by a force feedback device but simply by combining a combining a passive isometric input device with a visual passive isometric input device with a visual feedback.feedback.

The overall system returns a force cue called pseudoThe overall system returns a force cue called pseudo--haptic haptic feedback.feedback.

Probably Probably not replace an actual haptic one but can be useful not replace an actual haptic one but can be useful for some simple for some simple simulations.simulations.pp

All subjects were able to discriminate between a real spring All subjects were able to discriminate between a real spring All subjects were able to discriminate between a real spring All subjects were able to discriminate between a real spring and a virtual one without the proprioceptive information and a virtual one without the proprioceptive information from the from the thumb.thumb.

The visual dominance was used to influence the perception The visual dominance was used to influence the perception of the displacement of a virtual of the displacement of a virtual spring; people spring; people found that thefound that thevirtual spring model was virtual spring model was realistic.realistic.

Swamp ExperimentSwamp ExperimentSwamp ExperimentSwamp Experiment

Using mouse and Using mouse and SSpaceballpaceball..

The subjects were told to manipulate a virtual cube, The subjects were told to manipulate a virtual cube, displayed on the horizontal displayed on the horizontal plane.plane.

When over When over the “swamp”, the “swamp”, the speed of the cube was either the speed of the cube was either accelerated or slowed accelerated or slowed down.down.

Swamp ResultsSwamp ResultsSwamp ResultsSwamp Results

While using the While using the SSpaceballpaceball subjects felt something as they subjects felt something as they crossed the crossed the swamp.swamp.pp

Forces were more perceptible than with a 2D Forces were more perceptible than with a 2D mouse.mouse.

Results From Previous Results From Previous ExperimentsExperiments

Cues sent by different senses are interpreted Cues sent by different senses are interpreted together.together.

“Pluralistic“Pluralistic” nature of sensory ” nature of sensory perception.perception.

Work on visual dominance Work on visual dominance shows shows multiple cues offer a high multiple cues offer a high level of level of redundancy.redundancy.

Different Different material can easily be interpreted differently by material can easily be interpreted differently by Different Different material can easily be interpreted differently by material can easily be interpreted differently by blindfolded blindfolded subjects.subjects.

Thank YouThank YouAny Questions?Any Questions?