© 1998 Peter B.L. Meijer

On cross-modal sensory streams...

The vThe vOICOICeeby by

Peter B.L. Meijer Peter B.L. Meijer

The vThe vOICOICeeby by

Peter B.L. Meijer Peter B.L. Meijer

Philips Research LaboratoriesPhilips Research LaboratoriesPeter.B.L.Meijer@philips.comPeter.B.L.Meijer@philips.com

Invited Panel Presentation at SIGGRAPH '98, Orlando, Florida

© 1998 Peter B.L. Meijer

The vOICe: “Seeing with your Ears”

Basic facts:Basic facts:

• Blind people cannot Blind people cannot seesee......but usually they can but usually they can hearhear

• Multimedia computing allowsMultimedia computing allowsaudio-visual transformationsaudio-visual transformations

Basic facts:Basic facts:

• Blind people cannot Blind people cannot seesee......but usually they can but usually they can hearhear

• Multimedia computing allowsMultimedia computing allowsaudio-visual transformationsaudio-visual transformations

© 1998 Peter B.L. Meijer

The vOICe: “Seeing with your Ears”

Basic facts:Basic facts:

• Blind people cannot Blind people cannot seesee......but usually they can but usually they can hearhear

• Multimedia computing allowsMultimedia computing allowsaudio-visual transformationsaudio-visual transformations

Basic facts:Basic facts:

• Blind people cannot Blind people cannot seesee......but usually they can but usually they can hearhear

• Multimedia computing allowsMultimedia computing allowsaudio-visual transformationsaudio-visual transformations

VideoVideo

AudioAudio

MentalMentalVideoVideo

VideoVideo

AudioAudio

MentalMentalVideoVideo

The vOICeThe vOICeThe vOICeThe vOICe

Human brainHuman brainHuman brainHuman brain

So translate video to audio!?So translate video to audio!?So translate video to audio!?So translate video to audio!?

© 1998 Peter B.L. Meijer

The vOICe implementations

HardwareHardwareprototypeprototypeHardwareHardwareprototypeprototype

Java appletJava appletJava appletJava applet

WindowsWindows95/98/NT95/98/NT

WindowsWindows95/98/NT95/98/NT

http://www.seeingwithsound.com/http://www.seeingwithsound.com/

© 1998 Peter B.L. Meijer

The vOICe image-to-sound mapping

SumSum of ofsinusoidssinusoids

SumSum of ofsinusoidssinusoids

Fre

quen

cyF

requ

ency

Time

Time

AmplitudeAmplitude

© 1998 Peter B.L. Meijer

The vOICe image-to-sound mappingF

requ

enc

Fre

que

ncyy

TimeTime

Spectrographic Spectrographic synthesissynthesis

Spectrographic Spectrographic synthesissynthesis

© 1998 Peter B.L. Meijer

Could this really work?Could this really work?

CheckCheck:: map sounds back to images map sounds back to images

Could this really work?Could this really work?

CheckCheck:: map sounds back to images map sounds back to images

The vOICe image-to-sound mapping

Yes!Yes!(at(at leastleast technicallytechnically))

Yes!Yes!(at(at leastleast technicallytechnically))

TheoryTheoryTheoryTheory

© 1998 Peter B.L. Meijer

The vOICe image reconstruction

““Soundscape”Soundscape”““Soundscape”Soundscape”

The vThe vOICOICeeThe vThe vOICOICee

SpectrogramSpectrogramSpectrogramSpectrogram

© 1998 Peter B.L. Meijer

The vOICe image reconstruction

““SoundscapeSoundscape””““SoundscapeSoundscape””

© 1998 Peter B.L. Meijer

The vOICe and human speech

PronouncingPronouncingone imageone image

per secondper second

PronouncingPronouncingone imageone image

per secondper second

© 1998 Peter B.L. Meijer

The vOICe mental imagery?

RepresentationRepresentationmay dramaticallymay dramaticallyaffect affect recognitionrecognition!!

RepresentationRepresentationmay dramaticallymay dramaticallyaffect affect recognitionrecognition!!

© 1998 Peter B.L. Meijer

The vOICe limitations

Issues:Issues: Frequency-Time uncertaintyFrequency-Time uncertainty

• Physiology / Perception:Physiology / Perception:

just-noticeable difference (JND)just-noticeable difference (JND)

critical bands (human cochlea)critical bands (human cochlea)

Issues:Issues: Frequency-Time uncertaintyFrequency-Time uncertainty

• Physiology / Perception:Physiology / Perception:

just-noticeable difference (JND)just-noticeable difference (JND)

critical bands (human cochlea)critical bands (human cochlea)

© 1998 Peter B.L. Meijer

The vOICe limitations

Issues:Issues: Frequency-Time uncertaintyFrequency-Time uncertainty

• Physiology / Perception:Physiology / Perception:

just-noticeable difference (JND)just-noticeable difference (JND)

critical bands (human cochlea)critical bands (human cochlea)

? auditory streaming/segregationauditory streaming/segregation

? neural pathways & bandwidthneural pathways & bandwidth

? neural processing & plasticityneural processing & plasticity

Issues:Issues: Frequency-Time uncertaintyFrequency-Time uncertainty

• Physiology / Perception:Physiology / Perception:

just-noticeable difference (JND)just-noticeable difference (JND)

critical bands (human cochlea)critical bands (human cochlea)

? auditory streaming/segregationauditory streaming/segregation

? neural pathways & bandwidthneural pathways & bandwidth

? neural processing & plasticityneural processing & plasticity

© 1998 Peter B.L. Meijer

The vOICe limitations

Issues:Issues: Frequency-Time uncertaintyFrequency-Time uncertainty

• Physiology / Perception:Physiology / Perception:

just-noticeable difference (JND)just-noticeable difference (JND)

critical bands (human cochlea)critical bands (human cochlea)

? auditory streaming/segregationauditory streaming/segregation

? neural pathways & bandwidthneural pathways & bandwidth

? neural processing & plasticityneural processing & plasticity

Issues:Issues: Frequency-Time uncertaintyFrequency-Time uncertainty

• Physiology / Perception:Physiology / Perception:

just-noticeable difference (JND)just-noticeable difference (JND)

critical bands (human cochlea)critical bands (human cochlea)

? auditory streaming/segregationauditory streaming/segregation

? neural pathways & bandwidthneural pathways & bandwidth

? neural processing & plasticityneural processing & plasticity

• Psychology / Education:Psychology / Education:

? sounds (too) unpleasantsounds (too) unpleasant

? minimum required resultsminimum required results

? acceptable training effortacceptable training effort

? motivation for learningmotivation for learning

? training programstraining programs

• Psychology / Education:Psychology / Education:

? sounds (too) unpleasantsounds (too) unpleasant

? minimum required resultsminimum required results

? acceptable training effortacceptable training effort

? motivation for learningmotivation for learning

? training programstraining programs

© 1998 Peter B.L. Meijer

The vThe vOICOICe approache approach needs more researchneeds more research

is cheap technologyis cheap technology

is non-invasiveis non-invasive

The vThe vOICOICe approache approach needs more researchneeds more research

is cheap technologyis cheap technology

is non-invasiveis non-invasive

Conclusions

© 1998 Peter B.L. Meijer

The vThe vOICOICe approache approach needs more researchneeds more research

is cheap technologyis cheap technology

is non-invasiveis non-invasive

is available is available Now!Now!

The vThe vOICOICe approache approach needs more researchneeds more research

is cheap technologyis cheap technology

is non-invasiveis non-invasive

is available is available Now!Now!

Conclusions

http://www.http://www.seeingwithsoundseeingwithsound.com/.com/