5. Multimedia Data 5. Multimedia Data

22

Multimedia Data RepresentationMultimedia Data Representation

Digital Audio Digital Audio Sampling/Digitisation Sampling/Digitisation Compression (Details of Compression Compression (Details of Compression

algorithms – following chapters) algorithms – following chapters) Graphics/Image Formats Graphics/Image Formats Digital Video (Basics of Video, Didgiatl Digital Video (Basics of Video, Didgiatl

Video, closely entwined with Compression) Video, closely entwined with Compression)

33

Digital AudioDigital Audio

Application of Digital Audio - Selected Examples Application of Digital Audio - Selected Examples Music Production Music Production          

– Hard Disk Recording Hard Disk Recording – Sound Synthesis Sound Synthesis – Samplers Samplers – Effects Processing Effects Processing

Video Video - Audio Important Element: Music and Effects - Audio Important Element: Music and Effects

Web Web - Many uses on Web - Many uses on Web

– Spice up Web Pages Spice up Web Pages – Listen to Cds Listen to Cds – Listen to Web Radio Listen to Web Radio

Many More Uses Many More Uses - try and think of some? - try and think of some?

44

Digitization of Sound Digitization of Sound

Sound is a continuous wave that travels Sound is a continuous wave that travels through the air through the air

The wave is made up of pressure The wave is made up of pressure differences. Sound is detected by measuring differences. Sound is detected by measuring the pressure level at a particular location. the pressure level at a particular location.

Sound waves have normal wave properties Sound waves have normal wave properties (reflection, refraction, diffraction, etc.). (reflection, refraction, diffraction, etc.).

55

Source Source – Generates Sound Generates Sound

Air Pressure changes Air Pressure changes ElectricalElectrical - Loud Speaker - Loud Speaker AcousticAcoustic - Direct Pressure Variations - Direct Pressure Variations

Destination Destination – Receives Sound Receives Sound

ElectricalElectrical - Microphone produces electric signal - Microphone produces electric signal EarsEars - Responds to pressure - Responds to pressure hearhear sound sound

66

Sound is required input into a computer: it needs to sampled or Sound is required input into a computer: it needs to sampled or digitised: digitised:

Microphones, video cameras produce Microphones, video cameras produce analogue signalsanalogue signals (continuous- (continuous-valued voltages).valued voltages).

Continuous Analogue WaveformContinuous Analogue Waveform To get audio or video into a computer, we have to To get audio or video into a computer, we have to digitizedigitize it (convert it it (convert it

into a stream of numbers). into a stream of numbers). Need to convert Analogue-to-DigitalNeed to convert Analogue-to-Digital - - Specialised Hardware Specialised Hardware

So, we have to understand So, we have to understand discrete samplingdiscrete sampling (both time and voltage) (both time and voltage) SamplingSampling - divide the horizontal axis (the time dimension) into discrete - divide the horizontal axis (the time dimension) into discrete

pieces. Uniform sampling is ubiquitous. pieces. Uniform sampling is ubiquitous. QuantizationQuantization - divide the vertical axis (signal strength) into pieces. - divide the vertical axis (signal strength) into pieces.

Sometimes, a non-linear function is applied. Sometimes, a non-linear function is applied. – 8 bit quantization divides the vertical axis into 256 levels; 16 bit gives you 8 bit quantization divides the vertical axis into 256 levels; 16 bit gives you

65536 levels. 65536 levels.

77

Digitizing Audio Digitizing Audio

That is the basic idea of digitizing a sound That is the basic idea of digitizing a sound unfortunately things are (practically speaking) not unfortunately things are (practically speaking) not so simple. so simple.

Questions for producing digital audio (Analogue-Questions for producing digital audio (Analogue-to-Digital Conversion): to-Digital Conversion):

1. 1. How often do you need to sample the signal? How often do you need to sample the signal?

2. 2. How good is the signal? How good is the signal?

3. 3. How is audio data formatted? How is audio data formatted?

88

Computer Manipulation of SoundComputer Manipulation of Sound Once Digitised processing the digital sound is essentially Once Digitised processing the digital sound is essentially

straightforward although it depends on the processing you wish to do straightforward although it depends on the processing you wish to do ((e.g.e.g. volume is easier to code than accurate reverb) volume is easier to code than accurate reverb)

Sample Rates and Bit Size Sample Rates and Bit Size How do we store each sample value (How do we store each sample value (QuantisationQuantisation)? )? 8 Bit Value 8 Bit Value

– (0-255) (0-255) 16 Bit Value 16 Bit Value

– (Integer) (0-65535) (Integer) (0-65535) How many Samples to take?How many Samples to take? – 11.025 KHz 11.025 KHz Speech (Telephone 8KHz) Speech (Telephone 8KHz) – 22.05 KHz 22.05 KHz Low Grade Audio Low Grade Audio

(WWW Audio, AM Radio) (WWW Audio, AM Radio) – 44.1 KHz 44.1 KHz CD Quality CD Quality

99

Implications of Sample Rate and Bit SizeImplications of Sample Rate and Bit Size

Affects Quality of AudioAffects Quality of Audio Ears do not respond to sound in a linear fashion Ears do not respond to sound in a linear fashion Decibel (Decibel (dBdB) a logarithmic measurement of sound ) a logarithmic measurement of sound 16-Bit has a signal-to-noise ratio of 98 dB - 16-Bit has a signal-to-noise ratio of 98 dB -

virtually inaudible virtually inaudible 8-bit has a signal-to-noise ratio of 50 dB 8-bit has a signal-to-noise ratio of 50 dB Therefore, 8-bit is roughly 8 times as noisy Therefore, 8-bit is roughly 8 times as noisy

– 6 dB increment is twice as loud 6 dB increment is twice as loud

1010

Typical Audio Formats Typical Audio Formats

Popular audio file formats include .au (Unix Popular audio file formats include .au (Unix workstations), .aiff (MAC, SGI), .wav (PC, workstations), .aiff (MAC, SGI), .wav (PC, DEC workstations) DEC workstations)

A simple and widely used audio A simple and widely used audio compression method is Adaptive Delta compression method is Adaptive Delta Pulse Code Modulation (ADPCM). Based on Pulse Code Modulation (ADPCM). Based on past samples, it predicts the next sample past samples, it predicts the next sample and encodes the difference between the and encodes the difference between the actual value and the predicted value. actual value and the predicted value.

1111

Delivering Audio over a NetworkDelivering Audio over a Network

Trade off between desired fidelity and file Trade off between desired fidelity and file sizesize

Bandwidth Considerations for Web and other Bandwidth Considerations for Web and other media. media.

Compress Files: Compress Files:

Could affect live transmission on WebCould affect live transmission on Web

1212

Streaming AudioStreaming Audio

Buffered Data: Buffered Data: Trick get data to destination before it's needed Trick get data to destination before it's needed Temporarily store in memory (Buffer) Temporarily store in memory (Buffer) Server keeps feeding the buffer Server keeps feeding the buffer Client Application reads buffer Client Application reads buffer

Needs Reliable Connection, moderately fast too. Needs Reliable Connection, moderately fast too.

Specialised client, Steaming Audio Protocol (PNM Specialised client, Steaming Audio Protocol (PNM for real audio).for real audio).

1313

Introduction to MIDI (Musical Instrument Introduction to MIDI (Musical Instrument Digital Interface)Digital Interface)

Definition of MIDI:Definition of MIDI: a protocol that enables computer, a protocol that enables computer, synthesizers, keyboards, and other musical device to synthesizers, keyboards, and other musical device to communicate with each othercommunicate with each other

Components of a MIDI SystemComponents of a MIDI System::SynthesizerSynthesizer (a sound generator) (a sound generator)SequencerSequencer (a music editor) (a music editor)TrackTrack (to organize the recordings) (to organize the recordings)ChannelChannel (to separate information in a MIDI system) (to separate information in a MIDI system)TimbreTimbre (the quality of the sound) (the quality of the sound)PitchPitch (musical note that the instrument plays) (musical note that the instrument plays)VoiceVoice (the portion of the synthesizer that produces sound) (the portion of the synthesizer that produces sound)PatchPatch (the control settings that define a particular timbre) (the control settings that define a particular timbre)

1414

Digital Audio and MIDIDigital Audio and MIDI

There are many application of Digital Audio and MIDI being There are many application of Digital Audio and MIDI being used together: used together:

Modern Recording Studio - Hard Disk Recording and MIDI Modern Recording Studio - Hard Disk Recording and MIDI – Analogue Sounds (Live Vocals, Guitar, Sax etc) - DISK Analogue Sounds (Live Vocals, Guitar, Sax etc) - DISK – Keyboards, Drums, Samples, Loops Effects - MIDI Keyboards, Drums, Samples, Loops Effects - MIDI

Sound Generators: use a mix of Sound Generators: use a mix of – Synthesis Synthesis – Samples Samples

Samplers - Digitise (Sample) Sound then Samplers - Digitise (Sample) Sound then – Playback Playback – Loop (beats) Loop (beats) – Simulate Musical Instruments Simulate Musical Instruments

1515

Digital Audio, Synthesis, Midi and Digital Audio, Synthesis, Midi and

Compression - MPEG 4 Structured AudioCompression - MPEG 4 Structured Audio We have seen the need for compression already in Digital We have seen the need for compression already in Digital

Audio - Large Data Files Audio - Large Data Files Basic Ideas of compression (see following Chapters) used Basic Ideas of compression (see following Chapters) used

as integral part of audio format - MP3, real audio as integral part of audio format - MP3, real audio etc.etc. Mpeg-4 audio - actually combines compression synthesis Mpeg-4 audio - actually combines compression synthesis

and MIDI to have a massive impact on compression. and MIDI to have a massive impact on compression. MIDI, Synthesis encode what note to play and how to play MIDI, Synthesis encode what note to play and how to play

it with a small number of parameters - Much greater it with a small number of parameters - Much greater reduction than simply having some encoded bits of audio. reduction than simply having some encoded bits of audio.

Responsibility to create audio delegated to generation side. Responsibility to create audio delegated to generation side.

1616

MPEG 4 Structured AudioMPEG 4 Structured Audio

MPEG-4 covers the whole range of digital audio: MPEG-4 covers the whole range of digital audio: from very low bit rate speech from very low bit rate speech to full bandwidth high quality audio to full bandwidth high quality audio built in anti-piracy measures built in anti-piracy measures Structured AudioStructured Audio

Structured Audio ToolsStructured Audio Tools

MPEG-4 comprises of 6 MPEG-4 comprises of 6 Structured Audio toolsStructured Audio tools