MUSIC IN REAL LIFE VS MUSIC IN DIGITAL WORLD
AIST2010 Lecture 2
OUTLINE
SOUND AND MUSIC IN REAL LIFE
DIGITIZING SOUND AND MUSIC DATA
MUSICAL INSTRUMENTS
MUSIC HARDWARE AND SOFTWARE
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 2
SOUND AND MUSIC IN REAL LIFE
Vibration and soundThe vibration of air particles from the source into your ears Or via other media (e.g. water) Vibrations detected by human auditory system è “audio” signals
Different vibrations give rise to different kinds of sounds Some are harmonious, some are noise, some are high, some are low
MusicA clever combination of sounds and voids
Representing sounds It’s always an abstraction, e.g. Waveform (amplitude vs. time) Spectrum (frequency vs. amp. vs. time)
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 3
VIBRATION
For periodic signals, frequency measured in Hz à cycle per second (cps)
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 4
Image from: Fund. of Music Processing, p.20
Video from: https://youtu.be/XBP0jmpjzBw
ampl
itude
ampl
itude
time
freq
uenc
y
WAVEFORM
SPECTROGRAM
Note: There are other ways to represent the spectrum too
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 5
BASIC PERIODIC WAVEFORMS
A pitch is heard for periodic waveforms (at certain frequencies)Sine waveCosine wave Phase-shifted sine wave
Triangular waveSawtooth waveSquare wave
Commonly used in vintage synthesizers (i.e. electronics)
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 6
Image from: https://moogfoundation.org/learning-synthesis/synthesis-fundamentals/
HUMAN EAR AND PSYCHOACOUSTICS
Human sound perception is subjectiveGeneral range: ~20 Hz – ~20,000 Hz (20 kHz)Hearing range narrows with age, especially at high frequencies How well do animals hear? https://lsu.edu/deafness/HearingRange.html
Human is more sensitive to ~2 to 4 kHzA not-so-flat “frequency response”For the sake of human communication?Perception of loudness is also affected by age
Equal loudness contoursAIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 7
Image from: Fund. of Music Processing, p.26
MEASURING LOUDNESS
Sound power = (amplitude)2
Sound intensity = sound power per unit area (watt per sq. metre)
“Threshold of hearing” (TOH) vs “Threshold of pain” (TOP)!"#$ ≝ 10()* W/m*
TOP is roughly 1013 times of the intensity of TOH
Note: Power depends on distance!
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 8
Image from: Fund. of Music Processing, p.25
MEASURING LOUDNESS
Decibel (dB) is a logarithmic unit to measure the ratio
dB ! ≝ 10 ⋅ log)4!
!"#$ i.e. TOP is 130 dB when TOH is 0 dB
This value relative to TOH is also called the sound pressure level (SPL), i.e. dBSPL
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 9
Image from: Fund. of Music Processing, p.25
MEASURING LOUDNESS
Besides the intensity, the amplitude (which we discuss more) can also be represented in dB, usually we assume amplitude is 1 at 0dB
567 8 = 20 ⋅ log)4 8Do not mess up with:dBA or dB(A) represents the loudness perceived by human ear Compensation using the A-weighting basing on equal-loudness curves
10
This term is 20 because intensity is amplitude squared
Image from: https://en.wikipedia.org/wiki/A-weighting
MEASURING PITCH
Frequency range on a piano keyboard
Double frequency = “octave” higher The next occurrence of the same letter-name in music
“A440” = 440 Hz“Middle C” = 261 Hz
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 11
Freq
(Hz)
29.135
34.648
38.891
46.249
51.913
58.27
69.296
77.782
92.499
103.83
116.54
138.59
155.56
185
207.65
233.08
277.18
311.13
369.99
415.3
466.16
554.37
622.25
739.99
830.61
932.33
1108
.712
44.5
1480
1661
.218
64.7
2217
.524
89
2960
3322
.437
29.3
C1 C2 C3 D3E3 F3 G3 A3 B3 C4 D4E4 F4 G4 A4 B4 C5 C6 C7 C8
Freq
(Hz)
27.5
30.868
32.703
36.708
41.203
43.654
48.999
55 61.735
65.406
73.416
82.407
87.307
97.999
110
123.47
130.81
146.83
164.81
174.61
196
220
246.94
261.63
293.66
329.63
349.23
392
440
493.88
523.25
587.33
659.26
698.46
783.99
880
987.77
1046
.511
74.7
1318
.513
96.9
1568
1760
1975
.520
9323
49.3
2637
2793
.831
3635
2039
51.1
4186
An octave
MEASURING PITCH
Pitch, like dynamics, is perceived in a logarithmic mannerFeeling of doubling amplitude The change from 0.25 to 0.5 is similar to that from 0.1 to 0.2, a doubling of volume
Feeling of doubling frequency The change from 100 to 200 Hz is similar to that from 250 to 500 Hz, an octave higher
Perceptual scales of pitchMel-scaleCritical bands (Bark scale)Equivalent rectangular bandwidth (ERB) rate log Frequency
Empirical scales!
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 12
ASPECTS OF MUSIC
When sounds and rests are cleverlyput together, we have musicMelody combination of pitches (at different time)
Rhythm combination of long and short sounds
Harmony (chords) combination of pitches (at the same time)
Harmonic progression combination of chords (at different time)
Dynamics the expression in loudness
Structure how music sections are arranged
Texture the amount of different/similar sounds at times
Timbre different shapes of waveforms è different spectrum
There can be more!AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 13
DIGITIZING SOUND AND MUSIC
How can we represent audio/sound/music in a computer?Two main waysFaithful representation of what can be heard: audio recording and playbackTo improve the audio representation, research is on the improving sound fidelityReconstruction of music data (e.g. melody, rhythm, etc.): audio analysis and resynthesisTo improve the symbolic representation, research is on the analysis and synthesis methods
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 14
STORING AUDIO DATA
To represent audio signals in the digital world, digitizing is neededADC (analog-digital conversion) for recordingDAC (digital-analog conversion) for playing back
Sampling: the representation in timeThe higher sampling rate the better CD quality: 44.1 kHz è 44100 samples every second
Quantization: the representation in amplitudeThe more dynamic levels the better CD quality: 16-bit è 216=65536 different levels of amplitude
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 15
ADC DAC
Image from: Comp. Music Instruments, p.21
SAMPLING RATE
How many samples per second is the best?Storage requirements vs. sound quality
Sampling TheoremSignal reconstructable if frequency is not higher than Nyquist frequency: Ω = <=/2, where FS is the sampling rate Otherwise artifacts called aliasing will form (an extra freq. component)
Common rates
440Hz signal at 22,050Hz sampling rate
Aliasing occurs at (b) and (c)Image from: Fund. of Music Processing, p.62
Analog telephone: 8,000 Hz Pro digital video equipment: 48,000 Hz
Low quality audio: 22,050 Hz DVD-Audio, Blu-ray audio: 96,000 Hz
Audio CD, MP3: 44,100 Hz Super-Audio CD: 2,822,400 Hz
DYNAMIC LEVELS
The dynamic level is controlled by the bit depthTypically 8-, 16-, and 24-bit, corresponding to 28, 216, and 224 levelsQuantization error between analog value and quantized value
Represented as “Amplitude” Integer vs. floatingSigned vs unsigned
Common bit depthsAudio CD: 16-bit (~96 dB of range)Pro audio: 24-bit (~144 dB)
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 17
Rounding errors in quantizationImage from: Fund. of Music Processing, p.61
AUDIO CHANNELS
Different audio perception on our left and right ear helps determining the location of the sound source“Stereo” sound, with 2 channels Some professional recordings are shaped with Head Related Transfer Function (HRTF)
Surround sound systems5.1 or 7.1 with subwooferExtra channel = extra storage
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 18
Image from: https://lmkprod.com/what-is-hrtf-brief-explanation/HKBU LIATe theatre with
24.2-channel sound systemImage from: http://liate.hkbu.edu.hk/theatre.html
Dummy head microphone for immersive sound experience
STORING AUDIO DATA
The raw audio data, bit by bit, is called linear pulse-code modulation “linear PCM” encoding (usually as .wav files)Good enough for sound editing and manipulationConsuming too much storage CD quality: 44.1kHz 16-bit 2-channel = 80 minutes consuming ~700 MB
Compressed audio can fulfil requirements of the general public Lossy: MP3, AAC, WMA, … Lossless: FLAC, ALAC (.m4a), …Audio quality often measured as bandwidth requirement Spotify “normal quality”: AAC format at ~96 kbps
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 19
DEALING WITH MUSIC DATA
Musical Instrument Digital Interface (MIDI)Standardized in 1983Widespread technology to connect various musical hardware and software
Control data instead of actual audio dataSymbolic representationPlayback = Resynthesis on-the-flyBasic messages: “Note On/Off, note number, velocity”Only decides when and what should happen, but not HOW Synthesizer will make the actual sound (e.g. a violin sound, or a piano sound)
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 20
Image from: https://en.wikipedia.org/wiki/MIDI
BASIC MIDI MESSAGES
“Note On” denotes the starting point, where “Note Off” is to stopCalculating the note number (0-127)
> = 69 + 12×log 2( DEE4$F)
Velocity (0-127): How fast the key is pressed (correlates to playing intensity)
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 21
MIDI
22 25 27 30 32 34 37 39 42 44 46 49 51 54 56 58 61 63 66 68 70 73 75 78 80 82 85 87 90 92 94 97 99 102
104
106
C1 C2 C3 D3E3 F3 G3 A3 B3 C4 D4E4 F4 G4 A4 B4 C5 C6 C7 C8
MIDI
21 23 24 26 28 29 31 33 35 36 38 40 41 43 45 47 48 50 52 53 55 57 59 60 62 64 65 67 69 71 72 74 76 77 79 81 83 84 86 88 89 91 93 95 96 98 100
101
103
105
107
108
MIDI CHANNELS
16 MIDI channels on a MIDI connectionDriving 16 different instruments simultaneouslyEach having their own set of notes (note number, velocity, on/off) More than one note can happen at one time in a channel, but note on/off will correspond to note number
“Polyphony”What if you want a whole orchestra?
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 22
DEALING WITH MUSIC DATA
After “noteon”, the note can be further modified, e.g. with 7-bit values ofPitch bendModulationAftertouch/PressureControl change (CC) Damper pedal, sostenuto, expression, portamento, …
https://www.midi.org/midi/specifications-old/category/reference-tablesMIDI 2.0 draft is just announced in January 2019… Increasing bit precision: finer control of music expression
As always, the result depends on the synthesizer!
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 23
DISPLAYING MUSIC DATA
Musicians read music, just like we read textDifferent music notation software have their own proprietary standard
An “open” format: MusicXMLStoring music notes like MIDIYet it also stores how it should be rendered and displayed E.g. bar lines, key signatures, …
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 24
...<measure number="1" width="324.78">
<print><system-layout><system-margins><left-margin>0.00</left-margin><right-margin>752.71</right-margin></system-margins>
<top-system-distance>170.00</top-system-distance></system-layout>
</print><attributes><divisions>1</divisions><key><fifths>0</fifths></key>
<time><beats>4</beats><beat-type>4</beat-type></time>
<clef><sign>G</sign><line>2</line></clef>
</attributes><note default-x="83.07" default-y="-50.00"><pitch><step>C</step><octave>4</octave></pitch>
<duration>1</duration><voice>1</voice><type>quarter</type><stem>up</stem></note>
<note><rest/><duration>1</duration><voice>1</voice><type>quarter</type></note>
<note><rest/><duration>2</duration><voice>1</voice><type>half</type></note>
<barline location="right"><bar-style>light-heavy</bar-style></barline>
</measure>...
REAL WORLD MUSICAL INSTRUMENTS
How are audio signals generated in real life?Always vibrations…
Musical instruments are made to create regulated vibrationsUsually with a controllable pitch (periodic waveform)Different instruments show different spectra (frequency components)The vibration may be amplified with an acoustic body, e.g. Sound box of a violin Sound board of the piano
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 25
REAL WORLD MUSICAL INSTRUMENTS
Piano è ChordophonesKeys driving hammers to hit steel strings and make them vibrate
Strings è ChordophonesFriction between bow and strings create vibrations
Winds è AerophonesAir flow creates vibrations inside the tube of various length
Percussion è Membranophones/IdiophonesToo many possibilities! In general, an object vibrates when being struck
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 26
Note: We are not going to cover physics in this course!
ELECTRONIC MUSIC INSTRUMENTS
Since the development of electricity, musicians started to explore into new ways of expressing themselves, e.g.Telharmonium (1897) Tonewheels generates electric musical notes with electromagnets
Theremin (1920) Frequency/amplitude of the sound is controlled using the capacitance between hands and antennae
Analog synthesizers (1960 beyond) Sounds are generated and modified using op-amp, resistors, filters, and other electronics
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 27
Video from: https://youtu.be/pSzTPGlNa5U
INTERFACING WITH COMPUTERS
Audio interfaces“Sound cards”: providing signal conversion from analog to digital, and vice versaDAC + ADC
MIDI interfaces Input/output of MIDI messages between computers and instruments
Control surfacesProviding faders and buttons for direct control over DAW software UI
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 28
MIDI CONTROLLERS
With computers, now we can easily make music with software programsMIDI = prominent tool for music
MIDI controllers are the interface for musicians to make MIDI messagesMost controllers are in the shape of a piano keyboard 88-key, 72-key, … Weighted keys for realism of touching and expressiveness
Other shapes Winds, mallet percussion, drums, strings, …
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 29
NOVEL DESIGN OF INSTRUMENTS
There is a research conference for everything…“NIME — The International Conference on New Interfaces for Musical Expression” (since 2001): https://www.facebook.com/NIME2019 Check out publications if you are curious https://www.nime.org/archives
Roli Seaboard, LinnStrument, …Multi-dimensional, polyphonic
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 30
Image from: https://www.nime.org/past-nimes/
MUSIC SOFTWARE
SynthesizersEarliest software synthesizers simply emulate the analog synthesizersNow there are more complex kinds of synthesis which is only made possible by computer programsKeyboard workstations are controllers with an embedded system for synthesisSometimes exists as plugins for DAWs
VocaloidA special kind of sound synthesizer, which can mimic human voice by speech synthesis with controls of pitch and other vocal expressions
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 31
MUSIC SOFTWARE
Digital Audio Workstation (DAW)Allowing musicians to put together the tasks for recording, editing, and producingof audio dataMost of DAW supports VST plugins as a standard way to extend featuresTo name a few: Pro Tools, Logic Pro, Reason, GarageBand
Live PerformanceBesides using keyboard workstations, a computer with proper software can be used for live performance with maximum flexibility with MIDI controllers and VST pluginsExamples: MainStage, Ableton Live
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 32
MUSIC NOTATION AND RECOGNITION
Sheet music is still a major medium among musiciansNotation software allow easy typesetting with computer keyboard or MIDI controllersExamples: Sibelius, Finale, MuseScoreSome DAW can also output typeset music
While text can be recognized by OCR technology, handwritten or printed music can also be imported with proper recognition softwareMore and more popular on mobile devices
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 33
LECTURE REVIEW
We have discussed:How sounds are produced, and perceived by humanWays to digitize sound and music into computer systems Audio vs. symbolic representations
Music instruments and controllersMusic hardware and software
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 34
READ FURTHER
Section 1.3 “Audio Representation”, Fundamentals of Music Processing
Chapter 1 “Audio and Music Signals”, Computer Music Instruments
Section 8.1.1 “The MIDI protocol”, Computer Music Instruments
AIST2010 L2 – MUSIC IN REAL LIFE VS. MUSIC IN DIGITAL WORLD 35
Top Related