Music image processing Tim Bell Department of Computer Science and Software Engineering University...
76
Music image processing Tim Bell Department of Computer Science and Software Engineering University of Canterbury, Christchurch, New Zealand
-
date post
20-Jan-2016 -
Category
Documents
-
view
217 -
download
0
Transcript of Music image processing Tim Bell Department of Computer Science and Software Engineering University...
Music image processing
Tim Bell
Department of Computer Science and Software Engineering
University of Canterbury, Christchurch, New Zealand
Mu
sic
imag
e
pro
cess
ing
With…
o David Bainbridge (Waikato)
o Richard Lobb
o Dave Blizzard (Portland, OR)
o Richard Green
o John McPherson
o Karen Lin
o Annabel Church
o Simon Glass
Mu
sic
imag
e
pro
cess
ing
Overview
o OMR
o Digital music stand
o Page turning and image size
o Fast capture of music
o Music classification
Mu
sic
imag
e
pro
cess
ing
The vision...
o All music available on the web
o as score, recording and MIDI
o search by name, composer, lyrics, phrase, similarity, genre, novelty…
o remunerate those responsible
o culturally sensitive access
Mu
sic
imag
e
pro
cess
ing
Barlow and Morgenstern 1949
Mu
sic
imag
e
pro
cess
ing
Barlow and Morgenstern 1949
Mu
sic
imag
e
pro
cess
ing
Barlow and Morgenstern 1949
Mu
sic
imag
e
pro
cess
ing
Activities with music
o Composing
o Arranging
o Performance
o Teaching
o Musicology
o Recording
o Accompanying
o Transcribing...
Mu
sic
imag
e
pro
cess
ing
Digital music problems
o Cost of hardware and software
o Viewing music on small screen
o Loss of inspiration and creativity
o Loss of efficiency
o Learning curve
o Software compatibility
Mu
sic
imag
e
pro
cess
ing
Problems with paper
o Pen or pencil?
o Indexed retrieval
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
memorisedoral traditionoriginal ideas
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
live performancemp3, wav, cd
video?
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
sheet musicscreen
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
sheet musicscreen
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
sheet musicscreen
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
sheet musicscreen
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
sheet musicscreen
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
sheet musicscreen
D (lick #1) Dmin#5 Look at me now, will I ever learn?
D (lick #2) Dmin#5 GI don’t know how but I suddenly lose control.
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Live performance
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Live performance
read
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Live performance
play(interpret)
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Transcription
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Transcription
listen
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Transcription
write
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
BMP, GIF, JPEG
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
scanner,camera
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
print,display
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
MIDINIFF
MUSICXMLGUIDO
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Render(Sibelius, Lime, Guido, Tex etc.)
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
OMR
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Synthesis(audio rendering)
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Audio analysis(monophonic,polyphonic)
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Data entry
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Weak links
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Labour intensivelinks
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
QBH
Operations on music
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
ComposeArrange/orchestrate
RehearseReact
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Library (personal, shared)Music stand (rehearse, perform)
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Intermediate formArchive
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Suitable for:transpositionpart splitting
reductionsearching
theme detectionaccompaniment
performance following
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
listeningrecording studio
analysisthumbnailbacking
Mu
sic
imag
e
pro
cess
ing
Optical Music Recognition
Mu
sic
imag
e
pro
cess
ing
Mu
sic
imag
e
pro
cess
ing
Mu
sic
imag
e
pro
cess
ing
Wabot-2
o 1980-1984
o Read simple score
o Heavy processing requirements
Mu
sic
imag
e
pro
cess
ing
Mu
sic
imag
e
pro
cess
ing
Staff line removal/identification
o Horizontal projection
o Vertical slices
o Wobble/track
o Chords
o Template
Mu
sic
imag
e
pro
cess
ing
Mu
sic
imag
e
pro
cess
ing
Horizontal projection
Mu
sic
imag
e
pro
cess
ing
Piece at an angle
o Rotate until correct
Mu
sic
imag
e
pro
cess
ing
Vertical slices
Mu
sic
imag
e
pro
cess
ing
Vertical slices
Mu
sic
imag
e
pro
cess
ing
Wobble/track
Mu
sic
imag
e
pro
cess
ing
Object location
o Fragmentation
o Superimposed
o Touching objects
Mu
sic
imag
e
pro
cess
ing
Identifying objects
o Flood fill
o Template matching
o Hough transform
Mu
sic
imag
e
pro
cess
ing
Constructing musical features
o Grammars
o Decision tree
o Rules
Mu
sic
imag
e
pro
cess
ing
Musical semantics
o Treble clef determines pitches
o Accidentals change pitch
o Time signature changes note lengths
Mu
sic
imag
e
pro
cess
ing
Commercial systems
o Sharpeye
o Vivaldi
o Neuratron PhotoScore
o … and more
Mu
sic
imag
e
pro
cess
ing
Is 96% recognition good enough?
o One mistake in 24 notes
o No interpretation if playing music
o Time to set up, train and correct greater than typing?
Mu
sic
imag
e
pro
cess
ing
Not just notes…
Mu
sic
imag
e
pro
cess
ing
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Optical musicrecognitionin practice!
Mu
sic
imag
e
pro
cess
ing
The “gulf of interpretation”
o Classical tempo and dynamics
o Jazz improvisation
o Rock style (e.g. syncopation, articulation)
o Figured bass
o Cadenzas
o MIDI vs. Orchestra
Mu
sic
imag
e
pro
cess
ing
Interpretation
Mu
sic
imag
e
pro
cess
ing
Interpretation
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Pen-based musicdata entry
Mu
sic
imag
e
pro
cess
ing
Mu
sic
imag
e
pro
cess
ing
Mu
sic
imag
e
pro
cess
ing
Coloured staveline removal
o Scanned as RGB
o Convert to HSV and CMYK
o V indicates colour
o K indicates pencil, black pen
Mu
sic
imag
e
pro
cess
ing
Mu
sic
imag
e
pro
cess
ing
Mis-classified images
Mu
sic
imag
e
pro
cess
ing
Discussion?
