Learning to Generate High-Quality Music With Performance RNN

8
Final Project: Learning to Generate High-Quality Music With Performance RNN GCT634 Musical Applications of Machine Learning Spring 2019 KAIST Student: Pham Xuan Trung Kiheum Cho School of Electrical Engineering #14

Transcript of Learning to Generate High-Quality Music With Performance RNN

Page 1: Learning to Generate High-Quality Music With Performance RNN

1/8

Final Project:

Learning to Generate High-Quality Music

With Performance RNN

GCT634

Musical Applications of Machine Learning

Spring 2019

KAIST

Student: Pham Xuan Trung

Kiheum Cho

School of Electrical Engineering

#14

Page 2: Learning to Generate High-Quality Music With Performance RNN

2/8

Introduction

Expressive timing and dynamics are essential parts of music.

Expressive timing

refers to variation in performed durationsamong notes represented in a musical score with a single rhythmic value.

Dynamics

refer to the volume of sound. It relates to the variation of loudness of music over time. In piano, the loudness of a note depends on how hard the pianist hits the key.

Task – Generating piano music with expressive timing and dynamics.

Page 3: Learning to Generate High-Quality Music With Performance RNN

3/8

Why is it an interesting problem?

A sample from sheet music

Two main elements of music:

1. The composition focuses on the musical notes that define a song.2. The performance focuses on how the notes are played by the performer.

Can a machine perform like a human?

Can a machine produce the dynamics for digital sheet music and so, create stylized performances?

Using a RNN to do that task.

s

ot

w

Uxt

o0

U x0

s0

w

o1

U x1

s1

ot

U xt

st

…

Page 4: Learning to Generate High-Quality Music With Performance RNN

4/8

Related work

• We follow the idea from paper: “Learning to Create Piano Performances”, NIPS2017 conference. [1]

[1] “Learning to Create Piano Performances”, Sageev Oore, NIPS 2017; [2] https://magenta.tensorflow.org/performance-rnn

Music generation with score (16s) Music generation with performance

(16s) – Performance RNN tensorflow [2].

All previous work:Generating output is a sequence of notes

(scores).

This paper:Generating the notesGenerating expressive variations in the

timing and musical dynamics (loudness)

Page 5: Learning to Generate High-Quality Music With Performance RNN

5/8

• International Piano-e-Competition, recorded MIDI files

• Example

1) beethoven_opus90_2_format0.mid

2) schub_d960_2_format0.mid

…

N) schubert_d935_3_format0.mid

Dataset

http://www.piano-e-competition.com/midi_2018.asp

http://www.piano-e-competition.com/
http://www.piano-e-competition.com/midi_2018.asp
Page 6: Learning to Generate High-Quality Music With Performance RNN

6/8

Method

Previous

• Note-based

• Express midi with quantized note data

Performance RNN

• Event-based

• Convert note data to event data considering each note relationship

Page 7: Learning to Generate High-Quality Music With Performance RNN

7/8

Implementations

• Baseline• Our code is based on homework3. We use its network and some configurations.

• Data Preprocessing• Midi -> Note -> Event -> Array (w/ 256 label).

800

850

900

950

1000

1050

1100

0 200 400 600 800 1000 1200 1400

Tra

inin

g l

oss

Epochs

Learning curve

Example code to generate Event

Performance RNN in Pytorch

Page 8: Learning to Generate High-Quality Music With Performance RNN

8/8

Generated music (midi files)

note-based.midevent-based.mid

• Note-based (HW3) - Quantized• Event-based (Ours) - Time Shift


Dp-Rnn: Type Ii Diabetic Prediction Using Gkfcm And Rnn

Dp-Rnn: Type Ii Diabetic Prediction Using Gkfcm And Rnn

DeepCPU: Serving RNN-based Deep Learning Models 10x Fasterweb.eecs.umich.edu/~mosharaf/Readings/DeepCPU.pdf · DeepCPU: Serving RNN-based Deep Learning Models 10x Faster ... fluencing

DeepCPU: Serving RNN-based Deep Learning Models 10x Fasterweb.eecs.umich.edu/~mosharaf/Readings/DeepCPU.pdf · DeepCPU: Serving RNN-based Deep Learning Models 10x Faster ... fluencing

SongNet: Real-time Music Classificationcs229.stanford.edu/proj2018/poster/53.pdf · International Pop Experimental Hip-pop Folk qResults üUnderstand why C-RNN cannot perform well

SongNet: Real-time Music Classificationcs229.stanford.edu/proj2018/poster/53.pdf · International Pop Experimental Hip-pop Folk qResults üUnderstand why C-RNN cannot perform well

CUN PANNN DVELPN RNN UD - Denver

CUN PANNN DVELPN RNN UD - Denver

Blueprint wants to change Vancouver's music festival experience with Generate | BetaKit

Blueprint wants to change Vancouver's music festival experience with Generate | BetaKit

VGM-RNN: Recurrent Neural Networks for Video Game Music ...

VGM-RNN: Recurrent Neural Networks for Video Game Music ...

Computing the Gradient in an RNN - University at Buffalosrihari/CSE676/10.2.2 RNN-Gradients.pdfDeep LearningComputing BPTT gradients recursively Srihari • Start recursion with nodes

Computing the Gradient in an RNN - University at Buffalosrihari/CSE676/10.2.2 RNN-Gradients.pdfDeep LearningComputing BPTT gradients recursively Srihari • Start recursion with nodes

(final) RNN Implementation on FPGA

(final) RNN Implementation on FPGA

Recurrent Neural Networks (RNN) - Mines ParisTechperso.mines-paristech.fr/fabien.moutarde/ES... · Deep-Learning: Recurrent Neural Networks (RNN), Pr. Fabien MOUTARDE, Center for

Recurrent Neural Networks (RNN) - Mines ParisTechperso.mines-paristech.fr/fabien.moutarde/ES... · Deep-Learning: Recurrent Neural Networks (RNN), Pr. Fabien MOUTARDE, Center for

Stock Price Prediction Using RNN and LSTM

Stock Price Prediction Using RNN and LSTM

MAM-RNN: Multi-level Attention Model Based RNN for Video ... · MAM-RNN: Multi-level Attention Model Based RNN for Video Captioning Xuelong Li1, Bin Zhao2, ... video feature is encoded

MAM-RNN: Multi-level Attention Model Based RNN for Video ... · MAM-RNN: Multi-level Attention Model Based RNN for Video Captioning Xuelong Li1, Bin Zhao2, ... video feature is encoded

Deep Learning for Load Forecasting: Sequence to Sequence … · 2020-06-11 · a Sequence-to-Sequence (S2S) RNN which combines an encoder RNN and decoder RNN has shown a greater success

Deep Learning for Load Forecasting: Sequence to Sequence … · 2020-06-11 · a Sequence-to-Sequence (S2S) RNN which combines an encoder RNN and decoder RNN has shown a greater success

Movie Sentiment Analysis using Deep Learning RNN

Movie Sentiment Analysis using Deep Learning RNN

Music Technology, Idaho Content Standards...Idaho Fine Arts Standards – Music Technology Page 1, Music Technology Music - Music Technology Strand g Anchor Standard 1: Generate and

Music Technology, Idaho Content Standards...Idaho Fine Arts Standards – Music Technology Page 1, Music Technology Music - Music Technology Strand g Anchor Standard 1: Generate and

CNN RNN Autoencoder - UNISTisystems.unist.ac.kr/wp-content/uploads/sites/209/2017/... · 2018. 12. 28. · CNN RNN Autoencoder. Today 1. Machine*Learning 2. Deep*Learning –CNN –RNN

CNN RNN Autoencoder - UNISTisystems.unist.ac.kr/wp-content/uploads/sites/209/2017/... · 2018. 12. 28. · CNN RNN Autoencoder. Today 1. Machine*Learning 2. Deep*Learning –CNN –RNN

cs221 final poster - Stanford Universityweb.stanford.edu/class/cs224n/posters/15744482.pdfContext Final Representation RNN Self-Attention RNN Inter-Attention RNN Contextualized Word

cs221 final poster - Stanford Universityweb.stanford.edu/class/cs224n/posters/15744482.pdfContext Final Representation RNN Self-Attention RNN Inter-Attention RNN Contextualized Word

HNPRT LB (LN RNRN D PLNT DNT RNN - COREt Btn ltn 22: 42 l, HNPRT LB (LN RNRN D PLNT DNT RNN DVNT Rfl ZRT, blrd PR, nl NT ntn TRN RN. hnprt lb (Ltt rnrn d plnt dnt rnn dvnt. L rnn n

HNPRT LB (LN RNRN D PLNT DNT RNN - COREt Btn ltn 22: 42 l, HNPRT LB (LN RNRN D PLNT DNT RNN DVNT Rfl ZRT, blrd PR, nl NT ntn TRN RN. hnprt lb (Ltt rnrn d plnt dnt rnn dvnt. L rnn n

RNN LSTM and Deep Learning Libraries · RNN LSTM and Deep Learning Libraries UDRC Summer School Muhammad Awais m.a.rana@surrey.ac.uk

RNN LSTM and Deep Learning Libraries · RNN LSTM and Deep Learning Libraries UDRC Summer School Muhammad Awais [email protected]

Affective Music Generation and its effect on player experiencemarcoscirea.com/thesis/Scirea-Affective_Music_Generation.pdf · mes that generate music dynamically, which is the focus

Affective Music Generation and its effect on player experiencemarcoscirea.com/thesis/Scirea-Affective_Music_Generation.pdf · mes that generate music dynamically, which is the focus

RNN Slides

RNN Slides