JavaOne 2010, Rock Star winning presentation on Fugue and Log4JFugue

50
JFugue and Log4JFugue Music for Fun and Productivity David Koelle Brian Tarbox Author of JFugue Author of Log4JFugue Senior Software Engineer Principal Staff Engineer Charles River Analytics Inc. Wabi Sabi Software

Transcript of JavaOne 2010, Rock Star winning presentation on Fugue and Log4JFugue

JFugue and Log4JFugueMusic for Fun and Productivity

David Koelle Brian TarboxAuthor of JFugue Author of Log4JFugueSenior Software Engineer Principal Staff EngineerCharles River Analytics Inc. Wabi Sabi Software

2

Overview> Learn about JFugue API to program music in Java

> Discover how to use music to increase your productivity

3

Introduction to JFugue> JFugue is an open-source API for programming

music in JavaTM code Generates MIDI music using Java Sound Provides a high-level interface to Java Sound’s

low-level API> Allows you to specify music naturally

player.play(“C D E F G A B”);

4

Programming Music with JavaSound// Play a Middle-CSequencer sequencer = MidiSystem.getSequencer();Sequence sequence = sequencer.getSequence();Track track = sequence.createTrack();ShortMessage onMessage = new ShortMessage();onMessage.setMessage(ShortMessage.NOTE_ON, 0, 60, 128);MidiEvent noteOnEvent = new MidiEvent(onMessage, 0);track.add(noteOnEvent);ShortMessage offMessage = new ShortMessage();offMessage.setMessage(ShortMessage.NOTE_OFF, 0, 60, 128);MidiEvent noteOffEvent = new MidiEvent(offMessage, 200);track.add(noteOffEvent);sequencer.start();try { Thread.sleep(track.ticks());} catch (InterruptedException e) { Thread.currentThread().interrupt();}

5

Programming Music with JFugue

// Play a Middle-CPlayer player = new Player();player.play(“C”);

6

Introduction to JFugue> JFugue is a programmer’s tool for musical

innovation Parsing and transforming musical patterns Creating rhythms Creating microtonal music Capturing from, or sending to, external devices

> JFugue enables interaction with other music tools and formats Reads and writes MIDI, MusicXML Extensible parser/renderer architecture

Writing Music in JFugue

Pattern pattern = new Pattern(“C D E F G A B”);Player player = new Player();player.play(pattern)

Music StringPattern

8

Vivaldi’s “Spring” in JFugue

Free sheet music from http://www.classical-scores.com

9Free sheet music from http://www.classical-scores.com`

Vivaldi’s “Spring” in JFugue

Pattern p1 = new Pattern("G6i G6i G6i F6s E6s B6q. B6s A6s");

G note, 6th octave, 1/8 duration

w Whole Durationh Halfq Quarteri Eighth

s Sixteentht 32nd x 64th o 128th

10

Vivaldi’s “Spring” in JFugue

p1 p1

p1 p1

p2

p2

p3 p3

p4 p4

p5 p5

p6

p7

p8

11

Vivaldi’s “Spring” in JFugue

p1 p1 p2

p1 p1 p2

p3 p3

p4 p4

p5 p5

p6

p7

p8

Pattern p1 = new Pattern("G6i G6i G6i F6s E6s B6q. B6s A6s");Pattern p2 = new Pattern("G6i A6s B6s A6i G6i F6i D6i B5i E6i");Pattern violin1 = new Pattern(p0, p1, p1, p2);

Pattern p0 = new Pattern("T[Allegro] V0 I[Violin] KEmaj");

Player player = new Player();player.play(violin1);

T TempoV VoiceI InstrumentL Layer

K Key Signature& Pitch Wheel+, * Pressure EventsX Controller Event

12

Vivaldi’s “Spring” in JFugue

13

Making Music Sound Great> MIDI gets a bad rap for creating dinky music> The real problem is the soundbank> Soundbanks are now replaceable

Thanks to the Audio Synthesis Engine Project!http://openjdk.java.net/projects/audio-engine

> Steps to better music: Use Gervill as your synthesizer http://gervill.dev.java.net Get a great soundbank http://www.sonivoxrocks.com Load soundbank instruments into the synthesizer

14

Making Music Sound Great// Place gervill.jar in your classpath to get an instance// of the Gervill SynthesizerSoundbank soundbank = MidiSystem.getSoundbank(soundbank file));// Load instruments from the soundbank into the synthesizerSynthesizer synth = MidiSystem.getSynthesizer();synth.loadAllInstruments(soundbank);// Create a JFugue Player object that is attached// to the synthesizer with the new instrumentsPlayer player = new Player(synth);// Now play your music with the new soundbank!player.play(your music here);

15

> Other features…

> Not all synthesizers support all of these features!

Other Features of the MusicString> Chords are easy to specify

Cmaj Amin7^^(or Amin7^E)

Attack and Decay Channel PressureController Events

Interval NotationPitch Wheel

Triplets and TupletsTies

Microtonal music

16

Rhythms> Build rhythms in a natural manner

#.##...#.###

> Easily re-assign instruments to notes> Combine chromatic notes with percussions

Create riffs within your beats!

17

RhythmsRhythm rhythm = new Rhythm();rhythm.setLayer(1, "OO..oO..OO..8OO.");rhythm.setLayer(2, "..*...*...*...*.");rhythm.setLayer(3, "......B.......B!");rhythm.setLayer(4, "..C.......C...CC");rhythm.setLayer(5, "^.^.^.^.^.^.^.^.");rhythm.addSubstitution('.', "Ri");rhythm.addSubstitution('O', "[BASS_DRUM]i");rhythm.addSubstitution('o', "Rs [BASS_DRUM]s");rhythm.addSubstitution('8', "[BASS_DRUM]s [BASS_DRUM]s");rhythm.addSubstitution('*', "[ACOUSTIC_SNARE]i");rhythm.addSubstitution('B', "[ACOUSTIC_BASS_DRUM]i");rhythm.addSubstitution('C', "[HAND_CLAP]i");rhythm.addSubstitution('^', "[PEDAL_HI_HAT]s Rs");rhythm.addSubstitution('!', "[CRASH_CYMBAL_1]s Rs");

Pattern pattern = rhythm.getPattern();pattern.repeat(4);Player player = new Player();player.play(pattern);

OO..oO..OO..8OO...*...*...*...*.......B.......B!..C.......C...CC^.^.^.^.^.^.^.^.

X ..xl...Y ..ym......Z ..zn

18

Rhythms with RiffsRhythm rhythm = new Rhythm();// Keep the code we just created, and add some hornsrhythm.setVoice(1, "X ..xl", "I[Trombone]");rhythm.setVoice(2, "...Y ..ym", "I[Alto_Sax]");rhythm.setVoice(3, "......Z ..zn", "I[Tenor_Sax]");rhythm.setVoiceOffset("................", 1);rhythm.addSubstitution('X', "Dwh<50");rhythm.addSubstitution('Y', "Fwi<50");rhythm.addSubstitution('Z', "Ahq<50");rhythm.addSubstitution('x', "Di"); rhythm.addSubstitution('l', "Ei");rhythm.addSubstitution('y', "Ai"); rhythm.addSubstitution('m', "Gi");rhythm.addSubstitution('z', "Fi"); rhythm.addSubstitution('n', "Bi");

19

Connecting to Musical Devices> Sending music to an external device is easy:DeviceThatWillReceiveMidi device = new DeviceThatWillReceiveMidi(MidiDevice.Info); sequence = player.getSequence(pattern);device.sendSequence(sequence);

> Reading music from an external device is easy:DeviceThatWillTransmitMidi device = new DeviceThatWillTransmitMidi(MidiDevice.Info);device.listenForMillis(5000);Pattern pattern = device.getPatternFromListening();

> Each of these involves lots of JavaSound code!

20

Using JFugue as “Musical Glue”> JFugue uses event-driven parsers and listeners

Current set: MusicString, MIDI, MusicXML> A Parser knows how to convert data into musical

events> A ParserListener (also known as a Renderer) knows how to

handle or render musical events

> Code to connect Parsers and ParserListeners:YourParser parser = new YourParser();YourParserListener listener = new YourParserListener();parser.addParserListener(renderer);parser.parse(whatever object the parser can parse);

21

Fun Things You Can Do With JFugue> Re-mix Johann Sebastian Bach

Read existing MIDI files into JFugue Patterns Perform Pattern Transformations on the music Recombine to come up with mutated Bach music

> Write “Piano Hero” that talks to a MIDI keyboard> Make a Virtual Instrument that creates microtones> Create a 3D animation that acts in anticipation of

upcoming music> Write a spam parser to turn junk email into music

and renders a light show

22

Music for Everyday Tasks> Having trouble keeping track of time?

GrandfatherClockp1 = "T110 I[TUBULAR_BELLS] E5qd120 D5qd120 C5qd120 G4h.d120";p2 = "T110 I[TUBULAR_BELLS] C5qd120 E5qd120 D5qd120 G4h.d120";p3 = "T110 I[TUBULAR_BELLS] C5qd120 D5qd120 E5qd120 C5h.d120";p4 = "T110 I[TUBULAR_BELLS] E5qd120 C5qd120 D5qd120 G4h.d120";p5 = "T110 I[TUBULAR_BELLS] G4qd120 D5qd120 E5qd120 C5h.d120";quarterPattern = new Pattern(p1);halfPattern = new Pattern(p2, p3);threeQuarterPattern = new Pattern(p4, p5, p1);hourPattern = new Pattern(p2, p3, p4, p5);hourGong = new Pattern("T110 I[TUBULAR_BELLS]

C5hqa100d120 Rs");

23

Music for Everyday Tasks

import org.apache.tools.ant.module.spi.AntLogger;public class BuildLogger extends AntLogger { @Override public void buildFinished(AntEvent event) { Player player = new Player(); Throwable t = event.getException(); if (t != null) { player.play("I[String_Ensemble_1] B3q Bb3q. G3i F3h"); } else { player.play("I[French_Horn] As E6h As E6i Rt As E6h"); } }}

> Want to know if your Ant build was successful?http://blogs.sun.com/geertjan/entry/ode_to_build_scripts

24

Music for Everyday Tasks> What if… you could listen to your log files?

Log4JFugue

2525

Log4JFugue> Enterprise scale applications are large and

complex

> Knowing what an application is doing is getting harder

> Many of us have spent sleepless nights looking at enormous log files trying to figure out the cause of a customer problem

2626

> Most efforts to make a program’s behavior more apparent involve Application Visualization

> That’s fine but we have more senses than that

> Use more of your natural senses, not just vision

> Convert your program’s log4j output into sound

Application sonification

2727

Sample application: Video on Demand> “Streams” refers to video content that is being

played

> Interesting events:

stream createstream failureTrick a stream

stream destroyAdd contentEmit SNMP trap

2828

Mapping stream events to music> Each interesting log message is associated with a

particular musical instrument

stream create [ACOUSTIC_BASS_DRUM]sstream destroy [ACOUSTIC_SNARE]sstream failure [CLOSED_HI_HAT]s

2929

Counting occurrences of stream events> Increment a counter for an instrument each time

an event is found in a log message

> Use double buffering to build one Pattern as another one is being played

> Count-based music lends itself naturally to rhythms

30

Log4JFugue architecture

Message scanner

Log msg listener

Pattern builder

JFugue player

Instrument counts

Instrument counts

Create and play music

Instrument count double

buffer

Parse log files

3131

Message counts become beat patterns> Higher message counts equate to faster beats> Each second of play is based on a 16-beat

measure

> Beat for a message that occurred once:rhythm.setLayer(1, ".......O........");

> Beat for a message for occurred four timesrhythm.setLayer(2, "..O...O...O...O.");

32

Demo – playing sample beat patterns

32

33

Audience participation demo

33

34

Why use rhythms?> Tonal music varies in both duration and pitch

But Log4JFugue’s count-based approach provides only one changing value

Tried making all notes the same pitch and vary the duration (whole / half / quarter notes)

Tried changing the pitch and duration based on the count

> Until we figure out how to generate pitch and duration from a single count variable we’ll stay with rhythms

34

3535

Performance considerations> Log4JFugue isn’t free

Performance overhead

> Every log message needs to be compared to every target string

> Uses a highly optimized MIT-based scanner

> Can run on a remote box or after the fact

3636

Main classes> DataGetter – gets the messages

SocketDataGetter, FileDataGetter, LogDelay> MessageReceiver – processes the messages

Music map High-speed string scanner from MIT

> SoundBuilder – builds the music string> Player – wrapper around JFugue’s Player

PretendPlayer – writes music to a file, for playing later

3737

Spring-based, loosely coupled> Log4JFugue is build as a set of very loosely

coupled, extensible classes

> Configure where to get your data Live application, existing log file, replay speed

> Configure which strings to search for and which instruments to associate with them

3838

Spring-based, loosely coupled> Configure how to build the music

Chromatic or achromatic

> Almost no glue code in main, the application is assembled via Spring

39

Setting up Log4JFugue> We assume your application already uses Log4j

> Change your appender to use SocketAppender

log4j.appender.SOCKET=org.apache.log4j.net.SocketAppender

log4j.appender.SOCKET.Port=4445log4j.appender.SOCKET.RemoteHost=192.168.1.3

39

40

Setting up Log4JFugue> Configure Log4JFugue to listen on the socket

<bean id="dataGetter" class="org.log4jfugue.SocketDataGetter">

<property name="writer" ref="writer"/> <property name="reader" ref="reader"/> <property name="port" value="4445"/></bean>

40

41

Setting up Log4JFugue> Configure Log4JFugue to play an existing log file

<bean id="dataGetter" class="org.log4jfugue.FileDataGetter">

<property name="writer" ref="writer"/> <property name="reader" ref="reader"/> <property name="fileName“ value="server.log"/>

</bean>

41

42

Configuring for different MusicString> Edit the spring.xml file> Rerun the program

<prop key="OPEN_HI_HAT">sendTrap</prop> <prop key="MARACAS">stream create</prop> <prop key="HI_WOOD_BLOCK">destroyImpl</prop>

42

43

Log4JFugue Conclusion> Log4JFugue allows you to turn a dull and difficult

task into something easy and fun Understand your own application’s behavior Solve problems faster Actually get to sleep at night!

> Highly configurable> Lots of room for improvement

44

Conclusion> JFugue makes music programming easy and fun

Create exciting new musical things! Get your kids interested in programming! Impress your co-workers! Rekindle your joy of programming!

> Project websites JFugue – http://www.jfugue.org Log4JFugue – http://www.log4jfugue.org

REMOVED SLIDES

47

Making Music Sound Great// Place gervill.jar in your classpathSoundbank soundbank = MidiSystem.getSoundbank(soundbank file);Pattern pattern = new Pattern(your music here);Patch[] instruments = pattern.getPatches();// Load instruments from the soundbank into the synthesizerSynthesizer synth = MidiSystem.getSynthesizer();synth.loadInstruments(soundbank, patches); // was “loadAllInstruments” // Create a JFugue Player object that is attached// to the synthesizer with the new instrumentsPlayer player = new Player(synth);// Now play your music with the new soundbank!player.play(pattern);

48

JFugueUI> New side project!> User interface components for creating music> Encourages the development of Virtual Instruments

VirtualInstrumentListener

> Currently in development: JFPiano

49

JFugue Musical Intelligence> Another new side project!> Heuristics for creating music

Suggestions for counterpoint Procedural generation of musical phrases Generation of chord progressions using common

chord leading rules> An open-ended experiment in programming music> Participation welcome!

Just try ideas and see what works

5050

A Groovy side note> Most of Log4JFugue is written in Java…> …But some of it is written in Groovy

> For now there is no compelling reason that the Groovy modules had to be Groovy, basically did it because I could

> On the other hand, this will keep my eyes open to further use of Groovy in the future