1

The General Temporal Workbench

A Universal System for Exploring Time-based Phenomena

Donald ByrdWoodrow Wilson Indiana Teaching Fellow

andResearch Technologies & School of Informatics

Indiana University Bloomington

2 May 2013

1 May 2013 2

Introduction (1): Time Is of the Essence

• Gandini Juggling’s Mozart (Symphony 25, 1st mvmt) performance

• How created? How did jugglers learn it?

• I don’t know, but I do know was a huge pain!

• Likewise for doing almost anything w/ complex temporal phenomena

• …and timescale can be yoctoseconds or billions of years

• Natural phenomena– Inflation of early universe, chemical reactions, animal movement,

weather, illnesses, climate; development of individuals, of species, of geographic features, & of galaxies

• Artificial & cultural phenomena– Movies, behavior of computer programs, football games, multimedia

shows, operas, political crises, wars, development of civilizations

3 July 2009 3

Introduction (2)

• Complex temporal phenomena difficult to study or (for performances) to create, because don’t “sit still”

• Can “play” repeatedly, but at expense of context• Usual answer: visualization• Good start, but we can (and need to) do much better than

existing systems• For music, General MusicVisualizer (GMV)• Music ex: Hendrix/Star Spangled Banner: how to notate?• Musical audio: Shakuhachi solo: how to segment? (Nina

Fales)– Might be a good solution to notating Hendrix!– Studying speech or animal sounds is a similar problem

20 Apr. 2009 4

GMV “demo”: Segment shakuhachi solo

• Scenario: music-informatics researcher or ethnomusicologist comparing two audio-segmentation algorithms

• …or composer comparing input & output of synthesis programs

1 May 2013 5

The Problem for “Multimedia”

• “Multimedia” = performance w/ music & sync’ed movement (dance, etc.) or visuals (fireworks, etc.)

• Requirements– 1. People need several ways to “look” at an instance

simultaneously

– 2. People need to be able to compare instances

– 3. Has specialized notation (at least for music)

– 4. Media: video, audio, still images, text• Audio independent of video is often important

28 Mar. 2010 6

The Solution for Multimedia

• General MusicVisualizer => General Multimedia Workbench (GMW)

• Early prototype of GMW: AWB Plus– Based on EVIA Annotator’s Workbench

– Java program, developed for ethnography

• Queen of the Night aria, in “GMW”• Fireworks show with music, in AWB Plus

4 Apr. 2009 7

GMW “Screenshot”: Queen of the Night aria

• Scenario: singer (or conductor) comparing videos of performances to learn role in opera, musical, etc.

• …or stage director, choreographer, or lighting designer comparing previous versions to own ideas

• …or scholar studying performances (perhaps juggling w/ music!)

28 Mar. 2010 8

Fireworks show with music, in AWB Plus

• From video & still photos of show by Mike Durbin•All timestamped => (semi)automatic alignment

• Music added afterwards!

1 May 2013 9

The General Problem

• Long-standing, difficult problems in all fields– Affect all uses: research, creation, “aesthetics”,

organization• …plus deluge of data in many fields

– Even arts & humanities are getting lots of hard data– Data promises to help, but not much help so far!

• Need insight, not data; how to get there?– Widely recognized as important goal

1 May 2013 10

Towards a Solution (1)

• Review: multimedia features– 1. People need several ways to “look” at instance simultaneously

– 2. People need to be able to compare instances

– 3. Has specialized notation (at least for music)

– 4. Media: video, audio, still images, text

• Observations– Multimedia features #1 & 2 are true of every (and 3 & 4 of many)

complex temporal phenomenon

– Every temporal phenomenon is complex

– People often want to relate phenomenon of one type to another

1 May 2013 11

Towards a Solution (2)

• The cross-discipline argument– Problems in all fields have much in common => potential for

synergy

• But people rarely share ideas or software across disciplines– Issue of “disjoint technical vocabulary/literature” (cf. Swanson

1988)– Ex: Are there periodic patterns in public opinion? => political

scientists should try Fourier analysis, but…• The research vs. creation vs… argument

– Problems of researchers, creators, etc. in a field have much in common

• => A general temporal system could greatly reduce “reinventing the wheel” or doing without one

1 May 2013 12

The General Solution

• General temporal system applicable to very many disciplines is possible

• General temporal system applicable to all types of use is possible

• GMusicW needs so much flexibility, a GAnythingW wouldn’t need much more

• Oh, yeah? With what magic?

• The magic of Software Architecture!

15 Mar. 2009 13

Architecture for a General System (1)

• Main tasks for framework: (temporal) synchronization, UI support• Software for common audio & video uses:

15 Mar. 2009 14

Architecture for a General System (2)

For sequential art & movies based on sequential art (John Walsh):

5 Mar. 2010 15

General Temporal Toolkit/Workbench (GTT/W)

• “Temporal”, not just “Multimedia”• Supports multiple: coordinated, editable, interactive

visualizations & sonifications (eventually “tactilizations”, “olfactizations”?)

• …of multiple instances• …of any combination of temporal phenomena• …plus data mining & analysis (Solution 2)• For all applications (research, creative, other)

• Myosin V, in AWB Plus

7 Apr. 2009 16

Important Feature 1: Better Human/Machine Partnerships

Figure slightly modified from Yu et al (2008)

• Integrate info visualization & analysis/data mining (Shneiderman 2002; Yu et al 2008) => closed loop: use visual perception to generate hypotheses for analysis; present results of analysis visually– “HCIR”, visual analytics

• …or substitute synthesis for analysis, e.g., for composers

2 May 2013 17

Important Feature 2 & 3: Allow All Sensory Modes; Automatic Synchronization

• Sensory Modes– Visual: visualization is most generally useful, but not the only answer– Auditory: sound is central for many applications

• sonification is surely valuable for some non-audio phenomena

– Tactile: important for the blind– Other (olfactory): important for ??– Don’t rule out support for all sensory modalities

• Automatic Synchronization– Of audio (at least) has made great progress in last 10 years

28 Mar. 2010 18

Myosin V, in AWB Plus

• From nanobioscience work by Craig & Linke• Myosin V crawling along a strand of acetin

1 May 2013 19

Big Issue: Time Representation

• Resolution/precision– Cosmology: Planck time to quadrillions(?) of years

• Precision & handling of uncertainty– Historical forms: “1685?”, “fl. mid-15th century”, etc.

– Damaged sources: “25th of [unreadable] 1743”, etc.

– Handle with Probability Density Functions?• Absolute & relative time

– Calendars are incredibly messy• Design of universal time representation is well

along

1 May 2013 20

Huge Issue: Getting Off the Ground (1)

• AWB Plus prototype, based mostly on EVIA Annotators Workbench; some of Variations– may also use CIShell, Chen Yu’s system for “visual mining of

multimedia data”– all from IUB

• Other possible open-source starting points– Sonic Visualiser, SyncPlayer, SIMILE Timeline, etc.

• Connections to general tools for nontemporal visualization– Network Workbench (Katy Börner/IU SLIS): networks– Google Map API: “space” (surface of the earth)

• Connections to semi-general tools– Ex: Max/MSP, Pd: musical audio, video– Ex: imagej for image display/processing

1 May 2013 21

Getting Off the Ground (2)

• Support from communities of users is almost essential• Have seen strong interest for:

– Ethnomusicology field work– Opera scholarship

• “Could revolutionize study of opera”

– Conducting– Teaching electronic music – Study of computer-aided musical improvisation– Study of comic books/movies– Research on artificial life– Software engineering, research software– Nanoscience/nanotechnology– Study of French

1 May 2013 22

Conclusions

• How do I know applications are realistic?– Many probably aren’t, but many, many possibilities exist!– Have ca. 30 usage scenarios, ca. half written/endorsed by experts– Some examples (all experts in respective fields)

• Ruth Stone: ethnomusicology field work• Philip Gossett, Nick Cook: study of music/opera performance• Paul Nadler: conducting• John Gibson: teaching electronic music • Elaine Chew: annotating video of computer-aided musical

improvisation• John Walsh: comic books/movies• Larry Yaeger: artificial life• Fabian Fagerholm, Ben Kovitz: software engineering, research S/W• Amar Flood: nanoscience/nanotechnology

– Personal knowledge/experience for a few

1 May 2013 23

End

• Thanks to Eric Isaacson, Geoff Chirgwin, Will Cowan, Allen Winold, Paul Sturm, Ben Kovitz, Craig Stewart, Eric Wernert

• THE END

rev. 18 Feb. 2009 24

Extra Slides

• Following slides (with background like this one) are just in case…

rev. 18 Feb. 2009 25

Good Design for Music Can Be Good for Many Things

• Cf. “Why Studying Music is Both Difficult and Important” (Byrd 2009)

1. Music is an art => people use elements in unusual ways

2. Music is a performing art => performances & symbolic representations

3. Much music has complex synchronization requirements

4. Music involves many different instruments, often in groups. Leads to:– Arrangements/transcriptions for other instruments

– Versions for players with different levels of skill

– Notation may represent sounds or actions

5. Music is often combined with text via singing, narration, etc.

6. Music is extremely popular, so:1. Some works exist in many versions, arrangements for different ensembles, etc.

2. Handling challenges is important, even on purely economic grounds

7 Mar. 2009 26

HCI: Searching, Browsing, & Visualization

• Visualization is essential for browsing, merely helpful for searching• In browsing, user finds everything; the computer just helps• Browsing is obviously good because it gives user more control, but

few systems emphasize it. Why?– “Users are not likely to be pleasantly surprised to find that the library has

something but that it has to be obtained in a slow or inconvenient way. Nearly all items will come from a search, and we do not know well how to browse in a remote library.” (Lesk, p. 163)

• For “and”, read “as long as”• Searching is more natural on computer, browsing in real world

– Effective browsing takes very fast computers—widely available now– Effective browsing has subtle UI demands

• Cf. HCIR, visual analytics, visual searching, etc.

Why juggling? Who cares?

• A surprising domain, but realistic– Features 1 & 2 apply– Feature 3 applies in part: has established (though not graphic)

notation, “siteswap”• Many juggling programs available• GTW framework has support for:

– Control of tempo, including pausing or going backwards– UI for (temporal, not spatial) zooming in on details– Synchronization of multiple videos and/or animations– Framework for auto. synchronization– Framework for combining independent visualizations

• Animal motion in general is much more complex => more need for GTW!– Ex: dancing (Labanotation, etc.)

10 Feb. 09 28

Structure in Basic Representations of Music & Audio

Western Music Notation: very complex, irregular structure; some parts well-defined, some not—and what’s well-defined isn’t well-defined

Audio: no explicit structure

MIDI: simple, regular, well-defined structure

10 Feb. 09 29

Basic Representations of Music & Audio

1. Audio (e.g., CD, MP3): like speech

2. Time-stamped Events (e.g., MIDI file): like unformatted text

3. Music Notation: like text with complex formatting

• Time scales of graphs: #1, milliseconds; #2 & 3, seconds

• Essential difference among forms: “knowledge representation” = explicit structure

12 Feb. 09 30

“Isn’t it a mistake to use music notation this way?”

• Chris Raphael’s question about Hendrix transcription• It’s obviously useful: easy to find phrases, “Taps”, etc.• …but seriously misleading in places• But CMWN is “always” misleading!• Is it useful enough to justify danger of misleading?• Knowledge representation has inevitable bias (Davis et al

1993); notation has more bias (Wiggins et al 1993)• Fundamental issue of transcription in ethnomusicology• Conclusion: use it, but be careful

– Cf. my “Logician General’s Warning” on classification– …in fact, transcribing requires classifying constantly

18 Feb. 2009 31

Sequential-Art/Movie: The Hard Goodbye (1)

• From Frank Miller’s “Sin City” series• John Walsh (SLIS): want to compare comics, movies of them, etc.

18 Feb. 2009 32

Sequential-Art/Movie: The Hard Goodbye (2)

• From Frank Miller’s “Sin City” series• John Walsh (SLIS): want to compare comics, movies of them, etc.

18 Feb. 2009 33

Types of Visualizations of Music (and more)

• Is visualization static or dynamic?– Dynamic = time represented by time– Static = time represented by space

• What features are visualized?• What basic representation? Audio, symbolic,

both?– Easy to generalize to plays (score = script) & other text

phenomena, dance, etc.

26 Feb. 2009 34

Types of Visualizations of Music (and more)

• Hendrix example uses coordinated visualizations– Generalization of parallel, aligned, synchronized, etc.

• How are multiple visualizations coordinated?1. Parallel panes of a single window2. Superimposed in a single window3. Separate coordinated windows

• Forms 1 & 2 apply directly to audio (incl. sonification)• Easy to interpolate between forms 1 & 2

– Categories in the real world are rarely discrete

20 May 08 35

The Ultimate Music System

• Original goal: visualizer that can do anything with music– Handle any no. & combination of visualizations

– Static visualizations: audio, any kind of notation, structural diagrams, etc.

– Dynamic visualizations: video, etc.

– Automatic (or near-automatic) synchronization

– Support OS-level technologies (QuickTime, etc.)

– Easy-to-learn UI allowing high degree of control• User may want frequent extreme zoom changes => help with

• If it could do all that, should be useful for lots (domains with >=2 Features) besides music!