SoundsfromTheGardenButtMusicfromHell

AlBiles

Overview

• AudioinInteractiveMedia• CaseStudy:TheGarden,including…• TheButtMusicfromHell• Advicefordoingaudioinyourproject

Audio

• Criticaltoimmersiveuserexperience– Nolongeranenhancement,nowanexpectation– Barapproachingqualityofmoviesound

• Lastsetofassetsaddedtomostprojects– Getleftoverresources(memory&machinecycles)– Absorboverflowsfromearlierphases(money&time)– Havetoreacttowhat’salreadydone

• Audiogetssqueezed

Linearvs.InteractiveMedia

• Linear(movies,videos,etc.)– Onlyonepaththroughcontent,fixedtiming– Canfinetuneaudiotofitvisuals

• Interactive(simulations,games,VR,installations)–Multiple,semi-unpredictablepathsthroughcontent– Timingdependsonuseractions– Audiohastobeagile,morereactive

ConventionalAudioAssets

• Soundeffects/Foleyart– Visiblesourceinthenarrativespace

• Ambientsound– Can’tseesource,butcanhearit&expectit

• Voicework– Dialog,narration

• Music(cues)– Theme,background,source

• Interface(UI)sounds– Auralfeedbackforuseractions

Diegesis

• Diegetic:Heardbycharactersinnarrativespace– Soundeffects,ambient,dialog,exclamations– Sourcemusic:Musiciansplayinginscene,radio,etc.– Placeinnarrativespace:L-Rpan,level,reverb,etc.

• Non-diegetic:Heardonlybyuser/player– Narration,voiceover– Theme,background,incidentalmusic– Interfacesounds– Placeinuser/player’shead

PointofPerception(PoP)

• PoP =PointofView+PointofAudition• 3rd-person(Objectivecamera)– PoP isobserverofnarrativeworldthroughwindow– Diegeticsoundsallcomefromwithinthewindow

• 1st-person(Subjectivecamera)– PoP isuser/player’spositioninside narrativeworld– Soundscancomefromanywhere,evenoffscreen

• Almostallmoviesare3rd-person• Lotsofgamesare1st-person

SoundEffects/FoleyArt

• Trigger– User/playeractions,gameevents– Simple:Justcallplay()

• Localization– Diegetic=>placesoundatvisiblesource– UseLRpan,gain,EQ,reverb,Doppler(ifmoving)– Devenvironmentmightprovidereal-time3D

• Placesoundatsourcelocationinvirtualworld• Placemicatcameralocation• Physicallymodelwhatmichears

AmbientSounds

• Trigger–Whenenteringnewsetting,changinggamestate– Looptextures(croakingfrogs,wind,streetnoise)– Randomlyplaydiscretesounds(birdcalls,siren)

• Localization– Diegetic=>placethemsomewhereintheworld– Loopedtexturesplaced“everywhere”– Discretesoundsplaced“anywhere”

VoiceWork– Narration

• Trigger– Usuallyatagamestatechange– Playspeechtocompletion

• Localization– Non-diegetic=>Usuallymonaural,minimaleffects– Supposedtosoundinsideuser’shead

VoiceWork– Dialog

• BetweenNPCs(NonPlayingCharacters)– Triggeralinewhenplayerspeaks– LocalizetoNPClocation,withappropriateeffects– Oftensynchedtoanimation(especiallyincutscenes)

• BetweenNPCandplayer– NPCaddressingplayernoproblem(sameasabove)– Player’sspeechisaproblem

• Speechrecognitionnotreadyforprimetime• Mostdialogsaretext-based,atleastforplayerinput• Messy,tosaytheleast

Music

• Thememusic– Foreground:Makememorablefirstimpression– Usuallyastand-alone“tune”

• Background/underscoring– Establishsetting,enhancemood– Nota“tune”:Shouldn’tdrawattention

Music

• Incidentalmusic– Literallyaccompaniesanincident/event– Can“magnify”theevent(stinger)–Maybeforeshadowtheevent?

• Sourcemusic– Diegetic–Musiciansinnarrativespace,musiconradio– Usuallylocalizedtooriginofmusic

InteractiveMusic

• Musicshouldchangewithin-gameevents– Triggeredbynarrativeorbyuser/playeractions– Can’tjustplaycueonceandbedone

• Usuallyrequiresdeconstructingthecue– Horizontally:intoloopablephrases– Vertically:intolayerable parts

• Eventstriggertransitionswithin thecue• Differentcompositionalprocess

CaseStudy:GardenofEarthlyDelights

TheTriptych

TheGame• GamesetintheHellpanel

“Trappedinaworldtowhichtheydonotbelong,playersmustfindawaytoescape.Hell isastrangeplace,withlostsoulslockedintohellsoftheirowncreation.”

• PlayerGoalBecomethenewoverlordoftheregion

• Mechanics:2ModesSpirit:Movefreelybutcan’tinteractwithNPCsFlesh:PossessNPCbodies, interact,usestuff

• Gameplay:1st personPoPExploreandsolvepuzzlestounlockNPCbodiesandpossessthem

TheProject

• 2years:Mostworkinspring2014&spring2015• 12+faculty,30+students from5+academicunits• Lotsofteams– Gamedesign– Conceptart– 3Dmodelling&animation– Textures&UI– Development(programmers)– Audio– Research

QuickProjectCritique

• Greatlearningexperience…• Goalmorphed– From:serious/educationalgameaimedatthePrado– To:anadult-ish gamethatembracedthehellpart

• Scope:Tooambitious(Whatelseisnew...)• Semester:Artificialtimelineconstraint– Studentsdoingfinalprojectsforaclass– Allcameinatendofsemester:Notimetointegrate

• Bottomline:Needsmorepolish

AudioAssets

• 140+individualassetsused,alloriginal– LotsofSFX,voicework– Someambient,interfacesounds– Onlyonepieceofmusic

• Simpleinteractivity=>Nomiddlewareneeded• UsedUnitylocalizationforsomesounds

AudioTeam

• Facultylead• StudentsfromInteractive&GameAudioclass– 2studentsfromspring2014,4fromspring2015– Theirfinalprojects

• Idefinedinitiallistofneededassets,servedasconsultant,basicallystayedoutoftheway

• Studentsdidallproductionwork &deliveredassetstoprogrammers,except…

• …Ididthetune

AudioStudents!

AngelaMuscariello

DustinKochenspargerLindseyEllis TomFarrell

JohnMilliganRyanGochnauer

SFX/Foley

• Nolibrarysoundsinfinalgame• Recordedeitherinsoundlaborinfield(literally)• Soundsforalloccasions– Footsteps(walkcycles)– Cards,coins,knifesounds,mugsclanking– Openingdoors,crates;pickingthingsup,dropping– Heartbeats,dismembering,soulsucking,possession– Drinking,pouring,potionsounds– Interfacesounds

Forexample:WalkCycles

• Recordlotsoffootstepsindividuallyortogether• Editthemtogetherintoaloop–Maybehavealternativeloops–Maybehavealternativefootstepswithinaloop

• Differentsurfaces– Leaves–Mud–Water–Wood

MoreFoleyExamples

• ScrollOpening• FireonSpawn• Happiness-SuckingPotion• DeathGargle• SwanDeath

AmbientExamples

• Ghostwhispering• Wind• Demonhordeskittering• ScaryVoice

VoiceWork

• Dialogshandledwithtext– Userinteractionusesdialogboxes– DecidednottopronounceNPCtext

• Didlotsofvoiceactingforseveralmechanics– Reassembling/disassemblingwife’sbody– Bossbattlewiththerabbit

• Lotsofambientspeech:Voicesinamob

VoiceWorkExamples

• IntroAnnouncement• Wife(dis)assembly

• Crowdhecklingatbossbattle

• Oooohhh!

Music

• Nothemeorbackgroundmusic• Onlyonetune:Linear&diegetic• Usedinmechanicwhereplayersolvesaudiopuzzle tounlocktune&bestownewability

• Leveragedirresistiblefeatureofpainting…

TheBand!

HurdyGurdy

Drum

Shawm

Lute

Harp

Sacbut

CornettFlute

TheButtMusic

ButtMusicAligned

• PlainchantNotation• Four-linestafffornotepitches

• Nonotelengths• Nomeasurelines• Notimesignature• Supposedtohavelyrics

• GregorianChant

RealPlainchant

Cclef:ThislineisC

VerticalLine:PauseorrestLongerShorter

Climacus:aNeume

Punctum

• Neumes tellhowtosingasyllableofthetext

• Pitch[sequences]forasinglesyllable

• Severaldifferentsystems

• Replacedbyourfamiliar5-linestaff

• Stillusedinsomechurchestoday

BoschWasNoMusician• NoLyrics:Blasphemy!• Clefpaintedout?• Neumes don’tmatchanyknownsystem(justabrushgesture?)

• Neumes don’tlineupwellwithlines,sopitchesareambiguous

• Boschlikelyusingmusicas“decoration,”notasactualmusic

MusicasDecoration

WhattheHell…

• Transcribeitanyway!• Othershavedonethis• RecentlyinFebruary,2014– AmeliaHamrick– StudentatOklahomaChristianU.– Straightforwardinterpretation– Impose4/4timebydefault– Initialblogpostwentviral– Gotrecordedwithchoral“lyrics”

Amelia’sTranscription

ChoralRecording

• Gregorianchantinareally,reallybigcathedral• Onrepeat,imposed3/4timewithguitar&bass• Allegedlyrics:

Buttsongfromhell,Thisisthebuttsongfromhell.WesingfromourasseswhileburninginpurgatoryThebuttsongfromhell,Thebuttsongfromhell.Butts!

Al’sInterpretation

• AssumeCclef• Assumeallneumesarepunctums

• Interpretnotesasliterallyaspossible

• Handleambiguitiesasmusicallyaspossible

• Usebuttcrackasrestsinbottomtwostaves

• Useinstrumentsfromthepainting

Al’sScoreAssumeallquarternotesinCmajor(whitekeysonpiano)

SetdronestoC

Twobutt-crackrests

Addapitcheddrumlinetoprovideanunevenrhythm

Holdthelastnote

RenderedwithMIDI

HurdyGurdy

• DecidedtoplaythemelodiclineonaHurdyGurdy because:– It’sareallycoolperiodinstrument– Thisisthefirstdepictionofabuzzingdronestring(calledadog)

• Youcanallegedlybuyreplicasofthisonefor$5000(!)

AnatomyofaHurdyGurdy• Rosinedwheelexcitesstringsasit’scranked

• Keysshortenstringslikeanautoharp

• Keysdon’taffectdronestrings

• Buzzingbridgeaddsdifferenttimbretoonedronestring

• Basicallyastringversionofabagpipe(alsoexisted)

FakingaHurdyGurdy

• SynthesizedwithMIDIinstruments• UsedaYamahaMU-128tonegenerator(synth)• Melodiclinedoubledontwoinstruments:– Aweird,physicallymodeledbowedinstrument– Adetunedsitar

• DroneistwooctavesusingaChineseJingHu• DrumisaJapaneseTaikoDrum(pitched)• RecordedaMIDI“performance”

Al’sRecording

Monauralmixwithastereoreverb

Notlocalizedingame,justplayedback

Gamemechanic:Playerdiscoversfirstfewnotesofthetune,whichtriggersentiretune& unlocksnewability

SeguetoAdvice:AudioWorkflow

• Workflowiscentraltoproductiveproduction– Especiallywithmultiplepeopleonateam– Especiallyespeciallywithmultipleteams– Especially3withbig,complex,multi-facetedprojects

• Helpsgetahandleonsoftwarefordifferenttasks• Connectionsamongtasks

GardenAudioWorkflow:Simple

ProgrammersUnityC#

GitHub

AudioFolksAudacityProTools

Ableton Live

MusicDude(Al)MIDIEnvironment

AudacityMuseScore

mp3files

mp3fileVoiceActors

ScriptWriters

Linestorecord

Performlines

FoleyProps

Foleyart

AGeneralizedAudioWorkflow

ProgrammersInteractiveDevEnvScriptingLanguage(s)

VersionControl

SFX/FoleyDAW

Composer/ArrangerScoreEditor

Soundfiles

SoundEditorDAW

MusicEditorDAW

VoiceDAW

RecordingEngDAW

Soundfiles

DAWProjectsStems&Triggers

Soundfiles&Triggers

AudioWorkflowwithMiddleware

ProgrammersInteractiveDevEnvScriptingLanguage(s)

VersionControl

SFX/FoleyDAW

Composer/ArrangerScoreEditor

SoundfilesSoundEditor

DAW

MusicEditorDAW

VoiceDAW

RecordingEngDAW

Soundfiles

DAWProjectsStems&Triggers

MapSoundstoTriggers

MiddlewareWwise orFMODTriggers

&Localization

Scores&/orMIDI

Low-RoadAudioWorkflow

SimpleDevEnvGameMaker orProcessing

AudioDudeAudacity

Loop-BasedMusicGarageBand

mp3files

Individualloopfiles

VoiceActors

ScriptWriters

Linestorecord

Performlines

FoleyProps

Foleyart

LibrarySounds

LotsofAudio-RelatedSoftware!

AudioProductionTools

• AudioEditors:Audacity(Opensource)– Goodnews

• Easytouse• Lotsofeffects• Largeusercommunity• FineforsimpleSFX,voiceacting

– Badnews• Noeffectschains:Verycumbersometomix• Noloopfeature:Havetocopy/paste• NoMIDI:onlyaudioformusic• Verylimitedformusic

AudioProductionTools

• DigitalAudioWorkstations:ProTools,Logic– Goodnews

• Industrialstrength• Effectschains• MIDI• Almostrequiredfornon-trivialmusic

– Badnews• Licensed(notfree)• Learningcurve• Overkillforsimplethings

AudioProductionTools

• Loop-basedDAWs:FLStudio,Ableton Live– Goodnews

• Loop-basedcomposition/production• Full-featuredDAW• Can“compose”withminimalmusicalexperience

– Badnews• Steep learningcurve• Licenseissues

– GarageBand:Goodforbeginners• Flatterlearningcurve• LimitedasaDAW

AudioTools

• ScoreEditors(Finale,MuseScore)– Goodnews

• Generatesheetmusicforperformancegroups• GenerateMIDIversionsofthescore• Greatfortraditionalcomposers

– Badnews• Musiconly,traditionalmusicatthat• Noaudio• Notveryusefulfornon-musicians

AudioTools

• AudioMiddleware(Wwise,FMOD)– SitsbetweenDAW&game/audioengine– Goal:Connectcomposers,audiofolks,programmers

• Composerscanbuildinteractivemusicw/oprogramming• Audiofolkscanedit/mixaudioingamew/oprogramming• Programmerscanplacetriggerpointsincode

– Powerfulwaytostreamlineaudioworkflow– Greatonprojectswithcomplexinteractiveaudio– Overkillonsmallprojectswithsimpleaudio

AdviceforYourProjects

• Dospottingsessionsasearlyaspossible– Fromstoryboards,prototypes,earlyversions– Listallthesoundsyouneed– Determinetriggerpoints&sequences– Timethevisuals&synchpointswithin

• Preproductionwork– Decidewhichsoundsyou’llbuyvs.make– ExperimentwithFoleyprops–Writescriptsforvoicework– Findvoiceactors(amongyourteam,mostlikely)

Advice– Recording

• Recordingsession– Atleasttwopeople

• Atleastonetomakesounds• Atleastonetotwiddleknobs&pushbuttons

– Quietplace(turnofffans,appliances,etc.)– Decentmic(USBmicorstereorecorder)– Setlevelscorrectly(useheadroom,butnever clip)– Saveeverythinguncompressed(wav,aiff)– Archiveoriginalsandwrite-protectthem

Advice– Editing

• Startwithcopiesofarchivedoriginalrecordings• Workuncompressed(atleast44.1kHz,16bit)• Don’tbestingywithtracks• Savethemulti-trackproject,notjustmixedresult• Savefrequently,makealternativeversions• Deconstructcomplexsoundswhenpossible–Mayneedtodeliverstems,notfullymixedsound– Keepambienttexturesanddiscreetsoundsseparate– Interactivemusicrequiresindividualparts,phrases

Advice– VoiceWork

• Alwayshaveascript,atleasttostart!• Recorddialogsinsequenceinrealtime• Recordmonaurally,preferablyonemicperactor• Directingtheactors– Leavespacebetweenlinesunlessdialogmustoverlap– Goforgoodenunciationwithappropriatefeeling– Getdifferentemotionaltakes– Recordmultipletakes,butremember…– Youarenot StanleyKubrick!!!

Advice– Effects

• Whentoapply?– BakedintoassetatproductiontimeOR– Appliedin-gamebyaudioengine(activelocalization)

• Orderinwhichtoapplyeffects(guideline)– Compression(squeezedynamicrange)– Level(reallycouldbeanytime)– EQ(Treble/bassonsteroids)– Chorus/phase/flange/distortion– Panning(L-R)– Reverb

Advice– PuttingItAllTogether

• Lotsofaudioassetsatsametimeisaproblem– Canmaskeachother– Canbedistracting,overwhelming,breakimmersion

• Goal:Hearmostimportantsoundsclearly• Solutions– Ducklessimportantsounds(dialthemdownorout)– Useaudiodimensionstospreadsoundsout

AudioPriorities

1. Soundsneededforuser/playeractions2. Soundsthatmatchvisuals3. Soundsthatyou’dexpecttohear4. Soundsthatenhanceevents5. Soundsthatconveyemotions6. Soundsthatsetamood7. Soundsthatfillgaps

AudioDimensions

• Location:L-Rpan– Pancompetingsoundstodifferentlocations– Notalwaysfeasibleiftiedtoon-screensources

• Frequency:Centerpitch,bandwidth– ControlfrequencyrangeofsoundswithEQ– Importantsoundonlyoneusingaparticularband– Avoidloud“noise”sounds(theyeatallfrequencies)

• Timbre/envelope/origin– Soundswithdifferentattributescancoexistbetter

DevelopmentProcess

• Shouldbespiralmodel,notwaterfall– Fixingoneissueoftencreatesotherissues–Whack-a-Moleproblem– KeepDAWprojectsaroundtoremix

• Middlewarecanhelp– CanaddWwise toUnitydevinterface– Tweaklevels,otherparametersofspecificassetsinrealtimeduringgameplay

Overall

• Listenwithfreshears– Takelotsofbreaks&dootherthings– Getotherfolkstolisten,playtest

• DocumentEverything!– Eachassetshouldhaveadevtrail– Blog/Wikicanbeuseful(iffolksuseit)– Contentmanagementveryuseful:Learnhowtouseit– Librarian:Veryusefulrollforsomeonetoplay

• Havefun!!!!

Questions?