Game Audio Post-Production
-
Upload
karen-collins -
Category
Entertainment & Humor
-
view
5.918 -
download
1
description
Transcript of Game Audio Post-Production
Video Game Audio:The who, what, when, where, hows
and whys.
Karen CollinsCanada research chair
1
OutlineOutline
1. WHAT: What is Game Audio?2. WHO: Who cares? Who makes it?3. WHEN: When does the audio team get
involved? WHEN did these processes/techniques develop?
4. WHERE do games differ from linear media?5. HOW and WHY do we fix the problems?
2
WHAT is game audio?WHAT is game audio?
Interface soundsMusicSound EffectsDialogueAmbience Cinematics
all of these can function on several layers (like diegetic/non-diegetic)
3
WHO cares about game WHO cares about game audio?audio?
83% of Adult Gamers listed sound as one of the most important video game console elements
47% of Game Console owners (18-25 yr) hook up their game console to a home theatre system
48% of Hardcore Gamers said surround sound is a purchase driver for next generation consoles
WHO makes it?WHO makes it? The Game Audio Production The Game Audio Production ProcessProcess
Sound DirectorComposer(s)Sound Designer(s)Programmer/Engineer(s)Dialogue/VO director, actors.Licensing/Contracting directorsImplementation specialists
(emerging role)
5
Audio “Post”: IntegrationAudio “Post”: Integration
Sound design, mixing, integration typically handled by the same person.
Director/designer must make decisions regarding implementation of all audio assets (including music)
Often is a programmer as well
6
WHEN does the audio team WHEN does the audio team get involved?get involved?
Sometimes at end of game: populate game with sound
Sometimes at start or in middle: play a larger role in implementation of audio in the game, and can make critical decisions in regards to the development of the game and its sound
7
Music Music drivingdriving gameplay elements.gameplay elements.
8
New Super Mario Bros (Nintendo DS 2006)(Koji Kondo)
WHEN: A Brief History of
Game Audio Innovations
(What’s so different about game audio??)
9
Computer Space!(Nutting Associate
1971)First game to have
sound.
10
Space InvadersSpace Invaders (Midway 1978) (Midway 1978)
First use of continuous “background” music.
11
Technological ConstraintsTechnological Constraints
Up ‘N Down (Sega 1983)
12
Atari VCS (2600)Atari VCS (2600)
Up ‘N Down (Sega 1984)
13
Working With Constraints: Nintendo Working With Constraints: Nintendo NESNES
Metroid (Nintendo 1987) (Hip Tanaka)
vibrato (pitch modulation), tremolo (volume modulation), slides, portamento, echo effects
14
15
BallblazerBallblazer (LucasArts 1984) (Peter Langston) (LucasArts 1984) (Peter Langston)
Algorithmic generation“Riffology” method (Optimized randomness) by
Peter Langston32 eight-note melody fragmentsAlgorithm chooses how fast, how loud, when to
omit notes, when to insert rhythmic break
16
Working With Constraints: Commodore Working With Constraints: Commodore 6464
Track 1 (e.g. “Level_One”)
Instrument 1(envelope, waveform,effect filters, etc.)
Instrument 2(envelope, waveform,effect filters, etc.)
Rob Hubbard’s Module Format
Pattern 1 (sequence of notes)
Pattern 2 (sequence of notes)
Pattern 3 (sequence of notes)
17
Combining modules (in MIDI) with control statements
MIDI and the Creation of iMUSE
Land, Michael Z. and Peter N. McConnell. Method and Apparatus for Dynamically Composing Music and Sound Effect Using a Computer Entertainment System. US Patent No. 5,315,057. 24 May, 1994.
Super Mario WorldSuper Mario World (Nintendo 1991) (Koji Kondo) (Nintendo 1991) (Koji Kondo)
18
Musical layering techniqueMario jumps on Yoshi & gets extra layer of music(SNES).
Legend of Zelda: Ocarina of TimeLegend of Zelda: Ocarina of Time (Nintendo 1999) (Nintendo 1999) (Koji Kondo) (N64)(Koji Kondo) (N64)
Proximity-based algorithms control cross-fades
19
State-of-the-Art TodayState-of-the-Art Today
7.1 to 8.1 surround soundCombination of synth with orchestra, choirAt least 512 channels of sound
God of War (Gerard Marino, Sony 2006)
20
Techniques we can borrow Techniques we can borrow from Filmfrom FilmSpotting a gameEmotional reinforcementUse of sound FX librariesField recording techniques, FoleyMixing of cutscenes/cinematic sequencesUsing multichannel surround*Certain techniques from, for instance,
Ride films—e.g. Prominent sub-woofer, etc.—”physicality” of sound to shock and awe
21
Mix Goals (shared with linear media)Balance: among and between elements—games can’t
undwerstand dialogue through music, explosions, etc.IntelligibilityBelievability: realistic and emotionally effective reduces illusion;
characters for instance suddenly come in much more clearly, or too loud, etc.
Effective communication: what is the prioirity in the game right
now > focus listener’s attentionPrioritiesGameplay objectives Emotional delivery: audio tells player how to feel.Avoid distractionsAvoid competitions to player’s attention
22
Where games differ from FilmWhere games differ from FilmLinearity Vs. Non-Linearity/ unpredictability!Interactivity with player/multiplayerNo “production” soundTemporality (length)LocalizationBudgetsDelivery methods/technologyListening environmentMixing, dynamic range and the concept of
“post”
What is Interactive Audio?What is Interactive Audio?
sound events occur in direct reaction to a player’s movements. The player triggers the cue, and can repeatedly activate it, such as by making a character jump up and down.
Interactive Audio:Interactive Audio:
FootstepsEnemy music cue
What is What is adaptiveadaptive audio? audio?
“adaptive” audio is generally referred to as sound that reacts to transformations in the gameplay environment—such as a change from day to night set by a game’s timer mechanism. Adaptive audio is not directly triggered by a player
Adaptive AudioAdaptive Audio
Change in music day to nightWolf, crow
IMPACT OF IMPACT OF ADAPTIVE/ ADAPTIVE/ INTERACTIVITY ON INTERACTIVITY ON MUSICMUSIC
28
Problem: transitions from cue to cue◦ No “post” =
unpredictable timings
◦ Abrupt jumps are jarring
Solution: transitions: Cross Solution: transitions: Cross fadefade
Cross FadesCross Fades
31
Other solution: The Other solution: The Stab/SFXStab/SFXSound effect (e.g. Explosion,
gunshots, etc.) or stab (quick shock chord) used to cover transition time
32
Solution: Variable Solution: Variable structurestructure
Create “transition matrix”: chart out all possible directions for each sequence, and create transitions for marker points/jumps
Parameter Based Music: ParametersParameter Based Music: Parameters
Number/action of non-playing charactersNumber/action of playing charactersActionsLocations (place, time of day, etc.)Scripted or unscripted eventsPlayer health or enemy healthDifficultyTimingPlayer properties (skills, endurance)Bonus objects Movement (speed, direction, rhythm)“Camera” angle
34
The transition matrix approach and the creation of transitional units
Example: Parameter-Based MusicExample: Parameter-Based Music
No One Lives Forever (Guy Whitmore 2000)Six standard music states are based on
number of NPC enemies:
1. Silence 2. Super ambient 3. Ambient 4. Suspense/sneak 5. Action/combat 1 6. Action/combat 2
35
Example: Example: No One Lives ForeverNo One Lives Forever
Earth Orbit: Ambush theme starts in music state 5 (combat 1), transitions to music state 2 (ambient: in
elevator) then transitions to
music state 6 (combat 2)
36
Problems: Multi-player interactivityUnscripted events!parameter-based music is difficult:
◦Music cannot be tied to specific events or locations
◦Music cannot be tied to specific parameters
◦To some extent, this is a general audio problem in MMOGs—as audio is as much about emotion as realism.
IMPACT OF IMPACT OF INTERACTIVITY ON INTERACTIVITY ON SOUND EFFECTSSOUND EFFECTS
38
Problems with Game Sound Problems with Game Sound EffectsEffects
Ambience: Loops are boring. Loops are boring. Loops are boring.
Brain picks out patterns.Users get bored with hearing same
sounds BUT sound designers can’t possibly record enough variations of sounds (time, budget)
Users need a new experience every time they play the game (promised by LucasArts’ Euphoria technology)
Audio not responding to physics
Solution: Granular SynthesisSolution: Granular Synthesis
40
Granular Synthesis ExamplesGranular Synthesis Examples
1. Crowd2. Tennis3. Speech
41
Crowd and speech examples borrowed from Leonard Paul at Vancouver Film School
Granular: Remaining Open QuestionsGranular: Remaining Open Questions
What elements in a sound effect can be varied while still maintaining the “meaning” of the sound?
How little of a sound can be changed to alter perception?
How can we create AI systems that are aware of these potential meanings, and make real-time adjustments to sounds in a game?
How to develop an “audio physics engine”: e.g. footsteps change based on how much player is carrying, etc.
42
MIXING IN GAMESMIXING IN GAMES
43
MixingMixingProblems:Problems:-too many sounds-too many soundspoor dynamic rangepoor dynamic range-poor variation-poor variation-unpredictable -unpredictable timingstimings -no “post -no “post production” mixing production” mixing : results in “muddy”, : results in “muddy”, clash of soundsclash of sounds
Dynamic rangeDynamic range
45
… in a popular film
… in a popular game
Graphics adapted from those supplied by Rob Bridgett of Swordfish Studios.
Dynamic Range: ProblemsDynamic Range: ProblemsCompetition with wildly varying levels of
other games: reference levels not effective for games—no widespread adoption of standards
producer/game designer tends to always want it “louder”
Listening environment in games more competition for attention—larger “noise floor” = more competition in acoustic environment- restricts dynamic range
listener fatigue
46
Mixing: Dynamic Range Mixing: Dynamic Range SolutionsSolutions
Reduce amount of soundsVariable VolumeEvangelize need for good dynamic
rangeUse what standards you can find
(e.g. Xbox Boot sound)Be aware of possible noise-makers
◦Non-game sounds: e.g. windows messenger pops up
Variable volumeVariable volume
Solutions: Listener fatigueSolutions: Listener fatigueLearn how to use silence!Noise floors—what is loudest background
noise in environment?Balance impact opportunity with intelligibilityAudio ‘breaths’ (overall/in specific
frequencies—lose some sound e.g. bass to give a rest in particular frequency ranges)
Timeouts or gradual ducking for some elements (if it’s not important, time it out)
VARIABILITY!
Mixing Problem: Dynamic Mixing Problem: Dynamic Range/ VariationRange/ VariationChange DSP rather than just
volume to change audio ◦E.g. reverb to make it softer,
dreamier◦Phasing to create a “dazed” effect◦Overdrive to make it more
aggressive, etc.
50
Solution: Location-Based Run-Time Solution: Location-Based Run-Time MixingMixing
Real-time DSP to adjust sound E.g. bottle drop on hard floor of kitchen or in
next carpeted roomFactor in 5.1 surround to adjust real-time
panning
REQUIREMENT: audio engine to pass parameters from game and from player back and forth to engine.
51
Problem: Mixing: Unpredictability, Problem: Mixing: Unpredictability, VariabilityVariability
52
Problem: Ambience Problem: Ambience repetitionrepetitionNo “production sound” recorded–
ambience assembled from scratch.
Repetition of certain aspects can be detected by brain: patterns easy to discern and takes away from immersion
53
Solutions: Real-time Weighted Solutions: Real-time Weighted MixingMixing
Weighted permutations ◦ Predict which sounds can recur without making
obvious.Example:
◦ Dialogue, Sound FX A. Sound FX B, player sounds, music, ambience
◦ If dialogue = “run!”, set parameter to 1◦ If gunshot is coming towards us, set parameter to 2◦ If no action, fade out music and raise ambience
REQUIREMENT: “intelligent” Engine to predict and set weighting
54
The big problem: LogjamsThe big problem: LogjamsLittle definition for which
elements are acoustically ‘most important
Too much in similar frequency ranges
Intelligibility vs. impact
55
Logjam solutionsLogjam solutionsControl frequencies of soundsAssign priorities in code
◦E.g. dialogue = priority “1” Ambience = priority “4” Gunshot = priority “2” Music = priority “3” Footsteps = priority “5”
Real-time ducking: Use notch filters to cut out frequencies that conflict (must organize in advance!)
56
Steps to organizationSteps to organizationGroup sounds into multi-level categories:
e.g. first-person sounds, third-person, which are first-person weapon, third-person dialogue, etc.
Enforce RMS/peak levels (level limits) per category of sounds—e.g. enemies can never get above -10dB, first person sounds never above -15dB, etc.
Balance categories of sounds amongst themselves—within each category—check to maintain balance level-wise for dialogue, etc.+ choose reference piece
57
Steps to organizationSteps to organizationPrioritize Sound Groups by Function
◦E.g. Goal-relevant sounds, Instructive dialogue, Sounds that impact the player, Enemy weapon fire, Sounds that provide environment or emotional support, Musical score, Ambience, etc.
Dynamic Priorities (Per-Sound)◦E.g. Distance / distance ranking, Total
attenuation, Actual current peak/RMS, Number of playing instances, Game-specific priority schemes
58
Steps to organization: the Steps to organization: the MixMixDucking
◦Basic: Notification-driven◦Duck when new higher priority sound
starts◦Duration/speed to ducked state◦Target volume/percentage◦Advanced: Peak/RMS-driven◦Duck/unduck based on current
monitored levels of higher priority category (attack/release)
59
The mix: culling/mutingThe mix: culling/mutingCulling/Muting
◦Alternative to simple ducking◦Thins out sound field, Brings more
focus to new sounds◦Stop or mute using release
envelopes/fades◦Bandwidth concerns (mute)◦Restart accuracy (stop)
60
The Mix; EQ and FilteringThe Mix; EQ and Filtering◦Per-category critical frequency range(s)◦Can use offline or realtime analysis◦Notch out these frequencies for other
categories (duration for smoothing)‘Blur’ lower priority soundsCompression/Limiting
◦Can combine attenuation with compression/limiting
◦Keeps low-pri sounds from falling off too significantly or obviously
◦Can conflict with ducking desire
61
MIXING: SurroundMIXING: SurroundSurround as game tool: player
can interact/depend onHRTF: where is the player’s
character> Where are they facing, etc.?
Remember the CHARACTER may be changing position, but the PLAYER may not be! How do you adapt to this??
62
A final point on mixingA final point on mixingMIXING IS MORE THAN REALISM.
◦What are the psychological aspects of mixing?
◦The most effective mix is not always the most realistic.
63
Audio MiddlewareAudio Middleware
64
Wwise Wwise prototype the integration of audio in a game
simulation before the game is even finished.Environmental effects can be rendered in real-time,
and occlusion and obstruction effects can be managed within the software, mixing up to four simultaneous environments per object.
Sound prioritization for real-time mixing is also included, as is randomization of various elements for effects such as pitch or volume, to enhance realism.
Real-time game parameter controls can also be set, to adjust sound properties based on parameter value changes in a game
Validate profiles (how expensive is your audio!?)
65
TECHNOLOGICAL TECHNOLOGICAL CONSTRAINTS CONSTRAINTS (MOBILE)(MOBILE)
66
Problem: technological Problem: technological constraintsconstraintsMOBILE AUDIO:
◦Trying to write songs and create sounds in a few kB for constrained technology such as mobile phones
Cheese RacerCheese Racer
Muting and unmuting multiple tracks in a MIDI file to produce various mixes.◦2 drum loop samples,◦three tones: bass, melody and chords
In the code for the first level, ten different combinations of tracks: percussion + bass, percussion + bass + melody, bass + melody + chords, etc.
(cont..)(cont..)
Notice how the mix changes every time the mouse gets a piece of cheese.
Music loop 40 seconds. 10 different combinations: therefore, each level has more than 6 minutes of different music mixes. “Because the mix changes depending on gameplay, the music will never play exactly the same way twice, thereby increasing variation and decreasing "ear fatigue."”
In fact, there are two more sets of tracks playing two more styles of music, using the same tempo and percussion tracks as the first level.
Entire game contains almost 20 minutes of various music mixes, using only 68K of compressed sample and MIDI data.
Solutions: technological Solutions: technological constraints: constraints:
Playing a single sample at different pitches to produce variation.
The "pitch it up, play it down" technique for saving space.
Sampling to save sizeSampling to save sizeThis game contains only a single "trumpet fall"
sound effectmodify the playback sample rate so that each
time you pick up a piece of cheese, the sound is played at a different pitch
randomly vary the pitch during gameplay, so there's no repeating pattern to it
The sound tends to mask the transition from one mix to another, helping to create a more seamless audio experience.
same kind of pitch-shifting effect applied to car horn beeps: (fig 1)