SE in RT Audio Applications
description
Transcript of SE in RT Audio Applications
SE in RT Audio ApplicationsSE in RT Audio Applications
Bert Schiettecatte [[email protected]]
Promotor: Prof. D. Vermeir
Co-promotor: Prof. R. Lauwereins
Advisors: S. Himpe & T. Vander Aa
IntroductionIntroduction
Sound: infinite discrete sequence of normalized numbers = ‘samples’
RT Audio Synthesis: real-time (RT) computation and playback of synthetic sound
Synthetic Sound: sound generated using a mathematical model
RT Deadlines: buffer scheduling, event handling
Goals & LimitationsGoals & Limitations
Not: – Detailed treatment of digital signal processing
(DSP) arithmetic for audio processing– Experiments on the performance of OO in RT
applicationsInstead:
– Realistic exposure to application of some SE techniques in a RT application
– Roadmap for building a RT audio synthesizer
Case StudyCase Study
Real-time audio synthesizer on handheld device: StrongARM running PocketPC
To be used by musicians on the roadGoals:
– Feasibility– Exposure to realistic RT application– Exposure to DSP– SE principles realistic in RT applications
ImplementationImplementation
Trade-offs:– Floating-point VS fixed-point– Double buffering, tripple buffering or
ring buffering of sound– Static VS dynamic networks– OO VS best-fit implementation
ImplementationImplementation
Strategy: bench & correctness of1. Arithmetic
2. Synthesis Algorithm
3. Buffer Scheduling & OS Timing
4. Event Scheduling
Fixed VS float arithmeticFixed VS float arithmetic
Arithmetic: major impact on overall performance & sound quality (quantization noise)
Benchmark for comparing floating-point / fixed-point performance: LINPACK
Case study HW: factor 10 speedup (assembly-level analysis reveals float arithmetic implementation: library)
Fixed-point arithmetic and GPFixed-point arithmetic and GP
Floating-point arithmetic: sometimes regarded as ‘unlimited range’ by developers
Fixed-point arithmetic: limited per definition, requires detailed range analysis in some algorithms, resulting in confusing code
Solution: use generative programming to derive best precision for a variable
Generalized: explicit type for a variable can be substituted for a set of constraints
Synthesis AlgorithmSynthesis Algorithm
Additive synthesis: sum (harmonically related) weighted sine waves
Envelope: piecewise linear function changing weight over time
Requirements: – Oscillator: efficient sine
algorithm (e.g. CORDIC)– Envelopes: linear
interpolation
x
+
x
x
.............
amplitude
amplitude
amplitude
f
2*f
n*f
Synthesis AlgorithmSynthesis Algorithm
Use fixed-point arithmetic to implement algorithm
Dynamic network: runtime sequencing of DSP blocks, buffer granularity
Static network: compile time sequencing of DSP blocks, sample granularity
Here: static network because of fixed-point arithmetic & inlining
Buffer Scheduling: OS APIBuffer Scheduling: OS API
In general: 1. Check device capabilities
2. Open device
3. Prepare buffer
4. Queue buffer in the OS for playback
5. Wait until playback done
6. Goto 3
7. Unprepare buffer
8. Close device
Buffer SchedulingBuffer Scheduling
Algorithms:– Naive– Double/Triple buffering– Ring buffering
Buffer Scheduling: NaiveBuffer Scheduling: Naive
Algorithm:1. Prepare buffer
2. Queue buffer in the OS for playback
3. Wait until OS signals playback done
4. Goto 1
Problems: – sound pauses (prepare + queue)– OS event is overhead
Buffer Scheduling: Double/TrippleBuffer Scheduling: Double/Tripple
Algorithm:1. Prepare buffer 12. Queue buffer 1 in the OS for
playback3. Wait until OS signals buffer 2 done4. Prepare buffer 25. Queue buffer 26. Wait until OS signals buffer 1 done7. Goto 1
Problems:– OS event overhead
W
S
Buffer Scheduling: Ring BufferBuffer Scheduling: Ring Buffer
Algorithm:1. Fill ½ & queue buffer for looping
playback
2. Get playing position of the OS in buffer
3. Fill buffer from the last known rendering position up to the OS position
4. Update rendering position
5. Wait ¼ buffer size
6. Goto 2
Problems:– Tricky: sleep accuracy (WinCE 3.0: 1 to
2 ms VS Win98: 20ms!)– Choosing wait time is hard
W S
Buffer Scheduling: RealityBuffer Scheduling: Reality
Scheduling: very hard– Requires inspiration and experimenting
(e.g. buffer size problems in WinCE 3.0)– Buffer underrun detection/avoidance
(e.g. MP3)
Event SchedulingEvent Scheduling
Possibilities: – Seperate event thread updating settings– Handle events in the synthesizer class: keep a song
pointer & update when necessary
Seperate thread: requires additional CPU time for scheduling, synchronization
In the synth class: better, keep a song pointer (in samples) and update settings based on the song’s tempo (e.g. 120 BPM = update every 0.5s * 22050 Hz = 11025 samples)
Event SchedulingEvent Scheduling
Latency: the delay on events caused by the buffer scheduler
Latency:– In seperate event thread: because
of synchronization, at least 1 sound buffer
– In synthesis loop: almost none, since events can be handled almost every sample
Implementation: Best-fitImplementation: Best-fit
In general: mix paradigms to write clear code
OO not always the best choice: depends on levels of re-use (e.g. OS timer)
Machine-level inspection of code: crucial when using OO in RT applications
OO not per definition a bad thing
Implementation: Object-orientednessImplementation: Object-orientedness
Here: OO is the right paradigm when– It doesn’t keep the compiler from optimizing – It entirely disappears after compilation (e.g.
inlining): no run-time OO– It simplifies code
In general: OO languages without real ‘compile-time’ tend to have performance problems
Meta-programming: welcome addition
ContributionContribution
This dissertation: one of the few detailed available roadmaps for RT audio synthesis implementation
Not much new here (but still large amount of research, everything proprietary)
Application of generative programming to audio processing and fixed-point arithmetic
ConclusionConclusion
Implementing a RT audio synthesizer is hard Series of benchmarks & tests crucial OO design has to be applied with common sense
and is not always the best choice Generative programming can be a powerful
mechanism to abstract at little or no cost A meta-programming level adds expressive power Compiler features (inlining, templates, …)
essential
Synthesis engine demonstrationSynthesis engine demonstration
Sets up the additive synthesis engine, starts the sound buffer scheduler, loads events from a text file and feeds them to the additive synthesizer
81600 0 1 130.8120 0 0 11 0 1 184.9981 0 0 12 0 1 220.00...
Time Component ID
Parameter ID ParameterValue