Core Audio in iOS 6 (CocoaConf San Jose, April 2013)

Core Audio in iOS 6Chris Adamson • @invalidname

CocoaConf San JoseApril 20, 2013

Sides and code available on my blog:http://www.subfurther.com/blog

Monday, April 29, 13

The Reviews Are In!

Legitimate copies!

• Amazon (paper or Kindle)

• Barnes & Noble (paper or Nook)

• Apple (iBooks)

• Direct from InformIT (paper, eBook [.epub + .mobi + .pdf], or Bundle)

What You’ll Learn

• What Core Audio does and doesn’t do

• When to use and not use it

• What’s new in Core Audio for iOS 6

Simple things should be simple,complex things should be possible.

–Alan Kay

AV Foundation, Media Player

–Alan Kay

AV Foundation, Media Player

Core Audio

• Low-level C framework for processing audio

• Capture, play-out, real-time or off-line processing

• The “complex things should be possible” part of audio on OS X and iOS

Chris’ CA Taxonomy

• Engines: process streams of audio

• Capture, play-out, mixing, effects processing

• Helpers: deal with formats, encodings, etc.

• File I/O, stream I/O, format conversion, iOS “session” management

Helpers: Audio File

• Read from / write to multiple audio file types (.aiff, .wav, .caf, .m4a, .mp3) in a content-agnostic way

• Get metadata (data format, duration, iTunes/ID3 info)

Helpers: Audio File Stream

• Read audio from non-random-access source like a network stream

• Discover encoding and encapsulation on the fly, then deliver audio packets to client application

Helpers: Converters

• Convert buffers of audio to and from different encodings

• One side must be in an uncompressed format (i.e., Linear PCM)

Helpers: ExtAudioFile

• Combine file I/O and format conversion

• Read a compressed file into PCM buffers

• Write PCM buffers into a compressed file

Helpers: Audio Session

• iOS-only API to negotiate use of audio resources with the rest of the system

• Deetermine whether your app mixes with other apps’ audio, honors ring/silent switch, can play in background, etc.

• Gets notified of audio interruptions

• See also AVAudioSession

Engines: Audio Units

• Low-latency (~10ms) processing of capture/play-out audio data

• Effects, mixing, etc.

• Connect units manually or via an AUGraph

• Much more on this topic momentarily…

Engines: Audio Queue

• Convenience API for recording or play-out, built atop audio units

• Rather than processing on-demand and on Core Audio’s thread, your callback provides or receives buffers of audio (at whatever size is convenient to you)

• Higher latency, naturally

• Supports compressed formats (MP3, AAC)

Engines: Open AL

• API for 3D spatialized audio, implemented atop audio units

• Set a source’s properties (x/y/z coordinates, orientation, audio buffer, etc.), OpenAL renders what it sounds like to the listener from that location

Engines and Helpers• Audio Units

• Audio Queue

• Open AL

• Audio File

• Audio File Stream

• Audio Converter

• ExtAudioFile

• Audio Session

Audio Units

Audio Unit

AUSomething

Types of Audio Units

• Output (which also do input)

• Generator

• Converter

• Effect

• Mixer

• Music

Pull Model

AUSomething

Pull Model

AUSomething

AudioUnitRender()

Pull Model

AUSomethingAUSomethingElse

Buses (aka, Elements)

AUSomething

AUSomethingElse

AUGraph

AUSomething

AUSomethingElse

Render Callbacks

AUSomething

AUSomethingElse

OSStatus converterInputRenderCallback (void *inRefCon,AudioUnitRenderActionFlags *ioActionFlags,const AudioTimeStamp *inTimeStamp,UInt32 inBusNumber,UInt32 inNumberFrames,AudioBufferList * ioData) {

CCFWebRadioPlayer *player = (__bridge CCFWebRadioPlayer*) inRefCon;

// read from bufferioData->mBuffers[0].mData = player.preRenderData;

return noErr;}

AURemoteIO

• Output unit used for play-out, capture

• A Core Audio thread repeatedly and automatically calls AudioUnitRender()

• Must set EnableIO property to explicitly enable capture and/or play-out

• Capture requires setting appropriate AudioSession category

Create AURemoteIOCheckError(NewAUGraph(&_auGraph), ! ! "couldn't create au graph"); !CheckError(AUGraphOpen(_auGraph), ! ! "couldn't open au graph"); !AudioComponentDescription componentDesc;componentDesc.componentType = kAudioUnitType_Output;componentDesc.componentSubType = kAudioUnitSubType_RemoteIO;componentDesc.componentManufacturer =

kAudioUnitManufacturer_Apple; !AUNode remoteIONode;CheckError(AUGraphAddNode(_auGraph, ! ! ! ! ! ! &componentDesc, ! ! ! ! ! ! &remoteIONode), ! ! "couldn't add remote io node");

Getting an AudioUnit from AUNode

! CheckError(AUGraphNodeInfo(self.auGraph, ! ! ! ! ! ! ! remoteIONode, ! ! ! ! ! ! ! NULL, ! ! ! ! ! ! ! &_remoteIOUnit), ! ! ! "couldn't get remote io unit from node");

AURemoteIO Buses

AURemoteIO

AURemoteIO Buses

AURemoteIObus 0

to output H/W

AURemoteIO Buses

AURemoteIObus 0

to output H/Wbus 0

from app

AURemoteIO Buses

AURemoteIObus 0

to output H/W

bus 1from input H/W

bus 0from app

AURemoteIO Buses

AURemoteIObus 0

to output H/W

bus 1from input H/W

bus 1to app

bus 0from app

EnableIO ! UInt32 oneFlag = 1; ! UInt32 busZero = 0; ! CheckError(AudioUnitSetProperty(self.remoteIOUnit, ! ! ! ! ! ! ! ! ! kAudioOutputUnitProperty_EnableIO, ! ! ! ! ! ! ! ! ! kAudioUnitScope_Output, ! ! ! ! ! ! ! ! ! busZero, ! ! ! ! ! ! ! ! ! &oneFlag, ! ! ! ! ! ! ! ! ! sizeof(oneFlag)), ! ! ! "couldn't enable remote io output"); ! UInt32 busOne = 1; ! CheckError(AudioUnitSetProperty(self.remoteIOUnit, ! ! ! ! ! ! ! ! ! kAudioOutputUnitProperty_EnableIO, ! ! ! ! ! ! ! ! ! kAudioUnitScope_Input, ! ! ! ! ! ! ! ! ! busOne, ! ! ! ! ! ! ! ! ! &oneFlag, ! ! ! ! ! ! ! ! ! sizeof(oneFlag)), ! ! ! "couldn't enable remote io input");

Pass Through

AURemoteIO

bus 1from input H/W

bus 0to output H/W

Connect In to Out

! UInt32 busZero = 0; ! UInt32 busOne = 1; ! CheckError(AUGraphConnectNodeInput(self.auGraph, ! ! ! ! ! ! ! ! ! remoteIONode, ! ! ! ! ! ! ! ! ! busOne, ! ! ! ! ! ! ! ! ! remoteIONode, ! ! ! ! ! ! ! ! ! busZero), ! ! ! "couldn't connect remote io bus 1 to 0");

Pass-Through with Effect

bus 0to output H/W

AURemoteIO

AUEffect

bus 1from input H/W

Demo: Delay EffectNew in iOS 6!

Creating the AUDelay ! componentDesc.componentType = kAudioUnitType_Effect; ! componentDesc.componentSubType = kAudioUnitSubType_Delay; ! componentDesc.componentManufacturer =

kAudioUnitManufacturer_Apple; ! ! AUNode effectNode; ! CheckError(AUGraphAddNode(self.auGraph, ! ! ! ! ! ! ! &componentDesc, ! ! ! ! ! ! ! &effectNode), ! ! ! "couldn't create effect node"); ! AudioUnit effectUnit; ! CheckError(AUGraphNodeInfo(self.auGraph, ! ! ! ! ! ! ! effectNode, ! ! ! ! ! ! ! NULL, ! ! ! ! ! ! ! &effectUnit), ! ! ! "couldn't get effect unit from node");

The problem with effect units

• Audio Units available since iPhone OS 2.0 prefer int formats

• Effect units arrived with iOS 5 (arm7 era) and only work with float format

• Have to set the AUEffect unit’s format on AURemoteIO

Setting formats ! AudioStreamBasicDescription effectDataFormat; ! UInt32 propSize = sizeof (effectDataFormat); ! CheckError(AudioUnitGetProperty(effectUnit, ! ! ! ! ! ! ! ! ! kAudioUnitProperty_StreamFormat, ! ! ! ! ! ! ! ! ! kAudioUnitScope_Output, ! ! ! ! ! ! ! ! ! busZero, ! ! ! ! ! ! ! ! ! &effectDataFormat, ! ! ! ! ! ! ! ! ! &propSize), ! ! ! "couldn't read effect format"); ! CheckError(AudioUnitSetProperty(self.remoteIOUnit, ! ! ! ! ! ! ! ! ! kAudioUnitProperty_StreamFormat, ! ! ! ! ! ! ! ! ! kAudioUnitScope_Output, ! ! ! ! ! ! ! ! ! busOne, ! ! ! ! ! ! ! ! ! &effectDataFormat, ! ! ! ! ! ! ! ! ! propSize), ! ! ! "couldn't set bus one output format");

Then repeat AudioUnitSetProperty() for input scope / bus 0

AUNewTimePitch

• New in iOS 6!

• Allows you to change pitch independent of time, or time independent of pitch

• How do you use it?

AUTimePitch

! AudioComponentDescription effectcd = {0}; ! effectcd.componentType = kAudioUnitType_FormatConverter; ! effectcd.componentSubType = kAudioUnitSubType_NewTimePitch; ! effectcd.componentManufacturer = kAudioUnitManufacturer_Apple; ! ! AUNode effectNode; ! CheckError(AUGraphAddNode(self.auGraph, ! ! ! ! ! ! ! &effectcd, ! ! ! ! ! ! ! &effectNode), ! ! ! "couldn't get effect node [time/pitch]");

Notice the type is AUFormatConverter, not AUEffect

AudioUnitParameters.h// Parameters for AUNewTimePitchenum { ! ! // Global, rate, 1/32 -> 32.0, 1.0 ! kNewTimePitchParam_Rate!! ! ! ! ! ! = 0, ! ! // Global, Cents, -2400 -> 2400, 1.0 ! kNewTimePitchParam_Pitch! ! ! ! ! ! = 1, ! ! // Global, generic, 3.0 -> 32.0, 8.0 ! kNewTimePitchParam_Overlap! ! ! ! ! ! = 4, ! ! // Global, Boolean, 0->1, 1 ! kNewTimePitchParam_EnablePeakLocking! ! ! = 6};

This is the entire documentation for the AUNewTimePitch parameters

AUNewTimePitch parameters

• Rate: kNewTimePitchParam_Rate takes a Float32 rate from 1/32 speed to 32x speed.

• Use powers of 2: 1/32, 1/16, …, 2, 4, 8…

• Pitch: kNewTimePitchParam_Pitch takes a Float32 representing cents, meaning 1/100 of a musical semitone

Pitch shifting

• Pitch can vary, time does not

• Suitable for real-time sources, such as audio capture

Demo: Pitch ShiftNew in iOS 6!

Rate shifting

• Rate can vary, pitch does not

• Think of 1.5x and 2x speed modes in Podcasts app

• Not suitable for real-time sources, as data will be consumed faster. Files work well.

• Sources must be able to map time systems with kAudioUnitProperty_InputSamplesInOutput

Demo: Rate ShiftNew in iOS 6!

AUSplitter

AUSomethingElse

New in iOS 6!

AUMatrixMixer

AUSomethingElse

New in iOS 6!

Audio Queues(and the APIs that help them)

AudioQueue

• Easier than AURemoteIO - provide data when you want to, less time pressure, can accept or provide compressed formats (MP3, AAC)

• Recording queue - receive buffers of captured audio in a callback

• Play-out queue - enqueue buffers of audio to play, optionally refill in a callback

Audio Queue

Common AQ scenarios

• File player - Read from file and “prime” queue buffers, start queue, when called back with used buffer, refill from next part of file

• Synthesis - Maintain state in your own code, write raw samples into buffers during callbacks

Web Radio

• Project from Thursday’s workshop

• Use Audio File Stream Services to pick out audio data from a network stream

• Enqueue these packets as new AQ buffers

• Dispose used buffers in callback

Parsing web radio

NSData NSData

Packets Packets Packets Packets Packets

NSURLConnection delivers NSData buffers, containing audio and framing info. We pass it to Audio File Services.

Parsing web radio

NSData NSData

Packets Packets

Packets Packets Packets

Audio File Services calls us back with parsed packets of audio data.

Parsing web radio

NSData NSData

Packets Packets

Packets Packets Packets

012Packets

PacketsPackets

Packets

Audio File Services calls us back with parsed packets of audio data.

We create an AudioQueueBuffer with those packets and enqueue it for play-out.

A complex thing!

• What if we want to see that data after it’s been decoded to PCM and is about to be played?

• e.g., spectrum analysis, effects, visualizers

• AudioQueue design is “fire-and-forget”

AudioQueue Tap!

http://www.last.fm/music/Spinal+TapMonday, April 29, 13

AudioQueueProcessingTap

• Set as a property on the Audio Queue

• Calls back to your function with decoded (PCM) audio data

• Three types: pre- or post- effects (that the AQ performs), or siphon. First two can modify the data.

• Only documentation is in AudioQueue.h

Creating an AQ Tap

! ! // create the tap ! ! UInt32 maxFrames = 0; ! ! AudioStreamBasicDescription tapFormat = {0}; ! ! AudioQueueProcessingTapRef tapRef; ! ! CheckError(AudioQueueProcessingTapNew(audioQueue, ! ! ! ! ! ! ! ! ! ! ! tapProc, ! ! ! ! ! ! ! ! ! ! ! (__bridge void *)(player), ! ! ! ! ! ! ! ! ! ! ! kAudioQueueProcessingTap_PreEffects, ! ! ! ! ! ! ! ! ! ! ! &maxFrames, ! ! ! ! ! ! ! ! ! ! ! &tapFormat, ! ! ! ! ! ! ! ! ! ! ! &tapRef), ! ! ! ! "couldn't create AQ tap");

Notice that you receive maxFrames and tapFormat. These do not appear to be settable.

AQ Tap Procvoid tapProc (void * inClientData, ! ! ! AudioQueueProcessingTapRef inAQTap, ! ! ! UInt32 inNumberFrames, ! ! ! AudioTimeStamp * ioTimeStamp, ! ! ! UInt32 * ioFlags, ! ! ! UInt32 * outNumberFrames, ! ! ! AudioBufferList * ioData) { ! CCFWebRadioPlayer *player =

(__bridge CCFWebRadioPlayer*) inClientData; ! UInt32 getSourceFlags = 0; ! UInt32 getSourceFrames = 0; ! AudioQueueProcessingTapGetSourceAudio(inAQTap, ! ! ! ! ! ! ! ! ! ! inNumberFrames, ! ! ! ! ! ! ! ! ! ! ioTimeStamp, ! ! ! ! ! ! ! ! ! ! &getSourceFlags, ! ! ! ! ! ! ! ! ! ! &getSourceFrames, ! ! ! ! ! ! ! ! ! ! ioData);// then do something with ioData// ...

So what should we do with the audio?

Let’s apply our pitch-shift effect

Shouldn’t this work?

AUEffect

Shouldn’t this work?

AUEffectAudioUnitRender()

AudioUnitRender()

• Last argument is an AudioBufferList, whose AudioBuffer members have mData pointers

• If mData != NULL, audio unit does its thing with those samples

• If mData == NULL, audio data pulls from whatever it’s connected to

• So we just call with AudioBufferList ioData we got from tap callback, right?

Psych!

• AQ tap provides data as signed ints

• Effect units only work with floating point

• We need to do an on-the-spot format conversion

invalidname’s convert-and-effect recipe

AUGenericOutputAUConverterAUEffectAUConverter

OSStatus converterInputRenderCallback (void *inRefCon,AudioUnitRenderActionFlags *ioActionFlags,const AudioTimeStamp *inTimeStamp,UInt32 inBusNumber,UInt32 inNumberFrames,AudioBufferList * ioData) {

CCFWebRadioPlayer *player = (__bridge CCFWebRadioPlayer*) inRefCon;

// read from bufferioData->mBuffers[0].mData = player.preRenderData;

return noErr;}

Note: red arrows are float format, yellow arrows are int

How it works

• AUGraph: AUConverter → AUEffect → AUConverter → AUGenericOutput

• Top AUConverter is connected to a render callback function

The trick!

• Copy mData pointer to a state variable and NULL it in ioData

• Call AudioQueueRender() on output unit. The NULL makes it pull from the graph.

• Top of the graph pulls on render callback, which gives it back the mData we copied off.

Yes, reallyThis is the rest of tapProc()

! // copy off the ioData so the graph can read from it // in render callback ! player.preRenderData = ioData->mBuffers[0].mData; ! ioData->mBuffers[0].mData = NULL; ! ! OSStatus renderErr = noErr; ! AudioUnitRenderActionFlags actionFlags = 0; ! renderErr = AudioUnitRender(player.genericOutputUnit, ! ! ! ! ! ! ! ! &actionFlags, ! ! ! ! ! ! ! ! player.renderTimeStamp, ! ! ! ! ! ! ! ! 0, ! ! ! ! ! ! ! ! inNumberFrames, ! ! ! ! ! ! ! ! ioData); ! NSLog (@"AudioUnitRender, renderErr = %ld",renderErr);}

Yes, really

OSStatus converterInputRenderCallback (void *inRefCon, ! ! ! ! ! ! ! ! ! AudioUnitRenderActionFlags *ioActionFlags, ! ! ! ! ! ! ! ! ! const AudioTimeStamp *inTimeStamp, ! ! ! ! ! ! ! ! ! UInt32 inBusNumber, ! ! ! ! ! ! ! ! ! UInt32 inNumberFrames, ! ! ! ! ! ! ! ! ! AudioBufferList * ioData) { ! CCFWebRadioPlayer *player =

(__bridge CCFWebRadioPlayer*) inRefCon; ! ! // read from buffer ! ioData->mBuffers[0].mData = player.preRenderData;

! return noErr;}

This is the render callback that supplies data to the int→float converter

Demo: AQ Tap + AUNewTimePitch

New in iOS 6!

Meanwhile in a mini-bus near

Copenhagan…

Audiobus

• Allows multiple audio apps to exchange data in realtime

• Works by sending raw data in MIDI

• Actually approved by Apple

• Actually supported in GarageBand

Supporting Audiobus

• Get the SDK from audiob.us

• Enable background mode, add an audiobus-compatible URL scheme, get API key from audiob.us

• Create and use ABAudiobusController, ABOutputPort/ABInputPort, and ABAudiobusAudioUnitWrapper

Wrapping up…

Takeaways

• Core Audio fundamentals never change

• New stuff is added as properties, typedefs, enums, etc.

• Watch the SDK API diffs document to find the new stuff

• Hope you like header files and experimentation

• Slides will be posted to slideshare.net/invalidname

• Code will be linked from there and my blog

• Watch CocoaConf glassboard, @invalidname on Twitter/ADN, or [Time code]; blog for announcement

• Thanks!

Core Audio in iOS 6 (CocoaConf San Jose, April 2013)

Technology

Transcript of Core Audio in iOS 6 (CocoaConf San Jose, April 2013)

Building A Streaming Apple TV App (CocoaConf DC, Sept 2016)

IP SLAs Configuration Guide, Cisco IOS Release 12 · IP SLAs Configuration Guide, Cisco IOS Release 12.2SX Americas Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose,

lopezobrador.org.mx · Notario Público No. 168 ... VICTOR SALVADOR MARIN, IGNACIO JUAREZ JOSE ALBERTO PEÑARAN LOPEZ, ... ASí mismo comparecen IOS señores JOSE LUIS HERNANDEZ SANCHEZ

IBDesignable - CocoaConf Seattle 2014

CocoaConf Austin 2014 | Demystifying Security Best Practices

UCI Hospital Emergencias Jose Casimiro Ulloa...Juegos Panamericanos y Ios Sextos Juegos Parapanamericanos, Ios cuales han sido analizados, logrando la identificación de la vulnerabilidad

Cisco IOS HTTP Services Command · PDF fileCisco IOS HTTP Services Command Reference Americas Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA

Video Killed the Rolex Star (CocoaConf Columbus, July 2015)

How to Optimize Apps for Apple iOS Search and iOS 9 Universal Links - SMX West, San Jose 2016

· Laboral de Coronel Portillo, ADMIT E a trámite la demanda Contenciosa Administrativa, interPuesta por IOS administrados JUAN MANUEL RODRIGUEZ VEGA, LINO GUEVARA RODRIGUEZ, JOSE

Revenge of the 80s: Cut/Copy/Paste, Undo/Redo, and More Big Hits (CocoaConf Chicago, 2015)

Building A Streaming Apple TV App (CocoaConf San Jose, Nov 2016)

GPU Programming: Chicago CocoaConf 2015

App Store Tips & Tricks - CocoaConf Boston 2014

Cisco IOS IP Routing: OSPF Command Reference IOS IP Routing: OSPF Command Reference Americas Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA

3D Math Primer: CocoaConf Chicago

RAD Studio XE4 - iOS Tutorials: Delphi iOS Application ...docs.embarcadero.com/products/rad_studio/radstudioXE4/iOS Tutorial... · iOS Tutorials: Delphi iOS Application Development

Medley Jose Jose

Brainstorming Creative Uses of iOS 8 Extensions / CocoaConf Columbus 2014

CocoaConf DC - Automate with Swift - Tony Ingraldi