CocoaConf Chicago 2017: Media Frameworks and Swift: This Is Fine

Media Frameworks and Swift:

This is FineChris Adamson • @invalidname CocoaConf Chicago, April 2017

Slides available at slideshare.net/invalidname Code available at github.com/invalidstream

http://slideshare.net/invalidname

http://github.com/invalidstream

Who the what, now?

@invalidname

import Cocoa import AVFoundation import CoreMediaIO

if let devices = AVCaptureDevice.devices(), let avDevices = devices.filter( {$0 is AVCaptureDevice}) as? [AVCaptureDevice] { for device in avDevices { print("\(device.description)") } }

<AVCaptureHALDevice: 0x100b16ab0 [Loopback Simulator][com.rogueamoeba.Loopback:E8577B20-0806-4472-A5E6-426CABCD6C8E]> <AVCaptureHALDevice: 0x100c1a7c0 [Loopback Line-In][com.rogueamoeba.Loopback:A00F38FD-C2B6-43FD-98B7-23BAA6FACB03]> <AVCaptureHALDevice: 0x100c16910 [iMic USB audio system][AppleUSBAudioEngine:Griffin Technology, Inc:iMic USB audio system:220000:2,1]> <AVCaptureHALDevice: 0x100d13900 [Loopback Keynote][com.rogueamoeba.Loopback:1936D2A3-6D0B-428E-899E-0ABE46628EA4]> <AVCaptureHALDevice: 0x100a26850 [Soundflower (64ch)][SoundflowerEngine:1]> <AVCaptureHALDevice: 0x100a26310 [HD Pro Webcam C920][AppleUSBAudioEngine:Unknown Manufacturer:HD Pro Webcam C920:1218B05F:3]> <AVCaptureHALDevice: 0x100d13660 [Soundflower (2ch)][SoundflowerEngine:0]> <AVCaptureDALDevice: 0x100a348f0 [iGlasses][iGlasses]> <AVCaptureDALDevice: 0x100a28d00 [HD Pro Webcam C920][0x244000046d082d]> Program ended with exit code: 0

CMIOObjectPropertyAddress prop = { kCMIOHardwarePropertyAllowScreenCaptureDevices, kCMIOObjectPropertyScopeGlobal, kCMIOObjectPropertyElementMaster }; UInt32 allow = 1; CMIOObjectSetPropertyData( kCMIOObjectSystemObject, &prop, 0, NULL, sizeof(allow), &allow );

var prop = CMIOObjectPropertyAddress( mSelector: CMIOObjectPropertySelector(

kCMIOHardwarePropertyAllowScreenCaptureDevices), mScope: CMIOObjectPropertyScope(kCMIOObjectPropertyScopeGlobal), mElement: CMIOObjectPropertyElement(

kCMIOObjectPropertyElementMaster)) var allow : UInt32 = 1 CMIOObjectSetPropertyData(CMIOObjectID(kCMIOObjectSystemObject), &prop, 0, nil, UInt32(MemoryLayout<UInt32>.size), &allow)





This is

fine


kCMIOHardwarePropertyAllowScreenCaptureDevices), mScope: CMIOObjectPropertyScope(

kCMIOObjectPropertyScopeGlobal), mElement: CMIOObjectPropertyElement(

kCMIOObjectPropertyElementMaster))

public typealias CMIOObjectPropertySelector = UInt32

public typealias CMIOObjectPropertyScope = UInt32

public typealias CMIOObjectPropertyElement = UInt32

public struct CMIOObjectPropertyAddress { public var mSelector: CMIOObjectPropertySelector

public var mScope: CMIOObjectPropertyScope

public var mElement: CMIOObjectPropertyElement

public init()

public init(mSelector: CMIOObjectPropertySelector, mScope: CMIOObjectPropertyScope, mElement: CMIOObjectPropertyElement)

}

extension CMIOObjectPropertySelector { static let allowScreenCaptureDevices = CMIOObjectPropertySelector( kCMIOHardwarePropertyAllowScreenCaptureDevices) }

extension CMIOObjectPropertyScope { static let global = CMIOObjectPropertyScope(kCMIOObjectPropertyScopeGlobal) }

extension CMIOObjectPropertyElement { static let master = CMIOObjectPropertyElement(kCMIOObjectPropertyElementMaster) }


kCMIOHardwarePropertyAllowScreenCaptureDevices), mScope: CMIOObjectPropertyScope(

kCMIOObjectPropertyScopeGlobal), mElement: CMIOObjectPropertyElement(

kCMIOObjectPropertyElementMaster))

var prop = CMIOObjectPropertyAddress( mSelector: .allowScreenCaptureDevices, mScope: .global, mElement: .master)

Demo

http://github.com/invalidstream/audio-reverser

Reversing Audio

1. Decode the MP3/AAC to LPCM

Reversing Audio

2. Grab a buffer from the end

Reversing Audio

3. Reverse its samples in memory

Reversing Audio

4. Write it to the front of a new file

Reversing Audio

5. Repeat until fully baked

API Needs

• Convert from MP3/AAC to LPCM

• Write sequentially to audio file (.caf, .aif, .wav)

• Random-access read from audio file

Plan A (Swift)

• AV Foundation

• AVAssetReader/Writer can do format conversion while reading/writing audio files

• Can’t (easily) read from arbitrary packet offsets; meant to process everything forward

Plan B (C, Swift?)

• Audio Toolbox (part of Core Audio)

• ExtAudioFile can do format conversions while reading/writing audio files

• AudioFile can read from arbitrary packet offset

// declare LPCM format we are converting to AudioStreamBasicDescription format = {0}; format.mSampleRate = 44100.0; format.mFormatID = kAudioFormatLinearPCM; format.mFormatFlags = kAudioFormatFlagIsPacked | kAudioFormatFlagIsSignedInteger; format.mBitsPerChannel = 16; format.mChannelsPerFrame = 2; format.mBytesPerFrame = 4; format.mFramesPerPacket = 1; format.mBytesPerPacket = 4;

// declare LPCM format we are converting to var format = AudioStreamBasicDescription( mSampleRate: 44100.0, mFormatID: kAudioFormatLinearPCM, mFormatFlags: kAudioFormatFlagIsPacked + kAudioFormatFlagIsSignedInteger, mBytesPerPacket: 4, mFramesPerPacket: 1, mBytesPerFrame: 4, mChannelsPerFrame: 2, mBitsPerChannel: 16, mReserved: 0)

// open AudioFile for output AudioFileID forwardAudioFile; err = AudioFileCreateWithURL(forwardURL, kAudioFileCAFType, &format, kAudioFileFlags_EraseFile, &forwardAudioFile); IF_ERR_RETURN

#define IF_ERR_RETURN if (err != noErr) { return err; }

// open AudioFile for output var forwardAudioFile: AudioFileID? err = AudioFileCreateWithURL(forwardURL, kAudioFileCAFType, &format, AudioFileFlags.eraseFile, &forwardAudioFile) if err != noErr { return err }


1. Uses a free function, rather than a method on AudioFile


2. Errors are communicated via the return value, rather than throws


3. Some parameters are UInt32 constants, some are enums


4. Audio format is passed as an UnsafePointer<AudioStreamBasicDescription>


5. Created object is returned via an in-out parameter

To say nothing of…

Pointer arithmetic!

// swap packets inside transfer buffer for i in 0..<packetsToTransfer/2 { let swapSrc = transferBuffer.advanced(by: Int(i) * Int(format.mBytesPerPacket)) let swapDst = transferBuffer.advanced(by: transferBufferSize - (Int(i+1) * Int(format.mBytesPerPacket))) memcpy(swapBuffer, swapSrc, Int(format.mBytesPerPacket)) memcpy(swapSrc, swapDst, Int(format.mBytesPerPacket)) memcpy(swapDst, swapBuffer, Int(format.mBytesPerPacket)) }

Couldn’t you just…

extension AudioFileID { init? (url: URL, fileType: UInt32, format: AudioStreamBasicDescription, flags: AudioFileFlags) { var fileId : AudioFileID? var format = format let err = AudioFileCreateWithURL(url as CFURL, fileType, &format, flags, &fileId) guard err != noErr, let createdFile = fileId else { return nil } self = createdFile } }

Been there, done that• The Amazing Audio Engine !

• Novocaine (!?)

• EZAudio !

• AudioKit

• Superpowered

• etc…

/** Convert a source audio file (using any Core Audio-supported codec) and create LPCM .caf files for its forward and backward versions. - parameter sourceURL: A file URL containing the source audio to be read from - parameter forwardURL: A file URL with the destination to write the decompressed (LPCM) forward file - parameter backwardURL: A file URL with the destination to write the backward file */ OSStatus convertAndReverse(CFURLRef sourceURL, CFURLRef forwardURL, CFURLRef backwardURL);

AudioReversingC.h

// // Use this file to import your target's public headers that you would like to expose to Swift. //

#import <CoreFoundation/CoreFoundation.h> #import <AudioToolbox/AudioToolbox.h>

OSStatus convertAndReverse(CFURLRef sourceURL, CFURLRef forwardURL, CFURLRef backwardURL);

AudioReverser-Bridging-Header.h

if USE_SWIFT_CONVERTER { err = convertAndReverseSwift(sourceURL: source as CFURL, forwardURL: self.forwardURL as! CFURL, backwardURL: self.backwardURL as! CFURL) } else { err = convertAndReverse(source as! CFURL, self.forwardURL as! CFURL, self.backwardURL as! CFURL) }

C APIs on iOS/macOS• Core Foundation

• Core Audio

• Core Media

• Video Toolbox

• Keychain

• IOKit

• OpenGL

• SQLite

• Accelerate

• OpenCV

• BSD, Mach

• etc…

Going deeper…

Audio Units

• Discrete software objects for working with audio

• Generators, I/O, Filters/Effects, Mixers, Converters

• Typically combined in a “graph” model

• Used by Garage Band, Logic, etc.

Demo

R(t) = C(t) x M(t)

https://github.com/invalidstream/ring-modulator-v3audiounit

Ring Modulator

• Multiplication of two signals

• One is usually a sine wave

• Originally implemented as a ring-shaped circuit

0 0.8 1.6 2.4 3.2 4 4.8 5.6 6.4 7.2

-2.4

-1.6

-0.8

0.8

1.6

2.4

R(t) = C(t) x M(t)

Modulate! Modulate!

• Ring modulator best known as the “Dalek” voice effect on Doctor Who (circa 1963)

• Also used in early electronic music

Wait, what?

-(instancetype)initWithComponentDescription:(AudioComponentDescription)componentDescription options:(AudioComponentInstantiationOptions)options error:(NSError **)outError { self = [super initWithComponentDescription:componentDescription options:options error:outError]; if (self == nil) { return nil; } // ... return self; }

MyAudioUnit.m

(macOS only)

Swift 3

Swift 4

https://github.com/apple/swift/blob/master/docs/ABIStabilityManifesto.md

“Given the importance of getting the core ABI and the related fundamentals correct, we are going to defer the declaration of ABI stability out of Swift 4 while still focusing the majority of effort to get to the point where the ABI can be declared stable.”

—Ted Kremenek, Feb. 16, 2017 “Swift 4, stage 2 starts now”

https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20170213/032116.html

// Block which subclassers must provide to implement rendering.

- (AUInternalRenderBlock)internalRenderBlock { // Capture in locals to avoid Obj-C member lookups. // If "self" is captured in render, we're doing it wrong. See sample code. return ^AUAudioUnitStatus(AudioUnitRenderActionFlags *actionFlags, const AudioTimeStamp *timestamp, AVAudioFrameCount frameCount, NSInteger outputBusNumber, AudioBufferList *outputData, const AURenderEvent *realtimeEventListHead, AURenderPullInputBlock pullInputBlock) {

// Do event handling and signal processing here. return noErr; }; }

Don’t do this

• An audio unit’s render block is called on a realtime thread

• Therefore it cannot perform any action that could block:

• I/O (file or network)

• Waiting on a mutex or semaphore

Also, don’t do this

• Call objc_msg_send()

• Capture any Objective-C or Swift object

• Allocate memory

Basically, if you touch anything in the block other than a pre-allocated C struct or primitive, you’re asking for trouble.

// Block which subclassers must provide to implement rendering. - (AUInternalRenderBlock)internalRenderBlock { // Capture in locals to avoid Obj-C member lookups. // If "self" is captured in render, we're doing it wrong. See sample code. AUValue *frequencyCapture = &frequency; AudioStreamBasicDescription *asbdCapture = &asbd; __block UInt64 *totalFramesCapture = &totalFrames; AudioBufferList *renderABLCapture = &renderABL; return ^AUAudioUnitStatus(AudioUnitRenderActionFlags *actionFlags, const AudioTimeStamp *timestamp, AVAudioFrameCount frameCount, NSInteger outputBusNumber, AudioBufferList *outputData, const AURenderEvent *realtimeEventListHead, AURenderPullInputBlock pullInputBlock) {

// Do event handling and signal processing here.

// BLOCK IMPLEMENTATION ON NEXT SLIDE

return noErr; };

// Block which subclassers must provide to implement rendering. - (AUInternalRenderBlock)internalRenderBlock { // Capture in locals to avoid Obj-C member lookups. // If "self" is captured in render, we're doing it wrong. See sample code. AUValue *frequencyCapture = &frequency; AudioStreamBasicDescription *asbdCapture = &asbd; __block UInt64 *totalFramesCapture = &totalFrames; AudioBufferList *renderABLCapture = &renderABL; return ^AUAudioUnitStatus(AudioUnitRenderActionFlags *actionFlags, const AudioTimeStamp *timestamp, AVAudioFrameCount frameCount, NSInteger outputBusNumber, AudioBufferList *outputData, const AURenderEvent *realtimeEventListHead, AURenderPullInputBlock pullInputBlock) {

// Do event handling and signal processing here.

// BLOCK IMPLEMENTATION ON NEXT SLIDE

return noErr; };

❌

// pull in samples to filter pullInputBlock(actionFlags, timestamp, frameCount, 0, renderABLCapture);

// copy samples from ABL, apply filter, write to outputData size_t sampleSize = sizeof(Float32); for (int frame = 0; frame < frameCount; frame++) { *totalFramesCapture += 1; for (int renderBuf = 0; renderBuf < renderABLCapture->mNumberBuffers; renderBuf++) { Float32 *sample = renderABLCapture->mBuffers[renderBuf].mData + (frame * asbdCapture->mBytesPerFrame); // apply modulation Float32 time = totalFrames / asbdCapture->mSampleRate; *sample = *sample * fabs(sinf(M_PI * 2 * time * *frequencyCapture)); memcpy(outputData->mBuffers[renderBuf].mData + (frame * asbdCapture->mBytesPerFrame), sample, sampleSize); } }

return noErr;

Obj-C Instance Variables!

❌AUValue *frequencyCapture = &frequency; AudioStreamBasicDescription *asbdCapture = &asbd; __block UInt64 *totalFramesCapture = &totalFrames; AudioBufferList *renderABLCapture = &renderABL;

https://github.com/apple/swift/blob/master/docs/OwnershipManifesto.md

Certain kinds of low-level programming require stricter performance guarantees. Often these guarantees are less about absolute performance than predictable performance. For example, keeping up with an audio stream is not a taxing job for a modern processor, even with significant per-sample overheads, but any sort of unexpected hiccup is immediately noticeable by users.

—“Swift Ownership Manifesto”,February 2017

We believe that these problems can be addressed with an opt-in set of features that we collectively call ownership. […]

Swift already has an ownership system, but it's “under the covers”: it's an implementation detail that programmers have little ability to influence. What we are proposing here is easy to summarize:

• We should add a core rule to the ownership system, called the Law of Exclusivity […]

• We should add features to give programmers more control over the ownership system […]

• We should add features to allow programmers to express types with unique ownership […]

And yet…

“[Swift] is the first industrial-quality systems programming language that is as expressive and enjoyable as a scripting language.”

https://developer.apple.com/library/content/documentation/Swift/Conceptual/Swift_Programming_Language/

So… when?

Waiting…

• ABI stability — will not be in Swift 4

• Ownership — unclear

• Are these traits sufficient?

Strategies

• Use AV Foundation if you can

• Learn to balance C and Swift

• “Render undo C-sar what is C-sar’s…”

• The goal is to have idiomatic Swift, not Swift that may work but looks like C

Media Frameworks and Swift:

This is FineChris Adamson • @invalidname CocoaConf Chicago, April 2017

Slides available at slideshare.net/invalidname Code available at github.com/invalidstream

http://slideshare.net/invalidname

http://github.com/invalidstream

CocoaConf Chicago 2017: Media Frameworks and Swift: This Is Fine

Technology

Transcript of CocoaConf Chicago 2017: Media Frameworks and Swift: This Is Fine