Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

58
Core Image The Most Fun API You’re Not Using Chris Adamson • @invalidname CocoaConf Columbus, August 2014

description

Graphics on iOS and OS X isn't just about stroking shapes and paths in Core Graphics and trying to figure out OpenGL. The Core Image framework gives you access to about 100 built-in filters, providing everything from photographic effects and color manipulation to face-finding and QR Code generation. It can leverage the power of the GPU to provide performance fast enough to perform complex effects work on real-time video capture. But even if you're not writing the next Final Cut Pro or Photoshop, it's easy to call in Core Image for simple tasks, like putting a blur in part of your UI for transitions or privacy reasons. In this session, we'll explore the many ways Core Image can make your app sizzle.

Transcript of Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Page 1: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Core Image The Most Fun API You’re Not Using

Chris Adamson • @invalidname CocoaConf Columbus, August 2014

Page 2: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)
Page 3: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)
Page 4: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

“Core Image is an image processing and analysis technology designed to provide near real-time

processing for still and video images.”

Page 5: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Agenda

• Images, Filters, and Contexts

• The Core Image Filter Gallery

• Neat Tricks with Built-In Filters

• Core Image on OS X

Page 6: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Core Image, Core Concepts

• Core Image is of the time

• A chain of filters describes a “recipe” of processing steps to be applied to one or more images

• “Stringly typed”

• You only get pixels when you render

Page 7: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Typical Workflow

• Start with a source CIImage

• Apply one or more filters

• Render resulting CIImage to a CIContext, or convert CIImage out to another type

• A few filters take or produce types other than CIImage (CIQRCodeGenerator)

Page 8: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

CIImage

• An image provided to or produced by Core Image

• But no bitmap of pixel data!

• Immutable

• -imageByCroppingToRect,-imageByApplyingTransform

• -extent — a CGRect of the image’s size

Page 9: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

CIImage sources

• NSURL

• CGImageRef

• Bitmap or JPEG/PNG/TIFF in NSData

• OpenGL texture

• Core Video image/pixel buffer

Page 10: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

CIContext• Rendering destination for a CIImage (-

[drawImage:inRect:fromRect:])

• This is where you get pixels (also, this is the processor-intenstive part)

• On iOS, must be created from an EAGLContext. On Mac, can be created with CGContextRef

• Can also produce output as a CGImageRef, bitmap data, or a CVPixelBuffer (iOS only)

Page 11: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

????

Page 12: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

CIFilter

• Performs an image processing operation

• Typically takes and produces a CIImage

• All parameters are provided via -[setValue:forKey:]

• Stringly-typed!

• Output is retrieved with -[valueForKey:]

Page 13: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

–Core Image Cat

“I can has filterz?

Page 14: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Yes, you can has Filterz!

Page 15: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)
Page 16: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Core Image Filter ReferenceFilter NameParameters

Note the type & number to provide

CategoriesWatch for CICategoryBuiltIn

and CICategoryVideo

Example Figure

AvailabilityWatch for versioning and

OS X-only filters

Page 17: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Filter Categories

• Group filters by functionality: CICategoryBlur, CICategoryGenerator, CICategoryCompositeOperation, etc.

• Also group filters by availability and appropriateness: CICategoryBuiltIn, CICategoryVideo, CICategoryNonSquarePixels

Page 18: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

CICategoryGenerator

• No input image, just produces an output

• CICategoryGradient is also output-only

• Example: CICheckerboardGenerator

Page 19: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

CICategoryBlur

• Algorithmically spreads/blends pixels

• CICategorySharpen offers an opposite effect

• Example: CIGaussianBlur

Page 20: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

CICategoryColorAdjustement

• Changes distribution of color throughout an image

• Example: CIColorControls (adjusts saturation, brightness, contrast)

Page 21: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

CICategoryColorEffect

• Color changes that affect the subjective nature of the image

• Example: CIPhotoEffectNoir

Page 22: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

CICategoryDistortionEffect

• Moves pixels to achieve an effect

• Example: CITorusLensDistortion

Page 23: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

CICategoryStylize

• Various stylistic effects

• Example: CIPointillize

Page 24: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

CICategoryGeometryAdjustment

• Moves pixels via cropping, affine transforms, etc.

• Example: CICrop

Page 25: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

CICategoryTileEffect

• Repeatedly copies all or part of an image

• Example: CIAffineTile

Page 26: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

CICategoryCompositeOperation

• Combines multiple images

• Example: CISourceOverCompositing

Page 27: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Demo

Page 28: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Creating CIColorControls Filter

_colorControlsFilter

Page 29: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Setting input values

[self! ! ! ! ! ! ! ! ! ! ! ![self! ! ! ! ! ! ! ! ! ! ! ![self! ! ! ! ! ! ! ! ! ! ! !

Page 30: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Setting input image

CIImage! ! ! ! ! ! ! ! ! ![self! ! ! ! ! ! ! !

Page 31: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Getting output image

ciImage = [UIImageself

Page 32: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Other output options

• Use a CIContext

• -[drawImage:inRect:fromRect:] draws pixels to the EAGLContext (iOS) or CGContextRef (OS X) that the CIContext was created from.

• CIContext can also render to a void* bitmap

• On iOS, can create a CVPixelBufferRef, typically used for writing to a file with AVAssetWriter

Page 33: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Chaining filters

• Use the output of one filter as the input to the next

• This doesn’t cost anything, because the CIImages just hold state, not pixels

Page 34: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Demo

Page 35: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Creating CIContext

if (! ! !! ! ! ! ! !! !}!! ! !// make CIContext from GL context, clearing out default color spaceself! ! ! ! ! ! ! ! ! ! ! ! !! ! ! !! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !!

Note: This is in a subclass of GLKView

Page 36: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Set up Sepia Tone filter

_sepiaToneFilter[_sepiaToneFilter! ! ! ! ! ! ! ! !

Page 37: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Set up Mask to Alpha filter

UIImage_circleMaskFilterCIImage! ! ! ! ! ! ! ! ! ! ! ! circleImageUI.[_circleMaskFilter! ! ! ! ! ! ! _circleMask

circle-mask-100x100.png

Page 38: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Set up Blend with Mask filter_constantColorGeneratorFilter! ! ! ! ! ! ! ! ! ![_constantColorGeneratorFilter! ! ! [! ! ! ! ! ! ! ! ! ! ! ! ! ! forKey_backgroundAlphaFill! ! ! ! ! ! ! ! ! ! !!_blendWithMaskFilter[_blendWithMaskFilter! ! ! ! ! ! ! ![_blendWithMaskFilter! ! ! ! ! ! ! !

Page 39: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Apply filters

// Get CIImage from source imageCGImageRef! ! ! ! ! ! ! !loupeImage = [!// Apply sepia filter[selfloupeImage = [!// Set sepia-filtered image as input to blend-with-mask[_blendWithMaskFilter! ! ! ! ! !loupeImage = [

Page 40: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Render in CIContextif ([! [}!! ![self!// GL-on-Retina fixCGRectdrawBoundsInPoints.drawBoundsInPoints.! !// drawing to CIContext draws to the EAGLESContext it's based on[self! ! ! ! !! ! !! ! ! ! ! ! ! !!// Refresh GLKView contents immediately[self

Page 41: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Working with Video

• AVFoundation AVCaptureVideoDataOutput and AVAssetReader deliver CMSampleBuffers

• CMSampleBuffers have timing information and CVImageBuffers/CVPixelBuffers

• +[CIImage imageWithCVPixelBuffer:]

Page 42: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Demo

Page 43: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Chroma Key (“green screen” recipe

• Use a CIColorCube to map green-ish colors to transparent

• Use CISourceOverCompositing to draw this alpha’ed image over another image

Page 44: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

CIColorCube

Maps colors from one RGB “cube” to another

http://en.wikipedia.org/wiki/RGB_color_space

Page 45: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Using CIColorCube

CIColorCube maps green(-ish) colors to 0.0 alpha, all other colors pass through

Page 46: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

CISourceOverCompositing

inputBackgroundImage inputImage

outputImage

Page 47: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

CIColorCube Dataconst unsigned int size = 64;!size_t cubeDataSize = size * size * size * sizeof (float) * 4;!float *keyCubeData = (float *)malloc (cubeDataSize);!//! float *alphaMatteCubeData = (float *)malloc (cubeDataSize);! //! float rgb[3], hsv[3], *keyC = keyCubeData, *alphaC = alphaMatteCubeData;! float rgb[3], hsv[3], *keyC = keyCubeData;!// Populate cube with a simple gradient going from 0 to 1!for (int z = 0; z < size; z++){!! rgb[2] = ((double)z)/(size-1); // Blue value! ! for (int y = 0; y < size; y++){! ! ! rgb[1] = ((double)y)/(size-1); // Green value! ! ! for (int x = 0; x < size; x ++){! ! ! ! rgb[0] = ((double)x)/(size-1); // Red value! !! ! ! // Convert RGB to HSV! ! ! ! // You can find publicly available rgbToHSV functions on the Internet! !! ! ! RGBtoHSV(rgb[0], rgb[1], rgb[2],! ! ! ! ! ! &hsv[0], &hsv[1], &hsv[2]);! !! ! ! // RGBtoHSV uses 0 to 360 for hue, while UIColor (used above) uses 0 to 1.! ! ! ! hsv[0] /= 360.0;! ! ! ! ! ! ! ! // Use the hue value to determine which to make transparent! ! ! ! // The minimum and maximum hue angle depends on! ! ! ! // the color you want to remove! ! ! ! ! ! ! ! bool keyed = (hsv[0] > minHueAngle && hsv[0] < maxHueAngle) &&! ! ! ! (hsv[1] > minSaturation && hsv[1] < maxSaturation) &&! ! ! ! (hsv[2] > minBrightness && hsv[2] < maxBrightness);! ! ! ! ! ! ! ! float alpha = keyed ? 0.0f : 1.0f;! ! ! ! ! ! ! ! // re-calculate c pointer! ! ! ! keyC = (((z * size * size) + (y * size) + x) * sizeof(float)) + keyCubeData;! ! ! ! ! ! ! ! // Calculate premultiplied alpha values for the cube! ! ! ! keyC[0] = rgb[0] * alpha;! ! ! ! keyC[1] = rgb[1] * alpha;! ! ! ! keyC[2] = rgb[2] * alpha;! ! ! ! keyC[3] = alpha;! ! ! ! ! ! ! ! ! ! ! }! ! }! }!

See “Chroma Key Filter Recipe” in Core Image Programming Guide

Page 48: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Create CIColorCube from mapping data

// build the color cube filter and set its data to above!self.colorCubeFilter = [CIFilter filterWithName:@"CIColorCube"];![self.colorCubeFilter setValue:[NSNumber numberWithInt:size]!! !! ! ! ! ! ! ! ! forKey:@"inputCubeDimension"];! NSData *data = [NSData dataWithBytesNoCopy:keyCubeData!! !! ! ! ! ! ! ! length:cubeDataSize! ! !! ! ! ! ! freeWhenDone:YES];! [self.colorCubeFilter setValue:data forKey:@"inputCubeData"];!

Page 49: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Create CISourceOverCompositing

// source over filter!self.backgroundImage = [UIImage imageNamed:!! !! ! ! ! ! ! ! ! ! @"img_washington_small_02.jpg"]; ! self.backgroundCIImage = [CIImage imageWithCGImage:!! !! ! ! ! ! ! ! ! ! ! self.backgroundImage.CGImage];! self.sourceOverFilter = [CIFilter filterWithName:!! !! ! ! ! ! ! ! ! ! @"CISourceOverCompositing"];! [self.sourceOverFilter setValue:self.backgroundCIImage !! !! ! ! ! ! ! ! forKeyPath:@"inputBackgroundImage"];!

Page 50: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Apply Filters in Capture Callback

CIImage *bufferCIImage = [CIImage imageWithCVPixelBuffer:cvBuffer];!![self.colorCubeFilter setValue:bufferCIImage !! !! ! ! ! ! ! ! ! ! forKey:kCIInputImageKey];! CIImage *keyedCameraImage = [self.colorCubeFilter valueForKey:!! !! ! ! ! ! ! ! ! ! ! ! kCIOutputImageKey];! ![self.sourceOverFilter setValue:keyedCameraImage !! !! ! ! ! ! ! ! ! forKeyPath:kCIInputImageKey];! !CIImage *compositedImage = [self.sourceOverFilter valueForKeyPath:!! !! ! ! ! ! ! ! ! ! ! kCIOutputImageKey];

Then draw compositedImage to CIContext as before

Page 51: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Other Points of Interest• CIQRCodeGenerator filter — Converts data (e.g., a string) to

a QR Code

• CILenticularHaloGenerator filter — aka, lens flare

• CIDetector — Class (not a filter) to find features in images. Currently only supports face finding (returned as an array of CIFeatures). Optionally detects smiles and eye blinks within faces.

• CIImage has a red-eye enhancement that takes the array of face CIFeatures to tell it where to apply the effect

Page 52: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Core Image on OS X• Core Image is part of QuartzCore (or Image Kit), so

you don’t @import CoreImage

• Many more filters are available

• Can create your own filter with OpenGL Shading Language (plus some CI extensions). See CIKernel.

• Also available in iOS 8

• Filters can be set on CALayers

Page 53: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

CALayer Filters on OS X• Views must be layer-backed (obviously)

• Must also call -[NSView setLayerUsesCoreImageFilters:] on 10.9+

• CALayer has properties: filters, compositingFilter, backgroundFilters, minificationFilter, magnificationFilter

• These exist on iOS, but do nothing

Page 54: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Demo

Page 55: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Adding CIPixellate to layer’s filters

self! !! ! ! ! ! ! ! ! self[self! !! ! ! ! ! [ ! !! ! ! ! ! ! ! ! ![self! !! ! ! ! !! ! ! ! ! ! ! ! ! self! !! ! !

Page 56: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Updating a layer’s filters

-(void! [ ! !! ! ! ! ! !! ! ! ! ! ! ! ! ! ! ! !! ! ! ! }

Page 57: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Wrap Up: Stuff to Remember

• Get psyched about filters, but remember to check that they’re on your targeted platform/version.

• Drawing to a CIContext on iOS must be GL-backed (e.g., with a GLKView)

Page 58: Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Q&ASlides and code will be posted to:

http://www.slideshare.net/invalidname/!

@invalidname http://subfurther.com/blog