Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)
-
Upload
chris-adamson -
Category
Technology
-
view
868 -
download
0
description
Transcript of Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)
Core Image The Most Fun API You’re Not Using
Chris Adamson • @invalidname CocoaConf Columbus, August 2014
“Core Image is an image processing and analysis technology designed to provide near real-time
processing for still and video images.”
Agenda
• Images, Filters, and Contexts
• The Core Image Filter Gallery
• Neat Tricks with Built-In Filters
• Core Image on OS X
Core Image, Core Concepts
• Core Image is of the time
• A chain of filters describes a “recipe” of processing steps to be applied to one or more images
• “Stringly typed”
• You only get pixels when you render
Typical Workflow
• Start with a source CIImage
• Apply one or more filters
• Render resulting CIImage to a CIContext, or convert CIImage out to another type
• A few filters take or produce types other than CIImage (CIQRCodeGenerator)
CIImage
• An image provided to or produced by Core Image
• But no bitmap of pixel data!
• Immutable
• -imageByCroppingToRect,-imageByApplyingTransform
• -extent — a CGRect of the image’s size
CIImage sources
• NSURL
• CGImageRef
• Bitmap or JPEG/PNG/TIFF in NSData
• OpenGL texture
• Core Video image/pixel buffer
CIContext• Rendering destination for a CIImage (-
[drawImage:inRect:fromRect:])
• This is where you get pixels (also, this is the processor-intenstive part)
• On iOS, must be created from an EAGLContext. On Mac, can be created with CGContextRef
• Can also produce output as a CGImageRef, bitmap data, or a CVPixelBuffer (iOS only)
????
CIFilter
• Performs an image processing operation
• Typically takes and produces a CIImage
• All parameters are provided via -[setValue:forKey:]
• Stringly-typed!
• Output is retrieved with -[valueForKey:]
–Core Image Cat
“I can has filterz?
Yes, you can has Filterz!
Core Image Filter ReferenceFilter NameParameters
Note the type & number to provide
CategoriesWatch for CICategoryBuiltIn
and CICategoryVideo
Example Figure
AvailabilityWatch for versioning and
OS X-only filters
Filter Categories
• Group filters by functionality: CICategoryBlur, CICategoryGenerator, CICategoryCompositeOperation, etc.
• Also group filters by availability and appropriateness: CICategoryBuiltIn, CICategoryVideo, CICategoryNonSquarePixels
CICategoryGenerator
• No input image, just produces an output
• CICategoryGradient is also output-only
• Example: CICheckerboardGenerator
CICategoryBlur
• Algorithmically spreads/blends pixels
• CICategorySharpen offers an opposite effect
• Example: CIGaussianBlur
CICategoryColorAdjustement
• Changes distribution of color throughout an image
• Example: CIColorControls (adjusts saturation, brightness, contrast)
CICategoryColorEffect
• Color changes that affect the subjective nature of the image
• Example: CIPhotoEffectNoir
CICategoryDistortionEffect
• Moves pixels to achieve an effect
• Example: CITorusLensDistortion
CICategoryStylize
• Various stylistic effects
• Example: CIPointillize
CICategoryGeometryAdjustment
• Moves pixels via cropping, affine transforms, etc.
• Example: CICrop
CICategoryTileEffect
• Repeatedly copies all or part of an image
• Example: CIAffineTile
CICategoryCompositeOperation
• Combines multiple images
• Example: CISourceOverCompositing
Demo
Creating CIColorControls Filter
_colorControlsFilter
Setting input values
[self! ! ! ! ! ! ! ! ! ! ! ![self! ! ! ! ! ! ! ! ! ! ! ![self! ! ! ! ! ! ! ! ! ! ! !
Setting input image
CIImage! ! ! ! ! ! ! ! ! ![self! ! ! ! ! ! ! !
Getting output image
ciImage = [UIImageself
Other output options
• Use a CIContext
• -[drawImage:inRect:fromRect:] draws pixels to the EAGLContext (iOS) or CGContextRef (OS X) that the CIContext was created from.
• CIContext can also render to a void* bitmap
• On iOS, can create a CVPixelBufferRef, typically used for writing to a file with AVAssetWriter
Chaining filters
• Use the output of one filter as the input to the next
• This doesn’t cost anything, because the CIImages just hold state, not pixels
Demo
Creating CIContext
if (! ! !! ! ! ! ! !! !}!! ! !// make CIContext from GL context, clearing out default color spaceself! ! ! ! ! ! ! ! ! ! ! ! !! ! ! !! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !!
Note: This is in a subclass of GLKView
Set up Sepia Tone filter
_sepiaToneFilter[_sepiaToneFilter! ! ! ! ! ! ! ! !
Set up Mask to Alpha filter
UIImage_circleMaskFilterCIImage! ! ! ! ! ! ! ! ! ! ! ! circleImageUI.[_circleMaskFilter! ! ! ! ! ! ! _circleMask
circle-mask-100x100.png
Set up Blend with Mask filter_constantColorGeneratorFilter! ! ! ! ! ! ! ! ! ![_constantColorGeneratorFilter! ! ! [! ! ! ! ! ! ! ! ! ! ! ! ! ! forKey_backgroundAlphaFill! ! ! ! ! ! ! ! ! ! !!_blendWithMaskFilter[_blendWithMaskFilter! ! ! ! ! ! ! ![_blendWithMaskFilter! ! ! ! ! ! ! !
Apply filters
// Get CIImage from source imageCGImageRef! ! ! ! ! ! ! !loupeImage = [!// Apply sepia filter[selfloupeImage = [!// Set sepia-filtered image as input to blend-with-mask[_blendWithMaskFilter! ! ! ! ! !loupeImage = [
Render in CIContextif ([! [}!! ![self!// GL-on-Retina fixCGRectdrawBoundsInPoints.drawBoundsInPoints.! !// drawing to CIContext draws to the EAGLESContext it's based on[self! ! ! ! !! ! !! ! ! ! ! ! ! !!// Refresh GLKView contents immediately[self
Working with Video
• AVFoundation AVCaptureVideoDataOutput and AVAssetReader deliver CMSampleBuffers
• CMSampleBuffers have timing information and CVImageBuffers/CVPixelBuffers
• +[CIImage imageWithCVPixelBuffer:]
Demo
Chroma Key (“green screen” recipe
• Use a CIColorCube to map green-ish colors to transparent
• Use CISourceOverCompositing to draw this alpha’ed image over another image
CIColorCube
Maps colors from one RGB “cube” to another
http://en.wikipedia.org/wiki/RGB_color_space
Using CIColorCube
CIColorCube maps green(-ish) colors to 0.0 alpha, all other colors pass through
CISourceOverCompositing
inputBackgroundImage inputImage
outputImage
CIColorCube Dataconst unsigned int size = 64;!size_t cubeDataSize = size * size * size * sizeof (float) * 4;!float *keyCubeData = (float *)malloc (cubeDataSize);!//! float *alphaMatteCubeData = (float *)malloc (cubeDataSize);! //! float rgb[3], hsv[3], *keyC = keyCubeData, *alphaC = alphaMatteCubeData;! float rgb[3], hsv[3], *keyC = keyCubeData;!// Populate cube with a simple gradient going from 0 to 1!for (int z = 0; z < size; z++){!! rgb[2] = ((double)z)/(size-1); // Blue value! ! for (int y = 0; y < size; y++){! ! ! rgb[1] = ((double)y)/(size-1); // Green value! ! ! for (int x = 0; x < size; x ++){! ! ! ! rgb[0] = ((double)x)/(size-1); // Red value! !! ! ! // Convert RGB to HSV! ! ! ! // You can find publicly available rgbToHSV functions on the Internet! !! ! ! RGBtoHSV(rgb[0], rgb[1], rgb[2],! ! ! ! ! ! &hsv[0], &hsv[1], &hsv[2]);! !! ! ! // RGBtoHSV uses 0 to 360 for hue, while UIColor (used above) uses 0 to 1.! ! ! ! hsv[0] /= 360.0;! ! ! ! ! ! ! ! // Use the hue value to determine which to make transparent! ! ! ! // The minimum and maximum hue angle depends on! ! ! ! // the color you want to remove! ! ! ! ! ! ! ! bool keyed = (hsv[0] > minHueAngle && hsv[0] < maxHueAngle) &&! ! ! ! (hsv[1] > minSaturation && hsv[1] < maxSaturation) &&! ! ! ! (hsv[2] > minBrightness && hsv[2] < maxBrightness);! ! ! ! ! ! ! ! float alpha = keyed ? 0.0f : 1.0f;! ! ! ! ! ! ! ! // re-calculate c pointer! ! ! ! keyC = (((z * size * size) + (y * size) + x) * sizeof(float)) + keyCubeData;! ! ! ! ! ! ! ! // Calculate premultiplied alpha values for the cube! ! ! ! keyC[0] = rgb[0] * alpha;! ! ! ! keyC[1] = rgb[1] * alpha;! ! ! ! keyC[2] = rgb[2] * alpha;! ! ! ! keyC[3] = alpha;! ! ! ! ! ! ! ! ! ! ! }! ! }! }!
See “Chroma Key Filter Recipe” in Core Image Programming Guide
Create CIColorCube from mapping data
// build the color cube filter and set its data to above!self.colorCubeFilter = [CIFilter filterWithName:@"CIColorCube"];![self.colorCubeFilter setValue:[NSNumber numberWithInt:size]!! !! ! ! ! ! ! ! ! forKey:@"inputCubeDimension"];! NSData *data = [NSData dataWithBytesNoCopy:keyCubeData!! !! ! ! ! ! ! ! length:cubeDataSize! ! !! ! ! ! ! freeWhenDone:YES];! [self.colorCubeFilter setValue:data forKey:@"inputCubeData"];!
Create CISourceOverCompositing
// source over filter!self.backgroundImage = [UIImage imageNamed:!! !! ! ! ! ! ! ! ! ! @"img_washington_small_02.jpg"]; ! self.backgroundCIImage = [CIImage imageWithCGImage:!! !! ! ! ! ! ! ! ! ! ! self.backgroundImage.CGImage];! self.sourceOverFilter = [CIFilter filterWithName:!! !! ! ! ! ! ! ! ! ! @"CISourceOverCompositing"];! [self.sourceOverFilter setValue:self.backgroundCIImage !! !! ! ! ! ! ! ! forKeyPath:@"inputBackgroundImage"];!
Apply Filters in Capture Callback
CIImage *bufferCIImage = [CIImage imageWithCVPixelBuffer:cvBuffer];!![self.colorCubeFilter setValue:bufferCIImage !! !! ! ! ! ! ! ! ! ! forKey:kCIInputImageKey];! CIImage *keyedCameraImage = [self.colorCubeFilter valueForKey:!! !! ! ! ! ! ! ! ! ! ! ! kCIOutputImageKey];! ![self.sourceOverFilter setValue:keyedCameraImage !! !! ! ! ! ! ! ! ! forKeyPath:kCIInputImageKey];! !CIImage *compositedImage = [self.sourceOverFilter valueForKeyPath:!! !! ! ! ! ! ! ! ! ! ! kCIOutputImageKey];
Then draw compositedImage to CIContext as before
Other Points of Interest• CIQRCodeGenerator filter — Converts data (e.g., a string) to
a QR Code
• CILenticularHaloGenerator filter — aka, lens flare
• CIDetector — Class (not a filter) to find features in images. Currently only supports face finding (returned as an array of CIFeatures). Optionally detects smiles and eye blinks within faces.
• CIImage has a red-eye enhancement that takes the array of face CIFeatures to tell it where to apply the effect
Core Image on OS X• Core Image is part of QuartzCore (or Image Kit), so
you don’t @import CoreImage
• Many more filters are available
• Can create your own filter with OpenGL Shading Language (plus some CI extensions). See CIKernel.
• Also available in iOS 8
• Filters can be set on CALayers
CALayer Filters on OS X• Views must be layer-backed (obviously)
• Must also call -[NSView setLayerUsesCoreImageFilters:] on 10.9+
• CALayer has properties: filters, compositingFilter, backgroundFilters, minificationFilter, magnificationFilter
• These exist on iOS, but do nothing
Demo
Adding CIPixellate to layer’s filters
self! !! ! ! ! ! ! ! ! self[self! !! ! ! ! ! [ ! !! ! ! ! ! ! ! ! ![self! !! ! ! ! !! ! ! ! ! ! ! ! ! self! !! ! !
Updating a layer’s filters
-(void! [ ! !! ! ! ! ! !! ! ! ! ! ! ! ! ! ! ! !! ! ! ! }
Wrap Up: Stuff to Remember
• Get psyched about filters, but remember to check that they’re on your targeted platform/version.
• Drawing to a CIContext on iOS must be GL-backed (e.g., with a GLKView)
Q&ASlides and code will be posted to:
http://www.slideshare.net/invalidname/!
@invalidname http://subfurther.com/blog