iOS GPUImage introduction
17
Aug. 5, 2014 GPUImage Introduction Bruce Tsai
-
Upload
yi-lung-tsai -
Category
Mobile
-
view
269 -
download
1
description
An introduction to GPUImage that is an excellent image processing framework for iOS. The architecture, threading model and how to leverage OpenGL ES are addressed in the slides.
Transcript of iOS GPUImage introduction
Aug. 5, 2014
GPUImage IntroductionBruce Tsai
Core Image
✤ Included in iOS from 2011!
✤ 100+ built-in filters!
✤ Multithread by CPU
Drawbacks
✤ Customization limited!
- Subclass CIFilter in iOS!
- Create custom filters (shader) only in OSX!
✤ Performance not tuned!
- Address tuning tips in WWDC 2012 (session 511)3
GPUImage
✤ Open source (BSD) iOS framework!
✤ GPU-based image and video processing!
✤ From GPU-accelerated video processing project (2010)!
✤ https://github.com/BradLarson/GPUImage
4
Advantages
✤ Performance!
- High FPS in real-time preview!
✤ Customization!
- OpenGL ES shader code
5
Buffer Process Flow
VideoCamera Sepia
View
MovieWriter
BlendPicture
Source Filter Output
Code Snippet
GPUImageVideoCamera *videoCamera = [[GPUImageVideoCamera alloc] initWithSessionPreset:AVCaptureSessionPreset640x480 cameraPosition:AVCaptureDevicePositionBack];
GPUImageFilter *customFilter = [[GPUImageFilter alloc] initWithFragmentShaderFromFile:@“CustomShader”];
GPUImageView *filteredVideoView = [[GPUImageView alloc] initWithFrame:CGRectMake(0.0, 0.0, viewWidth, viewHeight)];
[videoCamera addTarget:customFilter];
[customFilter addTarget:filteredVideoView];
[videoCamera startCameraCapture];
Class Diagram
Procedures
1. Add effect to input buffer !
2. Save result to output buffer!
3. Set targets’ input buffer attributes!
4. Set targets’ input buffer!
5. Invoke targets’ operation9
Frame Buffer Object (FBO)
glBindFramebuffer(GL_FRAMEBUFFER, framebuffer);
glBindTexture(GL_TEXTURE_2D, renderTexture);
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, renderTexture, 0);
Buffer Binding in FBO
Frame BufferTexturePixel Buffer
Render destinationActual render target
Main memory buffer
Conversion to and from Texture
CVPixelBufferRef pxBufferRef = CMSampleBufferGetImageBuffer(imageBuffer);
CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault, videoTextureCache, pxBufferRef, NULL, GL_TEXTURE_2D, GL_RGBA, width, height, GL_BGRA, GL_UNSIGNED_BYTE, 0, &videoTextureRef);
...
// renderTargetRef is corresponding to renderTexture of FBO
[writerAdaptor appendPixelBuffer:renderTargetRef withPresentationTime:pts];
Threading Model
✤ Grand Central Dispatch (GCD)!
- Dispatch queue!
✤ runSynchronousOnVideoProcessingQueue!
✤ runAsynchronousOnVideoProcessingQueue!
✤ runSynchronousOnContextQueue!
✤ runAsynchronousOnContextQueue14
Demo
Reference
✤ https://github.com/BradLarson/GPUImage!
✤ http://blog.db-in.com/all-about-opengl-es-2-x-part-2/!
✤ https://developer.apple.com/library/ios/documentation/3DDrawing/Conceptual/OpenGLES_ProgrammingGuide/Introduction/Introduction.html#//apple_ref/doc/uid/TP40008793!
✤ https://developer.apple.com/library/ios/documentation/GraphicsImaging/Conceptual/CoreImaging/ci_intro/ci_intro.html#//apple_ref/doc/uid/TP30001185
