Chimera Computational Photography Architecture | GTC 2013
17
1 | Chimera Computational Photography Architecture Brian Cabral VP of Computational Imaging
Transcript of Chimera Computational Photography Architecture | GTC 2013
1 |
Chimera Computational Photography Architecture
Brian Cabral VP of Computational Imaging
2 |
What problem are we trying solve
! Accelerate a wide variety of CP applications
! Complement existing programming environments ! CUDA, OpenGL shaders
! Make it easy to program hetrogeneous functional HW units
! Don’t try to solve the “general” image processing data flow graph problem
! Make it fast
HDR Panorama
High Dynamic Range
Object Tracking
3 |
Chimera: Computational Photography Architecture
VI-M
ux
CSI
Sensor
State Bus
State
Images
S0 S1 Sn …
Kernels GPU
K0
K1
Kn …
Kernels GPU
K0
K1
Kn
…
BP DM LS AWB
CCM YUV EE γ
HW-ISP
Frame/Image Bus F0 F1 Fn …
Conceptual model
Kernels CPU
K0
K1
Kn
…
Kernels CPU
K0
K1
Kn
…
4 |
Side bar: Bayer and YUV Domains
Y
U
V
BP DM LS AWB
CCM YUV EE γ
HW-ISP
5 |
Chimera Pipeline Programming Model
YUV Domain Bayer Domain
CPU GPU
ISP
GPU CPU
Frame Bus
State Bus
Frame Bus
State Bus
Control Flow
Data Flow Module Private Data Stage image buffer
6 |
Chimera Pipeline General Example
YUV Domain Bayer Domain
CPU GPU
ISP
GPU CPU
Frame Bus
State Bus
Frame Bus
State Bus
Control Flow
Data Flow Module Private Data Stage image buffer
… … … …
7 |
Camera Core
Chimera
Where does Chimera live
ISP … … … …
EGL context manager Copy engine CPU thread manager Buffer manager
Native Camera Application
OpenGL/3D driver 2D Driver
HW
ISP Driver
GPU 2D CPU/Neon CSI data mux ISP
Chimera API Camera HAL
8 |
Chimera Pipeline HDR Example
YUV Domain Bayer Domain
GPU
ISP
CPU
Frame Bus
State Bus
Frame Bus
State Bus
Control Flow
Data Flow Module Private Data Stage image buffer
9 |
What’s Required for Instant HDR?
This is your focal point of the frame that you want to ensure is captured perfectly
Capturing both the background and foreground is the desired outcome
10 |
Basic “traditional” HDR fusion technique
Find Matching
pixels
Warp Images
Index Images
Blend Images
11 |
Basic “traditional” HDR fusion technique
Find Matching
pixels
Warp Images
Index Images
Blend Images
2 40ops * ( 8ops 10ops 2ops ) * 8mp = ~1.0 Gflops
12 |
Basic “traditional” HDR fusion technique It would take 30GFlops to do this on a video stream!
Find Matching
pixels
Warp Images
Index Images
Blend Images
2 40ops * ( 8ops 10ops 2ops ) * 8mp = ~1.0 Gflops
13 |
Chimera Pipeline HDR Example
YUV Domain Bayer Domain
GPU
ISP
CPU
Frame Bus
State Bus
Frame Bus
State Bus
Control Flow
Data Flow Module Private Data Stage image buffer
14 |
Using the API for HDR example ! Setup a GPU stage
! Easy to set up the shader
! Plus uniforms etc.
!!class MyGPUModule : public Chimera::Module!{!public:! MyModule(int myParam)! {!
! OglStage *GpuStage = new OglStage(); ! SetStage(0, GpuStage);! GpuStage->SetVertexShader(…);! GpuStage->SetVertexUniforms(…);! GpuStage->SetFragmentShader(…);! GpuStage->SetFragmentUniforms(…);! GpuStage->SetTexture(…);! }!}! !!!
15 |
Using the API - cont’d ! Setup a CPU stage
! Include your code
! Plus “params”
class MyCPUStage : public Chimera::CPUStage!{!public:! MyCPUStage(int someParam) : param(someParam) { }! virtual void MyCPUStage::process(const MetaData& meta, ! ImageBuffer* inBuf)! {! // loop over pixels in inputBuffer! /// modifying them using param, etc.! }!private:! int param;!};!!!class MyCpuModule : public Chimera::Module!{!public:! MyModule(int myParam)! {! setStage(0, new MyCPUStage(myParam * M_PI));! }!}!
16 |
Using the API
! Create a pipline
! Add in the domains
! Attach your modules
! Glue the pipe together
! Launch it
class MyPipeline: public Chimera::Pipeline!{!public:! errorHandler(int errorCode)! {! //…! }!};!!int main(int argc, char **argv)!{! Chimera::Domain gpuDomain();! Chimera::Domain cpuDomain();!! gpuDomain.setModule(myGpuModule);! cpuDomain.setModule(myCpuModule);!! //…! MyPipeline pipe = MyPipeline(someParam);! pipe.setDomainPreIsp(gpuDomain);! pipe.setDomainPostIsp(cpuDomain);! pipe.Launch(pipe)!}!
17 |
Chimera Recap
! Enable a wide variety of CP applications
! Bulid upon existing programming environments ! CUDA, OpenGL shaders
! Provide an easy to use API
! 90%-10% solution
! Be fast and agile
HDR Panorama
High Dynamic Range
Object Tracking
