Implementing Computational Camera

Implementing Computational Camera

Michael Mangan Product Manager, Senior Qualcomm Technologies, Inc.


Computational camera: What is it?


Computational camera Utilizes both the ISP HW and Qualcomm® Snapdragon™ processing to create new and exciting camera effects!

• Example: Image segmentation and filtering an image allow for the digital re-creation of a sophisticated portraiture effect, called the “Bokeh” effect.

Sens

or 1

Sens

or 2

ISP 1

ISP 2

Qualcomm® KRAIT CPU

Qualcomm® ADRENO™ GPU

Qualcomm® HEXAGON™ DSP

Qualcomm Snapdragon, Qualcomm Adreno, Qualcomm Krait and Qualcomm Hexagon are products of Qualcomm Technologies, Inc.


Computational camera Depth calculation designed to enable shoot once, focus later

Sens

or 1

Se

nsor

2

ISP 1

ISP 2

KRAIT CPU

ADRENO GPU

HEXAGON DSP


Computational camera Depth calculation designed to enable range finding for instant auto-focus

Sens

or 1

Se

nsor

2

ISP 1

ISP 2


Computational photography examples


Qualcomm® UbiFocus™ camera feature

Qualcomm UbiFocus is a product of Qualcomm Technologies, Inc.


UbiFocus UbiFocus

Description UbiFocus is a camera feature that generates an all in focus image from a mobile camera with a limited depth of field. Additionally, this feature allows users to selectively refocus their image in post-processing.

KPI

1. No fewer than 5 images. 2. For 13M images

1. Take no longer than 1.0s to capture images. 0.7s with further optimization 2. Take no longer than 0.5s after the capture to process input images generate an all in focus image.

OEM feature deliverable

1. UbiFocus available to licensees provided as part of the MSM8974 LA software release as optional software. 2. UbiFocus provided in binary form, and called by OEM enabled camera applications via an API interface defined by Qualcomm Technologies. 3. Use-Case

a) All-In-Focus Mode: a) When ‘All-in-Focus’ mode for UbiFocus is selected, the snapshot operation will result in an ‘All-in-Focus’ JPEG image from the

mobile camera. b) Dynamic Focus Mode:

a) User will need to select ‘Dynamic Focus Mode’ option within Camera App. b) UbiFocus will allow the end-user to touch an object in a scene and bring it in focus. c) The user can either share the refocus-able object on the web, or; d) The user can select the desired focus point and save the image as a final JPEG image.


UbiFocus • Mobile camera’s have a limited Depth-Of-Field

Out of focus In focus Out of focus

Input image 1 Input image 2

In focus


UbiFocus • From a Limited Depth-of-field Camera

• UbiFocus generates an all-in-focus image

• Refocus at a desired location as post processing

Out of focus Out of focus

Input image 1 Input image 2 UbiFocus

In focus In focus

In focus


Video: UbiFocus


UbiFocus

Use case

• Refocus at a user-desired location

• Share images that can be re-focused (the recipient does not need UbiFocus to perform dynamic re-focus)

• Generates an all-in-focus image

Benefits

• Eliminates the need for auto-focus

• Focus-on-demand “live” images

• Same effect as Lytro and Pelican Imaging, but with much higher resolution and no additional hardware

• Preserves resolution (unlike Lytro)

• Integrated with Snapdragon camera pipeline


OptiZoom camera feature

OptiZoom is a product of Qualcomm Technologies, Inc.


OptiZoom OptiZoom

Description

OptiZoom is a multi-frame super-resolution feature that enhances the resolution by 2x on each side (i.e., 4x number of pixels) that mimics an optical zoom. OptiZoom uses a Qualcomm Technologies-designed multi-frame super-resolution technique to allow an end-user to zoom into an image beyond normal digital zoom levels. OptiZoom designed to increase image resolution and also reduce noise. OptiZoom, can be used to super-resolve only a region of interest or it can be used to super-resolve the entire image.

KPI

1. Eight images of 12MP shall be taken in less than 0.6s. 2. Processing time for converting a 4MP image into a 16MP image is ~700ms on 8974. Processing time varies almost linearly

wrt the number of pixels.


1. OptiZoom, licensed to OEMs as optional software, as part of the MSM 8974 software release. 2. OptiZoom provided in binary form, and called by OEM enabled camera applications via an API interface defined by

Qualcomm Technologies. 3. Use Case:

a) The end-user zooms into the area of interest. b) The end-user initiates OptiZoom on the area of interest. c) The OptiZoom algorithm enhances the resolution of the area of interest.


OptiZoom use cases

Option 1 When user presses the “take image” button, the device takes N full-size images.

OptiZoom is invoked with the full frame

The user can zoom in to any area of the image without invoking OptiZoom again

Option 2 User first zooms-in in the preview and specifies RoI and then N images are taken.

OptiZoom is invoked with the RoI The user gets a 4x image of the RoI.

Option 3

When user presses the “take image” button, the device takes N full-size images. Then user zooms in and selects an RoI

OptiZoom is invoked with the RoI

The user gets a 4x image of the RoI. If the user want another region of the image to be super-resolved, OptiZoom can be invoked again.


OptiZoom - Disabled


OptiZoom - Enabled


OptiZoom Disabled Enabled


OptiZoom

Use case

• Increasing the image resolution by 4x number of pixels

• See previous slide for more details

• Useful for allowing users to zoom in on key regions within an image.

- Read fine text - See faces clearly

Benefits

• Increases photo resolution and reduces image noise.

• Fast and scalable

• Utilizes Snapdragon Hardware.



Qualcomm® ChromaFlash™ camera feature

Qualcomm Cromaflash is a product of Qualcomm Technologies, Inc.


ChromaFlash ChromaFlash

Description

ChromaFlash takes two images in quick succession: one with flash enabled and one without flash enabled to produce a beautiful composite image. Flash images usually suffer from over-exposedness and bluish tint and dark background. Our ChromaFlash technology overcomes these problems algorithmically and produces and image that preserves color, texture, and brightness.

KPI

1. ChromaFlash Images shall be taken in less than 0.135s on top of flash image capture time to capture 8/13MP input images. 2. ChromaFlash algorithm processing shall take no more than 700ms for two 13MP images on 8974. Processing time varies

almost linearly with the number of pixels.


1. ChromaFlash, licensed to OEMs as optional software, as part of the MSM8974 LA software release. 2. ChromaFlash provided in binary form, and called by OEM enabled camera applications via an API interface defined by

Qualcomm Technologies. 3. Use-Case/Default Operation

a) ChromaFlash will engage whenever normal Flash is used, effectively replacing ‘normal’ flash operation. b) ChromaFlash 2.0 can handle dynamic scenes (i.e., scenes that contain motion) as well as camera motion due to

handshake


No-flash image Flash image ChromaFlash image

Another ChromaFlash example

Note: Projector quality may limit ability to see enhancements visible on laptop monitor.


ChromaFlash

Use case

• Flash that preserves color

• Enhances Low-light imaging

• Enhances Dynamic Range

Benefits

• No extra hardware is required

• Reduces the noise observed in no-flash images taken in low-light.

• Maintains color correctness lost using the flash.

• Increases the dynamic range



Low-light video enhancement


Low light vision enhancements

Description

Low-light video enhancement performs two main functions: - Adaptively brightens regions of the video frame that are too dark. If the scene is sufficiently bright already, Low-light video enhancement does not interfere. - Reduces noise

KPI Processing time is ~7ms on a 1080p frame on 8974. Processing time varies almost linearly wrt the number of pixels.

Tier 1 OEM feature deliverable

1. Low-light video enhancement, licensed to OEMs as optional software, as part of the MSM8974 LA software release. 2. Low-light video enhancement shall be provided in binary form, and called by OEM enabled camera applications via an API

interface defined by Qualcomm Technologies. 3. Use-Case: Low-light video

Competition

1. Arcsoft NightHawk 2. Morpho


What is Low-light video enhancement? Low-light video enhancement is an algorithm for adaptively brightening areas of video/images that are too dark

Attributes

• It works for dark scenes and scenes that are generally bright, but include dark regions

• It is a low-power and low-complexity algorithm. As a result, it can run comfortably on even low-tier chips @ 30fps for 1080p resolution

• It also includes a fast and efficient noise-reduction to attenuate noise in the areas that were under-exposed in the original frame


Video: Low light enhancement


Low-light video enhancement comparison Original Revision-LTM


Low-light video enhancement comparison Original Low-light video enhancement


Noise reduction

No Temporal NR With Temporal NR


Video: Preserving sharpness


Low-light video enhancement enhancing low light video

Original Low-light video enhancement


Portrait enhancement


Portrait enhancement

Description

A a camera feature that takes a regular portrait image taken with a camera phone and synthesizes shallow depth-of-field (aka Bokeh effect) as if the image is produced by an DSLR camera. This feature also brings a static image to life by automatically generating 3D parallax and zoom effect (see the videos in upcoming slides) The input image does not even have to be taken by a camera equipped with portrait enhancement.

KPI

1. Requires an image with 1-3 people in the scene (the max number of people is adjustable if needed) 2. Processing time on 8974:

a) Initial segmentation: 1.2s b) Re-segmentation after user edit: 200ms c) Bokeh effect: 1.1s for 13MP output d) Parallax effect: 250ms e) Dolly Zoom effect: 250ms f) Horizontal motion blur effect: 1.1s for 13MP output g) Sketch effect: 550ms for 13MP output h) Fusion effect: 450ms for 13MP output


1. Portrait enhancement licensed as optional software as part of the MSM8974 LA software release. 2. Portrait enhancement shall be provided in binary form, and called by OEM enabled camera applications via an API interface defined by Qualcomm

Technologies. 3. Use-Case:

- Synthesize a shallow-depth-of-field automatically even on images that have been taken previously without portrait enhancement - Generates 3D Parallax and Dolly Zoom effects


Portrait enhancement • What is it?

− It takes a single image and automatically segments out humans in the image. Then one of several effects (see following slides) can be applied applied to the background

• Attributes

− portrait enhancement can be applied even to images that are taken previously using an ordinary camera

− If needed, it allows the user to edit the foreground and background by providing small hints


Portrait enhancement vs. DSLR

Camera phone (input image)

Portrait enhancement DSLR w/ expensive lens


Input image


Shallow depth-of-field (mimics portrait mode of DSLR cameras)


Parallax


Dolly-zoom


Horizontal motion blur


Sketch


Still camera image enhancement


Still camera image enhancement

Description

Takes a single image and enhances the dynamic range of the still image by automatically brightening regions of the image that are too dark and remove haze from the image. It allows the user to move a slider and change the amount of adaptive enhancement applied to the image. The input image does not even have to be taken with a camera that is equipped with Still camera image enhancement.

KPI Processing time for 13MP image on 8974 is ~80ms. Processing time varies almost linearly wrt the number of pixels.


1. Still camera image enhancement, licensed as optional software, provided as part of the MSM8974 LA software release. 2. Still camera image enhancement shall be provided in binary form, and called by OEM enabled camera applications via an API

interface defined by Qualcomm Technologies. A sample app is included as a reference to guide OEM’s implementation of the feature

3. Use-Case: - Enhance the image quality dramatically by adaptively and automatically removing haze and brightening regions of the image that are too dark


Video: Still camera image enhancement


General image segmentation


Image segmentation

• Accurate image segmentation with user interactive

• Applications include

– Photo editing for re-focusing

– Multi-layer color editing

– Multi-layer image filtering

• Available – October 2014

Segm

enta

tion

Mat

ting


Video: Image segmentation


Touch to Track A general object tracker


Touch to Track (T2T) v2.0

Track, detect, and learn

• Lucas-Kanade optical flow tracking

• Decision Forest and NCC detection

• Online learning bootstrapping binary classifier

• Leverages Motion Vector Information from Venus Video Core.

Applications

• Camera and video auto focus, exposure, zoom, etc.

• Robot control with T2T

• Advanced video recording / snapshot


Video: Touch to Track (T2T) v2.0


Touch to Track – Power optimized performance • Below are the power consumption in performance-mode* and power-optimized-mode** of Touch to Track on 8974

using the same test sequences

• In performance mode, Touch to Track power consumption is about 200-300% of those in the power-optimized-mode. The ratio could be higher in full frame detection

382

367

210

395

109 123 113

183

0

50

100

150

200

250

300

350

400

450

video1 video2 video3 video4

mW

performance mode power-optimized mode

* Performance mode (CPU Only) and power-optimized (DSP & CPU) mode selectable ** Power saving achieved by utilizing multiple processors including DSP on 8974 Note: No tracking performance differences between performance and power-optimized modes.


Touch to Track

Use case

• Tracker enhanced 3A algorithms • Single Camera Picture in Picture in preview and

recording • Artistic special effects in camera preview and video

recording

Benefits

• Multiple object tracking - Track 2, up to 4 objects

• Robust tracking algorithm tightly integrated with camera 3A and auto-zoom algorithms, enhances: - Auto Exposure - Auto Focus - Auto Zoom.

• Allow selected region of image to be displayed in single-camera PIP (Picture in Picture)

• Tracker is Snapdragon HW accelerated • Tracking on 1080p @30fps


Object removal & cloning


Object removal and cloning 2.0 – single frame This feature is different from multiframe motion based object removal • Content based on-device image editing

• Object Removal allows the user to select and remove and object

• Cloning allows users to replicate and insert an image Object removal Object cloning


Object removal and cloning

Use case

Object Removal

• Allows on-device, user selectable, removal of blemishes within the image.

• Uses advanced background matching algorithm to match the background.

Object Cloning

• Allows on-device, user selectable, cloning of new images onto existing photographs.

Benefits

• Low power, high quality, image processing features • Easily integrated into existing Camera Application to

fit existing User Interface. • Written using OpenCL to fully utilize Adreno GPU


For more information on Qualcomm, visit us at: www.qualcomm.com & www.qualcomm.com/blog

©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved. Qualcomm, Snapdragon, Adreno, and Hexagon are trademarks of Qualcomm Incorporated, registered in the United States and other countries. Krait, Uplinq, UbiFocus and ChromaFlash are trademarks of Qualcomm Incorporated. All Qualcomm Incorporated trademarks are used with permission. Other products and brand names may be trademarks or registered trademarks of their respective owners. References in this presentation to “Qualcomm” may mean Qualcomm Incorporated, Qualcomm Technologies, Inc., and/or other subsidiaries or business units within the Qualcomm corporate structure, as applicable. Qualcomm Incorporated includes Qualcomm’s licensing business, QTL, and the vast majority of its patent portfolio. Qualcomm Technologies, Inc., a wholly-owned subsidiary of Qualcomm Incorporated, operates, along with its subsidiaries, substantially all of Qualcomm’s engineering, research and development functions, and substantially all of its product and services businesses, including its semiconductor business, QCT.

Thank you FOLLOW US ON:

Implementing Computational Camera

Technology

Transcript of Implementing Computational Camera