PL-4043, Accelerating OpenVL for Heterogeneous Platforms, by Gregor Miller

OpenVL: A Developer-Friendly Abstraction forComputer Vision on APU, GPU and CPU

Gregor Miller and Sidney Fels

Human Communication Technologies LaboratoryUniversity of British Columbia

Vancouver, Canada

13th November 2013AMD Developer Summit

OpenVL: A Developer-Friendly Abstraction for Computer Vision on APU, GPU and CPU

OpenVLVision Everywhere

2

Source: AFDS Keynote by Phil Rogers, AMD Inc. (12/06/12)


OpenVLWhat’s happening in the real world?

Quote from Hacker News about FastCV and OpenCV (28/10/11):

“I hate how this (and to some extent OpenCV) expectsyou to understand computer vision to use the library.

I sympathise with your frustration, but being in vision myself, I have an idea why this is the case: vision only barely works!What this means is that most things you would want to do are only barely possible, if at all, and it is highly application- and input-dependent. To get it to work the way you want, you have to understand some vision, so that you can tweak things the right way.I think once vision matures more and things become more ‘off-the-shelf’, you'll find more libraries aimed at non-experts.”

3Source: http://news.ycombinator.com/item?id=3167295

http://news.ycombinator.com/item?id=3167295

http://news.ycombinator.com/item?id=3167295


OpenVL

4 13 | The Promise of Parallel: Today’s State of Heterogeneous Computing | June 12, 2012

Dev

elop

er R

etur

n (P

erfo

rman

ce, F

eatu

res,

etc

.)

Developer Investment (Effort, Time, New skills)

GPU DEVELOPERS (NINJAs)

WHAT WILL IT TAKE TO HAVE MORE PROJECTS LIKE “CINEMADNG->OPENEXR CONVERSION”? HOW DO WE ATTRACT THE MAINSTREAM DEVELOPER?

~4M+ Apps

Good User Experiences

MAINSTREAM DEVELOPERS

~200+ Apps

Significant Niche Value

~10+M* CPU

Coders

~100K GPU

Coders

*IDC Source: AFDS Keynote by Tom Malloy, Senior Vice President and Chief Software Architect, Adobe Systems Inc. (12/06/12)

Mainstream Developers are Key to Technology Adoption


OpenVLHigher-Level Abstractions are Needed

522 | The Promise of Parallel: Today’s State of Heterogeneous Computing | June 12, 2012

SOFTWARE CONSIDERATIONS PLATFORMS (AND STANDARDS)

Claim 3: OpenCL is a valuable standard for ninjas, but … Higher level abstractions that provide performance portability

needed for mainstream developers May be evolutionary

See Khronos’ OpenCL Overview http://www.khronos.org/assets/uploads/developers/library/overview/opencl-overview.pdf

… or it may be something completely different

Source: AFDS Keynote by Tom Malloy, Senior Vice President and Chief Software Architect, Adobe Systems Inc. (12/06/12)


OpenVLCurrent Computer Vision Frameworks

• OpenCV: Probably the most widely-used vision framework

• Not aimed at mainstream developers: use face detection as an example

• We do not consider it developer-friendly for three reasons:

– The specific algorithm is used as the function name - cvHaarDetectObjects

– The arguments are specific to the algorithm– No indication of the conditions under which it

operates: e.g. how much of a rotation is allowed, what’s the smallest/largest face, etc.

• Most common level of abstraction for vision, similar found in Matlab IPT/CVT, ...

6


OpenVLHCI for Vision, focussing on developers; Why?

7

• Example from FastCV: Qualcomm’s vision library optimised for mobile and embedded platforms

– fcvCornerFast9InMaskScoreu8(const uint8_t *__restrict src, unsigned int srcWidth, unsigned int srcHeight, unsigned int srcStride, int barrier, unsigned int border, uint32_t *__restrict xy, uint32_t *__restrict scores, unsigned int nCornersMax, uint32_t *__restrict nCorners, const uint8_t *__restrict mask, unsigned int maskWidth, unsigned int maskHeight)

– fcvNCCPatchOnCircle8x8u8_v2(const uint8_t *__restrict patch, const uint8_t *__restrict src, unsigned short srcWidth, unsigned short srcHeight, unsigned short search_center_x, unsigned short search_center_y, unsigned short search_radius, int filterLowVariance, uint16_t *best_x, uint16_t *best_y, uint32_t *bestNCC, int findSubPixel, float *subX, float *subY)

– fcvTrackBMOpticalFlow16x16u8(const uint8_t *__restrict src1, const uint8_t *__restrict src2, uint32_t srcWidth, uint32_t srcHeight, uint32_t srcStride, uint32_t roiLeft, uint32_t roiTop, uint32_t roiRight, uint32_t roiBottom, uint32_t shiftSize, uint32_t searchWidth, uint32_t searchHeight, uint32_t searchStep, uint32_t usePrevious, uint32_t *numMv, uint32_t *locX, uint32_t *locY, uint32_t *mvX, uint32_t *mvY)

– fcvDescriptor17x17u8To36s8(const uint8_t *__restrict patch, int8_t *__restrict descriptorChar, int32_t *__restrict descriptorNormSq)

– fcvMserExtu8(void *mserHandle, const uint8_t *__restrict srcPtr, unsigned int srcWidth, unsigned int srcHeight, unsigned int srcStride, unsigned int maxContours, unsigned int *__restrict numContours, unsigned int *__restrict numPointsInContour, unsigned int *__restrict pointsArray, unsigned int pointsArraySize, unsigned int *__restrict contourVariation, int *__restrict contourPolarity, unsigned int *__restrict contourNodeId, unsigned int *__restrict contourNodeCounter)


OpenVLA Case for the Problem Description Layer

8

Each of these layers plays a role for widespread adoption of computer vision.

Applications

Computer Vision Solutions

Computer Vision Frameworks

Computer Vision Problem Description

Hardware Acceleration API

Physical Hardware

Consumer

Vision Ninja

Vision Researcher

Mainstream Developer

Optimisation Specialists

Users


OpenVLHigher-level abstraction for computer vision

9

Performance

Precision

Power

Performance

Precision

Power

Performance

Precision

Power ?


OpenVL Analysis using OpenVL: ‘What’ vs ‘How’

• Analysis using OpenVL– High-level abstraction targeted towards mainstream

developers

• API based on problem description– The description is interpreted to select algorithm(s) to

find the required solution

• Expressive interface: ‘what’ vs. ‘how’– Describe what the task is, not how to solve it

• Example: face detection– Description: pose, 2D rotation, expression, age,

occlusion, size quantity, species

10


OpenVLOpenVL

• Input / Output– Images: input from developer (via VU)– Segments: basic unit; distinct regions within the image

e.g. red circle– Matches/Detections: representation of matches and

detections e.g. a face– Transforms: geometric representation e.g. vector of

motion between segments

11

•Operations– Segment(constraints, images, segments)– Detect(constraints, template, segments, detections)– Match(constraints, variances, segments, matches)– Solve(constraints, matches/detections, transforms)– Select(constraints, conditions, segments, segments)

•Descriptors– Template: Description of well-known (and usually

arbitrary) ‘object’ to find in images e.g. face, eyes, hand, body

– Variances: Differences between segments such as intensity

– Conditions: Property-based requirements such ascolour == red

•Constraints– Operation-specific hints e.g.

– Quantity of segments/detections/matches– Shape/regularity of segments– Type of match e.g. unique


OpenVLUsing Segments to Abstract Pixels

12

Definition:A segment is a distinct continuously bounded region within a bounded 2D plane


OpenVLSegmentation using a Problem Description

13


OpenVLImage Registration Example

14

Segment - Match - Solve(Image)

Visualized OutputInput


OpenVLFlexibility of Operation Sequences

• Segment - Match(variances) - Solve(Image)defines image registration

– Match variances: Intensity: HDR registration Blur: Multi-focal registration

• Segment - Match(variances)defines correspondence, as shown

15


OpenVLOpenVL Matching Demo

16


OpenVLOptical Flow (Video Motion)

17

• Change Solve(Image) to Solve(Segment) defines Optical Flow– Find the transform of each pair of matched segments

Segment - Match - Solve(Segment)


OpenVLCombined Example: Stitching and Detection

18


OpenVLExample accelerated back-end to OpenVL using HSA

19


OpenVLSummary and Future Work

•A problem description layer fills the gap between applications and algorithms

– Critical for widespread adoption of computer vision– Allows replacement of implementation without impacting developer– High-level abstraction aimed at mainstream developers - not ninjas!

•Expressive API (based on ‘What’, not ‘How’)

•An abstraction for all platforms

•Prototype reference implementations for CPU, GPU and HSA platforms

20


OpenVLOpenVL has been released! (Image segmentation)

21

[email protected] | http://www.openvl.org

mailto:[email protected]

mailto:[email protected]

http://www.openvl.org

http://www.openvl.org


OpenVLAcknowledgements

22

PL-4043, Accelerating OpenVL for Heterogeneous Platforms, by Gregor Miller

Technology

Transcript of PL-4043, Accelerating OpenVL for Heterogeneous Platforms, by Gregor Miller