Soft Edge Smoothness Prior for Alpha Channel Super Resolution

Shengyang Dai1, Mei Han2, Wei Xu2, Ying Wu1, Yihong Gong2

1. EECS Department, Northwestern University, Evanston, IL2. NEC Laboratories America, Inc., Cupertino, CA

Prior is needed Minimize the reconstruction error

Degraded HR → LR input Efficient solution by Back-projection (Irani’93)

However … SR is under-determined (Baker&Kanade’02,

Lin&Shum’04) Image prior is needed for regularization

What kind of prior?

LR input Back-projection Bicubic interpolation

Smooth edges are preferred

What kind of smoothness?

binary value real value

Hard edge vs. Soft edge

Soft smoothness is preferred

LR input Hard + Smooth Soft + Smooth

QuestionsHow to describe an edge? How to obtain a soft smooth edge?

Our solutionsAlpha channel edge description Soft edge smoothness prior

QuestionsHow to describe an edge? How to obtain a soft smooth edge?

Our solutionsAlpha channel edge description Soft edge smoothness prior

Image synthesis

Alpha matting

F B

(1 )B-

1 +

alpha matting

A closed form solution with smoothness assumption (Levin etc. ’06)

LR patches

HR patchesMatting for SR

alpha matting

convolution down-sampling

synthesis

LR patches

HR patches

alpha matting

convolution down-sampling

synthesis

SR?SR?

alpha matting

SRSR

synthesis

Matting for SR

BicubicBicubic BicubicBicubic

Matting for SR

? SR

QuestionsHow to describe an edge? How to obtain a soft smooth edge?

Our solutionsAlpha channel edge description Soft edge smoothness prior

Objective

Regularity term prefers soft smooth edge

Hard edge Soft edge

Geocut Our method

Objective

Hard edge Soft edge

Geocut Our method

Pixel neighborhood

Image grid graph

Hard edge smoothnessGeocut (Boykov&Kolmogorov’03)

Curve C

Cut metric:

10 : Neighborhood order of the image grid

: Edge weight of the grid graph

: Set of the pixels pairs of order k

: Binary indication function on grid

Hard edge smoothness

A regularity term prefers hard smooth edge Application

Segmentation problem Reducing the metrication artifact

Euclidean lengthCut metric

Objective

Hard edge Soft edge

Geocut Our method

Soft edge smoothness

Soft edge Level lines

Soft edge is equivalent to a set of image level lines

Soft edge smoothness

Geocut Our method

Binary indication function Real-valued function

Cut metric Soft cut metric

Soft edge smoothness

A regularity term prefers soft smooth edge Application

Super resolution Reducing the jaggy effect

For SR Objective function

Efficient optimization by steepest descent Critical parameters

Two options Alpha channel SR for each edge segment Color channels SR over entire image

…

Effect of

……

only upper plane is shown

Zoom factor

Image size

Effect ofBack-projection

Note: color channels are processed separately

LR input Our result Bicubic interpolation

Color channels SR

Color channels vs. alpha channel

Color channels SR on the entire image

Alpha channel SRon edge segments

Corner detection (He etc.

04)

Alpha channel SR

Process each edge segment separately

Comparison

Bicubic Our method Back-projection

Comparison – reconstruction based

Bicubic Our method Back-projection

Comparison – exemplar based

Our method Learning low level vision Neighbor embedding (courtesy to Bill Freeman) (courtesy to Dit-Yan Yeung)

Conclusion Soft smoothness prior

With specific geometric explanation Applicable to super resolution

Alpha channel super resolution

Limitation The smoothness prior may not hold for texture