Robust Face Hallucination using Quantization-Adaptive Dictionaries

Reuben FarrugiaChristine Guillemot

IEEE Int. Conf. on Image Processing, Arizona, USA 26th September 2016

Outline

• Limitations of existing Face Hallucination Methods• Analysis of Compression artefacts on faces• Proposed Robust Face Hallucination Methods• Experimental Results• Robust Face Hallucination in the Wild

Limitations of existing Face Hallucination methods

Face Hallucination

The low-quality face image 𝑿 isdivided into overlapping patchesෝ𝒙1,ෝ𝒙2,…ෝ𝒙𝑷.

These weights 𝒘𝑝are then used to

combine the corresponding high-

quality dictionary 𝑯𝑝 to restore 𝒚𝑝.

The optimal weights 𝒘𝑝 suitable to

reconstruct ෝ𝒙𝑝 are derived using the

low-quality dictionary 𝑳𝑝.

Face Hallucination Dictionary Construction

Let X and Y denote the high- andlow- quality facial images,respectively. Face Hallucinationmethods formulate the acquisitionmodel as

𝑿 =↓ 𝑩𝒀 + 𝜖 where 𝑩 is the blurring function and ↓ is thedownscaling operator.

However, this formulation is valid for uncompressed images and is noteffective when the low-quality image is compressed.

Non-Robust Face Hallucination

Comparing the original high quality face image (left), the downscaled and compressed low-quality face image (centre) and the super-resolved image using a state-of-the-art face hallucination method (right). The original image is down-scaled such that they have inter-eye distance of 10 pixels and compressed using H.264/AVC Intra with a quantization parameter of 20.

Analysis of Compression artefacts on Faces

Compression Artefacts on FacesH

.26

4/A

VC

Co

de

c d

x =

20

H.2

64

/AV

C C

od

ec d

x =

10

QP = 10 QP = 35 QP = 10 QP = 35

Compression Artefacts on FacesJP

EG C

od

ec

dx

= 2

0

JPEG

Co

dec

dx

= 1

0

Quality = 70 Quality = 30 Quality = 70 Quality = 50

Proposed Robust Face Hallucination Method

Robust Face Hallucination Dictionary Const.

The acquisition model considered in this work to construct thecoupled dictionaries includes the codec following the blurring anddownscaling operators and can be formulated using𝑿 = 𝑻−1𝑄 𝑻 ↓ 𝑩𝒀 + 𝜖 where T is the transform and Q is thequantization function.

Robust Position Patch (RPP)

The Robust Position Patch derives the combination weights byminimising the following optimization

𝑤𝑝 = 𝑎𝑟𝑔𝑚𝑖𝑛𝑤𝑝ො𝑥𝑝 − 𝐿𝑝𝑤𝑝 2

𝑠. 𝑡. 𝑤𝑝 2= 1

Robust Position Patch (RPP)

The Robust Position Patch derives the combination weights byminimising the following optimization

𝑤𝑝 = 𝑎𝑟𝑔𝑚𝑖𝑛𝑤𝑝ො𝑥𝑝 − 𝐿𝑝𝑤𝑝 2

𝑠. 𝑡. 𝑤𝑝 2= 1

Robust Sparse Position Patch (RSPP)

The Robust Sparse Position Patch derives the combination weights byminimising the following optimization

𝑤𝑝 = 𝑎𝑟𝑔𝑚𝑖𝑛𝑤𝑝𝑤𝑝 1

𝑠. 𝑡. ො𝑥𝑝 − 𝐿𝑝𝑤𝑝 2< 𝜖

Experimental Results

PSNR Analysis

20 25 30

Baseline 10 27.59 27.33 26.63

PP 10 25.74 25.73 25.53

RPP 10 30.19 29.73 28.65

SPP 10 25.52 25.43 25.20

RSPP 10 30.12 29.54 28.51

LINE 10 25.32 25.22 25.01

RLINE 10 29.95 29.39 28.30

Baseline 20 29.63 29.46 29.03

PP 20 34.13 33.29 31.70

RPP 20 34.11 33.36 32.05

SPP 20 33.91 32.89 31.22

RSPP 20 34.01 33.21 31.88

LINE 20 33.90 32.88 31.25

RLINE 20 33.94 33.09 31.64

Quantization ParameterMethod dx

H.264/AVC Intra Codec

50 60 75

Baseline 10 26.78 27.02 27.48

PP 10 27.36 27.71 28.50

RPP 10 28.34 28.58 29.18

SPP 10 26.82 27.16 27.83

RSPP 10 28.13 28.40 28.96

LINE 10 26.67 27.02 27.74

RLINE 10 27.99 28.23 28.83

Baseline 20 30.05 30.30 30.75

PP 20 31.07 31.39 32.23

RPP 20 31.83 32.09 32.75

SPP 20 30.54 30.86 31.59

RSPP 20 31.62 31.88 32.47

LINE 20 30.52 30.84 31.65

RLINE 20 31.51 31.81 32.47

Quality ParameterMethod dx

JPEG Codec

Recognition Analysis

20 25 30

Baseline 10 0.538 0.498 0.354

PP 10 0.416 0.377 0.281

RPP 10 0.553 0.486 0.318

SPP 10 0.469 0.399 0.304

RSPP 10 0.619 0.561 0.385

LINE 10 0.485 0.424 0.293

RLINE 10 0.607 0.556 0.394

Baseline 20 0.722 0.693 0.625

PP 20 0.748 0.714 0.640

RPP 20 0.745 0.700 0.623

SPP 20 0.780 0.732 0.667

RSPP 20 0.761 0.723 0.634

LINE 20 0.779 0.744 0.681

RLINE 20 0.770 0.724 0.654

Quantization ParameterMethod dx

H.264/AVC Intra Codec

50 60 75

Baseline 10 0.362 0.380 0.450

PP 10 0.424 0.433 0.517

RPP 10 0.382 0.436 0.479

SPP 10 0.415 0.436 0.525

RSPP 10 0.428 0.461 0.526

LINE 10 0.444 0.481 0.532

RLINE 10 0.417 0.455 0.523

Baseline 20 0.663 0.685 0.699

PP 20 0.673 0.689 0.719

RPP 20 0.668 0.672 0.711

SPP 20 0.675 0.707 0.731

RSPP 20 0.687 0.705 0.727

LINE 20 0.683 0.719 0.726

RLINE 20 0.679 0.698 0.726

Quality ParameterMethod dx

JPEG Codec

Subjective Evaluation

Comparing the original high quality face image (first column), the downscaled and compressed low-quality face image(second column), the super-resolved image using a state-of-the-art face hallucination method (third column) and thesuper-resolved face image using the proposed robust method, where the LM-CSS face SR method is used. The originalimage is down-scaled such that dx = 10 and compressed using H.264/AVC Intra with a quantization parameter of 20.

Robust Face Hallucination in the Wild

Robust Face Hallucination in the WildTraining Images

Low-quality image

Upscale

Warping

HallucinationDegradation

Robust Face Hallucination in the Wild

HQ Image LQ Image RLM-CSS HQ Image LQ Image RLM-CSS

Conclusion

• This work presents a new dictionary construction methodwhich employs the syntax available in the image file tomodel the distortions affecting the low-quality image.

• Experimental Results show a significant gain in terms ofquality with respect to the non-robust hallucinationmethods.

• The proposed method can be extended to super-resolvefacial images in the wild.

Thank You

Reuben Farrugiahttps://www.um.edu.mt/staff/reuben.farrugiareuben.farrugia@um.edu.mt

More information about this project can be found onhttps://www.um.edu.mt/staff/reuben.farrugia/projects/face_hallucination