GROUP MAD COMPETITION - A NEW M COMPARE O I Q M › ~zduanmu › cvpr16_gmad › paper ›...
Transcript of GROUP MAD COMPETITION - A NEW M COMPARE O I Q M › ~zduanmu › cvpr16_gmad › paper ›...
GROUP MAD COMPETITION
- A NEW METHODOLOGY TO COMPARE
OBJECTIVE IMAGE QUALITY MODELS
Kede Ma, Qingbo Wu, Zhou Wang, Zhengfang Duanmu,Hongwei Yong, Hongliang Li and Lei Zhang
June 28, 2016
Image Quality Assessment (IQA)
PurposeCreate objective models to predict human perception of image quality.
QuestionWith a significant number of IQA models available, how to fairly comparetheir performance?
2 / 13
Evaluating IQA Models
Conventional Evaluation MethodologyProve them by computing correlation metrics between subjective assessmentand objective model predictions.
ProblemEnormous image space
……
……
……
……
Subjective test
Unaffordable
3 / 13
MAximum Differentiation (MAD) Competition
Merits of MADDisprove IQA models by synthesizing strongest “counter-examples”.
Counter-examples search
�������������� �������������������� �������� �� ����������������� ��
�� �������������������� ��� �������������������� ��� �������������������� �
4 / 13
MAximum Differentiation (MAD) Competition
MAD Competition
Reference
image
B
C
D EA
Best SSIM for
fixed MSE
Worst SSIM for
fixed MSE
Best MSE for
fixed SSIM
Worst MSE for
fixed SSIM
Initial
distortion
5 / 13
Group MAD (gMAD) Competition
Attacking-Defending Game between Models
(a) An image collection
(b) Subset of images that
have the same PSNR
(c1) Best MS-SSIM image
(c2) Worst MS-SSIM image
(d1) Best BIQI image
(d2) Worst BIQI image
(e1) Best M3 image
(e2) Worst M3 image
image grouping by
PSNR (defender)
Pair sampling by other
models (attackers)
MS-SSIM
(attacher1)
BIQI
(attacher2)
M3
(attacher3)
6 / 13
Group MAD (gMAD) Competition
Subjective Testing
7 / 13
Group MAD (gMAD) Competition
Performance Measures1 Aggressiveness: How successful of a model at attacking another model?2 Resistance: How successful of a model at defending the attacks from
another model?
Global RankingThe Global rankings obtained by aggregating the aggressiveness matrix A andResistance matrix R.
8 / 13
Waterloo Exploration Database
4K+ source and ∼100K distorted images
Human Animal Plant Landscape
Cityscape Still-life Transportation
9 / 13
Applying gMAD to Waterloo Exploration Database
Pairwise Comparison between 16 Models
MS-SSIM
CORNIAFSIM
PSNRNIQE
ILNIQESSIM
TCLTLPSI
QAC
BRISQUENFERM
BIQI
BLIINDS-II M3
DIIVINE
MS-SSIM
CORNIA
FSIM
PSNR
NIQE
ILNIQE
SSIM
TCLT
LPSI
QAC
BRISQUE
NFERM
BIQI
BLIINDS-II
M3
DIIVINE
-60
-40
-20
0
20
40
60
Figure: Aggressiveness matrix
MS-SSIM
CORNIAFSIM
PSNRNIQE
ILNIQESSIM
TCLTLPSI
QAC
BRISQUENFERM
BIQI
BLIINDS-II M3
DIIVINE
MS-SSIM
CORNIA
FSIM
PSNR
NIQE
ILNIQE
SSIM
TCLT
LPSI
QAC
BRISQUE
NFERM
BIQI
BLIINDS-II
M3
DIIVINE -60
-40
-20
0
20
40
60
Figure: Resistance matrix
10 / 13
Applying gMAD to Waterloo Exploration Database
Global Ranking Result of 16 Models
Glo
bal r
anki
ng s
core
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
M3
DIIVINE
NFERM
BLIINDS-II
BRISQUEBIQI
QACTCLT
LPSISSIM
PSNRNIQE
ILNIQEFSIM
CORNIA
MS-SSIM
ResistanceAggressiveness
11 / 13
Applying gMAD to Waterloo Exploration Database
Observations1 FR-IQA models are more competitive;2 MS-SSIM and FSIM are top performing FR-IQA models;3 CORNIA and ILNIQE are top performing NR-IQA models;4 Machine learning based IQA models generally do not perform well.
12 / 13
Thank you
13 / 13