Post on 26-Mar-2020
Fei Gaogaofei@hdu.edu.cn
http://camalab.hdu.edu.cn
Hangzhou Dianzi University
Image Quality Assessment via Deep Learning 2
Contents
• Background
• Image Quality Assessment (IQA)
• Photo Quality Assessment (PQA)
• Biometric Quality Assessment (BQA)
• Discussions
Image Quality Assessment via Deep Learning 3
Contents
• Background
• Image Quality Assessment (IQA)
• Photo Quality Assessment (PQA)
• Biometric Quality Assessment (BQA)
• Discussions
Image Quality Assessment via Deep Learning 4
Background
• Dramatically increasing amount of imagesn Devices: computers, mobile phones, cameras, monitors, …
n Applications: medias, websites, IM clients, …
Image Quality Assessment via Deep Learning 5
Background
• Extensively existed distortionsn Processing: acquisition, compression, transmission,
reconstruction, …
n Distortions: blurring, JPEG compression, Gaussian noise, …
Image Quality Assessment via Deep Learning 6
What is Quality?
• Various aspectsn Athletics: perceived beauty or image appeal
n Fidelity: deviation from the undistorted version
n Intelligibility: discriminability of image content
[Keelan,HandbookofImageQuality,2002][JanssenandBlommaert,JIST,1997]
[Janssen,Proc.IEEE,2001][Halonen,etal.,Proc.SPIE,2011]
Athletics FidelityUndistortedDistorted
Intelligibility
Image Quality Assessment via Deep Learning 7
Background
• Demands for IQA metrics
Evaluation, control, and improve the perceptual quality of multimedia content, quality of service (QoS), image processing systems, …
[Z. Wang, IEEE SPM’11]
Diagram of IQA-based feedback optimization
method.
ImageProcessingSystem
UpdatingAlgorithm
IQAEvaluation
Input Output
Image Quality Assessment via Deep Learning 8
Background
• Common Subjective IQA Method
1005 Excellent
4 Good
3 Fair
2 Poor
1 Bad
80
60
20
0
40
Quality Scale/Score
Categorical Continuous
Test image k Test image k+1
T1 = 10s Reference image
T2 = 3s Mid-gray image
T3 = 10s Test image
T4 = 10s Mid-gray image
[ITUR BT.500-13., 2012]
Image Quality Assessment via Deep Learning 9
Contents
• Background
• Image Quality Assessment (IQA)
• Photo Quality Assessment (PQA)
• Biometric Quality Assessment (BQA)
• Discussions
Image Quality Assessment via Deep Learning 10
Image Quality Assessment (IQA)
• Traditional Objective Methodsn Mean Squared Error (MSE)
n Peak Signal-to-Noise Ratio
n Advantages
• Mathematically conventional
• Effective for white noise
n Disadvantage
• Lack of consideration of human visual property
∑∑= =
−=m
i
n
jjiRjiI
mnMSE
1 1
2),(),(1
⎟⎟⎠
⎞⎜⎜⎝
⎛⋅=
MSEIPSNR2max
10log10
[Wang&Bovik,IEEESPM09]
Brightened(MSE=309) JPEG(MSE=309)
Gaussiannoise(MSE=309) Gaussianblur(MSE=309)
Image Quality Assessment via Deep Learning 11
Image Quality Assessment (IQA)
• How to Construct Objective IQA Metrics?
KnowledgeaboutTheHVS
PerceptualModeling
VisualPhysology andPhychophysics
Knowledgemapexpressing theelementsrequiredtoconstructsuccessful IQAmodels.
[Wang&Bovik,SPM’11]
KnowledgeaboutImageSource
Deterministic Natural Scenestatistics (NSS)
KnowledgeaboutImageDistortions
ArtifactDetection
DistortionModelling
Model-basedImageQualityAssessment
DualProblems
Image Quality Assessment via Deep Learning 12
Image Quality Assessment (IQA)
• Classification of Objective IQA
Reduced-reference(RR)IQAmethods
No-reference(NR)IQAmethods
Full-reference(FR)IQAmethods
RR-IQAmethods
NR-IQAmethods
FR-IQAmethods
Referenceimage
Distortedimage
SubjectiveQualityScores
SupplementaryInformation
Distortedimage
SubjectiveQualityScores
DistortedimageReferenceimage
SubjectiveQualityScores
Image Quality Assessment via Deep Learning 13
Image Quality Assessment (IQA)
• FR-IQA: Structural Similarity Index (SSIM)
Referenceimage!
Distortedimage"
LuminanceMeasure:Meanintensity
#$ =1'( )*
+
*,-
ContrastMeasure:Standarddeviationofintensity
.$ =1
'−1( )* −#$ 0+
*,-
-0
StructureMeasure:Normalizedimage
)−#$ /.$
2 !, " = 2#5#6+ 8-#50 +#60 + 8-
9 !, " = 2.5.6 +80.50 +.60 +80
: !, " = 2.56 +8;.5.6 +8;
<<)= !," =2 !," > ? 9 !," @ ? : !," A
[Z.Wang,etal. TIP’04]
n Advantages: efficient and effective
n Extensions: CBM [X. Gao et al., LNAI’05], IW-SSIM[W. Zhou and Q. Li, TIP’11], RR-SSIM [A. Rehman
and Z. Wang, TIP’12], etc.
Image Quality Assessment via Deep Learning 14
Image Quality Assessment (IQA)
• FR-IQA: Deep Similarity (DeepSim) [F.Gao,etal. NEUCOM’17]
Image Quality Assessment via Deep Learning 15
Image Quality Assessment (IQA)
• RR-IQAn Application: wireless communication scenario
n Framework of RR-IQA System
Distortion Channel
Feature Extraction
Ancillary Channel
Feature Extraction
RR Quality Analysis
Sender Side Receiver Side
original image
quality meature
RR features
distorted image
Image Quality Assessment via Deep Learning 16
Image Quality Assessment (IQA)
• NR-IQAn Convolutional Neural Networks for BIQA (Kang et al. CVPR’14)
50 feature mapsInput patch
32
32 77
minpooling
maxpooling
convolutional 26
26
50
50800
ReLUs ReLUs
linear regressio
noutput(score)
800
a) The input is locally normalized 32�32 image patches;
b) The convolutional layer produces 50 feature maps each of size 26�26, followed by a pooling operation that reduces each feature map to one max and one min;
c) The last layer is a simple linear regression with a one dimensional output that gives the score.
Image Quality Assessment via Deep Learning 17
Image Quality Assessment (IQA)
• Performance Evaluationn Color Image Database TID2013
• 25 reference color images (512*368 pixels);
• 120 distorted versions of each reference image;
• 24 types of distortions, 125 images of each type;
• Totally 3000 images;
• MOS of each image is available.
[N. Ponomarenko, et al. EUVIP’13]
Image Quality Assessment via Deep Learning 18
Image Quality Assessment (IQA)
• Performance Evaluationn Scatter plots
Image Quality Assessment via Deep Learning 19
Image Quality Assessment (IQA)
• Performance Evaluationn Pearson linear correlation coefficient (PLCC),
n Spearman’s rank- order correlation coefficient (SRCC), and
n Kendall rank-order correlation coefficient (KRCC)
Image Quality Assessment via Deep Learning 20
Image Quality Assessment (IQA)
• Applications: IQA Guided CNNs Training
Image Quality Assessment via Deep Learning 21
Image Quality Assessment (IQA)
• Applications: GAN
Image Quality Assessment via Deep Learning 22
Image Quality Assessment (IQA)
• Applications: Compression Artifact Removal
Galteri, Leonardo, et al. “Deep Generative Adversarial Compression Artifact Removal." arXiv preprint arXiv:1704.02518 (2017).
Image Quality Assessment via Deep Learning 23
Contents
• Background
• Visual Quality Assessment (VQA)
• Photo Quality Assessment (PQA)
• Biometric Quality Assessment (BQA)
• Discussions
Image Quality Assessment via Deep Learning 24
Photo Quality Assessment (PQA)
• What is Photo Aesthetic Quality?
Beautiful / Professional / Good Ugly / Amateurish / Bad
[Marchesotti et al. 2015]
Image Quality Assessment via Deep Learning 25
Photo Quality Assessment (PQA)
• Challengesn Aesthetic Principles
n Subjective PQA Mechanism
n Personalized
n Correlated with various image attributes (e.g. emotion, scene, semantic, style)
Image Quality Assessment via Deep Learning 26
Photo Quality Assessment (PQA)
• Research Groups
Prof. Tang Xiaoou, SeanFellow IEEE
Multimedia Laboratory
Xinmei TianAssociate Professor
Xin Lu
Image Quality Assessment via Deep Learning 27
Photo Quality Assessment (PQA)
• AVA database (250k images)n Ratings, Average ratings, Semantics, Styles
Image Quality Assessment via Deep Learning 28
Photo Quality Assessment (PQA)
• Crowd Opinion [Marchesotti et al. 2015]
Image Quality Assessment via Deep Learning 29
Photo Quality Assessment (PQA)
• Crowd Opinion [Marchesotti et al. 2015]
Image Quality Assessment via Deep Learning 30
Photo Quality Assessment (PQA)
• Crowd Opinion [Marchesotti et al. 2015]
Image Quality Assessment via Deep Learning 31
Photo Quality Assessment (PQA)
• Style
low depth-of-field, single salientobject and rule-of-thirds
[Marchesotti et al. 2015]
black and white
Image Quality Assessment via Deep Learning 32
PQA via Deep Learning
• Methodology
Y. Deng, C.-L. Chen, and X. Tang, “Image aesthetic assessment: An experimental survey,” 2016.
PQA via CNNs
Single-column CNNs
Multi-column CNNs
Multi-task CNNs
Image Quality Assessment via Deep Learning 33
PQA via Deep Learning
• Single-column CNNsn CNNs trained on ImageNet
n Transferred CNNs
• Pretrain in the PQA task
• Fine-tuning in the binary classification task of photo aesthetic (good or bad)
Y. Deng, C.-L. Chen, and X. Tang, “Image aesthetic assessment: An experimental survey,” 2016.
Image Quality Assessment via Deep Learning 34
PQA via Deep Learning
• Multi-column CNNsn global features + local features
n global features + attribute features
Y. Deng, C.-L. Chen, and X. Tang, “Image aesthetic assessment: An experimental survey,” 2016.
Image Quality Assessment via Deep Learning 35
PQA via Deep Learning
• Multi-task CNNsn Correlated with various image attributes (e.g.
emotion, scene, semantic, style)
• Task 1: Aesthetic prediction
• Task 2: Semantic prediction
• Task n: …
Y. Deng, C.-L. Chen, and X. Tang, “Image aesthetic assessment: An experimental survey,” 2016.
Image Quality Assessment via Deep Learning 36
PQA via Deep Learning
• Multi-task Multi-column CNNs Y. Deng, C.-L. Chen, and X. Tang, “Image aesthetic assessment: An experimental survey,” 2016.
Image Quality Assessment via Deep Learning 37
PQA via Deep Learning
• A-lampn Adaptive Layout-Aware Multi-Patch
Image Quality Assessment via Deep Learning 38
PQA via Deep Learning
• A-lamp
Pipeline of attribute-graphs construction. (a) Salient objects (labeled by yellow bounding boxes) are first detected by a trained CNN, and regarded as local nodes. The dashed green bounding box denote the overall scene, which served as a global node. (b) Local and global attributes are extracted from these nodes to capture the object topology and the image layout. (c) Attribute-graphs are constructed and (d) concatenated into an aggregation layer.
Image Quality Assessment via Deep Learning 39
PQA via Deep Learning
• Performance
Y. Deng, C.-L. Chen, and X. Tang, “Image aesthetic assessment: An experimental survey,” 2016.
Image Quality Assessment via Deep Learning 40
PQA via Deep Learning
• Applications
n Enhancement
n Retargeting
n Retrieval
n Mobile APPs
Image Quality Assessment via Deep Learning 41
Image Enhancement
• EnhanceGAN
• Deng, Yubin, Chen Change Loy, and Xiaoou Tang. "Aesthetic-Driven Image Enhancement by Adversarial Learning." arXiv preprint arXiv:1707.05251(2017).
Image Quality Assessment via Deep Learning 42
Image Enhancement
• Creatism
• Fang, Hui, and Meng Zhang. "Creatism: A deep-learning photographer capable of creating professional work." arXiv preprint arXiv:1707.03491(2017).
A panorama (a) is cropped into (b), with saturation and HDR strength enhanced in (c), and with dramatic mask applied in (d).
Image Quality Assessment via Deep Learning 43
Contents
• Background
• Image Quality Assessment (IQA)
• Photo Quality Assessment (PQA)
• Biometric Quality Assessment (BQA)
• Discussions
Image Quality Assessment via Deep Learning 44
Biometric Quality Assessment (BQA)
• Definitionn Quality of a biometric
sample is a measure of its efficiency in aiding recognition of an individual, ideally, irrespective of the recognition system in use.
Sample images of varying quality.
(a) Fingerprint, (b) iris (from WVU multimodal database), and (c) face (from SCface and CAS-PEAL face databases)
Image Quality Assessment via Deep Learning 45
Biometric Quality Assessment (BQA)
• Pipeline of a typical biometric system.
Image Quality Assessment via Deep Learning 46
Biometric Quality Assessment (BQA)
• Face
Image Quality Assessment via Deep Learning 47
Biometric Quality Assessment (BQA)
• Face: ID
Image Quality Assessment via Deep Learning 48
Biometric Quality Assessment (BQA)
• Four primary image features for BQA
Image Quality Assessment via Deep Learning 49
Biometric Quality Assessment (BQA)
• BQA via LightCNN
Image Quality Assessment via Deep Learning 50
Biometric Quality Assessment (BQA)
• Face recognition/identification/verification
Outline of the DeepFace architecture.
Image Quality Assessment via Deep Learning 51
Biometric Quality Assessment (BQA)
Image Quality Assessment via Deep Learning 52
Biometric Quality Assessment (BQA)
• Applications: Utilizing BQA for context switching
Image Quality Assessment via Deep Learning 53
Contents
• Background
• Image Quality Assessment (IQA)
• Photo Quality Assessment (PQA)
• Biometric Quality Assessment (BQA)
• Discussions