Post on 13-Jan-2016
description
1
ACM MultimediaOctober 22nd, 2009, Beijing, China
Quality of Multimedia Experience
Past, Present and Future
Prof. Dr. Touradj Ebrahimi
Touradj.Ebrahimi@epfl.ch
2
ACM MultimediaOctober 22nd, 2009, Beijing, China
What is “quality” of multimedia content? How is multimedia content “quality” measured
today? What are trends in assessment of “quality” in
multimedia? What are the challenges ahead?
Today we will talk about…
3
ACM MultimediaOctober 22nd, 2009, Beijing, China
Quality – a simple yet difficult concept
Like many human sensations quality is easy to understand but difficult to define
According to Wikipedia:– A quality (from Latin - qualitas) is an attribute or a
property. – Some philosophers assert that a quality cannot be
defined. – In contemporary philosophy, the idea of qualities and
especially how to distinguish certain kinds of qualities from one another remains controversial.
4
ACM MultimediaOctober 22nd, 2009, Beijing, China
A fundamental yet largely under-investigated concept
Aristotle classified every object of human apprehension into 10 Categories– Substance, Quantity,
Quality, Relation, Place, Time, Position, State, Action, Affection
Aristotle 384 BC – 322 BC
5
ACM MultimediaOctober 22nd, 2009, Beijing, China
Some definitions according to dictionary
Definition 1– General : Measure of excellence or state of
being free from defects, deficiencies, and significant variations.
– ISO 8402-1986 standard defines quality as "the totality of features and characteristics of a product or service that bears its ability to satisfy stated or implied needs"
6
ACM MultimediaOctober 22nd, 2009, Beijing, China
Some definitions according to dictionary
Definition 2– Manufacturing : Strict and consistent
adherence to measurable and verifiable standards to achieve uniformity of output that satisfies specific customer or user requirements.
7
ACM MultimediaOctober 22nd, 2009, Beijing, China
Some definitions according to dictionary
Definition 3– Objective : Measurable and verifiable aspect
of a thing or phenomenon, expressed in numbers or quantities, such as lightness or heaviness, thickness or thinness, softness or hardness.
8
ACM MultimediaOctober 22nd, 2009, Beijing, China
Some definitions according to dictionary
Definition 4– Subjective : Attribute, characteristic, or
property of a thing or phenomenon that can be observed and interpreted, and may be approximated (quantified) but cannot be measured, such as beauty, feel, flavor, taste.
9
ACM MultimediaOctober 22nd, 2009, Beijing, China
Definition according to ISO 9000
ISO 9000: a family of standards for quality management systems
Quality of something can be determined by comparing a set of inherent characteristics with a set of requirements– High quality: if characteristics meet requirements– Low quality: if characteristics do not meet all
requirements
Quality is a relative concept– Degree of quality
10
ACM MultimediaOctober 22nd, 2009, Beijing, China
Quality – is in fact an elephant
The blind men and the elephant: Poem by John Godfrey Saxe
11
ACM MultimediaOctober 22nd, 2009, Beijing, China
Quality of Service in computer networks and communications
Quality of Service (QoS) refers to a collection of networking technologies and measurement tools that allow for the network to guarantee delivering predictable results
12
ACM MultimediaOctober 22nd, 2009, Beijing, China
Quality in QoS framework
Network QualityCapacity
CoverageHandoff
Link QualityBitrate
Frame/Bit/Packet lossDelay
User QualitySpeech fidelity
Audio fidelityImage fidelityVideo fidelity
13
ACM MultimediaOctober 22nd, 2009, Beijing, China
Quality of Service in computer networks and communications
Quality of Service (QoS)– Resource reservation control mechanisms– Ability to provide different priority to different
applications, users, or data flows– Guarantee a certain level of performance
(quality) to a data flow
(Service) Provide-centric concept Tightly related to the concept of Mean
Opinion Score (MOS)
14
ACM MultimediaOctober 22nd, 2009, Beijing, China
What is Mean Opinion Score (MOS)?
Widely used in many fields:– Politics/Elections– Marketing/Advertisement– Food industry– Multimedia– …
The likely level of satisfaction of a service or product as appreciated by an average user
Should be performed such that it generates reliable and reproducible results– Subjective evaluation methodology– More complex and difficult that it a priori seems
15
ACM MultimediaOctober 22nd, 2009, Beijing, China
What is behind a MOS?
16
ACM MultimediaOctober 22nd, 2009, Beijing, China
A subjective tests aiming at producing MOS is a delicate mixture of ingredients and choices:
Subjective evaluation
• Test/lab environment• Test material• Test methodology• Analysis of the data
17
ACM MultimediaOctober 22nd, 2009, Beijing, China
Test/lab environment
Type of Monitors/Speakers and other test equipments Lighting /Acoustic conditions Laboratory architecture, background, … Viewing distance /Hearing position …
18
ACM MultimediaOctober 22nd, 2009, Beijing, China
Test material
Meaningful content for the envisaged scenario/application– Typical content– Worst case content– …
p01 p06 p10 bike cafe woman
19
ACM MultimediaOctober 22nd, 2009, Beijing, China
Test methodology
Subjects– Naïve or Expert?
Instructions– Which questions to ask subjects and how– Training
Presentation– Single or double stimulus– Sequential or simultaneous
Grading scale– Numerical– Categorical
…
20
ACM MultimediaOctober 22nd, 2009, Beijing, China
• Test conditions and methodologies are specified in:
Recommendation ITU-R BT. 500-11 “Methodology for the subjective assessment of the quality of television pictures” (1974-2002).
Recommendation ITU-T P. 910 “Subjective video quality assessment methods for multimedia applications” (1999).
Recommendation ITU-R BT. 1788 “Methodology for the subjective assessment of video quality in multimedia applications” (2007).
Based on television scenario!
ITU Recommendations for test methodologies
21
ACM MultimediaOctober 22nd, 2009, Beijing, China
Single Stimulus (SS)
Test methodology (I)
Non-categorical adjectival or numerical grading scale
5 Excellent 4 Good
3 Fair
2 Poor
1 Bad
5 Imperceptible
4 Perceptible but not annoying
3 Slightly annoying
2 Annoying
1 Very annoying
100
0
Excellent
Bad
Categorical adjectival grading scale: Categorical numerical grading scale:
“Rate from 1 to 11”
22
ACM MultimediaOctober 22nd, 2009, Beijing, China
Double Stimulus Impairment Scale (DSIS)
Test methodology (II)
Categorical impairment grading scale:
5 Imperceptible
4 Perceptible but not annoying
3 Slightly annoying
2 Annoying
1 Very annoying
23
ACM MultimediaOctober 22nd, 2009, Beijing, China
Double Stimulus Continuous Quality Scale (DSCQS)
Test methodology (III)
Sample 1Sample 2
Non-categorical adjectival or numerical grading scale:
100
0
Excellent
Bad
Sample 1 Sample 2
100
0
Excellent
Bad
24
ACM MultimediaOctober 22nd, 2009, Beijing, China
Stimulus Comparison (SC)
Test methodology (IV)
Categorical adjectival comparison scale:
“same or different” much worse
worse
slightly worse
the same
slightly better
better
much better
Non-categorical judgement:
Much worse
Much better
25
ACM MultimediaOctober 22nd, 2009, Beijing, China
Single Stimulus Continuous Quality Evaluation (SSCQE)
Test methodology (V)
(Very annoying)
(Imperceptible)
26
ACM MultimediaOctober 22nd, 2009, Beijing, China
Simultaneous Double Stimulus for Continuous Evaluation (SDSCE)
Test methodology (VI)
(Much better)
(Much worse)(Reference) (Test sequence)
27
ACM MultimediaOctober 22nd, 2009, Beijing, China
Analysis of the data
• Scores distributions across subjects is assumed to be close to normal distribution
• Outlier detection and removal• Mean Opinion Scores (MOS) and 95% confidence intervals
N
mMOS
N
i ijj
1
NNtCI j
j
),2/1(
mij = score by subject i for test condition j.
N = number of subjects after outliers removal.
t(1-α/2,N) = t-value corresponding to a two-tailed t-Student distribution with N-1 degrees of freedom and a desired significance level α (α=0.05 in our case).
σj = standard deviation of the scores distribution across subjects for test condition j.
28
ACM MultimediaOctober 22nd, 2009, Beijing, China
What is behind a MOS?
29
ACM MultimediaOctober 22nd, 2009, Beijing, China
Relationship between estimated mean values
• Hypothesis test to find out whether the difference between two MOS values are statistically significant
Two-sided t-test:
• t statistic:
• Decision rule to reject H0:
BA MOSMOSH :0
BAa MOSMOSH :
NN
MOSMOSt
BA
BAobs 22
),2/1(),2/( NttORNtt obsobs
30
ACM MultimediaOctober 22nd, 2009, Beijing, China
MOS hypothesis testJPEG 2000 4:2:0
JPEG 2000 4:4:4
JPEG
JPEG XR MS
JPEG XR PS
JPEG 2000 4:2:0
JPEG 2000 4:4:4
JPEG
JPEG XR MS
JPEG XR PS
0.25 bpp
0.50 bpp
0.75 bpp
1.00 bpp
1.25 bpp
1.50 bpp
6
5
4
3
2
1
0Number of times
H0 is rejected
31
ACM MultimediaOctober 22nd, 2009, Beijing, China
• Subjective tests are time consuming, expensive, and difficult to design
• Objective algorithms, i.e. metrics, estimating subjective MOS with high level of correlation are desired• Full reference metrics• No reference metrics• Reduced reference metrics
Objective quality metrics
32
ACM MultimediaOctober 22nd, 2009, Beijing, China
FR, RR and NR scenarios Full Reference approach:
Reduced Reference approach:
No Reference approach:
Input/Reference signal
Output/Processed signal
signalprocessing
FR METRIC
Input/Reference signal
Output/Processed signal
signalprocessing
NR METRIC
Input/Reference signal
Output/Processed signal
signalprocessing
Features extraction
RR METRIC
33
ACM MultimediaOctober 22nd, 2009, Beijing, China
Full Reference scenario Metrics which look at the fidelity of the signal when compared to an
explicit reference:
processed signal=
perfect quality reference signal+
error signal
MOS predictors based on fidelity measures
34
ACM MultimediaOctober 22nd, 2009, Beijing, China
Examples Mean Square Error (MSE) Peak Signal to Noise Ratio (PSNR) Maximum Pixel Deviation (Linf) Weighted PSNR Masked PSNR Structural SIMilarity (SSIM) Multiscale Structural Similarity (MSSIM) Visual Information Fidelity (VIF) etc…
MOS predictors based on fidelity metrics
35
ACM MultimediaOctober 22nd, 2009, Beijing, China
Peak Signal to Noise Ratio
Widely used because of its simplicity and ease in formalizing optimization problems!
For image and video data (Y component), a correlation of circa 80% reported when compared to subjective MOS evaluation
MSE)12(
log10PSNR2B
10
M
1y
N
1x
2ba y)](x,Imy)(x,[Im
MN
1MSE
where:
M, N = image dimensions Ima , Imb = pictures to compare
B= bit depth
36
ACM MultimediaOctober 22nd, 2009, Beijing, China
PSNR for color images/video (I)
Several alternatives to compute PSNR for color images/video:
WPSNR = w1PSNR1 + w2PSNR2 + w3PSNR3
)MSEwMSEwMSE(w
1)(210log
332211
2B
10
WPSNR_MSE
WPSNR_PIX
M
y
N
xbbbaaa
B
)y,x(Imw)y,x(Imw)y,x(Imw)y,x(Imw)y,x(Imw)y,x(ImwMN
)(log
1 1
2332211332211
2
10 112
10
37
ACM MultimediaOctober 22nd, 2009, Beijing, China
PSNR for color images/video (II)
Which color space to use? RGB Y’CbCr other?
Which weights to use? w1=w2=w3=1/3 w1=0.8, w2=w3=0.1 other?
38
ACM MultimediaOctober 22nd, 2009, Beijing, China
PSNR for color images/video (III)
on R component:
bpp (bits/pixel) bpp (bits/pixel) bpp (bits/pixel)
on G component: on B component:
39
ACM MultimediaOctober 22nd, 2009, Beijing, China
on Y’ component: on Cb component: on Cr component:
bpp (bits/pixel) bpp (bits/pixel) bpp (bits/pixel)
PSNR for color images/video (IV)
40
ACM MultimediaOctober 22nd, 2009, Beijing, China
Human Visual System (HVS) based metrics simulating properties of the early stages of the HVS Examples: PSNR-HVS-M, etc… simulating high level features of the HVS Examples: Osberger’s metric, etc…
Better correlation with human perception. High complexity.
MOS predictors based on fidelity metrics
Computationalmodel of thevisual system
Visibility Map
41
ACM MultimediaOctober 22nd, 2009, Beijing, China
PSNR-HVS-M
H
2B
10 MSE)12(
log10MHVSPSNR
7-M
1i
7N-
1j
8
1m
28
1ncijH )n,m(T)n,m(XKMSE
where: M, N = image dimensions K = constant
= visible difference between DCT coefficients of the original and distorted based on a contrast masking
Tc = matrix of correcting factors based on standard visually optimized
JPEG quantization tables B= bit depth
ij)n,m(X
42
ACM MultimediaOctober 22nd, 2009, Beijing, China
Metrics based on the hypothesis that the HVS is highly adapted for extraction of structural information from the content of a still image or video. degradation of still images or video = perceived structural information
variation Structural Similarity by Wang et al.
MOS predictors based on fidelity measures
43
ACM MultimediaOctober 22nd, 2009, Beijing, China
Mean SSIM (MSSIM)
12
22
1
12121 C
C2)Im,(Iml
22
22
1
22121 C
C2)Im,(Imc
€
SSIM(Im1,Im2) = l(Im1,Im2)[ ]αc(Im1,Im2)[ ]
βs(Im1,Im2)[ ]
γ)0,0,0(
• Luminance comparison function: (C1=constant)
• Contrast comparison function: (C2=constant)
• Structure comparison function:
321
32,121 C
C)Im,(Ims
(C3=constant)
44
ACM MultimediaOctober 22nd, 2009, Beijing, China
MSSIM vs PSNR
PSNR = 24.9 dBfor all the images
Mean SSIM (MSSIM)
MSSIM=0.9168 MSSIM=0.6949MSSIM=0.7052
45
ACM MultimediaOctober 22nd, 2009, Beijing, China
At times one is interested in specific types of distortions that occur in multimedia systems
Examples Bluriness Blockiness Ringing/Mosquito noise Jerkiness etc…
Specific distortion metrics
46
ACM MultimediaOctober 22nd, 2009, Beijing, China
Blur metric
A perceptual quality blur metric without a reference image. Example:
Gaussian blurred image JPEG2000 compressed image
47
ACM MultimediaOctober 22nd, 2009, Beijing, China
NR blur metric
48
ACM MultimediaOctober 22nd, 2009, Beijing, China
NR blur metric
140 145 150 155 160 165 170 175 1800
50
100
150
200
250
P2 P1 P2' P4 P3 P4'
Pixel position
Pixel value
49
ACM MultimediaOctober 22nd, 2009, Beijing, China
Correlation subjective ratings / NR blur metrics
4 5 6 7 8 9 10 11 12 13 140
1
2
3
4
5
6
7
8
9
10
Objective blur measure
Su
bje
cti
ve
blu
r ra
tin
g
Gaussian blurred images
4 5 6 7 8 9 100
1
2
3
4
5
6
7
8
9
10
Objective blur measure
Su
bje
cti
ve
blu
r ra
tin
g
JPEG 2000 images
96% correlation 85% correlation
50
ACM MultimediaOctober 22nd, 2009, Beijing, China
Multimedia communication – a definition
Multimedia is about sharing experience (real or imaginary) with others
In a way it all started with story telling and wall drawing around the fire in the caves of early men
Modern multimedia systems are evolved versions of the good old story telling and wall drawing, which hopefully offer increasingly richer experience
The degree of richness of the experience is measured by Quality of Experience (QoE)
51
ACM MultimediaOctober 22nd, 2009, Beijing, China
Evolutions versus Revolutions in multimedia
Evolution: A given modality improves itself in terms of Quality of Experience:– B&W TV– Color TV– Stereo and CD quality audio TV– HDTV
Revolution: New modalities and expression are introduced bringing new dimensions in Quality of Experience– Photography– Cinema– Internet
52
ACM MultimediaOctober 22nd, 2009, Beijing, China
Some of the major milestones in multimedia
Story telling and cave drawing Books and written press Photography Telegraph Telephone Radio and music recording Cinema Television and video recording Internet (including VOIP, IPTV, etc.) Mobile communication Social networking (Web 2.0) (What is next?)
53
ACM MultimediaOctober 22nd, 2009, Beijing, China
Quality of Service vs Quality of Experience
Quality of Service: Value of the average user’s experience richness estimated by a service/product/content provider
Quality of Experience: Value (estimated or actually measured) of a specific user’s experience richness
Quality of Experience is the dual (and extended) view of QoS problem
– QoS=provider-centric– QoE=user-centric
54
ACM MultimediaOctober 22nd, 2009, Beijing, China
Factors impacting Quality of Experience
Context Context
55
ACM MultimediaOctober 22nd, 2009, Beijing, China
Quality of Experience in networked multimedia
56
ACM MultimediaOctober 22nd, 2009, Beijing, China
Quality Wheel
57
ACM MultimediaOctober 22nd, 2009, Beijing, China
Trends in QoE
Digital world has (re-)discovered the notion of quality– Lower quality content is less and less tolerated by
end-users– Digital technology can now rival and even surpass
the old analog systems performance while remaining cost effective
Increasing interest in QoE– Extending from device-centric and system-centric
quality optimization to end-to-end and especially user-centric optimization
58
ACM MultimediaOctober 22nd, 2009, Beijing, China
Trends in QoE community building
Increased interest in workshops and conferences around the notion of quality assessment and metrics– QoMEX: International Workshop on Quality of Multimedia
Experience (http://www.qomex.org)– VPQM: International Workshop on Video Processing and
Quality Metrics for Consumer Electronics (http://www.vpqm.org)– IMQA: International Workshop on Image Media Quality and its
Applications (http://www.mi.tj.chiba-u.jp/imqa2008/)
QoE is one of the issues referred to in future funding programs by the EC– Workshop on “The Future of Networked Immersive Media”
(http://cordis.europa.eu/fp7/ict/netmedia/workshop/ws20090610_en.html)
59
ACM MultimediaOctober 22nd, 2009, Beijing, China
Trends in standardization
Standardization efforts in quality assessment and metrics– Video Quality Experts Group (VQEG)– ITU-T SG 12 (Performance, QoS and QoE)– ITU-R WP6C (Prog. production and quality assessment)– JPEG (Advanced Image Coding - AIC)– MPEG (High performance Video Coding – HVC)– …
60
ACM MultimediaOctober 22nd, 2009, Beijing, China
Challenges ahead
Some key issues in QoE need to be better addressed– Content-dependent quality assessment methods and metrics – Context-dependent quality assessment methods and metrics– Quality assessment methods and metrics beyond AV (haptics, …)– Multi-modal quality assessment methods and metrics (AV, …)– 3D quality assessment methods and metrics (3D sound, 3D video, …)– HDR content quality assessment methods and metrics – Interaction quality metrics (closely related to usability)– Presence/immersion quality metrics– …
Need for Quality Certification Mechanisms of multimedia services and products– Similar in idea to ISO 9000 series
61
ACM MultimediaOctober 22nd, 2009, Beijing, China
What does this all mean to you?
Era of user-centric multimedia has already started– It is not anymore sufficient to merely add new
features and functionalities to multimedia systems
– True added value in terms of impact on user’s experience of such features and functions should be demonstrated
– Quality of Experience plays a central role in this new game
62
ACM MultimediaOctober 22nd, 2009, Beijing, China
Thank you for your attentionQuestions?
63
ACM MultimediaOctober 22nd, 2009, Beijing, China
Acknowledgements
Some of the concepts and illustrations presented in this talk are the results of collaborations with individuals with whom I have had the pleasure of working. In particular:
– Francesca de Simone (EPFL/MMSPG)– Frederic Dufaux (EPFL/MMSPG)– Lutz Goldmann (EPFL/MMSPG)– Jong-Seok Lee (EPFL/MMSPG)– Andrew Perkis (NTNU/Q2S)– Ulrich Hoffmann (NTNU/Q2S)– Fitri Rahayu (NTNU/Q2S)
Part of the work presented here are the fruits of the research projects:
– EC funded NoE Petamedia– Swiss NSF funded NCCR IM2