State of the Art of Multimedia Quality Assessment Methods · State of the Art of Multimedia Quality...
22
International Telecommunication Union ITU-T VICA Workshop 22-23 July 2005, ITU Headquarters, Geneva State of the Art of State of the Art of Multimedia Quality Multimedia Quality Assessment Methods Assessment Methods Takanori Hayashi NTT Service Integration Labs.
Transcript of State of the Art of Multimedia Quality Assessment Methods · State of the Art of Multimedia Quality...
International Telecommunication Union
ITU-T VICA Workshop22-23 July 2005, ITU Headquarters, Geneva
State of the Art ofState of the Art ofMultimedia Quality Multimedia Quality
Assessment MethodsAssessment Methods
Takanori HayashiNTT Service Integration Labs.
2ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Outline
o Present state of multimedia quality assessment methods
o Future trends of quality assessment research
• Multimodality• Multiparty• Wideband
o Examples of NTT’s studies on multimedia quality assessment for audiovisual communication services
3ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Relevant ITU-T/R Recommendationson Multimedia Quality Assessment
ObjectiveSubjective
J.148, P.931P.911, P.920Multimedia
J.144P.910, BT.500Video
P.563, P.862, BS.1387P.800, BS.562Audio
Quality assessment methodMedia
o Multimedia quality assessment is at a reasonably advanced stage.
4ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Existing Subjective Multimedia Quality Assessment Methods (P.911)
o Multimedia quality is assessed in a similar way to individual audio/video qualities.
• Absolute Category Rating (ACR)• Degradation Category Rating (DCR)• Pair Comparison Method (PC)• Single Stimulus Continuous Quality
Evaluation (SSCQE)o Assessment paying attention to the
cross-modal influences is important.• Interactions between differing quality
levels in different modalities
5ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-To Several category judgment scales are
used to evaluate multimedia quality.• Overall audiovisual quality• Individual audio/video qualities• Effort needed to interrupt• Communication difficulty• Acceptability of communication
o Communication quality depends on tasks used in conversational test.
o Assessment considering interactivity and usability is also important.
Existing Subjective Multimedia Quality Assessment Methods (P.920)
6ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Existing Concept Model of Objective Multimedia Quality Evaluation (J.148)
Auditory source
Visualsource
Differential delay
Visual quality
Auditory quality
Multimediaquality
integrationfunction
Task
Multimedia quality
Aq
Aq(Vq)
Vq(Aq)
Vq
o Basic components of an objective multimedia quality model are defined.
7ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Examples of Multimedia Quality Evaluation Model
o ITU-T P.911 Annex• MOSMM=C1MOSAMOSV+C2
o Other multimedia quality evaluation models have been proposed. [1-7]
• MOSMM=C1MOSA+C2MOSV+C3MOSAMOSV+C4! Constants vary with tasks (audiovisual
contents, service environments, etc.).
MOSMM: MOS for multimediaMOSA: MOS for audioMOSV: MOS for videoC: Constant
8ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Framework for ConductingQuality Assessment Research [8]
Sing
le m
ediu
mSi
ngle
med
ium
Media
WidebandWideband
Bandwidth
N:N(Multiparty)(Multiparty)
Communication style
InterdestinationInterdestination
synchronizationsynchronization
InterdestinationInterdestination
quality balancequality balance
Quality Quality assessment for assessment for
••multimodalitymultimodality
••multiparty multiparty
••wideband wideband
IntermediaIntermediasynchronizationsynchronization
IntermediaIntermediaquality balancequality balance
(VoIP)
1:11:1
Conventional research field
Future research field
Mul
tim
odal
Mul
tim
odal
9ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Key Words for Future Trends of Multimedia Quality Assessment (1/3)
o Multimodality
• Combination of multiple media such as audio, video, text, graphics, fax, and telephony in the communication of information
• Key points of quality assessment! Intermedia synchronization! Intermedia quality balance
10ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Key Words for Future Trends of Multimedia Quality Assessment (2/3)
o Multiparty
• Communication style extending from 1:1 to N:N (e.g., instant messaging, teleconferencing, and distributed collaboration services)
• Key points of quality assessment! Interdestination synchronization! Interdestination quality balance
11ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Key Words for Future Trends of Multimedia Quality Assessment (3/3)
o Wideband
• Telecommunications applications having more bandwidth available for higher-quality multimodal services
• Key points of quality assessment! Psychological factors: We need to assess
the richness of high-quality services not only on a one-dimensional scale, like MOS, but also on a multi-dimensional scale.
12ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Examples of NTT’s Studies on Quality Assessment for “Multimodality” (1/3)
o Interactive multimodal quality model considering conversational delay [9]
• MOSOA=(C1MOSMM+C2)(C3MOSR+C4)• MOSR=C5exp(-D/C6)+C7
! Constants C5, C6, and C7 depend on conversational task.
1
2
3
4
5
1 2 3 4 5Subjective MOSOA
Esti
mat
ed M
OS O
A
R2: 0.94MOSOA: MOS for overall qualityMOSMM: MOS for multimediaMOSR: MOS for responseD: DelayC: Constant
13ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Examples of NTT’s Studies on Quality Assessment for “Multimodality” (2/3)
o Influence of differential delay is different when DA>DV or DA<DV.[10]
Acceptable quality region for response
One-way delay for audio: DA [ms]
One
-way
del
ay f
or v
ideo
: D
V[m
s]
800
600
400
200
2000 400 600 800
Quality can be improved by synchronizing
DA: 100 msDV: 300 ms
DA: 300 msDV: 300 ms
Subjects: 4 expertsTask: Free conversation
14ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Examples of NTT’s Studies on Quality Assessment for “Multimodality” (3/3)
o Opinion model for audiovisual communication services is now being discussed in ITU-T SG12. [11]
Multimediaquality
Opi
nion
mod
el f
or
audi
ovis
ual c
om.
serv
icesVideo codec
Video size/resolutionVideo coding bit rateVideo packet loss rateVideo delayVideo jitter
Audio codecAudio echoAudio packet loss rateAudio delayAudio jitter
Video qualityparameters
Audio qualityparameters
15ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Examples of NTT’s Studies on Quality Assessment for “Multiparty” (1/2)
o Quality imbalance is one of the multiparty quality degradation factors.
Point 1
Point 2 Point 3
Three-point audiovisual communication
Other-point audiovisual quality:
MOS1-2
Other-point audiovisual quality:
MOS1-3
Overall audiovisual quality: MOS1
Good No GoodQuality
16ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Examples of NTT’s Studies on Quality Assessment for “Multiparty” (2/2)
o Overall quality is strongly affected by the inferior quality at another point. [12]
o Other-point audiovisual quality depends on conversation task or roles.
MOS1-2
MO
S 1-3
Overall audiovisual quality: MOS1
Higher
qua
lity
Higher quality
Hig
her
qual
ity
17ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Examples of NTT’s Studies on Quality Assessment for “Wideband” (1/3)
o Interactive multimodal quality can be evaluated using a multi-dimensional scale of psychological factors. [11]
Multimediaquality
Video qualityparameters
Audio qualityparameters
Perceptual quality indices
Parameter A
Parameter B
Parameter C
Psychological factor indices
Factor X
Factor Y
Factor Z
Video codecVideo size/resolutionVideo coding bit rateVideo packet loss rateVideo delayVideo jitter, …
Audio codecAudio echoAudio packet loss rateAudio delayAudio jitter, …
18ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Examples of NTT’s Studies on Quality Assessment for “Wideband” (2/3)
o Psychological factors were extracted by using the semantic differential (SD) technique and factor analysis. [13]
• Subject’s impression of an audiovisual communication service was evaluated on the basis of 25 pairs of bipolar adjectives on a seven-grade comparison scale.
Slow
NeutralSlightlyRatherExtremely ExtremelyRatherSlightly
Fast
Example of scale
19ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Examples of NTT’s Studies on Quality Assessment for “Wideband” (3/3)
o Multimedia quality was formulated as a function of two psychological factors expressing an aesthetic feeling and a feeling of activity. [13, 14]
Mul
tim
edia
qua
lity
g(f 1
,f2)
Video qualityparameters
Audio qualityparameters
Perceptual quality indices
Psychological factor indices
Blurring
Jerkiness
Distortion
Aesthetic feeling
f1(BR,PLRV,PLRA)Video coding bit rate: BRVideo frame rate: FRVideo packet loss rate: PLRVVideo delay: D
Audio packet loss rate: PLRAAudio delay: D
Delay
Lack of sound
Feeling of activity
f2(BR,PLRV, FR,D)
20ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
Conclusions
o Multimedia quality assessment is at an advanced stage.
o Perceptual quality assessment methodologies for multimedia communications systems of the next generation are being discussed.
o Three important characteristics of upcoming services have been revealed by recent studies on multimedia quality evaluation models:
• Multimodality, multiparty, and wideband.
21ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
References (1/2)[1] David S. Hands, “A Basic Multimedia Quality Model,” IEEE
Transactions on Multimedia, vol. 6, no. 6, pp. 806 - 816, Dec. 2004.
[2] C. Jones and D. J. Atkinson, “Development of opinion-based audiovisual quality models for desktop video-teleconferencing,”IEEE Sixth Int. Workshop on Quality of Service (IWQoS’98), pp. 196 - 203, May 1998.
[3] A. Joly, N. Montard, and M. Buttin, “Audio-visual quality and interactions between television audio and video,” Sixth International Symposium on Signal Processing and its Applications, Vol. 2, pp. 438 - 441, Aug. 2001.
[4] A. Watson, and M. A. Sasse, “Measuring perceived quality of speech and video in multimedia conferencing applications,”Proceedings of the sixth ACM international conference on Multimedia, Sept. 1998.
[5] ITU-T Contribution COM12-19-E, “Relations between audio, video, and audiovisual quality,” Dec. 1997.
[6] ITU-T Contribution COM12-61-E “Study of the influence of experimental context on the relationship between audio, video, and audiovisual subjective qualities,” Sept. 1998.
22ITU-T VICA Workshop
22-23 July 2005, ITU Headquarters, Geneva
ITU-T
References (2/2)[7] ITU-T Contribution COM12-64-E “Results of an audiovisual
desktop video teleconferencing subjective experiment,” Sept. 1998.
[8] A. Takahashi, H. Yoshino, and N. Kitawaki, “Perceptual QoSAssessment Technologies for VoIP,” IEEE Com. Magazine, pp. 28 - 34, July 2004.
[9] ITU-T Contribution COM12-D14-E, “Example of multimedia quality integration function for videophones,” Jan. 2005.
[10] S. Iai, T. Kurita, and N. Kitawaki, “Quality requirements for multimedia communication services and terminals – interaction of speech and video delays,” Globecom'93, pp. 394 – 398, 1993.
[11] ITU-T Contribution COM12-D13-E, “Proposal on basic concepts of a multimedia quality assessment model,” Jan. 2005.
[12] T. Kurita, “Effects of Conversation Roles on Quality of Multiparty Audiovisual Communication Services,” IEICE Tec. Rep. CQ, July 2005 (in Japanese).
[13] ITU-T Contribution COM12-D15-E, “Example of multimedia quality assessment model for videophones,” Jan. 2005.
[14] K. Yamagishi, and T. Hayashi, “Analysis of psychological factors for quality assessment of interactive multimodal service,”Electronic Imaging 2005, 5666-15, pp. 130 - 138, Jan. 2005.
