Objective and Subjective Evaluation for Telecommunications Services and Video Quality
description
Transcript of Objective and Subjective Evaluation for Telecommunications Services and Video Quality
![Page 1: Objective and Subjective Evaluation for Telecommunications Services and Video Quality](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dfd550346895dcc3827/html5/thumbnails/1.jpg)
Objective and Subjective Evaluation for
Telecommunications Services and Video Quality
Arthur Webster, NTIA/ITSRapporteur Q21/9, Co-Chair VQEG
IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 2004
![Page 2: Objective and Subjective Evaluation for Telecommunications Services and Video Quality](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dfd550346895dcc3827/html5/thumbnails/2.jpg)
Introduction
• Evaluation of Video Services Needed– Service Level Agreements, Contracts
• Subjective Most Accurate—but Expensive• Objective Methods Needed• Standardized Methods Critical
![Page 3: Objective and Subjective Evaluation for Telecommunications Services and Video Quality](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dfd550346895dcc3827/html5/thumbnails/3.jpg)
Standards Groups Involved in Video Quality Evaluation
• ITU-T SG9 and SG12– Q21/9 and Q4/9– G.1000, G.1010 in SG12
• ITU-R WP 6Q• T1A1 (Committee T1)• IEEE Broadcasting Technology
![Page 4: Objective and Subjective Evaluation for Telecommunications Services and Video Quality](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dfd550346895dcc3827/html5/thumbnails/4.jpg)
Video Quality Recommendations• BT.500 Methodology for the subjective assessment of the quality of
television pictures• P.910 Subjective video quality assessment methods for multimedia
applications• P.911 Subjective audiovisual quality assessment methods for multimedia
applications• P.920 Interactive test methods for audiovisual communications• P.930 Principles of a reference impairment system for video• P.931 Multimedia communications delay, synchronization and frame rate
measurement• J.143 User requirements for objective perceptual video quality
measurements in digital cable television• J.144 Objective perceptual video quality measurement techniques for
digital cable television in the presence of a full reference
![Page 5: Objective and Subjective Evaluation for Telecommunications Services and Video Quality](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dfd550346895dcc3827/html5/thumbnails/5.jpg)
Standardization of Objective Video Quality Evaluation
Methods ITU-T & ITU-R
T1A1 & IEEE VQEGValidation
Testing
Results
Industry & Academia
Standards and Reports
![Page 6: Objective and Subjective Evaluation for Telecommunications Services and Video Quality](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dfd550346895dcc3827/html5/thumbnails/6.jpg)
VQEG
• Video Quality Experts Group – Founded in 1997 as Rapporteurs’ Group– ITU-T SG12, ITU-T SG9, ITU-R WP11E(6Q)– Currently over 200 subscribers to the email list– Latest meeting in February 2002
• VQEG Website: www.its.bldrdoc.gov/vqeg
![Page 7: Objective and Subjective Evaluation for Telecommunications Services and Video Quality](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dfd550346895dcc3827/html5/thumbnails/7.jpg)
VQEG Phase I
• Ten Proponent Systems • Over 26,000 Subjective Opinion Scores • 20 Different Source Sequences Processed
by 16 Different Video Systems • 8 Independent Laboratories Worldwide.
![Page 8: Objective and Subjective Evaluation for Telecommunications Services and Video Quality](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dfd550346895dcc3827/html5/thumbnails/8.jpg)
VQEG Phase I Results
• No Single Winner—8 Way Tie• Resulted in J.144
– 8 Objective Models Described in Appendix
• 80 Gbytes of Subjectively Rated Video Available to the Research Community
![Page 9: Objective and Subjective Evaluation for Telecommunications Services and Video Quality](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dfd550346895dcc3827/html5/thumbnails/9.jpg)
VQEG Phase IICurrent Work
• Full Reference TV– Inclusion of Expert Viewer Test
• Reduced Reference – No-reference TV– Most Useful for Monitoring
• Multimedia– Videoconferencing, Streaming, Webcasting
• VQEG Will Not Determine Winners—Report Results to Standards Bodies
![Page 10: Objective and Subjective Evaluation for Telecommunications Services and Video Quality](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dfd550346895dcc3827/html5/thumbnails/10.jpg)
T1A1 Current WorkFamily of 5 Technical Reports
• T1.TR.PP.72-2001 Methodological Framework for Specifying Accuracy and Cross-calibration of Video Quality Metrics
• T1.TR.PP.73-2001 Video Normalization Methods Applicable to Objective Video Quality Metrics Utilizing a Full Reference Technique
• T1.TR.PP.74-2001 Objective Video Quality Measurement Using a Peak-Signal-to-Noise-Ratio (PSNR) Full Reference Technique
• T1.TR.PP.75-2001 Objective Perceptual Video Quality Measurement Using a JND-based Full Reference Technique
• T1.TR.PP.77-2002 Data and Sample Program Code to Be Used With the Method Specified in Technical Report [72] for the Calculation of Resolving Power of the Video Quality Metrics in Technical Reports [74] and [75]
• TRs Input to the ITU for Further Validation– T1.TR-72-2001 Used to Judge Accuracy of Metrics
![Page 11: Objective and Subjective Evaluation for Telecommunications Services and Video Quality](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dfd550346895dcc3827/html5/thumbnails/11.jpg)
0.95
0.90
0.85
0.80
0.75
0.70
0.65
0.60
0.550 0.1 0.2 0.3 VQM(2) - VQM(1) (Normalized Scale)
Ave
rage
con
fiden
ce th
at V
QM
(1) i
s w
orse
than
VQ
M(2
)
0.4 0.5 0.6 0.7
1
Using T1.TR-72-2001 to Assess VQM Resolving Power
![Page 12: Objective and Subjective Evaluation for Telecommunications Services and Video Quality](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dfd550346895dcc3827/html5/thumbnails/12.jpg)
Using T1.TR-72-2001 To Classify Errors for a
Particular VQM
![Page 13: Objective and Subjective Evaluation for Telecommunications Services and Video Quality](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dfd550346895dcc3827/html5/thumbnails/13.jpg)
IEEE Broadcasting SocietyIEEE G-2.1.6
• P1486-D06 Draft Standard on Subjective Measurement of Visual Impairments in Digital Video Using an IEEE‑JND Scale – Based on Work Done at NASA– Develops an Absolute Video Quality Scale– Can Be Used to Calibrate Video Quality Metrics
• Input to the ITU for review
![Page 14: Objective and Subjective Evaluation for Telecommunications Services and Video Quality](https://reader035.fdocuments.in/reader035/viewer/2022062520/56815dfd550346895dcc3827/html5/thumbnails/14.jpg)
Conclusions
• Video QOS Evaluation Is Important• Work Ongoing in Several Standards Bodies• VQEG Is Conducting New Validation Tests
• Work Progresses as Resources are Committed to Projects
IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 2004