Speech quality Learnings

Click here to load reader

  • date post

    07-Jul-2015
  • Category

    Engineering

  • view

    821
  • download

    2

Embed Size (px)

description

NA

Transcript of Speech quality Learnings

  • 1. Dingli (27/7/2013) DL1AMOSWP Rev1 1 / 37 Speech Quality Testing Solution (MOS) Whitepaper

2. Dingli (27/7/2013) DL1AMOSWP Rev1 2 / 37 Revision History Date Version Author Description 2013-05-06 1.0 Geng Xiaoming First Edition 2013-07-27 1.1 Zhang Qifei Change document organization, optimize MOS testing solution, and add POLQA parameters 3. Dingli (27/7/2013) DL1AMOSWP Rev1 3 / 37 Contents 1 Introduction...............................................................................................................................4 1.1 Document Organization ............................................................................................4 1.2 Abbreviations and Acronyms....................................................................................4 2 Development of Speech Quality Testing...................................................................................6 3 PESQ Algorithm .......................................................................................................................9 3.1 Introduction...............................................................................................................9 3.2 PESQ Algorithm Structure........................................................................................9 4 POLQAAlgorithm..................................................................................................................12 4.1 Introduction.............................................................................................................12 4.2 Evolution of POLQA Based on PESQ ....................................................................12 4.3 Super-Wideband and Narrowband in POLQA........................................................13 4.4 POLQA Perceptual Model and Structure................................................................14 4.5 Ideal POLQA MOS Values in Different Network Environment .............................15 5 Dingli MOS Testing Solution..................................................................................................16 5.1 Product History .......................................................................................................16 5.2 Added Value of Dingli MOS Testing System..........................................................16 5.3 Dingli MOS Test System.........................................................................................17 5.4 MOS Testing Solutions ...........................................................................................19 5.4.1 MOS Testing Solution Based on Pilot Pioneer.........................................19 5.4.2 MOS Analysis Solution Based on Pilot Navigator ...................................21 5.4.3 MOS Testing Solution Based on Pilot Fleet (RCU) .................................22 5.4.4 MOS Testing Solution Based on Pilot Walktour.......................................24 5.5 Details About Dingli MOS Testing Solution...........................................................25 5.5.1 Impact of Network Factors on PESQ MOS Value ...................................25 5.5.2 Parameters in POLQA Testing Result .......................................................29 5.5.3 Reference PESQ-LQ Value: Some practical examples..........................32 6 Appendix 1: ITUT P.862 Standard Relevant Information...................................................33 7 References...............................................................................................................................37 4. Dingli (27/7/2013) DL1AMOSWP Rev1 4 / 37 11 IInnttrroodduuccttiioonn This document describes the development and evolution of speech quality testing technologies in Telecommunications network, and focuses on two kinds of objective testing methods--PESQ and POLQA. 1.1 Document Organization Chapter 1: Introduction to document objectives, document organization, and abbreviations and acronyms Chapter 2: Basics and development of speech quality testing Chapter 3: Objective speech quality testing method PESQ algorithm Chapter 4: New speech quality testing method recommended by ITU POLQA algorithm Chapter 5: Dinglis speech quality evaluation solutions 1.2 Abbreviations and Acronyms Table 1 Abbreviations and acronyms Item Full Spelling Description ITU International Telecommunication Union MOS Mean Opinion Score MOS-LQO MOS Listening Quality Objective AMR Adaptive Multi-Rate Codec AAC Advance Audio Codec FER Frame Error Rate BER Bit Error Rate POTS Plain Old Telephone System GSM Global System for Mobile Communications CDMA Code Division Multiple Access LTE Long Term Evolution 5. Dingli (27/7/2013) DL1AMOSWP Rev1 5 / 37 Item Full Spelling Description PAMS Perceptual Analysis Measurement System Recommendation of ITU-T P.800 PSQM Perceptual Speech Quality Measure Recommendation of ITU-T P.861 PESQ Perceptual Evaluation of Speech Quality Recommendation of ITU-T P.862 POLQA Perceptual Objective Listening Quality Assessment Recommendation of ITU-T P.863 VoIP Voice Over IP P.56 ITU-T Rec. P.56 Recommendation of ITU-T P.56 6. Dingli (27/7/2013) DL1AMOSWP Rev1 6 / 37 22 DDeevveellooppmmeenntt ooff SSppeeeecchh QQuuaalliittyy TTeessttiinngg Speech quality testing on different networks with a unified standard can be a challenge due to different communication technologies. For example, in GSM network, RxQual (based on BER) is used to evaluate the speech quality; and in CDMA network, FER is used to evaluate the speech quality. In addition, even within the same network, a single RxQual or FER value cannot represent the true speech quality. A professional speech quality testing system should be an end-to-end and user-based system. Therefore, a general testing method is required to perform direct comparative testing for different networks. Based on the test subject, speech quality testing can be divided into two categories: subjective and objective test. Subjective test depends on listeners to evaluate the communication system performance. According to the studies in ITU-T P.800 and ITU-T P.830, about 40 to 60 trained listeners are required to perform subjective perceptual comparison for reference signal and degraded signal based on detailed criteria. Score on the degraded signal is in accordance with the MOS scoring standard (scoring from 0 to 5). This way, a final MOS value is obtained. Table 2 MOS scoring standard Degree MOS Value Users Satisfaction Excellent 5.0 Very clear No distortion No delay Good 4.0 Clear Small delay Little noise Fair 3.0 Unclear A certain of delay A certain of noise 7. Dingli (27/7/2013) DL1AMOSWP Rev1 7 / 37 Degree MOS Value Users Satisfaction A certain of distortion Poor 2.0 Unclear Big noise or interruption Serious distortion Bad 1.0 Silence or totally unclear Very big noise The subjective testing result is most reliable, and this method can be used to evaluate network performance and quality of speech with any speech coding mode. However, its disadvantages are obvious. In the test, factors (such as evaluation environment and listener) should be strictly controlled; the speech material must be carefully selected; otherwise, the final result may be affected. All these makes the test time-consuming, laborious, difficult to organize, and with poor repeatability. As a result, a more efficient and repeatable method is required in the actual test, that is, an objective testing method. In practice, objective testing depends on the parameter comparison (of reference and degraded speech signal) in time and frequency domain, while the test result is calculated by hardware or software. Some objective testing methods such as PAMS and PSQM are introduced during the research of objective speech quality testing. However, these methods have significant limitations. The test result is affected by particular speech codec, and in some cases, the result is much different from the MOS value in subjective testing. In ITU-T P.862-2001, the core speech quality testing method is upgraded to PESQ algorithm which integrates all advantages of previous algorithms. The PESQ test result is very close to the MOS value in the subjective testing, and PESQ algorithm is widely accepted and applied. Later, with the development and evolution of new communication technologies, POLQA algorithm is developed to support new speech codecs and super-wideband speech, and handle the time factor in VoIP. Compared with previous algorithms, introduction of POLQA algorithm to the unified and complex communication networks will achieve significant 8. Dingli (27/7/2013) DL1AMOSWP Rev1 8 / 37 improvements in accuracy and credibility. In addition, the capability of handling time factor makes POLQA algorithm applicable to any speech quality testing scenarios. POLQA algorithm includes two modes: NB (Narrow Band) and SWB (Supper Wideband), corresponding to different speech samples with different frequencies. POLQA algorithm has become the recommended algorithm in ITU-T P.863-2011. Figure 2.1 shows the evolution of ITU-T recommendations for speech quality testing. The information is obtained from http://www.polqa.info/index.html. Figure 2.1 Evolution of ITU-T recommendations for speech quality testing 9. Dingli (27/7/2013) DL1AMOSWP Rev1 9 / 37 33 PPEESSQQ AAllggoorriitthhmm 3.1 Introduction PESQ algorithm became an ITU-T recommendation (P.862) for objective speech quality testing in February 2001. Due to its innovative features and correlation accuracy, PESQ quickly becomes the mainstream speech quality testing method. PESQ algorithm is applicable to end-to-end speech quality testing. It takes into account various perceptual factors (such as codec distortion, errors, packet loss, variable delay, jitter and filtration) to objectively test the speech quality, and provides a fully quantifiable speech quality testing method. Figure 3.1 PESQ reference signal and degraded signal 3.2 PESQ Algorithm Structure Figure 3.2 shows the entire