Wp Testing Wireless Vq

8/2/2019 Wp Testing Wireless Vq

1/16

| W H I T E P A P E R |

Testing Wireless Voice Quality:

Capturing the Customer Experience

December 2006


2/16

Contents

Copyright 2007 Ditech Networks 2/16

Executive Summary.......................................................................................................................... 3

Voice Quality The Customer Experi ence.......................................................................................... 4

Increased Network Complexity...................................................................................................... 4

Rising Customer Expectations....................................................................................................... 5

Put Your Company Ahead o the Curve with Di tech Networks......................................................... 6

VQA....................................................................................................................................6

Experience Intelligenc (EXi)....................................................................................................6

Overview o Voice Quality Testing..................................................................................................... 7

Standards Governing Voice Quality Testing.................................................................................... 7

ITU P.800............................................................................................................................... 7

ITU P.862............................................................................................................................... 8

ITU G.107.............................................................................................................................. 9

Current Industry Tools or Asse ssing Voice Quality.........................................................................10

Intrusive Tests or Characterizing the Network..............................................................................10

Ditech Networks EXi Solution: Managing the Customers Live Voice Quality Experience....................... 11

Exposing External Impairments....................................................................................................11

Noise and Speech Level......................... ................................................................................. 12

Echo and Delay......................................................................................................................12

Delivering Meaningul Customer Experience Data......................................................................... 12A Future-Proo Solution..........................................................................................................14

Value or Multiple Quality Initiatives........................................................................................14

Control the Voice Quality Exper ience............................................................................................... 15

Testing Wireless Voice Quality


3/16


Copyright 2007 Ditech Networks /16

Executive Summary

A customers experience o voice quality extends beyond the physical network, as environmental issues and user

equipment can have a proound impact on how a call sounds. The users environment and equipment impact the

voice quality or a subscriber even in a perect RF environment because they directly aect the speech signal input.

Recognizing this issue is important because customers are becoming less tolerant o voice problems as they become

more sophisticated wireless users.

Existing industry voice quality tests ocus on the network, primarily radio requency (RF), and not on the speech signal.

The ollowing quotation rom the documentation o a leading wireless voice quality test tool illustrates this bias:

The Speech Quality Index (SQI) is a patented Ericsson algorithm which is built upon the act that

almost all speech distortions in a mobile network are due to problems with the radio transmission.1

RF- and codec-ocused tests are not unimportant, but they provide a severely limited assessment o the overall

customer experience. In addition to ocusing only on network component perormance, existing voice quality tests

are intrusive, requiring the injection o sampled speech signals and measurement o the post-processed signal. This

methodology provides no insight into the number o customer calls impaired by the users environment or sub-par

equipment.

Ditech Networks Experience Intelligence (EXi) experience intelligence solution provides the industrys frst and only

live monitoring o environmental anduser-equipment-related impairments on every call, all the time. Ditechs EXi

solution uses the ITU-Ts G.107 standard to provide Transmission Rating (R) Factor and Mean Opinion Score (MOS)

values on the combined measured impact o background noise, echo, and audio level impairments on every call.

EXi provides an unprecedented visibility into the customers real-lie call experience.

1. Speech Quality Index in CDMA2000, published by Ericsson.


4/16



Voice Quality The Customer ExperienceThe ollowing trends are impacting the customers voice quality experience:

1. Network complexity is introducing additional voice quality impairments.

2. Customer expectations o voice quality rom wireless phones are rising, so mere annoyances in years past

are now creating churn.

3. As mobile become more commonplace, more calls originate rom andend in noisy or acoustic-echo prone

situations, adding voice quality issues.

Figure 1 outlines the issues involved in managing the customers experience o voice quality:

Figure 1 : : Impairments Experienced by the Customer

Increased Network Complexity

The world o voice communications is becoming increasingly complex. Despite industry consolidation, device and

IP-telephony prolieration are making it more difcult or carriers to engineer high voice quality. These trends intro-

duce impairments beyond the physical boundaries o the network, but directly impacting the customer experience

o the network.

Carriers test handsets beore they are oered to customers. In the early days, carriers could exhaustively test the

small number o handsets in their oerings. But today, the number o devices on the market is increasing and device

release cycles are shortening, making it difcult or carriers to dedicate the resources to extensively test each new

model. Testing is urther complicated by the intense competitive pressures to quickly introduce new models and tailor

handset oerings to multiple market segments. The inability to tightly control voice quality in a wide range o hand-

sets rom multiple vendors has resulted in greater incidences o acoustic echo, inappropriate audio levels, and noise

introduction.

Phone

Environment Other Networks

SIP Phone

Ability to Manage and Mitigate

Phones

Network

Accessories

LOW

LIMITED

COMPREHENSIVE


5/16



I controlling voice quality among multiple handset oerings is difcult, controlling quality once atermarket accessories

are added to a handset is impossible. The market is awash with device accessories, which consumers purchase o the

shel and oten personalize. The millions o possible device interaction permutations cannot be tested economically.

One aspect o a call that a carrier can control is its own network, and the industry has developed a robust methodology

or network characterization. In network testing, two attributes are primary used to defne voice quality:

Radio requency (RF) Assesses the wireless link perormance using metrics such as rame erasure rate (FER), bit

error rate (BER), and carrier-to-intererence ratio (C/I).

Network-wide quality Assesses network delay and codec selection.

However, this network control ends at the network edge, where trafc is delivered to other carriers or routing within

their networks. Unortunately, these interconnections oten introduce acoustic and hybrid echoes, noise, varying audio

levels, and delay, which add to poor voice quality.

Rising Customer Expectations

In addition to the mounting complexity in simply maintaining the status quo in voice quality, there is evidence rom

a leading research survey that consumer expectations o voice quality are increasing as the wireless industry matures.

This has resulted in a marked reduction in tolerance or voice quality issues such as noise, echo, and level mismatch.

The survey went urther to identiy that this reduction in tolerance is directly related to churn. In eect, service quality

issues that did not cause a customer to consider switching services last year are now considered unacceptable and

cause churn.

Figure 2 shows the voice quality impairments discussed above and where they originate within the overall call path.

Figure 2 : : Voice Quality Impairment Origins in the Network

SIP Phone

Acoustic EchoNoise

Audio LevelHybrid Echo

CodecDelay

Acoustic EchoAmbient Noise

Acoustic EchoAudio Level

FERBERC/I

Environment Device RF Network Wide Other Networks

BSC MSC

PSTN


6/16



Put Your Company Ahead of the Curve with Ditech NetworksWhat can be done to mitigate the problems that come rom outside a network, but directly impact customer experi-

ence and cause churn?

With the ollowing products, Ditech Networks oers the most comprehensive solution or managing and improving

the customer experience beyond the reach o a wireless carriers network boundaries:

VQA The most comprehensive suite of voice improvement capabilities

Superior Voice Quality Improvement

- Advanced Noise Cancellation (ANC)

Cancels noise without distorting the speakers voice

Eective even in extremely low SNR environments

- Automatic Level Control (ALC)

Removes level mismatches caused by both the network and the device

Prevents clipped speech and distortion

- Enhanced Voice Intelligibility (EVI)

Improves speech clarity, especially or calls that use low bit-rate codecs in noisy environments

Speech-based signal processing without ampliication or distortion

Dramatic statistical word recognition improvement

Advanced Echo Control

- Industry-best Hybrid Echo Cancellation (HEC)

192 ms tail length, the highest in industry

- Bidirectional Acoustic Echo Control (AEC)

Cancels echo a ull 3 dB louder than the competition

DSP-enabled nonlinear processing to address unique acoustic echo challenges

EXi The industrys only real-time customer experience metrics that include external impairments

24/7 measurement o every call

Provides visibility to impairment issues outside o the carriers network

Measures noise, echo, delay, and audio level impairments

Delivers ITU-T G.107-based R Factor and MOS values or every call


7/16



Overview of Voice Quality TestingHow do wireless carriers test or voice quality impairments and remedy them? This section discusses the capabilities and

limitations o existing standards and the industry tests associated with voice quality testing.

Standards Governing Voice Quality Testing

The most common standards that assess the impact o various elements on voice quality are the International Telecom-

munication Unions Telecommunication Standardization Sector (ITU-T) Recommendations P.800, P.862, and G.107. These

standards were designed to address specifc voice quality issues, as outlined in Table 1.

Table 1 : :Focus o ITU-T Standards or Voice Quality Testing

Standard Summary Environment Device RF Network Other Network

ITU-T P.800 Subjective Voice Quality Test Level All Codec Noise LevelITU-T P.862 Objective Voice Quality Test PESQ All Codec

ITU-T G.107 E-Model or Voice Quality Analysis All All All All

ITU-T P.800

ITU-T Recommendation P.800 identifes a subjective standard that attempts to measure listening quality and listening eort

using Absolute Category Rating (ACR), Degradation Category Rating (DCR), and Comparison Category Rating (CCR) pro-

cedures. These procedures require a panel o listeners to assign voice calls a score based on specifc criteria. These scores

are then aggregated into Mean Opinion Scores (MOS) that are quantifed rom 1 (worst) to 5 (best).

ITU-T P.800 is limited in its ability to provide accurate results when assessing the perormance o a voice enhancement

device. Dynastat, a leader in the subjective testing industry stated the ollowing in an ITU-T submission:

Standard rating scale methodologies, e.g., ACR, DCR, and CCR, can give conounding and even

spurious results when used to evaluate the eects o [voice enhancement devices].2

The paper went on to postulate that the inability or people to reliably rate quality is that individuals cannot weigh the

value o the two variables in voice enhancement o modifcation to the original signal and noise reduction level in a

repeatable manner.

Thereore, ITU-T P.800 is o limited use in assessing the impact o noise in the network since it cannot be relied upon to

provide reliable results on the change in customer experience created by introducing noise reduction technology. ITU-T

P.800 is also impractical due to the high cost o perorming subjective testing or network monitoring purposes.

2. From ITU-T Submission AH-00.


8/16



ITU-T P.862

ITU-T P.862s Perceptual Evaluation o Speech Quality (PESQ) procedure is designed to be analogous to the ITU-T P.800

procedure, providing an objective method or predicting a subscribers experience o voice quality rom a signal that

passes through a communication system. PESQ uses a defned series o phrases to produce a score that can be directly

mapped to the MOS value. However, PESQ is only accurate or a limited range o circumstances, none o which impact

the speech signal itsel rom noise, echo, or level issues. Additionally, PESQ only measures during active speech no

weight is given to the listening experience in pauses or silence.

The three tables below are rom the ITU-T P.862 Recommendation and identiy the capabilities and limitations o PESQ.

PESQ has demonstrated acceptable accuracy or the applications listed in Table 2.

Table 2 : : Factors or Which PESQ Has Acceptable Accuracy (ITU-T P.862)

Codec evaluation

Codec selection

Live network testing using digital or analog connection to the network

Testing o emulated and prototype networks

Note: When environmental noise is present, the quality can be measured by passing PESQ the clean original without noise, and the degraded

signal with noise.

PESQ is known to provide inaccurate predictions when used in conjunction with the variables in Table 3, or is otherwise

not intended to be used with these variables.

Table 3 : : Factors or Which PESQ Has Inaccuracy (ITU-T P.862)

TestFactorsListening levels (see note below)

Loudness loss

Eect o delay in conversational tests

Talker echo

Sidetone

Coding Technologies

Replacement o continuous sections o speech making up more than 25% o active speech by silence

(extreme temporal clipping)

Applications

In-service non-intrusive measurements device

Two-way communications perormance

Note: PESQ assumes a standard listening level o 79 dB SPL and compensates or non-optimum signal levels in the input fles. The subjective

eect o deviation rom optimum listening level is thereore not taken into account.


9/16



PESQ has not been validated or measuring the test actors and applications listed in Table 4.Table 4 : :Factors or Which PESQ Is Not a Valid Test (ITU-T P.862)

Test Factors

Music as input to a codec

Listener echo

Eects/artiacts rom operation o echo cancellers

Eects/artiacts rom noise reduction algorithms

Coding technologies

CELP and hybrid codecs


10/16



Current Industry Tools for Assessing Voice QualityExisting testing tools are based primarily on the P.862 PESQ standard and on the Spirent (ormerly SwissQual) SQUAD-

LQ proprietary algorithm. These tools include intrusive tests, using recorded fles, to measure voice quality within the

deterministic network environment. These tests have two primary limitations or measuring aspects o the voice quality

customer experience outside o the network component:

1. The tests rely on speech iles to characterize the network perormance rather than capture actual call metrics.

2. The algorithms have a limited ability to accurately assess non-network-based impairments to customer experience

even or recorded speech iles due to the limitations o the PESQ and SQUAD tools in modeling speech signal

impairments (as was identiied in the previous discussion o ITU P.862).

Table 5 below identifes the capabilities o various toolsets with respect to the overall customer experience.

Table 5 : : Ability o Various Tools to Measure Overall Customer Experience

Tool Algorithm Live Calls Environment Device RF Network Other Network

TEMs PESQ No All Codec

TEMs SQI Yes All Codec

SwissQual SQUAD-LQ No Audio Level All Codec

Invex3G PESQ No All Codec

Invex3G SQUAD-LQ No Audio Level All Codec

QVoice PESQ No All Codec

Agilent PESQ No All Codec

GL PESQ No All Codec

Intrusive Tests for Characterizing the Network

The objective o intrusive tests is to characterize the network while minimizing introduced impairments. Intrusive testing

is a very eective tool or this because the network component o the voice quality experience is a controlled environ-

ment that will provide consistent output or consistent input. Thereore, under a given network load and with a defned

RF environment the network will deliver the same quality level every time.

The same is not true or impacts to the speech signal that occurs on actual customer calls. A noisy background can cause

degradation to the customer experience even in a high-quality RF environment. The same is true o acoustic echo or im-

proper speech level coming in rom another network or a poor device. These issues cannot be identifed using recorded

fles, but only through an assessment o actual calls.


11/16



SIP Phone

Acoustic EchoNoise

Audio LevelHybrid Echo

CodecDelay

Acoustic EchoAmbient Noise

Acoustic EchoAudio Level

FERBERC/I

Environment Device RF SNRINPLR

WAEPLEPL

MOSR-Value

VED

Network Wide Other Networks

BSC

MSC

PSTN

Ditech Networks EXi Solution: Managing the Customers Live Voice Quality Experience

Ditechs EXi solution delivers new visibility into impairments that occur during real calls, accurately identiying where and

when impairments occur, and quantiying the impairments impact on the subscribers experience.

Ditechs EXi provides the ollowing eatures:

Measurement o live customer experience

Visibility to impairments outside o the carriers network

R Factor and MOS value per call based on measured noise, echo, and level impairments

Table 6 l ists EXis current and uture capabilities and eatures.

Table 6 : :Ditech Networks EXi Capabilities

Tool Algorithm Live Calls Environment Device RF VED Network Other Network

Ditech EXi ITU-T G.107 Yes All All Future All Future All

Accurate Production Environment Measurements

Ditechs EXi provides 24-hour monitoring on a per-call basis. Live monitoring delivers accurate measurements o subscrib-

ers actual experiences using a production network.

Exposing External Impairments

With Ditechs EXi technology, or the frst time carriers have visibility into impairments that originate outside o their

network. Ditechs solution monitors or acoustic echo, noise, audio level, and hybrid echo pinpointing the impairments

and removing them. Ditechs EXi also helps the carrier identiy problems caused by other networks, urther aiding in

troubleshooting.

EXi measures the ollowing voice quality impairments:

Noise level

Speech level

Hybrid echo delay

Hybrid echo return loss

Acoustic echo delay


12/16



Noise and Speech Level

An increasing number o calls originate rom noisy urban environments, and background noise can be a major degrada-

tion actor to the listening quality o a call. Ditechs EXi measures average per-call speech level, noise level, and signal-to-

noise ratio (SNR) in both directions on every call.

Echo and Delay

The migration o voice networks rom circuit to packet technologies is resulting in greater and more variable latency. The

majority o legacy hybrid echo cancellers are unable to handle this added delay, leaving callers unprotected rom annoying

hybrid echo.

In addition, variability in terminal equipment (handsets, headsets, and handsree kits) continues to expand without

necessarily providing adequate acoustic isolation, exposing more users to acoustic echo.

Ditechs EXi provides comprehensive statistics about echo delay and return loss or both hybrid (linear) echo and acoustic

(non-linear) echo. It monitors rom 0 to 400 ms in both directions, ensuring complete and accurate measurement coverage.

For a statistical understanding o how echo and delay combine to aect customer satisaction, Ditechs EXi also provides

Echo Objection Rates per ITU-T G.131.

Delivering Meaningful Customer Experience Data

Ditechs EXi solution delivers a unifed MOS score o the actual subscriber voice quality experience, measuring all calls, all

the time, or multiple quality impairments. The solution also complements existing test equipment or easy deployment

into an existing carrier environment.

Per-call listening and conversational R Factors and MOS values are continuously and non-intrusively computed based on

the measured voice quality impairments (speech, noise, and echo), as well as codec type. Ditechs objective scores ollow

the ITU-T G.107 E-Model standard using an industry-unique approach o DSP-based measurements o the live voice signal.

Ditech has partnered with Telchemy, a leading provider o perormance monitoring technology, and employs their

VQmon computational model to provide R Factors and MOS values or the ollowing:

Listening Quality (LQ) This score includes noise and speech level. It does not include impairments that aect

conversation, such as delay.

Conversational Quality (CQ) This score includes the impairments measure or the LQ score and adds echo and delay,

which aect conversational quality.


13/16



The impact o the impairments is reported in any o the ollowing mechanisms:

Transmission Rating (R) Factors

o Listening Quality (R-LQ)

o Conversational Quality (R-CQ)

Mean Opinion Scores (MOS)

o Listening Quality (MOS-LQ)

o Conversational Quality (MOS-CQ)

Weighted acoustic echo path loss

Echo Objection Rate per ITU-T G.131

Ditech EXi data can be aggregated rom the local, regional, and national levels to gain visibility into the customer experi-

ence at any point in the network. Results are easily integrated into existing management and executive dashboards or

at-a-glance status reporting. Below is an example o how EXi data can be graphically presented to show the level o voice

quality customers are experiencing.

Figure 3 : : Sample EXi Data

MOS Conversational Quality

1 2 3 4 5

18,000

16,000

14,000

12,000

10,000

8,000

6,000

4,000

2,000

MOS Score

CallCount

2,4471,254

14,750 15,207

8,923

14,563

12,328

14,332

10,432

16,100

Rin

Rout


14/16



A Future-Proof Solution

Ditech EXi uses a Telchemy scoring algorithm that delivers an R Factor and MOS values per call. The Telchemy algorithm is

widely adopted in IP networks, with more than two million licenses issued. This ensures that as wireless networks evolve

to VoIP, the quality measurements initiatives put in place today will continue to be o value in the uture.

Value for Multiple Quality Initiatives

Ditech EXi is easily integrated with other tools rom multiple vendors to improve voice quality management and expand

visibility into call quality and the origin o impairments. MOS data can be correlated with other network data, such as cell

identifer and Electronic Serial Number (ESN).

Integrating these multiple data points provides wireless carriers the ability to recognize and identiy network trouble

points, receive automatic alerts when a problem arises, and provide advanced reporting or network planning and

customer Service Level Agreements (SLAs).

Figure 4 : : Managing Network-wide Voice Quality

Partners Ditech


15/16



Control the Voice Quality ExperienceDitech EXi provides the ability to manage the complete end-to-end voice quality experience, so that wireless carriers can

use their existing PESQ, SQUAD-LQ, or SQI tools or RF-related network quality measurements, and EXi or monitoring

their noise, echo, and level impairments (Figure 5).

Figure 5 : : Tools or Controlling the Customer Experience

Ditechs EXi and VQA solutions provide carriers with visibility into andmitigation o impairments that, until now, have

been outside o a carriers control. Used in conjunction with industry-standard voice quality network tests, wireless carriers

now have tools to identiy and mitigate all aspects o voice quality impairments impacting the customer experience.

Phone

Environment Other Networks

SIP Phone

Phones

Network

Accessories

Poor Fair Good

Poor Fair Good

Good Good Fair

PESQ SQUAD-LQ Ditech EXi


16/16


Corporate Headquarters

US Sales West

Ditech Networks

825 East Middleield Rd.

Mountain View, CA 94043

USA

1-800-234-0884 (Toll Free)

1-650-623-1300 (Direct)

1-650-564-9599 (Fax)

[email protected]

US Sales Office East and Central

Ditech Networks

8360 Greensboro Dr.

McLean, VA 22102

USA

1-800-234-0884 (Toll Free)

1-650-623-1300 (Direct)

1-650-564-9599 (Fax)

[email protected]

Canada Sales Office

Ditech Networks

2275 Lakeshore Blvd West, Suite 500

Toronto, ON M8V 3Y3

Canada

1-416-255-7776 (Phone)

[email protected]

Mexico Sales Office

Ditech Networks

Torcuato Tasso 245-6o. Piso

Col. Polanco

Mxico, D.F. C.P. 11570

Mxico

+52 (55) 5254-5422 (Main)

+52 (55) 5350-8679 (Sales)[email protected]

Brazil Sales Office

Ditech Networks

Alameda Araguaia, 933

Alphaville

Barueri, SP 06455-000

Brazil

+55 (11) 4208-6266 (Main)

+55 (21) 3521-5543 (Sales)

[email protected]

[email protected]

China Sales Office

Ditech Networks

Unit 3010, No. 500

Xiangyang South Road

Xuhui District

Shanghai 200031

PRC

+86 (21) 5456 0305 (Phone)

[email protected]

South-East Asia Sales Office

Ditech Networks

Lippo Plaza 3rd Floor

Jl. Jend. Sudirman Kav. 25

Jakarta 12920

Indonesia

+62 (21) 5291 3780 (Phone)

+62 (21) 522 1977 (Fax)

[email protected]

South Asia Sales Office

Ditech Networks

No. 8/11, Sarvapriya Vihar

New Delhi 110016

India+91 9810372555 (Phone)

[email protected]

Middle East/Africa Sales Office

Ditech Networks

21 El-Fawakeh St., 3rd Floor

Dokki, Giza 12311

Egypt

+20 2 336-5100 (Phone)

+20 2 761-8964 (Fax)

[email protected]

Spain Sales Office

Ditech Networks

Torres Quevedo, 1 (P.T.M.)

28760 Tres Cantos Madrid

Spain

+34 (91) 803 74 44 (Main)

+34 (91) 829 26 90 (Sales)

[email protected]

[email protected]

About Ditech NetworksDitech Networks supplies voice processing equipment or telecommunication networks around the world. Ditech Networks technology

solutions include voice, media processing, SIP, and security delivered on carrier-grade scalable platorms to enhance the delivery o

communications services over mobile, Voice over IP, and wireline networks. Ditech Networks customers are premier network operators

including Verizon, AT&T, Orascom Telecom, and others that collectively serve more than 150 million subscribers.

Wp Testing Wireless Vq

Documents

Transcript of Wp Testing Wireless Vq