Ascom Unattended Trial Report Dubai

8/19/2019 Ascom Unattended Trial Report Dubai

1/22

UADT Report

December 2013

Page 1

Unattended drive test trial reportME GD2.0 NPO Lever

Trial country – UAETrial network – du

Trial dates – 25th, 26th & 27th Nov 2013

Trial conducted in collaboration between Nasir Mahmood – NPO expert Sean Ettehadieh – NPO expert Usman Sharif – NPO expert Ashish Tandon – ME GD transition lead Faisal Ghazaleh – ASCOM Technical expert

Malick Noor –

ASCOM AM

Report prepared by Nasir Mahmood – NPO expert

Report reviewed by Sean Ettehadieh – NPO expert Usman Sharif – NPO expert Ashish Tandon – ME GD transition lead

Company confidentialNot to be distributed further without consent of the author & / or reviewers


2/22

Page 2

TABLE of Contents

1 Purpose................................................................................ 3

2 Executive Summary ............................................................. 4

3 Trial scope & setup .............................................................. 5

3.1 Scope of trial .................................................................................................. 5 3.2 Drive test sites ............................................................................................... 5 3.3 Equipment ..................................................................................................... 5 3.4 Setup ............................................................................................................. 5 3.5 Drive test route planning ................................................................................ 6

4 Test results .......................................................................... 7

4.1 Compliancy score rating matrix ...................................................................... 7 4.2 2G test results ............................................................................................... 8 4.2.1 Major observations ................................................................................... 9 4.2.2 Short term & permanent solutions for overcoming shortcomings .............. 9 4.2.3 Snapshot of test ....................................................................................... 9 4.3 3G test results ..............................................................................................10 4.3.1 Major observations ..................................................................................11 4.3.2 Short term & permanent solutions for overcoming shortcomings .............11 4.3.3 Snapshot of test ......................................................................................11

4.4 LTE test results .............................................................................................12 4.4.1 Major observations ..................................................................................13 4.4.2 Short term & permanent solutions for overcoming shortcomings .............13 4.4.3 Snapshot of test ......................................................................................13 4.5 Speech Quality test results ...........................................................................14 4.6 Tool automation test results ..........................................................................14 4.7 Trial results from Saudi trial ..........................................................................15 4.7.1 Major observations ..................................................................................16 4.7.2 Short term & permanent solutions for overcoming shortcomings .............16

5 TEMS Automated Road-map ............................................. 17

6 Frequently asked questions ............................................... 18

7 Glossary of Technical Terms .............................................. 21


3/22

Page 3

1 Purpose

This document serves to present an impartial view of trial results that have been conducted extensively (on

an existing site and on a new site) in UAE on the du network using ASCOM TEMS automated solution and

to showcase the proof of concept for automation in drive testing i.e. drive testing without a technician in the

car.

The report also aims to provide the local delivery teams all the relevant information in order to be fully

familiar with some shortcomings in automation of drive testing and potential short term workarounds and

long term solutions envisaged.

An overall technical score based on compliancy per tested scenario is also presented in the document only

for the purpose of benchmarking with the NEMO solution.


4/22

Page 4

2 Executive Summary

On the basis of the trial conducted involving region NPO experts & observations recorded; it can be

confirmed with reasonable confidence that TEMS automated solution is currently capable of fulfilling upto

76% of the project requirements for pre launch (new sites) DT scope

&

85% of the project requirements for post launch (routine & benchmarking) DT scope (refer

Saudi report in section 4.7)

It is important to note that there are current shortcomings of the tool as outlined below:

TD-LTE and CSFB

Instant live monitoring and Locking on target cell

MMS

However, the above shortcomings do not pose a serious challenge in respect of rolling out the solution

widely due to the proposed workarounds (see explanations in the subsequent sections) and permanent

solution already envisaged during Q1 and Q2 of 2014 (see relevant 5 on ASCOM roadmap)

Advantages

High efficiency gains in overall drive test process

o In the course of the trial, one of the clearly visible advantages of the solution was the quick

turnaround time starting from drive testing & ending with generating a drive test report.

During the drive tests, the raw data was received within 10mins (server based in Sweden) of

the start and a report was prepared within minutes thereafter using standardized templates

in TEMS discovery

New features & value add to customer

o TEMS Automated provides additional features related to “Speech Quality Assessment”

using PESQ / POLQA which can be used to detect echo delay, attenuation volume levels,

silent calls etc. TA will be a value add to our customers as it can be used for managing

special events like Hajj & Ramadan, World Cup, Expo etc by getting live feeds about

network health and KPIs as perceived by subscribers.

On the basis of the above observations, on a scale of 0 to 5, TEMS automated secured an overall technical

score of 3.81 for pre launch scope and 4.27 for post launch scope.

Scoring methodology & matrix is presented in the subsequent sections of this trial report.


5/22

Page 5

3 Trial scope & setup

3.1 Scope of trial

Drive test on a newly commissioned site & an existing site to simulate tests related to pre-

optimization, site rehomings, expansions & swap.

3.2 Drive test sites

Network – du

o DXB 1996 – existing site – 3G & LTE

o DXB 2349 – new site on air – 2G

3.3 Equipment

Hardware: RTU-Version 5 with three active probes and one idle probe

Software: 9.2.0

Voice Card: f3607

Data Card: MC7710

SIM Card: 3 SIMs (2G, 3G, LTE enabled) with 5GB, 5GB & 25GB data plan & 300voice min

3.4 Setup

B1 – ASCOM performed the drive tests with driver only

B2 – DT data was loaded automatically in the backend servers in Sweden

B3 – ASCOM engg controlled the DT remotely from the NSN Dubai office

B4 – DT log files were pulled via FTP & ASCOM engg showcased reporting & analysis

B5 – DT reports & analysis capabilities assessed by NSN NPO experts for technical compliancy


6/22

Page 6

3.5 Drive test route planning

Drive test route for both the sites DXB 1996 & DXB 2349 were clearly marked on a printout and

handed over to the driver Instructions were given to the driver to start driving from a particular location marked on the printout

on the pre defined route

Upon reaching the site, fleet manager (ASCOM engg) and the car driver maintained contact on

phone whenever it was needed

Additionally, the driver was a given an android based smart phone (GPS and Mobile data enabled)

with pre-load app called “LIFE360” and the location of the driver was monitored LIVE from the NSN

office using the same smart phone app installed an another android based device.

Figure 1: Drive test route for site DXB 2349 Figure 2: Drive test route for site DXB 1996

Figure 3: Life 360 app for monitoring driver location


7/22

Page 7

4 Test results

This section summarizes the drive test results per technology domain for which the trial was conducted.

Each test has been rated and given a compliance score as per methodology explained below.

The overall technical score awarded to the TA as mentioned in the executive summary chapter was an

average of the score awarded per feature test.

For each test availability of a short term solution (temporary workaround) and a permanent solution

(solution as per ASCOM roadmap) has also been highlighted.

4.1 Compliancy score rating matrix

Score Abbreviation Methodology

0 Non compliant (NC) Non compliant

1 Partially compliant (PC) Available short term solution = No

Expected permanent solution on ASCOM roadmap = No

2 Partially compliant (PC) Available short term solution = Yes

Expected permanent solution on ASCOM roadmap = No

3 Partially compliant (PC) Available short term solution = No

Expected permanent solution on ASCOM roadmap = Yes

4* Partially compliant (PC) Available short term solution = Yes


Will Stay as PC even with permanent solution based on manual

efforts needed to perform the test

4 Partially compliant (PC) Available short term solution = Yes


5 Compliant Fully compliant


8/22

UADT Report

December 2013

Page 8

4.2 2G test results

2G trial was conducted on site DXB2439 for validation of various testing scenarios related to pre-optimization of a newly commissioned 2G site. These tests

are also valid for site rehomings, TRX expansions and swaps. Results of this trial are summarized in table below:

Test

scenario

Scenario detail Compliance Short term solution

available

Permanent solution available Score

1 Idle mode coverage test for Rx level & footprint PC Yes Yes (2Q 2014) 4

2 Dedicated Mode Long Call & Short for checking Rx-Lev, Rx-Qual and

KPIs

PC Yes Yes (2Q 2014) 4

3 TRX Testing to find out faulty TRXs PC Yes Yes (2Q 2014) 4*

4 SMS Test PC Yes Yes (2Q 2014) 4

5 MMS Test NC No No 0

6 Sector Swap (Cross Feeder) Test PC Yes Yes (2Q 2014) 4*

7 Hand-over Tests PC Yes Yes (2Q 2014) 4

8 Data Testing (Ping, Throughput & Data KPIs) PC Yes Yes (2Q 2014) 4

9 Data Testing (DL Static Testing) PC Yes Yes (2Q 2014) 4


9/22

UADT Report

December 2013

Page 9

4.2.1 Major observations

1. TA Solution is capable of capturing information related to idle mode, dedicated mode (short & long

Calls), SMS, handover and data tests.

2. TA lacks capability of camping on a cell and visbility about current location, serving cell and RF

conditions which increase the probability of deviation from targeted area of test-drive & pose a

challenge for certain tests which require precise information about prevailing radio conditions.

3. TA is not capable of performing TRX and sector swap tests as per routines used by project teams due

to lack of ability to lock on targeted Cell. However, these tests could be performed with TA solution

using indirect ways and correlation of information captured during other tests.

4. TA Solution is not capable of performing MMS tests.

4.2.2 Short term & permanent solutions for overcoming shortcomings

Short-Term Solution: The challenges described above could be managed by using a TPR with

pre-fed drive-map and team-viewer application to provide the required visibility and control to central

fleet technician. This workaround is envisaged to be secured via commercial negotiations with

ASCOM

Permanent Solution: ASCOM plans to embed a feature (see roadmap section) in the TEMS

automated that would be capable of providing the basic on line cell information to the fleet manager

remotely before (& during) drive test.

4.2.3 Snapshot of test

RxLev

RxQual Foot-Print Data


10/22

UADT Report

December 2013

Page 10

4.3 3G test results

3G trial was conducted on an operational site DXB1996 for validation of various testing scenarios related to pre-optimization of a newly commissioned 3G site.

These tests are also valid for site rehomings and swaps. Results of this trial are summarized in table below:

Test

scenario


available


10 Idle Mode for CPICH RSCP, EcNo & Serving SC PC Yes Yes (2Q 2014) 4

11 Active Mode for Voice Call CPICH RSCP, EcNo & Serving SC PC Yes Yes (2Q 2014) 4


13 Sector Swap(Cross Feeder) Test PC Yes Yes (2Q 2014) 4*



16 Data UL/DL Test for Rel 99 PC Yes Yes (2Q 2014) 4

17 Data UL/DL Test for HSPA PC Yes Yes (2Q 2014) 4

18 Data Testing (DL Static Testing) PC Yes Yes (2Q 2014) 4

19 2G to 3G re-selection PC Yes (2Q 2014) 4*

20 3G to 2G HO (IRAT) PC Yes (2Q 2014) 4


11/22

UADT Report

December 2013

Page 11


1. TA Solution is capable of capturing information related to idle mode, active mode (short & long Calls),

SMS, handover and data tests.

2. TA lacks capability of camping on a cell and visbility about about current location, serving cell and RF

conditions which increases the probability of deviation from targeted area of test-drive & poses a

challenge for certain tests like static data DL/UL which require precise information about prevailing radio

conditions.

3. TA is not capable of performing sector swap test as per routines used by project teams due to lack of

ability to lock on targeted SC. However, this test could be performed with TA solution using indirect

ways and correlation of information captured during other tests.

4. TA Solution is not capable of performing MMS tests.





ASCOM



remotely before (& during) drive test.


CPICH RSCP CPICH EcNo Best Server SC


12/22

UADT Report

December 2013

Page 12

4.4 LTE test results

LTE trial was conducted on an operational site DXB2439 for validation of various testing scenarios related to pre-optimization of a newly commissioned LTE

site. These tests are also valid for site rehomings and swaps. Results of this trial are summarized in table below:

Test

scenario


available


21 Idle Mode PCI, RSRP, RSRQ, SINR PC Yes Yes (2Q 2014) 4

22 Connected Mode PCI, RSRP, RSRQ, SINR, CQI, RI & DL Throughput PC Yes Yes (2Q 2014) 4


24 Sector Swap Tests PC Yes Yes (2Q 2014) 4*



27 Ping Test PC Yes Yes (2Q 2014) 4

28 DL & UL throughput in static mode PC Yes Yes (2Q 2014) 4

29 CSFB (Circuit Switched Fall Back) PC No Yes (1Q 2014) 3

30 3G to LTE re-selection PC Yes Yes (2Q 2014) 4*


13/22

UADT Report

December 2013

Page 13


1. TA Solution is capable of capturing information related to idle mode, connected mode, SMS, handover

and data tests.

2. TA lacks capability of camping on a cell and visbility about current location, serving cell and RF

conditions which increases the probability of deviation from targeted area of test-drive & poses a

challenge for certain tests like which require precise information about prevailing radio conditions.

3. TA is not capable of performing sector swap tests as per routines used by project teams due to lack of

abaility to lock on targetted PCI. However these tests could be performed with TA using indirect ways

and co-relation of information captured during other tests.

4. TA Solution is not capable of performing CSFB and MMS tests.





ASCOM



remotely before (& during) drive test. CSFB is also part of official road-map from ASCOM (see road-

map section).


RxLev

RxQual SINR Data


14/22

Page 14

4.5 Speech Quality test results

The functionality related to Speech Quality Testing is not available in traditional drive test tools. It was

tested during trial and results of evaluation are summarized in table below:

Test

scenario

Scenario detail Compliance Short term

solution available

Permanent solution

available

Score

31 Additional – PESQ/POLQA

Quality of Voice Testing on

2G/3G

PC Yes Yes (2Q 2014) 4

4.6 Tool automation test results

In addition to the technical tests, TA was also evaluated for its capabilities on automation of drive test.

Results of this evaluation are summarized in table below:

Feature

scenario

Scenario detail Compliance Score

32 Transfer of log files while the drive test is ongoing C 5

33 Viewing and processing of log files while the drive test is ongoing C 5

34 Automated processing and DT report generation by TEMS discovery C 5

35 Defining automated routines via Fleet Manager for New Sites On-Air C 5

36 Preparing and downloading scripts to TA while the drive test is ongoing C 5

37 Ability to run multiple technology drive tests within one drive C 5


15/22

UADT Report

December 2013

Page 15

4.7 Trial results from Saudi trial

In addition to the trial that was conducted extensively in UAE, a trial was also conducted in Saudi in P9 2013, focusing primarily on routine optimization and

benchmarking related drive tests. Results of this evaluation are summarized in table below:

Test

scenarioTechnology Operator Test Cases Compliance

Permanent

Solution

Score

1 2G, 3G, LTE

STC,

Mobily,

Zain

STC-2G Idle Mobily-2G IdleZain-2G

Idle

STC-LTE

Idle

PC

Due to TD-LTE

Yes

(1Q 2014)

3

2 2G, 3G, LTE

STC,

Mobily,

Zain

STC-3G

Short Call

Mobily-3G Short

Call

STC-3G

FTP DL/UL

Zain-3G

FTP DL/UL

PC

Due to TD-LTE

Yes

(1Q 2014)

3

3 2G, 3G, LTE

STC,

Mobily,

Zain

STC-Dual

Mode Long

Call

Mobily-Dual

Mode Long Call

STC-LTE

FTP DL/UL

Zain-LTE

FTP DL/UL

PC

Due to TD-LTE

Yes(1Q 2014)

3

4 2G,LTESTC,

Mobily,

Zain

STC-2G

Short Call

STC-2G Long

Call

STC-LTE

Ping

Mobily-LTE

Ping

PC

Due to TD-LTE

Yes(1Q 2014)

3

The evaluation score related to speech quality tests and tool automation features listed in above sections are also valid for post launch testing for routine

optimization and benchmarking activities.


16/22

UADT Report

December 2013

Page 16


1. TA Solution is capable of peforming all tests related to routine optimization and benchmarking actvities

for 2G, 3G, and FDD-LTE.

2. TA Solution is not capable of conducting any tests on TD-LTE.

3. TA Solution is not capable of performing CSFB tests.


The major shortcomings reported during trial are already covered in official road-map provided by ASCOM.

The support for TD-LTE and CSFB will be available during 1Q of 2014.


17/22

UADT Report

December 2013

Page 17

5 TEMS Automated Road-map

Ascom has shared an official road-map confirming support for major shortcomings including TD-LTE,

CSFB, cell locking and real time monitoring of probes.

It is expected that these features will get embedded in TEMS automated during Q1 and Q2 of 2014 which

will drive the TEMS automated solution to a higher compliancy and ease of use in our NPO projects – pre

and post launch.

Details of the roadmap from ASCOM are attached below.

Road Map TEMS Automatic.pdf


18/22

Page 18

6 Frequently asked questions

1. Is it possible to simulate ‘in-car’ and ‘outdoor’ test scenarios using TEMS Automated?

Answer : Yes, it is possible to use TEMS Automated in both modes. TEMS Automated comes with ‘indoor’

and ‘outdoor’ antennas which can be used for ‘in-car’ and ‘outdoor’ tests based on project needs.

2. How are the log files transmitted over air during drive-test?

Answer: There are multiple options provisioned in TEMS Automated for transmission of log files during

drive test. Fleet technician can select any technology (i.e., GPRS, EDGE, 3G, HSPA or LTE) on one of the

chipsets of TEMS Automated and trigger automatic transmission of log files. It is possible to change the

technology as well as chipset during course of drive test. It is also possible to use a chipset in multi-RAB

mode in case all four chipsets are being used for testing.

3. Is there any interruption to the measurement collection procedure during transmission of log files?

Answer : No, there is no interruption to any measurement collection procedure. TEMS Automated uses a

technique of splitting log files in smaller files (~500 KB) depending upon type of test in progress (i.e., short

call, long Call etc). These files are transmitted over air at regular intervals using technology chosen by fleet

technician during course of drive test.

4. Will there be any impact on data and voice tests of targeted site(s) since TEMS Automated uses same

serving cell for uploading log-files during drive test?

Answer : Yes, there could be a direct or indirect impact of continuous uploading of log files using same

serving cell. It is strongly recommended to use alternative technology or alternative operator for

transmission of log files to avoid any impact on quality of information being collected from the targeted

site(s).

5. Is it possible to connect an external scanner with TEMS Automated?

Answer : Yes, TEMS Automated is provisioned with external ports for connection of scanners (like PCTEL)

for conducting all kinds of tests related to scanning.

6. Are the chipsets provisioned in TEMS Automated capable of conducting voice and data tests on 2G,

3G, HSPA and LTE?

Answer : Yes, all chipsets are capable of 2G, 3G, HSPA and FDD-LTE. However, the capability related to

data testing on HSPA and LTE is limited to 7.2Mbps. So, each TEMS Automated unit is equipped with one


19/22

Page 19

or more ‘category 3’ data-chipsets capable of data testing up to 100 Mbps. However it is expected that

ASCOM will introduce a new chipset MC7304 in Q1 2014 which will be capable of voice and data testing on

2G, 3G & LTE upto 100Mbps. Please also see the roadmap of ASCOM for further details on this aspect.

7. Is TEMS Automated capable of conducting tests for TD LTE?

Answer : No, TEMS Automated is not capable of perfuming any tests for TD-LTE. However, this

functionality is envisaged in TEMS Automated by Q1 2014.

8. Is it possible to retrieve log files from TEMS Automated in case of interruption in connectivity with back-

end server?

Answer : Yes, it is possible to retrieve log files from TEMS Automated by connecting it directly to a laptop.

9. Is it possible to enhance the configuration of TEMS Automated beyond 4 chipsets for benchmarking of

multiple operators for multiple technologies?

Answer : Yes, it is possible to connect multiple TEMS Automated units using ‘daisy-chain’ technique to

enhance TA configuration from four to eight and max upto twelve based on project needs.

10. Does TEMS Automated require special power arrangements inside the car?

Answer : No, there are no special requirements for TEMS Automated. A 12V socket in the car is sufficient

to keep TEMS Automated powered-up during drive test.

11. Does TEMS Automated require periodic calibration like other testing devices?

Answer : No, there is no need for periodic calibration of TEMS Automated. Ascom will provide after-sales

maintenance support in case any abnormal behavior is observed during measurement collection.

12. What are the technical specifications of data-chipset ‘MC7710’ used in TEMS Automated?

Answer : The technical specifications of MC7710 are exactly same as a category-3 device. It is capable of

testing up to 100Mbps.

13. Is it possible to reconfigure and change testing scenarios based on observations recorded during the

drive-test?

Answer : Yes, it is extremely easy to change test scenarios while drive-test is in progress. The entire

process takes few minutes and test routines are changed on-the-run.


20/22

Page 20

14. Do we need a special server with fixed-line connectivity for performing ‘speech quality test’ using

PESQ or POLQA techniques?

Answer : No, there is no need for any special server or fixed line connectivity. TEMS Automated performs

‘speech quality tests’ using two voice chipsets of the same unit.


21/22

Page 21

7 Glossary of Technical Terms

1. TEMS Test Equipment Mobile Station

2. TA TEMS Automated

3. TPR TEMS Pocket Remote

4. PESQ Perceptual Evaluation of Speech Quality

5. POLQA Perceptual Objective Listening Quality Assessment

6. PCI Physical Cell IDs

7. CSFB Circuit Switched Fall Back

8. FTP File Transfer Protocol

9. SMS Short Messaging Service

10. MMS Multimedia Messaging Service


22/22

Nokia Solutions and NetworksP.O. Box 1FI-02022Finland

Visiting address:Karaportti 3, ESPOO, FinlandSwitchboard +358 71 400 4000

Product code C401-00649-B-201008-2-EN

©2013 Nokia Solutions and Networks. All rights reserved.

PublicNSN is a trademark of Nokia Solutions and Networks. Nokia is a registeredtrademark of Nokia Corporation. Other product names mentioned in thisdocument may be trademarks of their respective owners, and they arementioned for identification purposes only.

Ascom Unattended Trial Report Dubai

Documents

Transcript of Ascom Unattended Trial Report Dubai