MPT Proposal_Customer Centric KPIs and PoC Data

Metricell PoC: Measuring Customer Centric KPIs(1) Plug-and-Play Test Tools (2) Crowd-sourcing Agents

Version 3Q3 2016

© 2016 Metricell Limited

- 2 -


- 2 -

Samadhan Myanmar Limited Strength


- 3 -


- 3 - Local Partnars


- 4 -


- 4 - Sister Concern


- 5 -


- 5 - Metricell


- 6 -


- 6 - The Areas We are Serve


- 7 -


- 7 -


- 8 -


- 8 -


- 9 -


- 9 -


- 10 -


- 10 -


- 11 -


- 11 -


- 12 -


- 12 - The ChallengesNetwork originating KPIs and traditional testing equipment is not always able to provide an accurate picture of end-user experience. This creates two unique challenges:

Customers may complain about poor network experience, although network KPIs are showing satisfactory levels. In this scenario, customer perception directly conflicts with the available network data.

Network teams may not be able to efficiently determine a root-cause of the problem – leaving customers, and the Support/Sales Teams they complain to, frustrated.

CHAL

LEN

GE O

NE

Traditional benchmarking is often focussed on road based coverage analysis, including some voice and data services. However, end-user perception and experiences are sometimes overlooked.

Network teams do not have the necessary tools to accurately benchmark competitors’ performance in relation to user experience based KQIs e.g. Video Re-buffering (%).

1st

2nd

3rd 3rd

1st

2nd

Road Measurements / Drive Testing

End-User Measurements / Device Based Testing

CHAL

LEN

GE T

WO


- 13 -


- 13 - The Challenges – Contributing FactorsChallenge 1: Customer Complaints Vs. Network DataKey factors contributing to the first challenge includes:

1) Lack of data measured from the end-user perspective2) Limited, technical, network KPIs for serving Sites / Cells3) Lack of relevant in-building and location based quality indicators4) Resource and time constraints on technical investigation

Challenge 2: Traditional Drive Testing Vs. Device Based MeasurementsKey factors contributing to the second challenge includes:

5) Drive testing is focussed on planning and optimisation, not customer experience

2) Public facing benchmarking reports have shifted towards‘real world KQIs’ and ‘crowd-sourced data’, which cannot be measured using traditional testing tools

3) Data must be collected on a mass scale, with standardised testing parameters and sufficient details for technical follow-up


- 14 -


- 14 - The Solution

Livetime

‘Livetime’ is the name given to the geospatial intelligence platform.

This platform is allowing operators to continuously measure network quality from a subscriber perspective. We utilise white-label Apps (Native/API Plug-In), as well as plug-and-play test units in order to replicate and directly measure end-user experience.

AutoMobile: An Android based plug-and-play testing tool, for dynamic measurementsDeployment: Dynamic (Road, Rail, Walk).

Beacon: An Android based plug-and-play testing tool, for location based monitoringDeployment: Static (Distributors / Stores, Points-of-Interest, VIP / Customer Offices).

MobileCustomerCare: An on-device agent with a range of active and passive features that enable the Operator to collect end-user KQIs.


- 15 -


- 15 - The Solution

Livetime

‘Livetime’ is the name given to the geospatial intelligence platform.

This platform is allowing operators to continuously measure network quality from a subscriber perspective. We utilise white-label Apps (Native/API Plug-In), as well as plug-and-play test units in order to replicate and directly measure end-user experience.

Area Contributing Issue to Telenor Challenges Tool / Deployment Option Solution

Quality Lack of data measured from the end-user perspective MobileCustomerCare – Deployed to customers temporarily, who can perform active tests and report issues as they arise. Passive KQIs also collected.

End-User Data Customer Engagement

Quality Limited, technical, network KPIs for serving Sites / CellsMobileCustomerCare – Deployed to distributors / staff/ partners to crowd-source data from end-user / device perspective e.g. Real Coverage / Signalling

New DevicePerspective Mass Coverage Data

Quality Lack of relevant in-building and location based quality indicators

Beacon – Temporarily deployed by Sales / Support Team directly to customer office or problem location for on-site investigation.

Joint Investigation Customer Engagement

Quality Resource and time constraints on technical investigation MobileCustomerCare / Beacon – Gives Network Teams greater visibility. Complete View

Bench-marking

Drive testing is focussed on planning and optimisation, not customer experience

AutoMobile – Based on commercial Smartphones which replicates active voice calls, sms, data sessions, video streaming and more.

Android Based Replicating End-User

Bench-marking

Public facing benchmarking reports have shifted towards ‘real world KQIs’ and ‘crowd sourced data’, which cannot be measured using traditional testing tools

AutoMobile – Focussed on customer centric Key Quality Indicators, including but not limited to: Service Availability / Unavailability (%); CSSR / Drop Call Rate (%); Web Page Loading Times (ms); Video Re-buffering (%); and more.

Customer Centric ‘Real World KQIs’ Device Measurements

Bench-marking

Data must be collected on a mass scale, with standardised testing parameters and sufficient details for technical follow-up

AutoMobile – The low cost units can be deployed to non-technical staff or external parties (e.g. Taxi’s), with all test setting remotely managed OTA.

Highly Scalable Fast / Easy Deployment


- 16 -


- 16 - Tools UtilisedBeacon(In-Building & Service Assurance)

Beacon is a remote Plug-and-Play Monitoring Unit. Once deployed, it continuously conducts network tests to ensure network Quality of Service (QoS) is maintained 24 hours a day, 7 days a week.

With automated reporting and KPI based alarming, it can be deployed at: retail stores, corporate office, points-of-interest and at new site locations.

AutoMobile(In-Vehicle & Benchmarking)

AutoMobile is a remote Plug-and-Play Test Unit. Deployed as a single device or in the form of a benchmarking suitcase; this tool allows Telenor to collect mass, near real-time network performance data.

As well as being able to benchmark against other Opcos, these units allow us to collect metrics based on true Data, Voice and Service customer experience.

Livetime

MobileCustomerCare (MCC)(Smartphone App)

MCC has a range of features, including active and passive testing.

Our focus is on leveraging the internal user community and sales distributors, by creating an army of crowdsouring network quality testers: • Speed Testing• Web Browsing Testing• Video Streaming Testing• Qualitative Employee Feedback and more


- 17 -


- 17 - Example KQIs Captured (Experience Critical)

Average Call Duration (s)

PASSIVE Voice (2G/3G)

KQIs

Call Success Rate (%)Call Setup Time (ms)Dropped Calls (Median -dBm)Dropped Call Rate (%)Most Used Voice CellWorst Voice CellCSFB Average DurationCSFR

Average Cell Handovers

Data Availability (%)

PASSIVE Data (2G/3G)

KQIs

Data Unaailability (%)Data Problems (Median -dBm)Data Amount Uploaded (Kb)Data Amount Downloaded (Kb)Data Technology Connection Time (%)Data Technology Availability Time (%)Application Download Amount (Kb)Application Upload Amount (Kb)Data Sessions Per Application (Count)Approx Speed of Data Session Per App (Mbps)Main Serving Technology per App (2G/3G)Most Used Data Cell (2G/3G)Worst Data CellWi-Fi Connection Time (hh:mm:ss)

Downlink Data Throughput

ACTIVE or PASSIVE Speed Test

KQIs

Uplink Data ThroughputDuration (s)Max Downlink Data Rate (Mbps)Max Uplink Data Rate (Mbps)Average Downlink Data Rate (Mbps)Average Uplink Data Rate (Mbps)Jitter

Latency (Ping Test)

Ping

ACTIVEHTTP Browsing

KQIs

Page Download Time (ms)Page Download SizeURL

Connection Setup Time (ms)

ACTIVE(144p, 240p, 360p, 480p, 720p, 1080p)

Video StreamingKQIs

Min Throughput (Kbps)Average Throughput (Mbps)Max Throughput (Mbps)Data Volume (Mb)Test Duration (s)Time to Start Play (ms)Min time buffering (ms)Number of Rebuffs (Count)

Initial Buffer Time (s)

ACTIVEYouTube Test

KQIs

Duration (s)Total Buffering Time (s)Rebuffer Count (Count)Initial Buffer SizeSetup Time (ms)

URL

Application Package Name (e.g. YouTube)

PASSIVEData Sessions

KQIs

Download Duration (ms) / SizeData Session Download SpeedUpload Duration (s) / SizeData Session Upload Speed

Service Availability (%)

PASSIVE Service

KQIs

Service Unavailability (%)Emergency Only Service (%)Service Unavailable Time (s)Emergency Only Service Time (s)Service State Connection Duration (s)Cell Connection Duration (s)Technology Type Connection Duration (s)Technology Unavailable Duration (s)Service Problems (Median -dBm)Most Used Service CellSignal, Service Problems (Median -dBm)Foreign Network Connection Time (s)Roaming Network OperatorTime on Roaming Network (s) MCC Changes

Battery Discharge Rate (%/h)

Other

Battery Charging Time (s)First installed DateData Last Seen DateScreen On TimeConnected AzimuthLocation Time OffsetTotal SMSSent SMS

VOICE

Performance Events

Successful CallDropped CallProblem CallCall Setup FailureDATANo DataSERVICENo ServiceEmergency OnlyRoaming


- 18 -


- 18 -

The next step is to run a Proof of Concept, focussed on Downtown Yangon. During the Proof of Concept, Metricell will share our experiences with other Operators, including Best Practices and Use Cases. The focus will be on overcoming Telenor’s current challenges.

Next Steps & Responsibilities Matrix

Task Metricell Auspicious Telenor

Supporting DataA) Site database, including 2G, 3G, 4G info of: Long/lat e.g. (51.064014 -0.334958), Cell name, B) Site name, RNC, CI, LAC, RAC, HBA, Azimuth, Rncld, PriScrCode, UARFCN, Mtilt, Etilt, and E-CGI for 4G.C) Geographic Information System polygons e.g. neighbourhoods, regions, engineering zones.

---

---

Responsible Responsible Responsible

Software Deliverablesi. Backend Platform for communicating with, configuring and analysing all device agent data: Livetime ii. Customer Experience On-Device Agent for ‘Last Mile Monitoring’: MobileCustomerCareiii. In-Building ‘Service Quality Monitoring’ Testing Solution: Beaconiv. Transportable ‘Customer Orientated’ Auto-Drive Testing Solution: AutoMobile

Responsible Responsible Responsible Responsible

SupportSupportSupportSupport

----

Operator provides unlimited voice/data testing SIMs (x4). Test settings and information needed includes: (1) Free-phone Voice Test Number, (2) URLs for Speed Tests, (3) Video File for Streaming and YouTube Tests,(4) Preferred Web Browsing Test URLs, (5) Translations (if needed)

- - Responsible

Project Access - Livetime has been designed as end-to-end platform, which means that we would expect a wide range of departments to be active users of the tool. We are happy to administer secure user access on your behalf and can provide you a report containing usage statistics at the end of the trial.

Support Support Responsible

Mobile App Distribution - Metricell will provide a download URL from where iOS and Android users will be able to install the application. The more people that install the app, the easier it will be to understand the benefits that the technology brings. We would expect an ordinary Pilot to achieve in excess of 1,000 installs.

Support - Responsible

Data Collection - The plug-and-play test devices will be provided on a temporary basis for the purpose of the PoC. Telenor will be responsible for deploying these Units and collecting data to test the intended Use Cases. Support Support Responsible

Local Support - Front-line Support will be provided, with both Metricell and Auspicious paying key roles. Support Responsible -

Project Management, Training and Data Analytics Support Responsible Responsible -


- 19 -

PoC Summary Report1/4 Month 60-115 MCC Users3 AutoMobiles1 Beacons

http://www.metricell.com/


- 20 -

‘Crowd-sourced’ MCC App Data

MobileCustomerCare (MCC)(Smartphone App)

MCC has a range of features, including active and passive testing.

Our focus is on leveraging the internal user community and sales distributors, by creating an army of crowdsouring network quality testers: • Speed Testing• Web Browsing Testing• Video Streaming Testing• Qualitative Employee Feedback and more



- 21 -


- 21 - MCC: Manual Problem Reports


- 22 -


- 22 - MCC: VOICE, SMS & SERVICE Experience


- 23 -


- 23 - MCC: DATA Experience – Speed

MSISDN Count Average D/L Rate (Mb/s)2,219 0.261,792 0.14

891 0.27

Technology CountAverage Signal (dBm)

Average Downlink Rate Avg

(Mb/s)

Average Uplink

Rate Avg (Mb/s)

Average Ping (ms)

Average Jitter (ms)

EDGE 3 -83.00 0.00 0.61 417.00 131.33GPRS 1 -75.00 0.00 0.00 35.00 7.00HSDPA 99 -87.33 2.99 1.36 63.12 6.19HSPA 81 -74.49 0.18 0.27 18.10 13.31HSPA+ 269 -71.70 3.48 2.22 86.23 60.38HSUPA 4 -72.00 1.42 0.39 17.50 31.75LTE 4 -92.25 7.78 7.21 48.25 5.50UMTS 1 -81.00 0.00 0.00 0.00 0.00Grand Total 462 -72.98 2.78 1.70 70.26 40.00

Cell CountAverage Signal (dBm)


(Mb/s)

Average Uplink

Rate Avg (Mb/s)

Average Ping (ms)

Average Jitter (ms)

15 -65.93 3.81 2.59 49.93 25.206 -65.33 8.35 1.20 50.00 18.836 -64.00 7.58 3.02 41.33 11.176 -59.33 5.47 2.24 41.17 29.506 -60.33 3.31 1.99 50.50 24.174 -59.50 5.51 2.70 64.25 92.254 -89.00 2.00 1.00 117.25 74.004 -83.00 1.94 2.02 286.75 130.004 -69.50 3.68 2.39 114.50 140.754 -77.50 8.98 2.08 85.75 21.75

MSISDN CountAverage Signal (dBm)


(Mb/s)

Average Uplink

Rate Avg (Mb/s)

Average Ping (ms)

Average Jitter (ms)

60 -53.97 0.38 1.66 47.77 47.5538 -74.74 2.92 1.55 102.37 90.3915 N/A 3.56 1.15 75.53 8.5314 N/A 6.51 1.53 51.00 5.3613 -80.23 4.13 1.32 19.69 3.1511 N/A 0.70 1.17 103.36 6.55

8 -78.75 4.66 3.94 34.88 18.008 -79.50 2.64 1.90 102.00 62.508 -53.00 4.09 2.17 54.63 7.137 -62.71 2.94 3.90 76.14 80.71

Handset Model CountAverage Signal (dBm)


(Mb/s)

Average Uplink

Rate Avg (Mb/s)

Average Ping (ms)

Average Jitter (ms)

60 -53.97 0.38 1.66 47.77 47.5532 -78.66 1.01 0.89 53.03 33.5328 -84.00 2.44 1.32 64.71 8.7125 3.95 1.37 74.04 5.8820 -86.30 4.75 1.72 82.20 33.2515 -76.60 3.23 1.61 46.40 16.2015 3.56 1.15 75.53 8.5313 -80.23 4.13 1.32 19.69 3.1511 -76.64 1.99 1.03 42.00 16.9110 -72.80 1.58 3.39 92.40 77.70

Spee

d b

y Te

chno

logy

Spee

d b

y Ce

ll

Spee

d b

y Ha

ndse

t Mod

elSp

eed

by

MSI

SDN

Data

Re

mov

ed


- 24 -


- 24 - MCC: DATA Experience – Browsing & Apps

All Technologies / Test URL Count

Sum Data Downloaded (Mb)

Average Data Downloaded (Mb)

Average Setup Time (s)

Average Load Time (s)

https://www.facebook.com/ 3295 6781.80 2.06 0.56 9.24https://twitter.com/ 3295 3990.41 1.21 0.57 8.81https://www.google.com 3295 5650.44 1.71 0.68 7.90http://www.house.com.mm 45 283.05 6.29 0.12 4.51http://www.telenor.com.mm 45 1222.36 27.16 0.16 4.49http://www.shop.com.mm 45 528.13 11.74 0.15 7.34http://www.mmtimes.com 34 103.62 3.05 0.31 2.39http://www.motors.com.mm/ 34 161.43 4.75 0.10 5.94http://www.motors.com.mm 11 70.85 6.44 0.02 7.85http://www.mmtimes.com 11 32.28 2.93 0.19 2.69Grand Total 10110 18824.37 1.86 0.59 8.57

Test URL CountSum Data Downloaded (Mb)




https://twitter.com/ 199 133.29 0.67 0.36 32.65https://www.google.com 199 225.49 1.13 0.26 28.91https://www.facebook.com/ 199 177.66 0.89 0.37 33.45Grand Total 597 536.44 0.90 0.33 31.67





https://www.facebook.com/ 3023 6604.14 2.18 0.58 7.87https://twitter.com/ 3023 3857.12 1.28 0.60 7.45https://www.google.com 3023 5424.95 1.79 0.73 6.71http://www.house.com.mm 42 272.36 6.48 0.12 4.64http://www.telenor.com.mm 42 1101.27 26.22 0.16 4.31http://www.shop.com.mm 42 485.64 11.56 0.15 7.23http://www.mmtimes.com 31 103.50 3.34 0.29 2.54http://www.motors.com.mm/ 31 146.70 4.73 0.11 6.05http://www.motors.com.mm 11 70.85 6.44 0.02 7.85http://www.mmtimes.com 11 32.28 2.93 0.19 2.69Grand Total 9279 18098.81 1.95 0.62 7.29





http://www.house.com.mm 2 7.66 3.83 0.16 2.73http://www.motors.com.mm/ 2 9.99 5.00 0.04 5.09http://www.telenor.com.mm 2 80.56 40.28 0.18 7.02http://www.mmtimes.com 2 0.08 0.04 0.22 0.77http://www.shop.com.mm 2 32.00 16.00 0.16 8.45Grand Total 10 130.29 10.02 0.12 3.70

Browsing Experience – 2G/3G/4G2G

3G

4G

Application(Android Only)

SampleCount

Sum ofDownloadSize

Sum ofDownloadDuration

AverageDownloadRate (Mbps)

Sum ofUploadSize

Sum ofUploadDuration

Average ofUploadRate (Mbps)

Android Email 240 2,538,012 687 0.03 1,657,901 629 0.01Viber 4,433 510,225,938 13,170 0.13 267,802,232 12,844 0.03Google Chrome 1,067 715,564,675 3,492 1.40 23,286,568 3,524 0.05Google Maps 201 9,017,637 295 0.13 1,177,727 308 0.02Samsung Internet 103 164,242,352 845 1.13 4,434,810 859 0.05WhatsApp 151 116,126,547 521 0.71 1,474,943 520 0.01YouTube 902 2,193,900,382 5,371 1.64 55,491,326 5,480 0.05Grand Total 7,201 3,713,726,427 24,539 0.53 355,734,789 24,296 0.03

App Experience – 2G/3G/4G


- 25 -

‘Benchmarking’ AutoMobile Data

AutoMobile(In-Vehicle & Benchmarking)

AutoMobile is a remote Plug-and-Play Test Unit. Deployed as a single device or in the form of a benchmarking suitcase; this tool allows Telenor to collect mass, near real-time network performance data.

As well as being able to benchmark against other Opcos, these units allow us to collect metrics based on true Data, Voice and Service customer experience.



- 26 -


- 26 - AutoMobile: SERVICE Benchmarking

Average Signal MPT Ooredoo TelenorAverage RxLev (dBm) - - -Average RSCP (dBm) -62.19 -59.45 -59.26Average RSRP (dBm) - - -

UM

TS R

SCP

Dist

ributi

on


- 27 -


- 27 -

1st for indoor coverage 2nd

for indoor coverage3rd for indoor coverage

AutoMobile: SERVICE ThresholdsCOVERAGE THRESHOLDS (dBm)Cov Thersholds dBm GSM GSM2 UMTS UMTS2 LTE LTE2 < > < > < >Indoor 0.0 -80.0 0.0 -75.0 0.0 -96.0Outdoor -80.0 -100.0 -75.0 -106.0 -96.0 -110.0

UMTS MPT Ooredoo TelenorIndoor 88.4% 95% 93.2%Outdoor 11.5% 5% 6.8%Fringe 0.1% 0.0% 0.0%

95.0%

5.0%

Ooredoo UMTS Service

IndoorOutdoorFringe

88.4%

11.5% 0.1%

MPT UMTS Service

IndoorOutdoorFringe

93.2%

6.8%

Telenor UMTS Service

IndoorOutdoorFringe


- 28 -


- 28 - AutoMobile: VOICE & DATA Summary

Voic

e KP

I Ben

chm

arki

ng

Data

KPI

Ben

chm

arki

ngCall Set Up

failuresDropped Calls Successful

Calls

0

10

20

30

40

50

60

70

MPTOoredooTelenor

Total Avg Download Rate0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

3G Download Rate

MPTOoredooTelenor

Total Avg Upload Rate (Mbps)

0.00

0.50

1.00

1.50

2.00

2.50

3G Upload Rate

MPTOoredooTelenor


- 29 -

‘Static In-Building’ Beacon Data

Beacon(In-Building & Service Assurance)

Beacon is a remote Plug-and-Play Monitoring Unit. Once deployed, it continuously conducts network tests to ensure network Quality of Service (QoS) is maintained 24 hours a day, 7 days a week.

With automated reporting and KPI based alarming, it can be deployed at: retail stores, corporate office, points-of-interest and at new site locations.



- 30 -


- 30 - Test Beacon 1: Overview of Experience


- 31 -

RFP Requirements



- 32 -


- 32 - Requirements


- 33 - Requirements


- 34 -

Local Partner (Myanmar)

Md. Joynal AbadenCoordination

Email: [email protected] Tel: +95 9969702355Skype: joynalabaden www.samadhanmm.com

Metricell AsiaLuke AlexanderGeneral Manager - Asia Pacific

Email: [email protected] Tel: +60 1129 023 249 Skype: lukehalexander9 www.metricell.com

Metricell UKGroup HeadquartersSales Enquiries

Email: [email protected] Tel: +44 1403 251 494Alt. Tel: +44 1403 750 151www.metricell.co.uk

Would you like to know more?

MPT Proposal_Customer Centric KPIs and PoC Data

Documents

Transcript of MPT Proposal_Customer Centric KPIs and PoC Data