ITU Workshop on “Performance, Quality of Service and Quality of Experience of Emerging Networks and Services”
(Athens, Greece, 7-8 September 2015)
Network Performance vs. Application Testing; findings and considerations
Henry Kwong
Senior Consultant, Omnitele [email protected]
© Omnitele Ltd. 2015 2
ag
end
a
1. Introduction
− About Omnitele
2. Beyond bitrates
Drive Testing vs. App Testing
3. OTT delivery chain
Understanding the QoS results
4. Key take-aways
Summary
© Omnitele Ltd. 2015 3
Maximised Experience, Minimised Cost
We provide consulting and expert services for telecom
operators and regulators in network strategy, design
and quality assurance. Our mission is to maximise
mobile subscriber quality of experience and minimise
operator network expenditures.
Delivering Omnitele Experience
The company was founded in 1988 to set up world’s
first GSM network. Since then we have completed over
1000 projects in over 80 countries around the globe.
Always delivering Omnitele Experience – a fact proven
by our long lasting client relationships.
International And Independent
Our headquarters is located in Helsinki, Finland. We
have local presence in the Netherlands. Our company
is owned by Finnish telecom investors and we are
independent of operator groups and network vendors.
The Omnitele Way
Our unique way of working sets us apart from the
competition and gives us a strong identity in the world
of telecommunications. We call this the Omnitele Way,
which means being Straightforward, Trusted and
Intelligent.
Omnitele 1-pager: company info in nutshell, can be used in all reports, proposals etc to add company info in concise fashion
www.omnitele.com
© Omnitele Ltd. 2015 4
content
content
title
Technology
Strategy
Design and
Optimisation
Audit and
Benchmark
Performance
Management
our services
maximised customer experience minimised network cost
© Omnitele Ltd. 2015 5
ag
end
a
1. Introduction
− About Omnitele
2. Beyond bitrates
Drive Testing vs. App Testing
3. OTT delivery chain
Understanding the QoS results
4. Key take-aways
Summary
© Omnitele Ltd. 2015 6
Monitoring End-User QoS in… …pre-HSPA networks
General user experience depends on two basic services
Voice Services Data Services
MOS, Success Rates, Drop Rates, Setup Time
Throughput
Generally available bitrates are not sufficient for data services. As throughput is the bottleneck, it becomes the
measure to define the QoS
© Omnitele Ltd. 2015 7
content
content
title
0
2
4
6
8
10
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20QoS:
WW
W p
age w
aitin
g t
ime [
s]
NW Performance: Bitrate [Mbit/s]
1000KB WWW Page Waiting Time vs. Bitrate
HSPA LTE
Customer Experience Challenge Bitrate is no longer the ultimate mobile
network quality indicator
© Omnitele Ltd. 2015 8
Customers mind service quality, not
network performance
72%
20%
7% 5% 5%
20.0 %
70.5 %
3.3 % 1.1 % 0.3 % 4.7 %
0%
20%
40%
60%
80%
General Browsing Video Streaming P2P Gaming VoIP M2M
Subscribers using (%) Share of data volume (%)Mobile data use cases and data volume
Customers desire smooth
experience, not average bitrate
WWW & Video dominate usage
© Omnitele Ltd. 2015 9
Competitive end-user QoS benchmark
OBJECTIVE: gain total visibility to mobile service experience & quality positioning
RESULT: a high quality benchmark report suitable for public distribution
METHODOLOGY: “be the customer” -approach to capture end-user experience
© Omnitele Ltd. 2015 10
Capturing the End-user Experience
HOW Terminals & test cases as per real subscriber behaviour
WHEN Test focus on peak hours, no empty network testing
WHERE Independently selected dense population areas & landmarks
© Omnitele Ltd. 2015 11
content
content
title Customer Experience Challenge
Vodafone higher bitrate, but KPN faster YouTube
0
10
20
30
40
T-Mobile Vodafone KPN
bitrate [Mbit/s]
big difference in NW performance…
0
0.5
1
1.5
2
T-Mobile Vodafone KPN
buffering time [s]
…thin margins in Quality of Service
Case study: Omnitele benchmark in Netherlands
© Omnitele Ltd. 2015 12
content
content
title Drive Testing vs. App Testing
App Testing
Insta
Estonia
FTP Downlink Average (Mbps)
Operator 1 43.8
Operator 2 24.3
Operator 3 20.3
Estonia
Dropbox DL Average (Mbps)
Operator 2 13.5
Operator 3 12.8
Operator 1 11.1
Case study – Omnitele benchmark in Estonia
Traditional Drive Testing
© Omnitele Ltd. 2015 13
content
content
title Drive Testing vs. App Testing
Case study – Omnitele benchmark in Estonia
Traditional Drive Testing App Testing
Insta
Estonia
LTE RB Utilisation Average (%)
Operator 1 91.2
Operator 2 80.0
Operator 3 72.8
Estonia
LTE RB Utilisation Average (%)
Operator 2 21.9
Operator 3 26.6
Operator 1 19.9
Results from Drive Testing and App Testing do not match at
all. Network usage patterns are completely different
© Omnitele Ltd. 2015 14
content
content
title
20 MHz bandwidth
Customer Experience Challenge all-IP PS network
300 Mbps 8x8 MIMO
adaptative modulation
OFDMA access scheme
Carrier Aggregation improved cell-edge performance
SC-FDMA uplink
spectral efficiency 15 bit/s/Hz
low latency
© Omnitele Ltd. 2015 15
content
content
title Customer Experience Challenge
20 MHz bandwidth
all-IP PS network
300 Mbps 8x8 MIMO
adaptative modulation
OFDMA access scheme
Carrier Aggregation improved cell-edge performance
SC-FDMA uplink
spectral efficiency 15 bit/s/Hz
low latency
news refreshing time?
facebook refreshing
time?
app download time?
it takes too long to upload a pic in Instagram…
© Omnitele Ltd. 2015 16
content
content
title Customer Experience Challenge
Where to focus your monitoring efforts?
Insta
Depends on the objectives that
you want to achieve
Network
Performance
End-User
QoS
© Omnitele Ltd. 2015 17
ag
end
a
1. Introduction
− About Omnitele
2. Beyond bitrates
Drive Testing vs. App Testing
3. OTT delivery chain
Understanding the QoS results
4. Key take-aways
Summary
© Omnitele Ltd. 2015 18
OTT Delivery Chain in Mobile Network environment
OTT Content
Delivery Platforms
Internet
Core
GGSN/PGW
Radio
Access
Insta
3G 4G
Request Response
Environment NOT controlled by service
provider
Environment CONTROLLED by service provider
© Omnitele Ltd. 2015 19
OTT Delivery Chain in Mobile Network environment
Internet
Core
GGSN/PGW
Radio
Access
Insta
3G 4G
Request
• Environment is not fully controlled by network operator
• OTT measurement results will not depend only on operator performance
• Special attention should be paid when analysing OTT performance results (Facebook
waiting times, YouTube buffering time, etc.)
• Multiple points of measurement within controlled environment would facilitate problem isolation
Response
OTT Content
Delivery Platforms
© Omnitele Ltd. 2015 20
ag
end
a
1. Introduction
− About Omnitele
2. Beyond bitrates
Drive Testing vs. App Testing
3. OTT delivery chain
Understanding the QoS results
4. Key take-aways
Summary
© Omnitele Ltd. 2015 21
Summary 1. Data throughput is no longer the unique KPI to take into
consideration when monitoring Mobile Data QoS
2. Network Performance vs. App Testing – depending on your
own objectives you may setup your measurements one way or
another
− Use of network resources differs and thus, results obtained
3. OTT delivery chain is not fully controlled by service provider,
thus results may be influenced by external factors
− Setting up different measurement points within controlled environment will
assist in troubleshooting potential issues
© Omnitele Ltd. 2015 22
content
content
content
title
title
can be 2 rows thank you for your attention!
Questions?
© Omnitele Ltd. 2015 23
Contact details Omnitele Ltd
Name Phone +358 9 695 991
Title Email [email protected]
Phone Website www.omnitele.com
Email Address Omnitele Ltd. Mäkitorpantie 3B P.O. Box 969 00101 Helsinki, Finland
We answer
Henry Kwong
Senior Consultant
+358 440 432 900
© Omnitele Ltd. 2015 24
maximised customer experience minimised network cost
w w w . o m n i t e l e . c o m
Top Related