IPTV Forum Eastern Europe 2009

Ensuring QoS/QoE throughout the network

Andrey AlekseyevDirector IP service networkCOMSTAR-UTS

Wednesday, October 21, 2009

The company

Leading supplier of integrated telecom solutions – Russia, CIS50.9% of COMSTAR-UTS owned by MTS, Russia’s top mobile operatorSubsidiaries in Ukraine, ArmeniaListed on the London Stock Exchange (LSE: CMST)Largest broadband customer base in MoscowOwns 56% of MGTS (Moscow PSTN), with 3.6 million of telephony subscribersOperates its own multi-service network across Russia and EuropeThe first Russian operator to launch commercial IPTV services in 2005Number of subscribers (Q2 2009):791,000 broadband Internet / 133,000 IPTV subscribers in Moscow48,000 corporate customers in Moscow

324,000 broadband customers / 1,953,000 pay-TV subscribers in the regions

Wednesday, October 21, 2009Hello, my name is Andrey Alekseyev and I work as the Director IP service network for COMSTAR-United TeleSystems. First of all, Iʼd like to apologize to the audience for not being able to attend this conference, thatʼs because I had run into some unfortunate complications with my European visa. About the company - COMSTAR-United TeleSystems is one of the biggest telecommunication companies in Russia providing a wide range of services to residential customers across Russia. Main customer base of COMSTAR-UTS is in Moscow where in tandem with MGTS, the local PSTN company belonging to the same group of companies, we provide broadband Internet services and IPTV. We were the first Russian operator to launch IPTV services in late 2005 and our IPTV customer base grew quickly.

The network

AS8359 – fully converged, IPv6 enabled IP/MPLS networkCore nodes in Moscow and Saint Petersburg (Russia)Main backbone nodes/PoP’s in Moscow, Saint Petersburg, Rostov-on-Don, Kiev, Stockholm, Frankfurt (am Main), Amsterdam, London, Paris, New York, Los AngelesRedundant n*10GE inter/intra-city linksBest of breed, proven network hardwareTens of gigabits of Internet and media traffic in the networkAggregating Internet traffic at multiple Internet ExchangesAccess network in Moscow built and operated by MGTS (local PSTN)Access based mostly on ADSL in Moscow and Ethernet in the regions

Wednesday, October 21, 2009About the network. COMSTAR-UTS has built its own powerful IP/MPLS network over the past 10 years. Obviously, being an ISP we should continuously develop it to meet the customersʼ expectations on the quality of services. Thus we typically utilize the latest technology and state-of-the art solutions. We have points of presence across Russia, Europe and US and aggregate lots of traffic. Our customers consume as much as they can, often just plainly saturating their ports (but not our uplinks!) with 24 hours downloads. In regards to the access technologies we are mostly ADSL with a steady growth of percentage of Ethernet-based networks weʼre building across Russia. On the slide you may find more specific information about autonomous system 8359 which is our flagship AS.

Moscow network – key elements

Over 200 access nodes (MGTS central offices), redundant 10GE to every nodeOver 4000 DSLAMs installedADSL access 6-20 Mbps80% of subscribers at 6 Mbps, 20% at 10 Mbps and upInternet traffic and IPTV unicast isolated and end-to-end prioritized in the network

Backbone Core Access

CO

CO

CO

n*10G

n*10G

10G

10G

IP/MPLS MGTSCOMSTAR

Wednesday, October 21, 2009Moscow part of the network is probably the most complicated one. Thatʼs because in Moscow to connect our subscribers we should have a PoP/node at every central office in the city. Whatʼs quite unique about Moscow is we have over 200 of central offices in the city. Network topology is mostly ring-based with a double-star in the heart of it. On this slide I put a brief outline of a single ring segment connected to the network core. Basically, we have every central office connected to the core via redundant 10GigE links. We have over 4000 of DSLAMs installed in the network with the typical access speed of 6 Megabits per second. It is essential for a network like this to utilize all the means available in the core and access equipment to ensure most strict policies in regards to quality of service. We isolated Internet and IPTV traffic from each other throughout the entire network all the way down to subscriberʼs CPE. We apply vendor specific solutions like prioritized queues and Differentiated Services architecture to achieve QoS in the network.

Challenges in the network

Several administrative domains (MGTS access, COMSTAR-UTS core/backbone/transport)ADSL infrastructure based on traditional telephony copper lines to homes passedVast majority of DSLAMs are at CO’s, typical distance to residential customer around 1.5 kmOld copper infrastructure introduces erratic behaviour to the service with high demand on qualityNo use to monitor service close to DSLAM’s – most problematic is the subscriber line (and CPE)CPE’s unmanaged from the very beginning, no enforcement on CPE model/typeTrue end-to-end monitoring solution required for IPTVWithout ensuring proactive QoE measurements service delivery and support turns into a nightmare

Wednesday, October 21, 2009We have many challenges to solve on the network side. The toughest ones are probably the quality of the copper lines and the fact we have deliberately chosen unmanaged CPEʼs from the very beginning. The fact we have a few companies inside the group working closely to deliver the services also attributes to the complexity of the service delivery and service support chains. To do things right we definitely should have adopted a number of process models, procedures and tools to monitor, analyze and solve problems.

IPTV infrastructure elements

Key IPTV infrastructure elements very similar to the vast majority of deploymentsMajor differences in streaming bitrates, VOD setup, middleware

Headend (receivers, encoders)

Network access/aggregation/load

balancingCore network

MiddlewareBackbone

Content encryption/DRM

Network access/aggregation/load

balancing

Access network

VOD central node

VoDCO ...

VoDCO B

VoDCO A

VOD edge nodesSubscriber

Subscriber

Content creation/ingest

Wednesday, October 21, 2009In regards to the IPTV infrastructure we donʼt have anything unusual in the architecture. We have antennas, satellite receivers, encoders, middleware and accompanying elements just as every other operator on the market does. However, in our opinion what differentiates our setup now is the current middleware platform and also probably our VOD infrastructure enhancements.

Challenges in the headend

Full re-coding for most channels and VOD titles to ensure predictable quality on the majority of ADSL linesOver 100 TV channels plus 70 radio channels to broadcastSDI input to encoders/streamersMPEG2 is still the main format with the legacy/new STB’s ratio of 95%CBR due to legacy issues (primarily because of the initial CAS/DRM setup)MPEG4 gradually introduced, requirement to maintain both MPEG2 and MPEG4 to ensure compatibility with legacy STB’sMost channels are from satellite (already compressed, with the obvious complications when re-coding to MPEG2 CBR)Even the channels produced locally suffer from compression issuesA whole variety of input formats for VOD content ingest (MPEG PS files, DVD’s, HDCAM etc.)

Wednesday, October 21, 2009Taking a closer look at our headend infrastructure you may find a number of factors which contribute to the complexity of ensuring the necessary quality of experience for the subscribers. First of all, being mostly an ADSL ISP we need to provide certain guarantees on the quality of the streamed content. And we canʼt do this unless we re-code our content before streaming. We have all of our receivers outputting video through SDI to the encoding part. We are also still on MPEG2 with the MPEG4 format gradually introduced with the increasing number of the new models of set-top-boxes. We also struggle with a number of problems related to the VOD content management and especially with the whole variety of formats we are getting our VOD content in. It is not a surprise that we have understood that ensuring QoE is not that easy but also that it is inevitable thing to do.

Headend – encoding

Standard definition videoMPEG2Profile MP@MLAround 2.5s compression delayVideo bitrate 3.5 Mbps, IP stream 4 Mbps, ADSL bandwidth around 4.5 MbpsResolution 544x576Constant IBBBP GOP structure and length

MPEG4/H.264Profile Main@Level3Around 2s compression delayVideo bitrate 1.8 Mbps, IP stream 2 Mbps, ADSL bandwidth around 2.5 MbpsResolution 544x576IBBBP GOP structure, GOP length 24 (+adaptive GOP)Hierarchical B

High definition videoMPEG4/H.264High@Level4 profileAround 2s compression delayVideo bitrate 7.3 Mbps, IP stream 8 MbpsResolution 1440x1080IBBBP GOP structure, GOP length 48 (adaptive GOP)

Sound streamed w/o re-coding, same bitrate as ingest. Typically MPEG1 Layer2 96-256 Kbps for stereo, AC3 for AC3

Wednesday, October 21, 2009This slide is probably purely technical. I will leave the thorough reading to the curious audience, maybe this one will be helpful for comparison to the setups you employ. Just a few things worth mentioning. We use a short GOP structure for MPEG2 and this kind of helps us to avoid a number or problems related to the channel change timings. We were once approached by then a major IPTV vendor soliciting their solution for fast channel change. The sales person actually failed to explain us how to beat the channel change time we already had at that time. For the high definition content we have to stick with around 8 Megabits per second of ip streaming mostly because of the limitations in the access technology.

Headend – evolution, findings

First deployed in 2004-2005 (MPEG2 only, receiver-encoder/streamer setup)Fully revamped in 2007-2008 (MPEG2/MPEG4/PiP of the same channel from a single encoder/streamer)Multiple vendor solutions tested before choosing a new setup/equipmentContrary to what was expected from testing, the result turned out to be that default factory settings for encoding are most effective onesReasons for the above

most of the channels contain very different content (for instance, old color movie followed by an old black and white movie, followed by a sport programme)most of the channels we receive are already compressed – often not very well, even if not coming from a satelliteSD channels/VOD content, which is still the biggest part of the streamed content come in 544/576 resolution, not much additional processing to dostatic picture enhancement options like noise reduction fail to maintain sustained level of acceptable video quality in a situation when channel content changes frequently and in unpredictable mannerwith so much of an old/edited/re-mastered content on most channels any specific picture enhancement options might actually degrade video/audio quality in comparison to a well-balanced factory set of options for encoding

Wednesday, October 21, 2009We never stopped to improve things, though. During 2007-2008 we actually completely revamped the headend encoding. We changed all of the bulky streaming encoders we had to the most recent chassis which currently does simultaneous streaming of MPEG2, MPEG4 and QCIF Picture-in-picture copies of every channel. What is quite funny and what may seem to be an unwise thing to do, is we still use the factory default settings (well, mostly) for almost all of the channels streamed. The reasons are simple. We have so different quality of the content - we just cannot quite match all of the enhancement features we have in the hardware to this content. We have tried a variety of the hardware from different vendors only to find out that most of the time any enhancement feature we looked into, fails to qualify as the cure for a mid to low quality content. There are certain specifics about locally available content which the vendors just cannot fix. Or maybe they decided to fix it with the factory defaults :)

VOD load balancing

Fault-tolerant central node at the headend premisesEdge nodes across the cityRTSP dynamically redirected to edge nodes based on the routing informationSubscribers get content from the nearest edge node – unicast video traffic localized in the access network

VODcentral node

VODedge node

Core

AccessVOD

edge node

Wednesday, October 21, 2009In respect to what we have done differently in the VOD part there is one thing which is particularly worth to mention. This one is about fault-tolerance and scalability. As I said the access network in Moscow is huge and we didnʼt quite like the idea of making the IPTV unicast flow through the entire network to reach the subscribers. Instead we chose a decentralized model with a dozen of VOD nodes located at the edge of the network. To make it fault-tolerant we invented our own algorithm to ensure the subscribers will be getting the content from the closest node. It is based on the BGP prefixes information and it is dynamic. This way we allow our neighbour company, MGTS, to manage the load on the access network without our intervention and we also provide enough means to fail over the damaged node. The redirector doing smart load distribution is actually part of the middleware.

Fault and performance management

With the variety of services provided obviously we need an industry standard solution for fault managementDeployed fault management solution allows to monitor network and service issues like link/router failures, application failures, IPTV infrastructure outages etc. This includes, for instance, monitoring service dependancies and multicast trees end-to-end with the root cause analysisWe monitor everything! Moreover, every other day we typically find more things to monitor :)We have our engineering development/operations people for both IP/MPLS network and IPTV completely freed from day-to-day monitoring activities – the work is done by our NOC staff who do 24x7x365 monitoring of the network and the headendWe have tools to free people from hard and boring tasks and keep them entertained with the new technologiesFor performance management we currently use a number of open source tools to mostly gather historical data and visualize it for reference purposes, planning and trend analysisFor the entire network we also use a separate real-time traffic analyzerWe supply all new CPE with TR-069 software enabled and can now monitor them

Wednesday, October 21, 2009With these many complexities, issues and sophistication in the network and IPTV infrastructure, we have been continuously evolving in our monitoring solutions. We actually use now the best fault management software available. And we make the most of it. We consider it very important to develop our people and to free them from abundant manual operations in regards to monitoring. On the other side we donʼt want them to be just plain watchers. We always try to provide them with the tools which are both fun to study and operate and at the same time convenient, scalable and robust. We still have lots of work to do in regards to the performance management, though. Also worth mentioning is the fact we try to use all of the mechanisms available to manage and control every element in the service delivery chain. Thatʼs why we reconsidered the paradigm of unmanaged CPEʼs and implemented TR-069 solution with all of the new CPEʼs shipped TR-069 enabled.

Headend monitoring

Industry standard solutions for fault and performance management are good though still not sufficient to monitor problems specific to IPTV

We complement it with a set of additional IPTV-related tools

We do additional monitoring/management of the headend equipment with the proprietary vendor software

We monitor MDI (DF and MLR) using hardware probes at the headend and inside the access network to keep a clear distinction between areas of responsibility in different administrative domains and to have a single metric to compare when there is a service degradation/loss

We intentionally keep MDI monitoring simple yet efficient – it should be easily recognized and accepted by the NOC engineers, it should work around the clock, it has to do clear visualization

With the deployed MDI probes we can partially detect even more complicated problems which stem from the receiving part due to weather conditions etc. (which are then reflected in the streams output from the headend)

Wednesday, October 21, 2009In regards to monitoring IPTV infrastructure it is essential to complement the well-known and proven fault and performance management solutions with a certain number of carefully selected instruments. We picked up a solution for MDI monitoring which we found to be the most effective for our NOC engineers. These are essentially MDI probes which we installed in the network and since that time analyzing daily MDI reports is a routine. What helps us a lot to resolve a certain number of complicated issues in the process of cooperation with our neighbour engineers is the fact we clearly separate the responsibility of keeping the service intact. And MDI probes contribute immensely.

MDI monitoring

We have probes deployed across headend infrastructure and access network for MDI monitoring

Access(DSLAM's,

BRAS)

Edge node

FE

GE

GE

Probe

VOD edge node

MDI probe

EncoderEncoder

Headend

Access (DSLAM's,

BRAS)

Edge node

Sub ProbeField

engineer

Controller

NOC

FE

n*GE

CORE

GRTUnicast

vpn

Multicast

Unicast mgmt

Wednesday, October 21, 2009This is the illustration of the process of MDI monitoring in our network. All of the key elements are seen here and you may envision how we do that.

Set-top-box monitoring

We never stop monitoring so we decided to go even further in order to be able to drill down to the source of the problems by doing service monitoring on the STB sideTraditional approach of monitoring IPTV services at DSLAM port doesn’t work for us – our problems are mostly at the very end of the service delivery chainWe found an innovative and clever solution on the market to do what we need – a tiny monitoring agent embedded inside STB and sending live statistics data in the real-time to the centralized server softwareAllows us to collect and analyze the following data per STB in the real-time:

Performance data (memory, CPU, network utilization)User actions (navigation, channel change etc.)STB actions in response to user activity (when actual IGMP join occurred, what came in response to RTSP request, what are the errors etc.)

Allows us to do network wide metrics analysis (e.g. actual mean channel change time across the entire subs base)Does the correlation between the infrastructure and the data collected from STB’s – can visualize what CO’s are prone to service degradation and whyDoes a variety of historical reports, including proactive suggestions (e.g. jeopardy of churn)

Wednesday, October 21, 2009A year ago we accidentally found a solution weʼve been thinking of, to monitor things even further. The problem with our service model is we have lots of trouble tickets related to the quality of the ADSL line and/or CPE or set-top-box configuration. The solution is actually simple - we now monitor set-top-boxes too! A tiny piece of software inside all of our set-top-boxes gather all the necessary data to analyze. What is important, the data is available to us in the real-time. We can select any of our STBʼs and check if there is anything wrong with it. Or was. We can now do fast correlation between what we get from the other monitoring tools and what we see from the subscribers side. Basically, we now have complete picture, seeing the network as the operator and also as one big subscriber as well. A flexible reporting is available of course. We can now do day-to-day analysis of the things like overall service quality, channel popularity and most important quality indicators as well.

Set-top-box monitoring

Live channel status allows to quickly evaluate how well STB’s in the field receive multicast/VOD

Network history tracks outages at CO’s

Helps to correlate monitoring data

Intuitive to use

Wednesday, October 21, 2009A couple of screenshots to illustrate how it looks like to check the live channel status and outages tracking.

Set-top-box monitoring

STB summary page allows to watch a particular STB state/behaviour in time

KPI statistics provide

useful information on indicators like channel change times and similar figures

Wednesday, October 21, 2009This ones are all about drilling down to the gory details of set-top-box statistics and also, overall kpi statistics.

Set-top-box monitoring

Event logs allow to check exactly what an STB was doing

Proven reporting mechanisms provide historical overview and easily read status information

Reports help manage content too!

Easily integrated with the existing

solutions (OSS/BSS and middleware)

Wednesday, October 21, 2009We can analyze it more thorougly with the detailed log of what was going on inside an STB. Like when it sent IGMP join, when the stream started to flow to be decoded and all the things like that. The reports are based on an industry standard tools and are clearly structured and ready to use.

IPTV service performance – middleware

Transition from a basic web-based application to a feature-rich UI client

Dramatic boost of interface performance

Ensures great usability even with

the navigation pane minimized

Provides the same experience on

legacy and new STB’s

Does bells and whistles like PiP (QCIF)

even on legacy STB’s

Wednesday, October 21, 2009Last but not least come the middleware enhancements we did in the past few years. In the beginning we had only a basic web-based client in the set-top-box and in two years it became really obsolete. We opted to change and in close cooperation with a local company a new middleware appeared. It is actually the solution we needed. It changed the usability and interface performance so drastically we were really afraid it will be a shock to our subscribers with the migration. Fortunately it wasnʼt. Of course we got complaints, but then again, an operator always gets some. What was really surprising we managed to migrate around 150,000 of STBʼs in two months period completely transparently. Now with the new middleware in place weʼre free to introduce new services (which we already do) and the subscribers really experience a better quality because their navigation is now extremely fast, they have more intuitive interface, they have lots of more things to come out of their TV screens too.Well, thatʼs about all I had to say today. To summarize: QoE becomes really important these days and lots of tools should be utilized wisely. Often, revamping of the infrastructure is unavoidable too. Many thanks to the audience for the attention to this presentation. Iʼm ready for the questions too if any of you have them. Iʼm generally reachable via e-mail and occasionally I visit Europe too.

7

Products and partners

Wednesday, October 21, 2009Here are the logos of our partners as the courtesy to them being most influential and helpful.