Ignacio Mulas Viela – Kappa Architecture in the Telecom Industry

Slide title 70 pt

CAPITALS

Slide subtitle minimum 30 pt

Kappa architecture in the telecom industry

Ignacio Mulas Viela Ericsson Research

Slide title 44 pt

Text and bullet level 1 minimum 24 pt

Bullets level 2-5 minimum 20 pt

Characters for Embedded font: !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~¡¢£¤¥¦§¨©ª«¬®¯°±²³´¶·¸¹º»¼½ÀÁÂÃÄÅÆÇÈËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿĀāĂăąĆćĊċČĎďĐđĒĖėĘęĚěĞğĠġĢģĪīĮįİıĶķĹĺĻļĽľŁłŃńŅņŇňŌŐőŒœŔŕŖŗŘřŚśŞşŠšŢţŤťŪūŮůŰűŲųŴŵŶŷŸŹźŻż�Ž�žƒȘșˆˇ˘˙˚˛˜˝ẀẁẃẄẅỲỳ–—‘’‚“”„†‡•…‰‹›⁄€™ĀĀĂĂĄĄĆĆĊĊČČĎĎĐĐĒĒĖĖĘĘĚĚĞĞĠĠĢĢĪĪĮĮİĶĶĹĹĻĻĽĽŃŃŅŅŇŇŌŌŐŐŔŔŖŖŘŘŚŚŞŞŢŢŤŤŪŪŮŮŰŰŲŲŴŴŶŶŹŹŻŻȘș−≤≥fifl

ΆΈΉΊΌΎΏΐΑΒΓΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΪΫΆΈΉΊΰαβγδεζηθικλνξορςΣΤΥΦΧΨΩΪΫΌΎΏ

ЁЂЃЄЅІЇЈЉЊЋЌЎЏАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯЁЂЃЄЅІЇЈЉЊЋЌЎЏѢѢѲѲѴѴҐҐәǽẀẁẂẃẄẅỲỳ№

Do not add objects or text in the footer area

Ericsson Internal | 2011-10-19 |

› Who? › What? › Why? › How?

– Architecture – Time handling

› Demo on local machine › Learning › Q&A

Proposed agenda

Flink-forward | Ignacio Mulas | 12-October-2015

Slide title 44 pt








› Ericsson Research – Management of Complex Systems – Analytics technologies applied to the ICT industry

› EIT ICT EUROPA project › Contributors:

Who?


Slide title 44 pt








ericsson MOBILE INFRASTRUCTURE

AND CLOUD

OPERATIONS & BUSINESS SUPPORT SOLUTIONS

TELECOM SERVICES

TV PLATFORMS

65,000

2.5 billion

1 billion Subscribers managed by

us

Subscribers supported by

us

Services professionals

Slide title 44 pt








One day in the life of a medium sized network (~10M customers)

Web pages 700,000,000

Videos 40,000,000

Radio sessions (RAB) 120,000,000

Handovers (HSDSCH-CC) 300,000,000

Internet sessions (PDP) 66,000,000

Sum data 10à100 TB/day

Real-time data rate 100,000à1,000,000 events/second

+200 more types of events


Slide title 44 pt








Why?

KAPPA ARCHITECTURE

Everything is a Stream

Immutable data sources

Single analytics framework Stream replay


Slide title 44 pt








›  Real-time analysis of metrics and logs of a cloud infrastructure – Analysis of system behavior

›  Extraction of statistical characteristics of system’s logs and metrics – Anomaly detection

›  Clustering of node states and deviations (anomalies)

What?

…


Analytics

Slide title 44 pt








How?

Kafka Elastic Search

Kibana

Flink Pre-processing Streaming k-means

Classification Time drift analysis

raw data pre-processed data

analysis results

… …

Data sources


Slide title 44 pt








Data pipeline

Raw data

Pre- processing

Feature Vectors

Streaming k-means

Centroids

Predictor Dashboard

Timedrift analysis

OK/NOK

Timedrifts


Slide title 44 pt








Data pipeline


Raw data

Pre- processing

Feature Vectors

Streaming k-means

Centroids

Predictor Dashboard

Timedrift analysis

OK/NOK

Timedrifts

Slide title 44 pt








Time challenge

How would you count 20 seconds without your own clock?

How would you count 20 seconds? Trivial! Looking at my clock…

13

15

14

Not trivial… Two things to consider:

- Cannot ask all at the same time

- Everyone has a different time

Slide title 44 pt








Processor Time (tp)

-  Use machine clock -  Things to consider:

-  Differences between hosts

-  Default

Time handling

Event Time (te)

›  Use time of events ›  Things to consider:

–  Time drifts –  Unordered events

› Watermarks

t0 t0 t0 te0 tp

VS


Slide title 44 pt








Event time e0 e1 e2 e3

…t0 t1 t2 t3

<t0,e0> <t1,e1> <t2,e2> <t3,e3>

<te0,e0> <te1,e1> <te2,e2> <te3,e3>

EventTimeExstractor()

enableTimestamps()

<te0,e0> <te1,e1> <te2,e2> <te3,e3>

w2 w1 w0

window()


Te0

+ window

+ watermark

Slide title 44 pt








Data pipeline


Raw data

Pre- processing

Feature Vectors

Streaming k-means

Centroids

Predictor Dashboard

Timedrift analysis

OK/NOK

Timedrifts

Slide title 44 pt








Tips and tricks

Slide title 44 pt








Local deployment

Kafka

Elastic Search

Kibana

Flink LOGSTASh

Data dump

VM1 VM2

vagrant up

Slide title 44 pt








Good ›  Flink is a natural candidate to address Kappa architecture (everything is a stream!) ›  Good integration with Kafka (StreamExecutionEnvironment) ›  Nice representation of jobs (using graph) ›  Easy to start running a cluster

Things to improve ›  API not consistent in all the cases

–  Timeout on sources and step function –  Reverse of other of parameters in different functions of the API

›  Easy to debug in a single machine but a bit hard in the distributed setup ›  Hard to see contextual information on slaves in the cluster

learnings


Slide title 70 pt

CAPITALS

Slide subtitle minimum 30 pt

Please, feel free to contact me if you have suggestions/comments/questions @ [email protected]

Thank you! quESTIONS?

Ignacio Mulas Viela – Kappa Architecture in the Telecom Industry

Technology

Transcript of Ignacio Mulas Viela – Kappa Architecture in the Telecom Industry