Big Data Pipeline and Analytics Platform Using NetflixOSS and Other Open Source Libraries
-
Upload
sudhir-tonse -
Category
Engineering
-
view
6.164 -
download
3
Transcript of Big Data Pipeline and Analytics Platform Using NetflixOSS and Other Open Source Libraries
Sudhir Tonse (@stonse)Danny Yuan (@g9yuayon)
Big Data Pipeline and Analytics Platform
Using NetflixOSS and Other Open Source Software
Data Is the most important asset at Netflix
If all the data is easily available to all teams, it can be leveraged in new and exciting ways
~1000 Device Types~500 Apps/Web Services~100 Billion Events/Day
3.2M messages per second at peak time
3GB per second at peak time
Dashboard
Type of EventsUser Interface EventsSearch Event (Matrix using PS3 )Star Rating Event (HoC : 5 stars, Xbox, US, )
Infrastructural EventsRPC Call (API -> Billing Service, /bill/.., 200, )Log Errors (NPE, Movie is null, , )
Other Events
Making Sense of Billions of Events
http://netflix.github.io+
A Humble Beginning
Evolution Scale!
Application
Application
Application
Application
Application
Application
Application
Application
Application
Application
Something happened. Our traffic turned into a hockey stick, and the number of applications exploded. So, log traffic also exploded. Simple log scraping wouldnt cut it any more.
We Want to Process App Data in Hadoop
Our Hadoop Ecosystem
@NetflixOSS Big Data Tools
Hadoop as a Service
Pig Scripting on Steroids
Pig Married to Clojure
Map-Reduce for Clojure
S3MPER
S3mper is a library that provides an additional layer of consistency checking on top of Amazon's S3 index through use of a consistent, secondary index.S3mper is a library that provides an additional layer of consistency checking on top of Amazon's S3 index through use of a consistent, secondary index.
Efficient ETL with Cassandra
Cassandra
Offline Analysis
Evolution Speed!
We Want to Aggregate, Index, and Query Data in Real Time
Interactive Exploration
For one thing: interactive exploration. Sometimes we want to get data in real time so we can act quickly. Some data is only useful in a small time window after all. Sometimes we want to perform lots of experimental queries just to find the right insights. If we wait too long for a query back, we wont be able to iterate fast enough. Either way, we need to get query results back in seconds.
Lets walk through some use cases
client activity event
*
/name = movieStarts
Pipeline ChallengesApp owners: send and forgetData scientists: validation, ETL, batch processingDevOps: stream processing, targeted search
Message Routing
Here is one example: we process more than 150 thousand events per second about user activities. What if wed like to know the geographically how many users started playing videos in the past 5 minutes? So I submit my query, and in a few seconds....
But this is an aggregated view. What if I want to drill down the data immediately along different dimensions? In this particular case, to find out failed attempts on our SilverLight players that run on PCs and Macs?
We Want to Consume Data Selectively in Different Ways
Message brokerHigh-throughputPersistent and replicated
There Is More
Intelligent Alerts
Note this is different from alerting based on monitoring metrics. Monitoring metrics are great and versatile. But it doesnt help us catch unexpected errors. When we build an application, we instrument our code diligently, yet its very likely we miss some critical instrumentation points. Theres one thing that we always catch, though: logged errors and unhandled exceptions. Its about The alert provides a precise entrypoint and the right context for people to drill down the right problems
Intelligent Alerts
Note this is different from alerting based on monitoring metrics. Monitoring metrics are great and versatile. But it doesnt help us catch unexpected errors. When we build an application, we instrument our code diligently, yet its very likely we miss some critical instrumentation points. Theres one thing that we always catch, though: logged errors and unhandled exceptions. Its about The alert provides a precise entrypoint and the right context for people to drill down the right problems
Guided Debugging in the Right Context
Guided Debugging in the Right Context
Guided Debugging in the Right Context
Ad-hoc query with different dimensionsQuick aggregations and Top-N queriesTime series with flexible filtersQuick access to raw data using boolean queriesWhat We Need
DruidRapid exploration of high dimensional dataFast ingestion and queryingTime series
Real-time indexing of event streamsKiller feature: boolean searchGreat UI: Kibana
The Old Pipeline
The New Pipeline
There Is More
Its Not All About Counters and Time Series
RequestIdParent IdNode IdService NameStatus4965-4a740123Edge Service2004965-4a74123456Gateway2004965-4a74456789Service A2004965-4a74e456abcService B200
Status:200
Distributed Tracing
Distributed Tracing
Distributed Tracing
A System that Supports All These
A Data Pipeline To Glue Them All
Make It Simple
Message ProducingSimple and Uniform APImessageBus.publish(event)
Consumption Is Simple Too consumer.observe().subscribe(new Subscriber() {@Overridepublic void onNext(Ackable ackable) {process(ackable.getEntity(MyEventType.class));ackable.ack();}});
consumer.pause();consumer.resume()
RxJavaFunctional reactive programming modelPowerful streaming APISeparation of logic and threading model
Design DecisionsTop Priority: app stability and throughput Asynchronous operationsAggressive bufferingDrops messages if necessary
Anything Can Fail
Cloud Resiliency
Fault Tolerance FeaturesWrite and forward with auto-reattached EBS (Amazons Elastic Block Storage)disk-backed queue: big-queueCustomized scaling down
Theres More to DoContribute to @NetflixOSS Join us :-)
Summary
http://netflix.github.io+
You can build your own web-scale data pipeline using open source components
Thank You!Sudhir Tonsehttp://www.linkedin.com/in/sudhirtonseTwitter: @stonseDanny Yuan http://www.linkedin.com/pub/danny-yuan/4/374/862Twitter: @g9yuayon