![Dhalion: Self-Regulating Stream Processing in Heron · at data center scale in recent years (see Apache Storm [26], Spark Streaming [10] and Twitter’s Heron [22], LinkedIn’s Samza](https://static.fdocuments.in/doc/165x107/5ec573d675aebe15262d3f8a/dhalion-self-regulating-stream-processing-in-heron-at-data-center-scale-in-recent.jpg)
SDI/ISTC SeminarLinkedIn and became the lead of the Apache Samza team, which provides a scalable...
1
SDI/ISTC Seminar Yi Pan LinkedIn Yi Pan graduated from UCI with a Ph.D. in Computer Science in 2008. Since then, he has worked in distributed platforms for Internet applications. He started at Yahoo! working on Yahoo!'s NoSQL database project, leading the development of multiple features, such as real-time notification of database updates, secondary index, and live-migration from legacy systems to NoSQL databases. Later, he led the development of the Cloud Messaging System, which is used heavily as a pub-sub service and transaction log for distributed databases at Yahoo!. Since 2014, he joined LinkedIn and became the lead of the Apache Samza team, which provides a scalable stream processing service for the whole company. Building a Lambda-less Stream Processing System using Local States and Windowing This talk will provide an overview of LinkedIn's distributed stream processing platform, including Samza/Kafka/Databus. It will first cover the high level scenarios for stream processing in LinkedIn, followed by detailed requirements around scalability, re-processing, accuracy of results, and ease of programmability; then we will focus on the requirements of stateful stream processing applications and explain how Samza’s state management allows us to build applications that meet all the above requirements. The key concepts, architecture and usage in LinkedIn's stream processing pipeline will be explained, including state management in Samza, the use and configuration of Kafka and Databus as input/output and as a change log. We will also discuss in detail how we leverage the reliable, replayable messaging system (i.e. Kafka) together with durable state management in Samza to build a Lambda-less stream processing platform. The key mechanism to achieve a unified process model between batch and real-time stream is windowing. We will dive into the requirements and our solutions to windowing a real-time stream in this talk as well. Thursday April 14, 2016 RMCIC 4th Floor Panther Hollow Room 12:00 - 1:00 pm VISITOR HOSTS: Majd Sakr, Garth Gibson VISITOR COORD: Majd Sakr, [email protected], 412-268-1161 For more information or questions: Karen Lindenfelser, 8-6716, [email protected] http://www.pdl.cmu.edu/SDI/ Partially funded by:
Transcript of SDI/ISTC SeminarLinkedIn and became the lead of the Apache Samza team, which provides a scalable...
SDI/ISTC Seminar
Yi PanLinkedIn
Yi Pan graduated from UCI with a Ph.D. in Computer Science in
2008. Since then, he has worked in distributed platforms for
Internet applications. He started at Yahoo! working on Yahoo!'s
NoSQL database project, leading the development of multiple
features, such as real-time notification of database
updates, secondary index, and live-migration from legacy
systems to NoSQL databases. Later, he led the development of
the Cloud Messaging System, which is used heavily as a
pub-sub service and transaction log for distributed databases at
Yahoo!. Since 2014, he joined LinkedIn and became the lead
of the Apache Samza team, which provides a scalable
stream processing service for the whole company.
Building a Lambda-less Stream Processing System using Local States and WindowingThis talk will provide an overview of LinkedIn's distributed stream processing platform, including Samza/Kafka/Databus. It will first cover the high level scenarios for stream processing in LinkedIn, followed by detailed requirements around scalability, re-processing, accuracy of results, and ease of programmability; then we will focus on the requirements of stateful stream processing applications and explain how Samza’s state management allows us to build applications that meet all the above requirements. The key concepts, architecture and usage in LinkedIn's stream processing pipeline will be explained, including state management in Samza, the use and configuration of Kafka and Databus as input/output and as a change log. We will also discuss in detail how we leverage the reliable, replayable messaging system (i.e. Kafka) together with durable state management in Samza to build a Lambda-less stream processing platform. The key mechanism to achieve a unified process model between batch and real-time stream is windowing. We will dive into the requirements and our solutions to windowing a real-time stream in this talk as well.
ThursdayApril 14, 2016
RMCIC 4th Floor Panther Hollow Room
12:00 - 1:00 pm
VISITOR HOSTS: Majd Sakr, Garth GibsonVISITOR COORD: Majd Sakr, [email protected], 412-268-1161
For more information or questions:Karen Lindenfelser, 8-6716, [email protected]
http://www.pdl.cmu.edu/SDI/
Partially funded by:
![Data-Trace Types for Distributed Stream Processing Systemsmamouras/papers/Trace-Transductions.pdf · and Apache Samza [2] ... During compilation and deployment, this dataflow graph](https://static.fdocuments.in/doc/165x107/5aad5ccd7f8b9a2e088e1781/data-trace-types-for-distributed-stream-processing-mamouraspaperstrace-transductionspdfand.jpg)
![Locality-Aware Routing in Stateful Streaming Applications · Apache Storm [23], Flink [1], S4 [18], Samza [2], and Twit-ter Heron [14], are examples of popular stream processing engines.](https://static.fdocuments.in/doc/165x107/5ed376124bb814069d356364/locality-aware-routing-in-stateful-streaming-applications-apache-storm-23-flink.jpg)
![Auto-sizing for Stream Processing Applications at LinkedIn...Stream processing systems such as Spark Streaming [39], Flink [3], Heron [31], Samza [35], and Puma [23] are widely employed](https://static.fdocuments.in/doc/165x107/609e0a4b756d1461767f538c/auto-sizing-for-stream-processing-applications-at-linkedin-stream-processing.jpg)