Real time analytics with Netty, Storm, Kafka
-
Upload
trieu-nguyen -
Category
Technology
-
view
1.442 -
download
1
description
Transcript of Real time analytics with Netty, Storm, Kafka
Real-time Analytics with Netty, Apache Kafka and StormCase study with “lambda architecture”
http://nguyentantrieu.infoUpdate: 07/06/2013
Agenda1. Overview Architecture2. Log HTTP-Handler and producer: Netty 43. Kafka 0.8 (Stream Data Log Storage)4. Storm Analytics Cluster
Overview System Architecture
Concept Flow
Concept FlowJavaScript Tracking
Mobile SDK
Http Log Server
Kafka
S2 HTTP Log Servernetty framework 4
Netty.ioNetty is a non-blocking I/O (NIO) client-server framework for the development of Java network applications such as protocol servers and clients. The asynchronous event-driven network application framework and tools are used to simplify network programming such as TCP and UDP socket servers.[2] Netty includes an implementation of the reactor pattern of programming.
http://en.wikipedia.org/wiki/Netty_(software)http://nguyentantrieu.info/blog/backend-system-with-netty-io
Apache Kafka (version 0.8)https://cwiki.apache.org/confluence/display/KAFKA/Index
In Production● Clustering ( 4 nodes)● Partitions
○ user-activity: 24 partitions● Producer and consumer are replication
aware
Core Producer Class (S2 HTTP Log Server)
Storm Analytics Cluster
The Storm Topology
Kafka Clustertopic: user-activity
Tokenizer Bolt
Parser Bolt
Aggregate Bolt
Redis Statistics Bolt
Save DWH Bolt
Raw Data
Kafka Consumer Spout
Useful Links● https://storm.incubator.apache.org● https://www.youtube.com/watch?
v=DRgs8L4AOgc● https://cwiki.apache.
org/confluence/display/KAFKA/Index● http://nguyentantrieu.info/blog/backend-
system-with-netty-io●