UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging +...
Transcript of UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging +...
![Page 1: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/1.jpg)
�1
UNIFYING MESSAGING, QUEUING, STREAMING & COMPUTE WITH APACHE PULSAR
KARTHIK RAMASAMYCO-FOUNDER AND CEO
![Page 2: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/2.jpg)
�2
Connected World
![Page 3: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/3.jpg)
�3
Ubiquity of Real-Time Data Streams & Events
![Page 4: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/4.jpg)
EVENT/STREAM DATA PROCESSING
�4
✦ Events are analyzed and processed as they arrive
✦ Decisions are timely, contextual and based on fresh data
✦ Decision latency is eliminated
✦ Data in motion
Ingest/Buffer
Analyze Act
![Page 5: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/5.jpg)
MICROSERVICES
MODEL INFERENCEWORKFLOWS ANALYTICS
MONITORING
EVENT/STREAM PROCESSING PATTERNS
![Page 6: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/6.jpg)
STREAM PROCESSING PATTERN
�6
ComputeMessaging
Storage
DataInges6on DataProcessing
ResultsStorageDataStorageData
Serving
![Page 7: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/7.jpg)
ELEMENTS OF EVENT/STREAM PROCESSING
�7
Aggregation Systems
Messaging Systems
Result Engine
HDFS Queryable Engines
![Page 8: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/8.jpg)
APACHE PULSAR
�8
Flexible Messaging + Streaming System
backed by a durable log storage
![Page 9: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/9.jpg)
Key Concepts
![Page 10: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/10.jpg)
Core concepts: Tenants, namespaces, topics
�10
Apache Pulsar Cluster
Tenants Namespaces Topics
Marketing
Sales
Security
Analytics
Campaigns
Data transformation
Data Integration
Microservices
Visits
Conversions
Responses
Conversions
Transactions
Interactions
Log events
Signatures
Accesses
![Page 11: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/11.jpg)
Topics
�11
TopicProducers
Consumers
Time
Consumers
Consumers
Producers
![Page 12: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/12.jpg)
Topic partitions
�12
Topic - P0
Time
Topic - P1
Topic - P2
Producers
Producers
Consumers
Consumers
Consumers
![Page 13: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/13.jpg)
Segments
�13
Time
Segment 1 Segment 2 Segment 3
Segment 1 Segment 2 Segment 3 Segment 4
Segment 1 Segment 2 Segment 3
P0
P1
P2
![Page 14: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/14.jpg)
Architecture
![Page 15: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/15.jpg)
APACHE PULSAR
�15
Bookie Bookie Bookie
Broker Broker Broker
Producer Consumer
SERVINGBrokers can be added independently Traffic can be shifted quickly across brokers
STORAGEBookies can be added independently New bookies will ramp up traffic quickly
![Page 16: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/16.jpg)
APACHE PULSAR - BROKER
�16
✦ Broker is the only point of interaction for clients (producers and consumers)
✦ Brokers acquire ownership of group of topics and “serve” them
✦ Broker has no durable state
✦ Provides service discovery mechanism for client to connect to right broker
![Page 17: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/17.jpg)
APACHE PULSAR - BROKER
�17
![Page 18: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/18.jpg)
APACHE PULSAR - CONSISTENCY
�18
Bookie
Bookie
BookieBrokerProducer
![Page 19: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/19.jpg)
APACHE PULSAR - DURABILITY (NO DATA LOSS)
�19
Bookie
Bookie
BookieBrokerProducer
Journal
Journal
Journal
fsync
fsync
fsync
![Page 20: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/20.jpg)
APACHE PULSAR - ISOLATION
�20
![Page 21: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/21.jpg)
APACHE PULSAR - SEGMENT STORAGE
�21
234…20212223…40414243…60616263…
Segment 1
Segment 3
Segment 2
Segment 2
Segment 1
Segment 3
Segment 4
Segment 3
Segment 2
Segment 1
Segment 4
Segment 4
![Page 22: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/22.jpg)
APACHE PULSAR - RESILIENCY
�22
1234…20212223…40414243…60616263…
Segment 1
Segment 3
Segment 2
Segment 2
Segment 1
Segment 3
Segment 4
Segment 3
Segment 2
Segment 1
Segment 4
Segment 4
![Page 23: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/23.jpg)
APACHE PULSAR - SEAMLESS CLUSTER EXPANSION
�23
1234…20212223…40414243…60616263…
Segment 1
Segment 3
Segment 2
Segment 2
Segment 1
Segment 3
Segment 4
Segment 3
Segment 2
Segment 1
Segment 4
Segment 4
Segment Y
Segment Z
Segment X
![Page 24: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/24.jpg)
APACHE PULSAR - TIERED STORAGE
�24
Low Cost Storage
1234…20212223…40414243…60616263…
Segment 3
Segment 2Segment 3
Segment 4
Segment 3
Segment 1
Segment 4 Segment 4
![Page 25: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/25.jpg)
Multi-tiered storage and serving
�25
Partition
Broker Broker Broker
... ... ... ...
Processing(brokers)
WarmStorage
ColdStorage
Tailing reads: served from in-memory cache
Catch-up reads: served from persistent storage layer
Historical reads: served from cold storage
![Page 26: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/26.jpg)
PARTITIONS VS SEGMENTS - WHY SHOULD YOU CARE?
�26
Legacy Architectures
● Storage co-resident with processing● Partition-centric● Cumbersome to scale--data
redistribution, performance impact
Logical View
Apache Pulsar
● Storage decoupled from processing● Partitions stored as segments● Flexible, easy scalability
Partition
Processing & Storage
Segment 1 Segment 3Segment 2 Segment n
Partition
Broker
Partition (primary)
Broker
Partition (copy)
Broker
Partition (copy)
Broker Broker Broker
Segment 1
Segment 2
Segment n
...Segment 2
Segment 3
Segment n
...Segment 3
Segment 1
Segment n
...Segment 1
Segment 2
Segment n
...
Processing(brokers)
Storage
![Page 27: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/27.jpg)
PARTITIONS VS SEGMENTS - WHY SHOULD YOU CARE?
�27
✦ In Kafka, partitions are assigned to brokers “permanently”
✦ A single partition is stored entirely in a single node
✦ Retention is limited by a single node storage capacity
✦ Failure recovery and capacity expansion require expensive “rebalancing”
✦ Rebalancing has a big impact over the system, affecting regular traffic
![Page 28: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/28.jpg)
UNIFIED MESSAGING MODEL - STREAMING
�28
Pulsar topic/partition
Producer 2
Producer 1
Consumer 1
Consumer 2
Subscription A
M4
M3
M2
M1
M0
M4
M3
M2
M1
M0
X
Exclusive
![Page 29: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/29.jpg)
UNIFIED MESSAGING MODEL - STREAMING
�29
Pulsar topic/partition
Producer 2
Producer 1
Consumer 1
Consumer 2
Subscription B
M4
M3
M2
M1
M0
M4
M3
M2
M1
M0
Failover
In case of failure in consumer 1
![Page 30: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/30.jpg)
UNIFIED MESSAGING MODEL - QUEUING
�30
Pulsar topic/partition
Producer 2
Producer 1
Consumer 2
Consumer 3
Subscription C
M4
M3
M2
M1
M0
Shared
Traffic is equally distributedacross consumers
Consumer 1
M4
M3
M2
M1
M0
![Page 31: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/31.jpg)
DISASTER RECOVERY
�31
Topic(T1) Topic(T1)
Topic(T1)
Subscrip6on(S1)
Subscrip6on(S1)
Producer(P1)
Consumer(C1)
Producer(P3)
Producer(P2)
Consumer(C2)
DataCenterA DataCenterB
DataCenterC
Integrated in the broker message flow
Simple configuration to add/remove regions
Asynchronous (default) and synchronous
replication
![Page 32: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/32.jpg)
• Two independent clusters, primary and standby
• Configured tenants and namespaces replicate to standby
• Data published to primary is asynchronously replicated to standby
• Producers and consumers restarted in second datacenter upon primary failure
Asynchronous replication example
�32
Producers (active)
Datacenter 1
Consumers (active)
Pulsar Cluster(primary)
Datacenter 2
Producers (standby)
Consumers (standby)
Pulsar Cluster(standby)
Pulsar replication
ZooKeeper ZooKeeper
![Page 33: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/33.jpg)
ZooKeeper
• Each topic owned by one broker at a time, i.e. in one datacenter
• ZooKeeper cluster spread across multiple locations
• Broker commits writes to bookies in both datacenters
• In event of datacenter failure, broker in surviving datacenter assumes ownership of topic
Synchronous replication example
�33
Producers
Datacenter 1
Consumers
Pulsar Cluster
Datacenter 2
Producers
Consumers
![Page 34: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/34.jpg)
Replicated subscriptions
�34
Producers
Datacenter 1
ConsumersPulsar Cluster 1
Subscriptions
Datacenter 2
ConsumersPulsar Cluster 2
Subscriptions
Pulsar Replication
MarkerMarker Marker
![Page 35: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/35.jpg)
MULTITENANCY - CLOUD NATIVE
�35
Apache Pulsar Cluster
Product Safety
ETL
FraudDetection
Topic-1 Account History
Topic-2 User Clustering
Topic-1 Risk Classification
MarketingCampaigns
ETL
Topic-1 Budgeted Spend
Topic-2 Demographic Classification
Topic-1 Location Resolution
Data Serving
Microservice
Topic-1 Customer Authentication
10 TB
7 TB
5 TB
✦ Authentication✦ Authorization✦ Software isolation
๏ Storage quotas, flow control, back pressure, rate limiting✦ Hardware isolation
๏ Constrain some tenants on a subset of brokers/bookies
![Page 36: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/36.jpg)
PULSAR CLIENTS
�36
Apache Pulsar Cluster
Java
Python
Go
C++ C
![Page 37: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/37.jpg)
PULSAR PRODUCER
�37
PulsarClient client = PulsarClient.create( “http://broker.usw.example.com:8080”);
Producer producer = client.createProducer( “persistent://my-property/us-west/my-namespace/my-topic”);
// handles retries in case of failure producer.send("my-message".getBytes());
// Async version: producer.sendAsync("my-message".getBytes()).thenRun(() -> { // Message was persisted });
![Page 38: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/38.jpg)
PULSAR CONSUMER
�38
PulsarClient client = PulsarClient.create( "http://broker.usw.example.com:8080");
Consumer consumer = client.subscribe( "persistent://my-property/us-west/my-namespace/my-topic", "my-subscription-name");
while (true) { // Wait for a message Message msg = consumer.receive();
System.out.println("Received message: " + msg.getData());
// Acknowledge the message so that it can be deleted by broker consumer.acknowledge(msg); }
![Page 39: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/39.jpg)
SCHEMA REGISTRY
�39
✦ Provides type safety to applications built on top of Pulsar
✦ Two approaches
✦ Client side - type safety enforcement up to the application
✦ Server side - system enforces type safety and ensures that producers and consumers remain synced
✦ Schema registry enables clients to upload data schemas on a topic basis.
✦ Schemas dictate which data types are recognized as valid for that topic
![Page 40: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/40.jpg)
PULSAR SCHEMAS - HOW DO THEY WORK?
�40
✦ Enforced at the topic level
✦ Pulsar schemas consists of
✦ Name - Name refers to the topic to which the schema is applied
✦ Payload - Binary representation of the schema
✦ Schema type - JSON, Protobuf and Avro
✦ User defined properties - Map of strings to strings (application specific - e.g git hash of the schema)
![Page 41: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/41.jpg)
SCHEMA VERSIONING
�41
PulsarClient client = PulsarClient.builder() .serviceUrl(“http://broker.usw.example.com:6650") .build()
Producer<SensorReading> producer = client.newProducer(JSONSchema.of(SensorReading.class)) .topic(“sensor-data”) .sendTimeout(3, TimeUnit.SECONDS) .create()
Scenario What happens
No schema exists for the topic Producer is created using the given schema
Schema already exists; producer connects using the same schema
that’s already storedSchema is transmitted to the broker, determines that it is already stored
Schema already exists; producer connects using a new schema that is
compatibleSchema is transmitted, compatibility determined and stored as new schema
![Page 42: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/42.jpg)
Processing framework
![Page 43: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/43.jpg)
HOW TO PROCESS DATA MODELED AS STREAMS
�43
✦ Consume data as it is produced (pub/sub)
✦ Light weight compute - transform and react to data as it arrives
✦ Heavy weight compute - continuous data processing
✦ Interactive query of stored streams
![Page 44: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/44.jpg)
LIGHT WEIGHT COMPUTE
�44
f(x)IncomingMessages OutputMessages
ABSTRACT VIEW OF COMPUTE REPRESENTATION
![Page 45: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/45.jpg)
TRADITIONAL COMPUTE REPRESENTATION
�45
DAG
%
%
%
%
%
Source 1
Source 2
Action
Action
Action
Sink 1
Sink 2
![Page 46: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/46.jpg)
REALIZING COMPUTATION - EXPLICIT CODE
�46
public static class SplitSentence extends BaseBasicBolt { @Override public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields("word")); }
@Override public Map<String, Object> getComponentConfiguration() { return null; }
public void execute(Tuple tuple, BasicOutputCollector basicOutputCollector) { String sentence = tuple.getStringByField("sentence"); String words[] = sentence.split(" "); for (String w : words) { basicOutputCollector.emit(new Values(w)); } }}
STITCHED BY PROGRAMMERS
![Page 47: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/47.jpg)
REALIZING COMPUTATION - FUNCTIONAL
�47
Builder.newBuilder() .newSource(() -> StreamletUtils.randomFromList(SENTENCES)) .flatMap(sentence -> Arrays.asList(sentence.toLowerCase().split("\\s+"))) .reduceByKeyAndWindow(word -> word, word -> 1, WindowConfig.TumblingCountWindow(50), (x, y) -> x + y);
![Page 48: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/48.jpg)
TRADITIONAL REAL TIME - SEPARATE SYSTEMS
�48
Messaging Compute
![Page 49: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/49.jpg)
TRADITIONAL REAL TIME SYSTEMS
�49
DEVELOPER EXPERIENCE
✦ Powerful API but complicated
✦ Does everyone really need to learn functional programming?
✦ Configurable and scalable but management overhead
✦ Edge systems have resource and management constraints
![Page 50: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/50.jpg)
TRADITIONAL REAL TIME SYSTEMS
�50
OPERATIONAL EXPERIENCE
✦ Multiple systems to operate
✦ IoT deployments routinely have thousands of edge systems
✦ Semantic differences
✦ Mismatch and duplication between systems
✦ Creates developer and operator friction
![Page 51: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/51.jpg)
LESSONS LEARNT - USE CASES
�51
✦ Data transformations
✦ Data classification
✦ Data enrichment
✦ Data routing
✦ Data extraction and loading
✦ Real time aggregation
✦ Microservices
Significant set of processing tasks are exceedingly simple
![Page 52: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/52.jpg)
EMERGENCE OF CLOUD - SERVERLESS
�52
✦ Simple function API
✦ Functions are submitted to the system
✦ Runs per events
✦ Composition APIs to do complex things
✦ Wildly popular
![Page 53: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/53.jpg)
SERVERLESS VS STREAMING
�53
✦ Both are event driven architectures
✦ Both can be used for analytics and data serving
✦ Both have composition APIs
๏ Configuration based for serverless
๏ DSL based for streaming
✦ Serverless typically does not guarantee ordering
✦ Serverless is pay per action
![Page 54: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/54.jpg)
STREAM NATIVE COMPUTE USING FUNCTIONS
�54
✦ Simplest possible API -function or a procedure
✦ Support for multi language
✦ Use of native API for each language
✦ Scale developers
✦ Use of message bus native concepts - input and output as topics
✦ Flexible runtime - simple standalone applications vs managed system applications
APPLYING INSIGHT GAINED FROM SERVERLESS
![Page 55: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/55.jpg)
PULSAR FUNCTIONS
�55
SDK LESS API
import java.util.function.Function; public class ExclamationFunction implements Function<String, String> { @Override public String apply(String input) { return input + "!"; } }
![Page 56: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/56.jpg)
PULSAR FUNCTIONS
�56
SDK API
import org.apache.pulsar.functions.api.PulsarFunction; import org.apache.pulsar.functions.api.Context; public class ExclamationFunction implements PulsarFunction<String, String> { @Override public String process(String input, Context context) { return input + "!"; } }
![Page 57: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/57.jpg)
PULSAR FUNCTIONS
�57
✦ Function executed for every message of input topic
✦ Support for multiple topics as inputs
✦ Function output goes into output topic - can be void topic as well
✦ SerDe takes care of serialization/deserialization of messages
๏ Custom SerDe can be provided by the users
๏ Integration with schema registry
![Page 58: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/58.jpg)
PROCESSING GUARANTEES
�58
✦ ATMOST_ONCE
๏ Message acked to Pulsar as soon as we receive it
✦ ATLEAST_ONCE
๏ Message acked to Pulsar after the function completes
๏ Default behavior - don’t want people to loose data
✦ EFFECTIVELY_ONCE
๏ Uses Pulsar’s inbuilt effectively once semantics
✦ Controlled at runtime by user
![Page 59: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/59.jpg)
DEPLOYING FUNCTIONS - BROKER
�59
Broker 1
Worker
Functionwordcount-1
Functiontransform-2
Broker 1
Worker
Function transform-1
Functiondataroute-1
Broker 1
Worker
Function wordcount-2
Functiontransform-3
Node 1 Node 2 Node 3
![Page 60: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/60.jpg)
DEPLOYING FUNCTIONS - WORKER NODES
�60
Worker
Functionwordcount-1
Functiontransform-2
Worker
Function transform-1
Functiondataroute-1
Worker
Function wordcount-2
Functiontransform-3
Node 1 Node 2 Node 3
Broker 1 Broker 2 Broker 3
Node 4 Node 5 Node 6
![Page 61: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/61.jpg)
DEPLOYING FUNCTIONS - KUBERNETES
�61
Functionwordcount-1
Function transform-1
Functiontransform-3
Pod 1 Pod 2 Pod 3
Broker 1 Broker 2 Broker 3
Pod 7 Pod 8 Pod 9
Functiondataroute-1
Function wordcount-2
Functiontransform-2
Pod 4 Pod 5 Pod 6
![Page 62: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/62.jpg)
BUILT-IN STATE MANAGEMENT IN FUNCTIONS
�62
✦ Functions can store state in inbuilt storage
๏ Framework provides a simple library to store and retrieve state
✦ Support server side operations like counters
✦ Simplified application development
๏ No need to standup an extra system
![Page 63: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/63.jpg)
DISTRIBUTED STATE IN FUNCTIONS
�63
import org.apache.pulsar.functions.api.Context; import org.apache.pulsar.functions.api.PulsarFunction;
public class CounterFunction implements PulsarFunction<String, Void> { @Override public Void process(String input, Context context) throws Exception { for (String word : input.split("\\.")) { context.incrCounter(word, 1); } return null; } }
![Page 64: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/64.jpg)
PULSAR - DATA IN AND OUT
�64
✦ Users can write custom code using Pulsar producer and consumer API
✦ Challenges
๏ Where should the application to publish data or consume data from Pulsar?
๏ How should the application to publish data or consume data from Pulsar?
✦ Current systems have no organized and fault tolerant way to run applications that ingress and egress data from and to external systems
![Page 65: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/65.jpg)
PULSAR IO TO THE RESCUE
�65
Apache Pulsar ClusterSource Sink
![Page 66: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/66.jpg)
PULSAR IO - EXECUTION
�66
Broker 1
Worker
Sink Cassandra-1
SourceKinesis-2
Broker 2
Worker
Source Kinesis-1
SourceTwitter-1
Broker 3
Worker
Sink Cassandra-2
SourceKinesis-3
Node 1 Node 2 Node 3
Fault tolerance Parallelism Elasticity Load Balancing On-demand updates
![Page 67: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/67.jpg)
INTERACTIVE QUERYING OF STREAMS - PULSAR SQL
�67
1234…20212223…40414243…60616263…
Segment 1
Segment 3
Segment 2
Segment 2
Segment 1
Segment 3
Segment 4
Segment 3
Segment 2
Segment 1
Segment 4
Segment 4
SegmentReader
SegmentReader
SegmentReader
SegmentReaderCoordinator
![Page 68: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/68.jpg)
PULSAR PERFORMANCE
�68
![Page 69: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/69.jpg)
PULSAR PERFORMANCE - LATENCY
�69
![Page 70: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/70.jpg)
APACHE PULSAR VS. APACHE KAFKA
�70
Mul$-tenancyAsingleclustercansupportmanytenantsandusecases
SeamlessClusterExpansionExpandtheclusterwithoutanydown$me
Highthroughput&LowLatencyCanreach1.8Mmessages/sinasinglepar$$onandpublishlatencyof5msat99pct
DurabilityDatareplicatedandsyncedtodisk
Geo-replica$onOutofboxsupportforgeographicallydistributedapplica$ons
UnifiedmessagingmodelSupportbothTopic&Queueseman$cinasinglemodel
TieredStorageHot/warmdataforreal$meaccessandcoldeventdataincheaperstorage
PulsarFunc$onsFlexiblelightweightcompute
HighlyscalableCansupportmillionsoftopics,makesdatamodelingeasier
![Page 71: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/71.jpg)
Examples of companies using Apache Pulsar
�71STREAMLIO CONFIDENTIAL
Strong market validation
!13
Open source adopters
Open source evaluators
Streamlio outreach
Growing funnel of validation and leads from outbound, inbound
and open source
![Page 72: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/72.jpg)
ScenarioNeed to collect and distribute user and data events to distributed global applications at Internet scale
Challenges• Multiple technologies to handle
messaging needs• Multiple, siloed messaging clusters• Hard to meet scale and performance• Complex, fragile environment
Yahoo!
�72
Solution• Central event data bus using Apache Pulsar• Consolidated multiple technologies and clusters into a
single solution• Fully-replicated across 8 global datacenter• Processing >100B messages / day, 2.3M topics
![Page 73: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/73.jpg)
APACHE PULSAR IN PRODUCTION @SCALE
�73
4+years
Serves2.3milliontopics
700billionmessages/day
500+bookienodes
200+brokernodes
Averagelatency<5ms
99.9%15ms(strongdurabilityguarantees)
Zerodataloss
150+applica6ons
Selfservedprovisioning
Full-meshcross-datacenterreplica6on-8+datacenters
![Page 74: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/74.jpg)
Growing ecosystem
�74
![Page 75: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/75.jpg)
Use Cases
![Page 76: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/76.jpg)
Example use cases
�76
Streaming data transformation
Data distribution
Real-time analytics
Real-time monitoring and notifications IoT analytics
!
Event-driven workflows
Interactive applications
Log processing and analytics
![Page 77: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/77.jpg)
Data-driven workflows
�77
Scenario
Application processes incoming events and documents that generate processing workflows
Challenges
Operational burdens and scalability challenges of existing technologies growing as data grows
Solution
Process incoming events and data and create work queues in same system
Decrypt, extract, convert, dispatch, process, store
![Page 78: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/78.jpg)
Data distribution
�78
Data collected from multiple sources
Normalized, enriched transformed and put
into topics
Delivered to applications and users
as data streams
Distribution and usage logged for auditing
Data Sources
![Page 79: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/79.jpg)
ScenarioRetail analytics software provider brings together operational and market research data for insights.
ChallengesExisting Kinesis + Spark + data lake infrastructure was unnecessarily complex and burdensome to operate and maintain.
Solution• Replaced Kinesis + Spark with Apache Pulsar• Simplified data transformation pipeline• Reduced operations burdens
Simplifying the data pipeline
�79
Data Lake
![Page 80: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/80.jpg)
Event sourcing
�80
SolutionDeploy Apache Pulsar for long-term retention and scalable processing and distribution of event data.Why Streamlio• Architected for scalable and efficient long-term storage• High performance, scalable processing and distribution of
data due to unique architecture
ProblemEvent-driven applications require long-term retention of data streams, but current technologies are cumbersome and expensive to use for data retention and cannot efficiently replay data.
![Page 81: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/81.jpg)
IOT ENVIRONMENT
�81
D SmartD
EdgeAggregator
Light Device Smart Device Edge Node
✦ Typically sensors
✦ Only one functionality
✦ Simple to configure
✦ Light weight protocols to communicate
✦ Typically ARM based
✦ Multiple functionality
✦ Basic but generic computational logic, limited storage
✦ Light weight and propriety protocols to communicate
✦ Multicore based
✦ Versatile functionality
✦ Complex and generic computational logic, decent amount of storage
✦ Light weight and propriety protocols to communicate
Cloud
Cloud
✦ Multiple machines
✦ Versatile functionality
✦ Complex and generic computational logic
✦ Lots of storage
![Page 82: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/82.jpg)
IOT DATA FABRIC WITH APACHE PULSAR
�82
Apache Pulsar CloudApache Pulsar Edge
Apache Pulsar Edge
Apache PulsarDevice
Apache PulsarDevice
Apache PulsarDevice
Apache PulsarDevice
D
D
D
D
Apache PulsarDevice D
filter-fn
Web Socket API
Web Socket API
Web Socket APIW
eb S
ocke
t API
xform-fn
xform-fn
Web Socket API
aggr-fn
xform-fn
aggr-fn
Data Replication Data Replication Data Replication
Data Replication
Data Replication
![Page 83: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/83.jpg)
ScenarioContinuously-arriving data generated by connected cars needs to be quickly collected, processed and distributed to applications and partners
ChallengesRequire scalability to handle growing data sources and volumes without complex mix of technologies
SolutionLeverage Streamlio solution to provide data backbone that can receive, transform, and distribute data at scale
Large Car Manufacturer: Connected vehicle
�83
![Page 84: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/84.jpg)
Large Car Manufacturer: Connected vehicle
�84
Telemetry data from connected vehicles
transmitted and published to Pulsar
Data cleansing, enrichment and refinement processed
inside Pulsar
Data made available to internal teams for analysis
and reports
Data feeds supplied to partners and partner
applications
![Page 85: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/85.jpg)
ScenarioContinuously ingest logs from big data system for distributed to appropriate teams with appropriate log transformations and enrichment
ChallengesRequire scalability to handle growing set of big data systems and larger log volumes
SolutionLeverage Streamlio Pulsar solution to provide logging backbone that can ingest, transform, and distribute logs at scale
Large Car Manufacturer: Big Data Logging System
�85
![Page 86: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/86.jpg)
Large Car Manufacturer: Big Data Logging System
�86
Pulsar functions to route and transform logs to different teams
Team 1 logs
Team 2 logs
![Page 87: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/87.jpg)
Connected consumer
�87
Connected consumer electronic
devices
Emit event data that is collected and
processed in Pulsar
Generating notifications and work requests
Distributed to microservices for
processing
Supporting connected services
and applications
![Page 88: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/88.jpg)
�88
✓ Understanding How Pulsar Works https://jack-vanlightly.com/blog/2018/10/2/understanding-how-apache-pulsar-
works
✓ How To (Not) Lose Messages on Apache Pulsar Cluster https://jack-vanlightly.com/blog/2018/10/21/how-to-not-lose-messages-on-an-
apache-pulsar-cluster
MORE READINGS
![Page 89: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/89.jpg)
MORE READINGS
�89
✓ Unified queuing and streaming https://streaml.io/blog/pulsar-streaming-queuing
✓ Segment centric storage https://streaml.io/blog/pulsar-segment-based-architecture
✓ Messaging, Storage or Both https://streaml.io/blog/messaging-storage-or-both
✓ Access patterns and tiered storage https://streaml.io/blog/access-patterns-and-tiered-storage-in-apache-pulsar
✓ Tiered Storage in Apache Pulsar https://streaml.io/blog/tiered-storage-in-apache-pulsar
![Page 90: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/90.jpg)
QUESTIONS
�90
![Page 92: UNIFYING MESSAGING, QUEUING, STREAMING ......2019/06/27 · APACHE PULSAR 8 Flexible Messaging + Streaming System backed by a durable log storage Key Concepts Core concepts: Tenants,](https://reader035.fdocuments.in/reader035/viewer/2022071103/5fdd45e6607c45709921428e/html5/thumbnails/92.jpg)
�92
@karthikz