Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh
Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout
-
Upload
confluent -
Category
Engineering
-
view
4.054 -
download
5
Transcript of Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout
![Page 1: Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout](https://reader036.fdocuments.in/reader036/viewer/2022062401/586f758f1a28ab10258b60dd/html5/thumbnails/1.jpg)
KAFKA +• Building the World's Realtime Transit Infrastructure
![Page 2: Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout](https://reader036.fdocuments.in/reader036/viewer/2022062401/586f758f1a28ab10258b60dd/html5/thumbnails/2.jpg)
For Illustration only
![Page 3: Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout](https://reader036.fdocuments.in/reader036/viewer/2022062401/586f758f1a28ab10258b60dd/html5/thumbnails/3.jpg)
![Page 4: Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout](https://reader036.fdocuments.in/reader036/viewer/2022062401/586f758f1a28ab10258b60dd/html5/thumbnails/4.jpg)
SURGE - CIRCA 2013
![Page 5: Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout](https://reader036.fdocuments.in/reader036/viewer/2022062401/586f758f1a28ab10258b60dd/html5/thumbnails/5.jpg)
SURGE - CIRCA 2016
![Page 6: Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout](https://reader036.fdocuments.in/reader036/viewer/2022062401/586f758f1a28ab10258b60dd/html5/thumbnails/6.jpg)
DATA CONSUMERS
Real-time, Fast Analytics
BATCH PIPELINE
Storm
ApplicationsData Science
AnalyticsReporting
KAFKA
VERTICA
RIDER APP
DRIVER APP
API / SERVICES
DISPATCH (gps logs)
Mapping & Logistic Ad-hoc
exploration
ELK
Samza Alerts,Dashboards
Debugging
REAL-TIME PIPELINE
HADOOP
Surge Mobile App
DATA PRODUCERS
KAFKA 8 ECOSYSTEM @UBER
![Page 7: Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout](https://reader036.fdocuments.in/reader036/viewer/2022062401/586f758f1a28ab10258b60dd/html5/thumbnails/7.jpg)
Product Features
Predictive Models
Operational Analytics
Business Intelligence
INFRASTRUCTURE ECOSYSTEM
![Page 8: Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout](https://reader036.fdocuments.in/reader036/viewer/2022062401/586f758f1a28ab10258b60dd/html5/thumbnails/8.jpg)
NEAR REALTIME PRICE SURGING
PRODUCT FEATURES
![Page 9: Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout](https://reader036.fdocuments.in/reader036/viewer/2022062401/586f758f1a28ab10258b60dd/html5/thumbnails/9.jpg)
FRAUD -ANOMALY DETECTION
PREDICTIVE MODELS
![Page 10: Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout](https://reader036.fdocuments.in/reader036/viewer/2022062401/586f758f1a28ab10258b60dd/html5/thumbnails/10.jpg)
PREDICTIVE MODELS
ETA
![Page 11: Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout](https://reader036.fdocuments.in/reader036/viewer/2022062401/586f758f1a28ab10258b60dd/html5/thumbnails/11.jpg)
OPERATIONAL ANALYTICS
![Page 12: Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout](https://reader036.fdocuments.in/reader036/viewer/2022062401/586f758f1a28ab10258b60dd/html5/thumbnails/12.jpg)
UberEATs
OPERATIONAL ANALYTICS
![Page 13: Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout](https://reader036.fdocuments.in/reader036/viewer/2022062401/586f758f1a28ab10258b60dd/html5/thumbnails/13.jpg)
XP
OPERATIONAL ANALYTICS
![Page 14: Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout](https://reader036.fdocuments.in/reader036/viewer/2022062401/586f758f1a28ab10258b60dd/html5/thumbnails/14.jpg)
BUSINESS INTELLIGENCE
![Page 15: Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout](https://reader036.fdocuments.in/reader036/viewer/2022062401/586f758f1a28ab10258b60dd/html5/thumbnails/15.jpg)
KAFKA 8KAFKA 7 MIGRATOR
Limited Availability
Difficult to Scale
Not multi-DC Multi-lang incompatibility Multi-DC, multi-languagesupport
2013
2014
2015 - 2016
KAFKA 7 WORLD
Difficult to Operate
Producer Scale Issues
High Availability
High Scalability
Kafka 7 + Mirrormaker
Deployed everywhere
Kafka 7 migratorDeployed
everywhere
New Kafka 8pipeline
![Page 16: Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout](https://reader036.fdocuments.in/reader036/viewer/2022062401/586f758f1a28ab10258b60dd/html5/thumbnails/16.jpg)
Kafka 7
Mirrormaker
2.0
Rest architectu
re
Data AuditAutomatedTopic Mgmt
![Page 17: Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout](https://reader036.fdocuments.in/reader036/viewer/2022062401/586f758f1a28ab10258b60dd/html5/thumbnails/17.jpg)
Logs Business events
Async REST library
Data Audit
Local spooling
High throughput custom protocol
REST ARCHITECTURE
Rest Proxy
![Page 18: Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout](https://reader036.fdocuments.in/reader036/viewer/2022062401/586f758f1a28ab10258b60dd/html5/thumbnails/18.jpg)
Automated Schema and Topic Management
![Page 19: Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout](https://reader036.fdocuments.in/reader036/viewer/2022062401/586f758f1a28ab10258b60dd/html5/thumbnails/19.jpg)
Mirrormaker 2.0
Robust
Data Audit
Dynamic topics
MIRROR MAKER 2.0
Destination DCSource DC
![Page 20: Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout](https://reader036.fdocuments.in/reader036/viewer/2022062401/586f758f1a28ab10258b60dd/html5/thumbnails/20.jpg)
Msg counts across multiple DCs
End-end latencies across multiple DCs
DATA AUDIT FOR KAFKA MESSAGES
![Page 21: Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout](https://reader036.fdocuments.in/reader036/viewer/2022062401/586f758f1a28ab10258b60dd/html5/thumbnails/21.jpg)
Mirrormaker
2.0
Rest architectu
re
Data Audit Kafka 8
AutomatedTopic Mgmt
![Page 22: Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout](https://reader036.fdocuments.in/reader036/viewer/2022062401/586f758f1a28ab10258b60dd/html5/thumbnails/22.jpg)
A ROBUST FUTURE
0 data loss messaging systemData discovery and lineageQuota managementSelf-correcting brokersActive active data pipelines
![Page 23: Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout](https://reader036.fdocuments.in/reader036/viewer/2022062401/586f758f1a28ab10258b60dd/html5/thumbnails/23.jpg)
Real-time Data
Dynamic SQL(ish)
Real-time decision
THE FUTURE
Real-time Data
Custom Application
Real-time decision
THE PRESENT
![Page 24: Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout](https://reader036.fdocuments.in/reader036/viewer/2022062401/586f758f1a28ab10258b60dd/html5/thumbnails/24.jpg)
TELEMATICS
![Page 25: Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout](https://reader036.fdocuments.in/reader036/viewer/2022062401/586f758f1a28ab10258b60dd/html5/thumbnails/25.jpg)
SELF DRIVING CAR
![Page 26: Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout](https://reader036.fdocuments.in/reader036/viewer/2022062401/586f758f1a28ab10258b60dd/html5/thumbnails/26.jpg)
![Page 27: Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout](https://reader036.fdocuments.in/reader036/viewer/2022062401/586f758f1a28ab10258b60dd/html5/thumbnails/27.jpg)
Thank you, Kafka Community!