Managing Capacity @ LinkedIn · Linkedin Espresso Hadoop Collection Ingestion Processing / Store...
26
Managing Capacity @ LinkedIn Anuprita Harkare Site Reliability Engineer, LinkedIn
Transcript of Managing Capacity @ LinkedIn · Linkedin Espresso Hadoop Collection Ingestion Processing / Store...
Managing Capacity @ LinkedIn
Anuprita Harkare
Site Reliability Engineer, LinkedIn
Agenda
To edit the table….
• Right click your mouse, select “Insert” or ”Delete” on drop down menu
Linkedin’s data footprint
The Data pipeline
Underlying Components
Tools & Metrics
Q&A
Entire Production Footprint
1.6MWInstalled power
2.5KServers
Equinix Chicago1.2K Servers | 800kW
Equinix Los Angeles 1.2K Servers | 800kW
2010
Production Application Footprint
29MWInstalled capacity
95KServers
Richardson, TX (LTX1)30K Servers | 7.2MW
Ashburn, VA (LVA1)34K Servers | 9.2MW
Hillsboro, OR (LOR1)20K Servers | 8MW
Singapore (LSG1)11K Servers | 4.2MW
2017
Storage By the Numbers
170PB
HDFS Storage
4.2PB
MySQL Storage
140TB 1.3PB
Oracle StorageEspresso Storage
1600Kafka Brokers
2T
Kafka messages per day
Linkedin’s pipeline
Gobblin
LumosEspresso
3rd Party Services
Collection
Oracle DB
Tracking
Lumos Landing
Zone
ODS Teradata
Ingestion Processing / Store Reporting
Proprietary tools
Tableau
MicroStrategy
Hadoop
Linkedin’s pipeline
Gobblin
Lumos
3rd Party Services
Oracle DB
Tracking
Lumos Landing
Zone
ODS Teradata Proprietary tools
Collection Ingestion Processing / Store Reporting
MicroStrategy
Tableau
LinkedinEspresso
Hadoop
Linkedin’s pipeline
Gobblin
Lumos
3rd Party Services
Oracle DB
Tracking
Lumos Landing
Zone
ODS Teradata Proprietary tools
LinkedinEspresso
Hadoop
Collection Ingestion Processing / Store Reporting
MicroStrategy
Tableau
EspressoLumos
Landing Zone
Tracking
Linkedin’s pipeline
Gobblin
Lumos
3rd Party Services
Oracle DB
Tracking
Lumos Landing
Zone
ODS Teradata Proprietary tools
LinkedinEspresso
Hadoop
Collection Ingestion Processing / Store Reporting
MicroStrategy
Tableau
EspressoLumos
Landing Zone
Tracking
Linkedin’s pipeline
Gobblin
Lumos
3rd Party Services
Oracle DB
Tracking
Lumos Landing
Zone
ODS Teradata Proprietary tools
LinkedinEspresso
Hadoop
Collection Ingestion Processing / Store Reporting
MicroStrategy
Tableau
EspressoLumos
Landing Zone
Tracking
Capacity planning for Major Components
Espresso
EspressoKey Value SOT
● Distributed Document Store
● Strongly Consistent
● Provide features between NoSQL and RDBMS
Stores LinkedIn’s Member Profile Data.
Espresso
Espresso
EspressoKey Value SOT
Espresso Footprint
237
Databases
4.2PB
Storage Capacity
Espresso Capacity Planning
Organic Growth New Projects/ DBs Unplanned Scenarios
Capacity model : Organic growth
InGraph (Disk Usage)
● Disk utilisation is not always linear
● Anomalies to be ignored
● Not max but percentile based calculation (~98th percentile)
● Quarterly projections are derived over weekly growth trend in previous quarter
● Buffer over projected capacity
● Distributed streaming platform
● Backbone of Linkedin’s data pipeline
● Metrics, member activity data, change data capture and much more
Kafka Footprint
Kafka Servers
2T
Kafka Events per Day
1600
Kafka Capacity Planning
● Disk usage bound
● Cap disk utilisation at 60%
● Horizontally scalable
● Configurable retention
● Kafka rebalances to keep utilization under 40%
Kcap - Kafka Capacity Auditor
● Distributed storage and processing framework
● It can store any kind of data
● Write once read many
Hadoop
Hadoop Footprint
170PB10K
Hadoop Nodes HDFS Storage
3.6K
Hadoop Users
145K
YARN jobs/per day
Hadoop Capacity Planning
Disk Utilization Memory Utilization CPU Utilization
Hadoop capacity management tools
Thank You
Q&A
