[@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ
-
Upload
indeedeng -
Category
Technology
-
view
5.506 -
download
0
description
Transcript of [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ
![Page 1: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/1.jpg)
How to Get a Job 35 Million Times a Day Using RabbitMQ
Ketan Gangatirkar and Cameron Davison
![Page 2: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/2.jpg)
One search. All jobs.
![Page 3: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/3.jpg)
Aggregation gets jobs
![Page 4: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/4.jpg)
Aggregation gets jobs soJobseekers get jobs
![Page 5: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/5.jpg)
Aggregation != Spidering
Spiders see pages.
Aggregation sees jobs.
![Page 6: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/6.jpg)
How spiders see job sites
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
Page
![Page 7: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/7.jpg)
How Indeed sees job sitesStart
Job List
Job Job Job
Job List
Job Job Job
Job List
Job Job Job
Navigation Navigation
JobJob
Job
![Page 8: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/8.jpg)
Aggregation != Spidering
Job sites have structure
Job pages have semantics
Navigation is more than following links
![Page 9: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/9.jpg)
![Page 10: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/10.jpg)
Rememberthis
![Page 11: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/11.jpg)
Aggevery
job
![Page 12: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/12.jpg)
![Page 13: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/13.jpg)
{ Url: http://www.applytracking.com/track.aspx/3VYzR Title: Senior Erlang Engineer Company: Machine Zone Location: Palo Alto,CA,US, 94301 Source Type: Employer Job Type: Full-time ... Description: The Senior Erlang Engineer is an integral ... ... Createdate: 2013-02-05 23:18:05 ...}
What's in a job
![Page 14: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/14.jpg)
location
description
Company
Title
![Page 15: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/15.jpg)
Title
salary
locationjob type
description
Company
![Page 16: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/16.jpg)
![Page 17: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/17.jpg)
How we build products
simple
fast
comprehensive
relevant
![Page 18: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/18.jpg)
Simple
Tough problems, simple solutions
![Page 19: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/19.jpg)
Fast
Discover the jobs quickly
Get them to jobseekers in minutes
![Page 20: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/20.jpg)
10% of jobseekers sort by date
![Page 21: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/21.jpg)
Do you want only new jobs?
![Page 22: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/22.jpg)
20% of jobseekers want only new jobs
![Page 23: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/23.jpg)
Daily new job emails
![Page 24: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/24.jpg)
Speed matters
![Page 25: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/25.jpg)
Comprehensive
Get every job
![Page 26: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/26.jpg)
Relevant
Semantic extraction
The job is still available
Ignore non-jobs
![Page 27: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/27.jpg)
This is a hard problem
Flaky sites
Site redesigns
Javascript
Missing or bad information
![Page 28: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/28.jpg)
Big N makes it even harder
Examine 38M jobs every day
![Page 29: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/29.jpg)
Do this in minutes
Search 100MJobseekersAggregation
EmployersJob BoardsStaffing firmsRecruiters
![Page 30: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/30.jpg)
Strawman* architecture
Datacenter B
MySQL
Engine
Datacenter A
Job site
Engine
Job site
Engine
Job site
Engine
Job site
Engine
Job site
Engine
Job site
Primary Datacenter
![Page 31: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/31.jpg)
Limitations
![Page 32: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/32.jpg)
N connections
MySQL
Job siteJob siteJob siteJob siteJob siteJob site
Primary Datacenter
EngineEngineEngineEngineEngineEngine
Datacenter BDatacenter A
![Page 33: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/33.jpg)
N concurrent writers
MySQL
Job siteJob siteJob siteJob siteJob siteJob site
Primary Datacenter
EngineEngineEngineEngineEngineEngine
Datacenter BDatacenter A
![Page 34: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/34.jpg)
High latency
MySQL
Job siteJob siteJob siteJob siteJob siteJob site
Primary Datacenter
EngineEngineEngineEngineEngineEngine
Datacenter BDatacenter A
![Page 35: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/35.jpg)
Limitation: failure points
Datacenter B
MySQL
Engine
Datacenter A
Job site
Engine
Job site
Engine
Job site
Engine
Job site
Engine
Job site
Engine
Job site
Primary Datacenter
X
X
![Page 36: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/36.jpg)
Scaling Patterns
What has worked for us so far?
![Page 37: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/37.jpg)
Service-Oriented Architecture
Engine
Engine
Engine
Job Write Service MySQL
RemoteDatacenter
PrimaryDatacenter
see http://go.indeed.com/boxcar
![Page 38: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/38.jpg)
Standard Service Interaction
Client Service Database
![Page 39: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/39.jpg)
Our Interaction
Client Service Database
![Page 40: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/40.jpg)
Does this do what we need?
● Lots of workers...● Sending lots of results...● Over a long distance...● That need to get processed fast...● Reliably?
![Page 41: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/41.jpg)
Engine Failure
Engine
Engine
Engine
Job Write Service MySQL
RemoteDatacenter
XPrimaryDatacenter
![Page 42: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/42.jpg)
Engine failure fix:Buffer to disk
Engine
Engine
Engine
Job Write Service MySQL
RemoteDatacenter
disk
disk
disk
PrimaryDatacenter
X
![Page 43: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/43.jpg)
Network Failure
Engine
Engine
Engine
Job Write Service MySQL
RemoteDatacenter
XPrimaryDatacenter
![Page 44: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/44.jpg)
Network failure fix:Disks solve that too
Engine
Engine
Engine
Job Write Service MySQL
RemoteDatacenter
disk
disk
disk
XPrimaryDatacenter
![Page 45: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/45.jpg)
Write Service Failure
Job Write Service MySQL
RemoteDatacenter
XEngine
Engine
Engine
PrimaryDatacenter
![Page 46: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/46.jpg)
Write Service Failure fix:Disks solve that too
Job Write Service MySQL
RemoteDatacenter
XEngine
Engine
Engine
PrimaryDatacenter
disk
disk
disk
![Page 47: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/47.jpg)
Write Service Failure fix:Redundancy
Job Write Service
MySQL
RemoteDatacenter
PrimaryDatacenter
XEngine
Engine
Engine
Job Write Service
Job Write Service
![Page 48: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/48.jpg)
Database Failure
Job Write Service MySQL
RemoteDatacenter
XEngine
Engine
Engine
PrimaryDatacenter
![Page 49: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/49.jpg)
Database Failure fix:Buffer to disk
Job Write Service
MySQL
RemoteDatacenter
XEngine
Engine
Engine
disk
PrimaryDatacenter
![Page 50: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/50.jpg)
Our new architectureJob Write Service
MySQL
RemoteDatacenter
PrimaryDatacenter
Engine
Engine
Engine
disk
disk
disk
Job Write Service
Job Write Service
disk
disk
disk
![Page 51: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/51.jpg)
We could build this...Job Write Service
MySQL
RemoteDatacenter
PrimaryDatacenter
Engine
Engine
Engine
disk
disk
disk
Job Write Service
Job Write Service
disk
disk
disk
![Page 52: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/52.jpg)
... maybe someone already hasJob Write Service
MySQL
RemoteDatacenter
PrimaryDatacenter
Engine
Engine
Engine
disk
disk
disk
Job Write Service
Job Write Service
disk
disk
disk
![Page 53: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/53.jpg)
We should use a message queue
![Page 54: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/54.jpg)
Cameron Davison
![Page 55: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/55.jpg)
Aggregation Requirements
● Durable
● Multi-Data Center (latency)
● 38 million jobs a day
● 2KB average job size○ 76 GB a day
● Target peaks of 1000 jobs / second
● Programming language agnostic
![Page 56: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/56.jpg)
Selection
![Page 57: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/57.jpg)
What we found
High Availability
Open Source/Free
Self-hosted
Performant
![Page 58: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/58.jpg)
Out-of-the-box Experience
![Page 59: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/59.jpg)
Advanced Message Queuing Protocol (AMQP)
● Open Standard
● Wire protocol
● Existing Clients in Multiple Languages
![Page 60: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/60.jpg)
Concepts
● Confirmation and Ack
● At least once
● Asynchronous Confirms
● Persistent
● Clustering
![Page 61: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/61.jpg)
Confirmation and Ack
MQ
Producer Consumer
msg
confi
rm
ackmsg
1
2 3
4
![Page 62: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/62.jpg)
At least once
MQ
At most once
Consumer
Message
Ack
MQ ConsumerMessage
Auto Ack
![Page 63: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/63.jpg)
Asynchronous Confirms1
2
3
4
5
6
7
8
9
1011
12
13
14
15
16
Producer
messages
confirm #6
![Page 64: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/64.jpg)
Persistent
MQ
Producer Consumer
![Page 65: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/65.jpg)
Persistent
MQ
Producer Consumer
![Page 66: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/66.jpg)
Persistent
MQ
Producer Consumer
X
![Page 67: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/67.jpg)
Persistent
MQ
Producer Consumer
![Page 68: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/68.jpg)
Persistent
MQ
Producer Consumer
![Page 69: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/69.jpg)
Clustering
SlaveMaster
Producer
1
2
3
4
![Page 70: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/70.jpg)
Testing
![Page 71: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/71.jpg)
Test RabbitMQ
● Send millions of 2KB messages
● 20 producers and 20 consumers
● 1000 messages / second
● Simulate multiple failures
![Page 72: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/72.jpg)
Test Consistency
Producers
RabbitMQ
RabbitMQ
Consumers
Slave
Master
![Page 73: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/73.jpg)
Test Consistency
Producers
RabbitMQ
RabbitMQ
Consumers
Master
Slave
![Page 74: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/74.jpg)
Test Consistency
Producers
RabbitMQ
RabbitMQ
Consumers
Master
Slave
![Page 75: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/75.jpg)
Test Consistency
Producers
RabbitMQ
RabbitMQ
Consumers
X
Master
![Page 76: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/76.jpg)
Test Consistency
Producers
RabbitMQ
RabbitMQ
Consumers
X
Master
![Page 77: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/77.jpg)
Test Consistency
Producers
RabbitMQ
RabbitMQ
Consumers
Master
Slave
![Page 78: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/78.jpg)
RabbitMQ Clustering
Master Slave
![Page 79: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/79.jpg)
RabbitMQ Clustering
Master Slave
![Page 80: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/80.jpg)
RabbitMQ Clustering
Master
X
![Page 81: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/81.jpg)
RabbitMQ Clustering
Master
X
![Page 82: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/82.jpg)
RabbitMQ Clustering
MasterSlave
![Page 83: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/83.jpg)
RabbitMQ Clustering
MasterSlave
![Page 84: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/84.jpg)
RabbitMQ Clustering
MasterSlave
![Page 85: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/85.jpg)
RabbitMQ Clustering
MasterSlave
![Page 86: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/86.jpg)
RabbitMQ Clustering
MasterSlave
![Page 87: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/87.jpg)
RabbitMQ Clustering
MasterSlave
![Page 88: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/88.jpg)
RabbitMQ Clustering
Master
X
![Page 89: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/89.jpg)
RabbitMQ Clustering
MasterSlave
![Page 90: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/90.jpg)
RabbitMQ Clustering
MasterSlave
![Page 91: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/91.jpg)
RabbitMQ Clustering
MasterSlave
![Page 92: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/92.jpg)
RabbitMQ Clustering
Master
X
![Page 93: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/93.jpg)
RabbitMQ Clustering
Master
XX
![Page 94: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/94.jpg)
RabbitMQ Clustering
Master
X
![Page 95: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/95.jpg)
RabbitMQ Clustering
Master
X
![Page 96: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/96.jpg)
RabbitMQ Clustering
Master Slave
![Page 97: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/97.jpg)
Non-persistent
15990 Messages / Second30 MB/s
![Page 98: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/98.jpg)
Persistent
2781 Message / Second5.5 MB/s
![Page 99: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/99.jpg)
Clustered and Persistent
1262 Message / Second2.5 MB/s
![Page 100: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/100.jpg)
Applying RabbitMQ
![Page 101: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/101.jpg)
Unreliable High Latency Connections
Engine
Engine
Engine
Job Write Service
Remote DC Primary DC
MySQL
![Page 102: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/102.jpg)
Replaced with RabbitMQ
Engine
Engine
Engine
Job Write ServiceRabbit
MQ
Remote DC Primary DC
MySQL
![Page 103: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/103.jpg)
Replaced with RabbitMQ
Engine
Engine
Engine
Job Write ServiceRabbit
MQ
Remote DC Primary DC
![Page 104: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/104.jpg)
Replaced with RabbitMQ
Engine
Engine
Engine
Job Write ServiceRabbit
MQ
Remote DC Primary DC
![Page 105: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/105.jpg)
Replaced with RabbitMQ
Engine
Engine
Engine
Job Write Service
Remote DC Primary DC
RabbitMQ
![Page 106: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/106.jpg)
Replaced with RabbitMQ
Engine
Engine
Engine
Job Write Service
Remote DC Primary DC
RabbitMQ
![Page 107: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/107.jpg)
Rabbit can talk to Rabbit
Shovel Plugin
Producer RabbitMQ 1 ConsumerRabbitMQ 2
![Page 108: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/108.jpg)
Replaced with RabbitMQ
Engine
Engine
Engine
Job Write ServiceRabbit
MQ
RabbitMQ
RabbitMQ
RabbitMQ
Remote DC Primary DC
![Page 109: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/109.jpg)
Replaced with RabbitMQ
Engine
Engine
Engine
Job Write ServiceRabbit
MQ
RabbitMQ
RabbitMQ
RabbitMQ
Primary DC
RabbitMQ
Remote DC
![Page 110: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/110.jpg)
Parallelize Job Write Service
RabbitMQ
Job Write Service
Job Write Service
Job Write Service
Job A
Job B
Job C
![Page 111: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/111.jpg)
Replaced with RabbitMQ
Engine
Engine
Engine
Job Write ServiceRabbit
MQ
RabbitMQ
RabbitMQ
RabbitMQ
Primary DC
RabbitMQ
Job Write Service
Remote DC
![Page 112: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/112.jpg)
Replaced with RabbitMQ
Engine
Engine
Engine
Job Write ServiceRabbit
MQ
RabbitMQ
RabbitMQ
RabbitMQ
Primary DC
RabbitMQ
Job Write Service
![Page 113: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/113.jpg)
Message Flow
![Page 114: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/114.jpg)
Message Flow
Engine
Engine
Engine
Job Write Service
Primary DC
Job Write Service
RabbitMQ
RabbitMQ
RabbitMQ
RabbitMQ
RabbitMQ
![Page 115: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/115.jpg)
Message Flow
Engine
Engine
Engine
Job Write Service
Primary DC
Job Write Service
RabbitMQ
RabbitMQ
RabbitMQ
RabbitMQ
RabbitMQ
![Page 116: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/116.jpg)
Message Flow
Engine
Engine
Engine
Job Write Service
Primary DC
Job Write Service
RabbitMQ
RabbitMQ
RabbitMQ
RabbitMQ
RabbitMQ
![Page 117: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/117.jpg)
Message Flow
Engine
Engine
Engine
Job Write Service
Primary DC
Job Write Service
RabbitMQ
RabbitMQ
RabbitMQ
RabbitMQ
RabbitMQ
![Page 118: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/118.jpg)
Jobs/minute
![Page 119: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/119.jpg)
Jobs/minute from one site
220,000 jobs6 hours
611 jobs / minute
![Page 120: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/120.jpg)
Jobs/minute from one site
251,000 jobs20 minutes
12550 jobs / minute
![Page 121: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/121.jpg)
RabbitMQ
Horizontal Scale
Engine
Engine
Engine Job Write ServiceRabbit
MQ
Job Write Service
RabbitMQ
RabbitMQ
RabbitMQ
RabbitMQ
RabbitMQ
Job Write Service
Job Write Service
![Page 122: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/122.jpg)
Horizontal Scale
![Page 123: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/123.jpg)
Horizontal Scale
![Page 124: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/124.jpg)
Today 1000 messages / second
![Page 125: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/125.jpg)
RabbitMQ 3
2486 Message / Second5MB/s
![Page 126: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/126.jpg)
RabbitMQ Configuration
● Confirmations - Fire and Forget
● Persistent Messages - Durable
● Shoveling - Multi-Data Center
● Mirrored Queues in Cluster - High Reliability
![Page 127: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/127.jpg)
Can we do more with RabbitMQ?
![Page 128: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/128.jpg)
Aggregation Viewer
Real-time browser-based view of job stream
![Page 129: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/129.jpg)
● Almost real-time● Exclusive queue● Transient messages
Aggregation Viewer Architecture
Agg JobsRabbit MQ
ClusterAgg ViewerRabbit MQ
Agg Viewer
Shovel* SubscribeJobs HTTP Browser
![Page 130: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/130.jpg)
Resume Contacts Billing
Pay-per-contact: limited budget
![Page 131: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/131.jpg)
![Page 132: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/132.jpg)
Resume Contacts BillingOriginal Path
Pacific
Asia DC US DC
Log repoResume Search
MySQL
see http://go.indeed.com/logrepo
![Page 133: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/133.jpg)
Resume Contacts BillingFast Path
Pacific
Asia DC US DC
RabbitMQ
MySQL
Log repo
RabbitMQ
Resume Search
X
![Page 134: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/134.jpg)
Company Page Edits
User-contributed content about companies
![Page 135: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/135.jpg)
Company Page
![Page 136: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/136.jpg)
Company Page EditsImplementation
Writing data AND reading it back
![Page 137: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/137.jpg)
Company Page EditsSingle Datacenter
Browser
Web Server MySQL
![Page 138: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/138.jpg)
Company Page Serving
Browser
Web Server
LSM Tree
Asia Datacenter
Memcached
see http://go.indeed.com/lsmtree
![Page 139: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/139.jpg)
Pacific
Company Page Edits
Browser
Web Server
RabbitMQ RabbitMQ MySQL
Primary US Datacenter
Asia Datacenter EU Datacenter
Atlantic
[Et cetera]
Memcached
![Page 140: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/140.jpg)
Pacific
Company Page Reads
MySQL
LSM TreeBuilderLSM Tree
Primary US Datacenter
Asia Datacenter
LSM Tree
EU Datacenter
Atlantic
[Et cetera]
![Page 141: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/141.jpg)
Memcached
Pacific
Company Pages System
Browser
Web Server
RabbitMQ RabbitMQ MySQL
LSM TreeBuilderLSM Tree
Primary US Datacenter
Asia Datacenter
LSM Tree
EU Datacenter
Atlantic
[Et cetera]
![Page 142: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/142.jpg)
Other applications
![Page 143: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/143.jpg)
Company Pages
![Page 144: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/144.jpg)
![Page 145: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/145.jpg)
![Page 146: [@IndeedEng] How to Get a Job 35 Million Times a Day Using RabbitMQ](https://reader034.fdocuments.in/reader034/viewer/2022051611/54b7b6084a7959c9688b46d2/html5/thumbnails/146.jpg)
Recap: The jobs must flow
● Durability● High throughput● Low latency● Partition-tolerance● Efficient use of the database● Minimal points of failure