Tubular Labs - Using Elastic to Search Over 2.5B Videos
-
Upload
tubular-labs -
Category
Software
-
view
126 -
download
1
Transcript of Tubular Labs - Using Elastic to Search Over 2.5B Videos
Talk structure
● 4 steps to make user experience great again
● 4 patterns to simplify architecture and reduce costs
© 2016 Tubular Labs
2
Data size
● 2.5B documents
● AVG doc size 2Kb, 4Tb total size
● 200M daily updates (~8% of the index)
● Constant indexing rate of 3k/s with spikes
● Querying rate 1-3 r/s (low concurrency)
© 2016 Tubular Labs
3
Hardware
● 52 x c3.4xlarge
● 128 shards
● 16 cores per node
● ~3 shards per node
● 832 cores, 16Tb
SSD, 1.5Tb RAM
© 2016 Tubular Labs
4
● 26 x c3.8xlarge
● 416 shards
● 32 cores per node
● 16 shards per node
● 832 cores, 16Tb
SSD, 1.5Tb RAM
Before After (25% bigger)
Optimize indexing
● Using bulk API• 1Mb per batch (500 docs), should be 5k docs/s
• Recommended 5-15Mb
● Increasing refresh interval• From 1 to 30 seconds
● Monitoring bulk.rejected• Increased bulk.queueSize from 50 to 2000
© 2016 Tubular Labs
6
Goal
© 2016 Tubular Labs
10
• Slow queries • From 15 to 5 seconds for 95th
• Seeking for 3x improvement
Problem Goal
Understand hardware utilization
© 2016 Tubular Labs
11
• Run the heaviest query
• No bottlenecks (CPU, disk IO, network)
• Thread pool search.size 25
• Max search.active is 3
Search & Aggregations
© 2016 Tubular Labs
15
• Searching and sorting
is fast
• 8 term aggregations
are slow
Check facet usage
© 2016 Tubular Labs
17
● Talk to your product manager
● Low product usage
● Remove networks and claims aggregations
● Replace facets with filters
Cardinality
© 2016 Tubular Labs
19
● Reduce cardinality
● Going from 200M to 5M (channels to creators)
● Reducing # of topics from 5M to 500
Split query and aggregations
© 2016 Tubular Labs
21
● Searching and aggregating separately
● Using shard-level query cache
● Showing results in UI asynchronously
Performance gain
© 2016 Tubular Labs
23
● From 15 to 4 seconds (<5 seconds)
● Overall improvement 3.7x
● What about costs?
Part 2. Goals
© 2016 Tubular Labs
25
● Reduce costs
● Improve reliability
● Simplify architecture
● Reduce variability in latency
Split data by hot/full (idea for future)
© 2016 Tubular Labs
30
● Cheaper
hardware on full
● Shard allocation
filtering