Elastic @ GetYourGuide Click to edit Master title style Elastic Meetup @ GYG … · By Floris...
Transcript of Elastic @ GetYourGuide Click to edit Master title style Elastic Meetup @ GYG … · By Floris...
By Floris Dekker – September 2nd, 2013Click to edit Master title styleBy Tobias Rein & Alex Eftimie – October 12th, 2016Elastic @ GetYourGuide
5 students wanted to build apeer-to-peer platform for tour guides.
But we also learned one thing:
There is a greater problem to be solved.
So the team started from scratch . . .
. . . and we relaunched
is me e too o
e ee p e
e
1
We built a unique culture.
@GetYourGuide
In 2012 …
… we turned MySQL into elasticsearch
0.19.80.90.01.0.02.3.1
Upgrading was usually painless, thanks to Elastica (A PHP client for elasticsearch.)
MySQL is still our primary storage.
We are using asynchronous messages to update the elasticsearch index
Our Setup
Our architecture
1. Location lookup[elasticsearch]
2. Tour search[elasticsearch]
3. Availability filter[MySQL]
Multi-step
Our Cluster
56 indexesfor customer search
26 indexesfor intranet
4 indexes in 14 languages up to 2.5 million docs per index
3 nodes 8.1 GB
2 replicas, no sharding 10 million documents
Our migration strategy
• Unique index namesusing timestamp
• An alias <name>_live points to the current index
• For migration we use temporary alias names <name>_temp
@GetYourGuide
ELK
What do we do with logs @ GYG ?
• Debugging • Business Intelligence
Motivation for a new infrastructure
• SSH to host no longer an option
• Local storage in the cloud is limited
• No one uses outdated tools
• Fault tolerant trustworthy logs are a must!
• Redundancy, Auto failover, Monitoring
Logging infrastructure - FLKLEK
Two m4.2xlarge AWS instances
(One tie breaker instance in Digitalocean)
The Numbers
120 GB 20 Mioelasticsearch store size log lines /day
42 5.5Knodes avg. logs /second
Lessons learned
• Large documents affect ES performance
• Dynamic mappings are a pain
• Beats don’t play well with file rotation
• Kibana is a powerful tool
29
Questions?