Elastic Computing and Engineering Elastic Applications in ...
Elastic @ GetYourGuide Click to edit Master title stylefiles.meetup.com/7646592/2016-10-12 Elastic...
Transcript of Elastic @ GetYourGuide Click to edit Master title stylefiles.meetup.com/7646592/2016-10-12 Elastic...
![Page 1: Elastic @ GetYourGuide Click to edit Master title stylefiles.meetup.com/7646592/2016-10-12 Elastic Meetup @ GYG Zurich.pdfBy Floris Dekker – September 2nd, 2013 Click to edit Master](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f243c1b01e43f112637d56a/html5/thumbnails/1.jpg)
By Floris Dekker – September 2nd, 2013Click to edit Master title styleBy Tobias Rein & Alex Eftimie – October 12th, 2016Elastic @ GetYourGuide
![Page 2: Elastic @ GetYourGuide Click to edit Master title stylefiles.meetup.com/7646592/2016-10-12 Elastic Meetup @ GYG Zurich.pdfBy Floris Dekker – September 2nd, 2013 Click to edit Master](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f243c1b01e43f112637d56a/html5/thumbnails/2.jpg)
![Page 3: Elastic @ GetYourGuide Click to edit Master title stylefiles.meetup.com/7646592/2016-10-12 Elastic Meetup @ GYG Zurich.pdfBy Floris Dekker – September 2nd, 2013 Click to edit Master](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f243c1b01e43f112637d56a/html5/thumbnails/3.jpg)
5 students wanted to build apeer-to-peer platform for tour guides.
![Page 4: Elastic @ GetYourGuide Click to edit Master title stylefiles.meetup.com/7646592/2016-10-12 Elastic Meetup @ GYG Zurich.pdfBy Floris Dekker – September 2nd, 2013 Click to edit Master](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f243c1b01e43f112637d56a/html5/thumbnails/4.jpg)
![Page 5: Elastic @ GetYourGuide Click to edit Master title stylefiles.meetup.com/7646592/2016-10-12 Elastic Meetup @ GYG Zurich.pdfBy Floris Dekker – September 2nd, 2013 Click to edit Master](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f243c1b01e43f112637d56a/html5/thumbnails/5.jpg)
But we also learned one thing:
There is a greater problem to be solved.
![Page 6: Elastic @ GetYourGuide Click to edit Master title stylefiles.meetup.com/7646592/2016-10-12 Elastic Meetup @ GYG Zurich.pdfBy Floris Dekker – September 2nd, 2013 Click to edit Master](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f243c1b01e43f112637d56a/html5/thumbnails/6.jpg)
So the team started from scratch . . .
![Page 7: Elastic @ GetYourGuide Click to edit Master title stylefiles.meetup.com/7646592/2016-10-12 Elastic Meetup @ GYG Zurich.pdfBy Floris Dekker – September 2nd, 2013 Click to edit Master](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f243c1b01e43f112637d56a/html5/thumbnails/7.jpg)
. . . and we relaunched
![Page 8: Elastic @ GetYourGuide Click to edit Master title stylefiles.meetup.com/7646592/2016-10-12 Elastic Meetup @ GYG Zurich.pdfBy Floris Dekker – September 2nd, 2013 Click to edit Master](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f243c1b01e43f112637d56a/html5/thumbnails/8.jpg)
is me e too o
![Page 9: Elastic @ GetYourGuide Click to edit Master title stylefiles.meetup.com/7646592/2016-10-12 Elastic Meetup @ GYG Zurich.pdfBy Floris Dekker – September 2nd, 2013 Click to edit Master](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f243c1b01e43f112637d56a/html5/thumbnails/9.jpg)
e ee p e
e
1
![Page 10: Elastic @ GetYourGuide Click to edit Master title stylefiles.meetup.com/7646592/2016-10-12 Elastic Meetup @ GYG Zurich.pdfBy Floris Dekker – September 2nd, 2013 Click to edit Master](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f243c1b01e43f112637d56a/html5/thumbnails/10.jpg)
![Page 11: Elastic @ GetYourGuide Click to edit Master title stylefiles.meetup.com/7646592/2016-10-12 Elastic Meetup @ GYG Zurich.pdfBy Floris Dekker – September 2nd, 2013 Click to edit Master](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f243c1b01e43f112637d56a/html5/thumbnails/11.jpg)
We built a unique culture.
![Page 12: Elastic @ GetYourGuide Click to edit Master title stylefiles.meetup.com/7646592/2016-10-12 Elastic Meetup @ GYG Zurich.pdfBy Floris Dekker – September 2nd, 2013 Click to edit Master](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f243c1b01e43f112637d56a/html5/thumbnails/12.jpg)
![Page 13: Elastic @ GetYourGuide Click to edit Master title stylefiles.meetup.com/7646592/2016-10-12 Elastic Meetup @ GYG Zurich.pdfBy Floris Dekker – September 2nd, 2013 Click to edit Master](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f243c1b01e43f112637d56a/html5/thumbnails/13.jpg)
@GetYourGuide
![Page 14: Elastic @ GetYourGuide Click to edit Master title stylefiles.meetup.com/7646592/2016-10-12 Elastic Meetup @ GYG Zurich.pdfBy Floris Dekker – September 2nd, 2013 Click to edit Master](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f243c1b01e43f112637d56a/html5/thumbnails/14.jpg)
In 2012 …
… we turned MySQL into elasticsearch
![Page 15: Elastic @ GetYourGuide Click to edit Master title stylefiles.meetup.com/7646592/2016-10-12 Elastic Meetup @ GYG Zurich.pdfBy Floris Dekker – September 2nd, 2013 Click to edit Master](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f243c1b01e43f112637d56a/html5/thumbnails/15.jpg)
![Page 16: Elastic @ GetYourGuide Click to edit Master title stylefiles.meetup.com/7646592/2016-10-12 Elastic Meetup @ GYG Zurich.pdfBy Floris Dekker – September 2nd, 2013 Click to edit Master](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f243c1b01e43f112637d56a/html5/thumbnails/16.jpg)
0.19.80.90.01.0.02.3.1
Upgrading was usually painless, thanks to Elastica (A PHP client for elasticsearch.)
![Page 17: Elastic @ GetYourGuide Click to edit Master title stylefiles.meetup.com/7646592/2016-10-12 Elastic Meetup @ GYG Zurich.pdfBy Floris Dekker – September 2nd, 2013 Click to edit Master](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f243c1b01e43f112637d56a/html5/thumbnails/17.jpg)
MySQL is still our primary storage.
We are using asynchronous messages to update the elasticsearch index
Our Setup
![Page 18: Elastic @ GetYourGuide Click to edit Master title stylefiles.meetup.com/7646592/2016-10-12 Elastic Meetup @ GYG Zurich.pdfBy Floris Dekker – September 2nd, 2013 Click to edit Master](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f243c1b01e43f112637d56a/html5/thumbnails/18.jpg)
Our architecture
![Page 19: Elastic @ GetYourGuide Click to edit Master title stylefiles.meetup.com/7646592/2016-10-12 Elastic Meetup @ GYG Zurich.pdfBy Floris Dekker – September 2nd, 2013 Click to edit Master](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f243c1b01e43f112637d56a/html5/thumbnails/19.jpg)
1. Location lookup[elasticsearch]
2. Tour search[elasticsearch]
3. Availability filter[MySQL]
Multi-step
![Page 20: Elastic @ GetYourGuide Click to edit Master title stylefiles.meetup.com/7646592/2016-10-12 Elastic Meetup @ GYG Zurich.pdfBy Floris Dekker – September 2nd, 2013 Click to edit Master](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f243c1b01e43f112637d56a/html5/thumbnails/20.jpg)
Our Cluster
56 indexesfor customer search
26 indexesfor intranet
4 indexes in 14 languages up to 2.5 million docs per index
3 nodes 8.1 GB
2 replicas, no sharding 10 million documents
![Page 21: Elastic @ GetYourGuide Click to edit Master title stylefiles.meetup.com/7646592/2016-10-12 Elastic Meetup @ GYG Zurich.pdfBy Floris Dekker – September 2nd, 2013 Click to edit Master](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f243c1b01e43f112637d56a/html5/thumbnails/21.jpg)
Our migration strategy
• Unique index namesusing timestamp
• An alias <name>_live points to the current index
• For migration we use temporary alias names <name>_temp
![Page 22: Elastic @ GetYourGuide Click to edit Master title stylefiles.meetup.com/7646592/2016-10-12 Elastic Meetup @ GYG Zurich.pdfBy Floris Dekker – September 2nd, 2013 Click to edit Master](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f243c1b01e43f112637d56a/html5/thumbnails/22.jpg)
@GetYourGuide
ELK
![Page 23: Elastic @ GetYourGuide Click to edit Master title stylefiles.meetup.com/7646592/2016-10-12 Elastic Meetup @ GYG Zurich.pdfBy Floris Dekker – September 2nd, 2013 Click to edit Master](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f243c1b01e43f112637d56a/html5/thumbnails/23.jpg)
What do we do with logs @ GYG ?
• Debugging • Business Intelligence
![Page 24: Elastic @ GetYourGuide Click to edit Master title stylefiles.meetup.com/7646592/2016-10-12 Elastic Meetup @ GYG Zurich.pdfBy Floris Dekker – September 2nd, 2013 Click to edit Master](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f243c1b01e43f112637d56a/html5/thumbnails/24.jpg)
Motivation for a new infrastructure
• SSH to host no longer an option
• Local storage in the cloud is limited
• No one uses outdated tools
• Fault tolerant trustworthy logs are a must!
• Redundancy, Auto failover, Monitoring
![Page 25: Elastic @ GetYourGuide Click to edit Master title stylefiles.meetup.com/7646592/2016-10-12 Elastic Meetup @ GYG Zurich.pdfBy Floris Dekker – September 2nd, 2013 Click to edit Master](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f243c1b01e43f112637d56a/html5/thumbnails/25.jpg)
Logging infrastructure - FLKLEK
Two m4.2xlarge AWS instances
(One tie breaker instance in Digitalocean)
![Page 26: Elastic @ GetYourGuide Click to edit Master title stylefiles.meetup.com/7646592/2016-10-12 Elastic Meetup @ GYG Zurich.pdfBy Floris Dekker – September 2nd, 2013 Click to edit Master](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f243c1b01e43f112637d56a/html5/thumbnails/26.jpg)
The Numbers
120 GB 20 Mioelasticsearch store size log lines /day
42 5.5Knodes avg. logs /second
![Page 27: Elastic @ GetYourGuide Click to edit Master title stylefiles.meetup.com/7646592/2016-10-12 Elastic Meetup @ GYG Zurich.pdfBy Floris Dekker – September 2nd, 2013 Click to edit Master](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f243c1b01e43f112637d56a/html5/thumbnails/27.jpg)
Lessons learned
• Large documents affect ES performance
• Dynamic mappings are a pain
• Beats don’t play well with file rotation
• Kibana is a powerful tool
![Page 28: Elastic @ GetYourGuide Click to edit Master title stylefiles.meetup.com/7646592/2016-10-12 Elastic Meetup @ GYG Zurich.pdfBy Floris Dekker – September 2nd, 2013 Click to edit Master](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f243c1b01e43f112637d56a/html5/thumbnails/28.jpg)
29
![Page 29: Elastic @ GetYourGuide Click to edit Master title stylefiles.meetup.com/7646592/2016-10-12 Elastic Meetup @ GYG Zurich.pdfBy Floris Dekker – September 2nd, 2013 Click to edit Master](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f243c1b01e43f112637d56a/html5/thumbnails/29.jpg)
Questions?