From Zero to Hero - Centralized Logging with Logstash & Elasticsearch
ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like...
Transcript of ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like...
![Page 1: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/1.jpg)
ELASTIC SEARCH,LOGSTASH, KIBANA & BEATS
![Page 2: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/2.jpg)
![Page 3: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/3.jpg)
![Page 4: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/4.jpg)
![Page 5: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/5.jpg)
![Page 6: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/6.jpg)
![Page 7: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/7.jpg)
![Page 8: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/8.jpg)
![Page 9: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/9.jpg)
![Page 10: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/10.jpg)
![Page 11: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/11.jpg)
![Page 12: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/12.jpg)
![Page 13: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/13.jpg)
![Page 14: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/14.jpg)
![Page 15: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/15.jpg)
![Page 16: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/16.jpg)
![Page 17: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/17.jpg)
![Page 18: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/18.jpg)
![Page 19: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/19.jpg)
![Page 20: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/20.jpg)
![Page 21: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/21.jpg)
Installing Elastic Search● Once the ubuntu 16 Server is up, install java using
‚apt-get install openjdk-8-jre-headless
● Create a directory and download elastic search package
mkdir pkg
cd pkg
wget
https://artifacts.elastic.co/downloads/elasticsearch/elastics
earch-5.0.0.deb
![Page 22: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/22.jpg)
Installing Elastic Search (Contd)● Execute command ‚dpkg -i elasticsearch-5.0.0.deb‛
● Elastic search configuration file is present at
‚/etc/elasticsearch/elasticsearch.yml‛
● Change cluster and node name in elasticsearch.yml
![Page 23: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/23.jpg)
Installing Elasticsearch (Contd)● Change network.host: <ip address>
● Increase the memory map count by ‚ sysctl -w
vm.max_map_count=262144‛
● Start elasticsearch cluster service by ‚service
elasticsearch start‛
● Test by executing curl http://<ipadress>:9200
● By default elastic search runs on port 9200
● To start elastic search on boot ‚systemctl enable
elasticsearch‛
![Page 24: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/24.jpg)
![Page 25: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/25.jpg)
![Page 26: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/26.jpg)
![Page 27: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/27.jpg)
![Page 28: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/28.jpg)
![Page 29: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/29.jpg)
![Page 30: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/30.jpg)
![Page 31: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/31.jpg)
Install LogSTASH● Install java much like elasticsearch installation step
● Run the following command to import the Elasticsearch
public GPG key into apt
wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch
| sudo apt-key add -
● Create the Elasticsearch source list:
echo "deb https://artifacts.elastic.co/packages/5.x/apt
stable main" | sudo tee -a
/etc/apt/sources.list.d/elasticsearch-5.x.list
![Page 32: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/32.jpg)
Install LogSTASH (contd..)● Execute ‚apt-get update && apt-get install logstash‛
● Logstash is stored in /usr/share/logstash and move to
this directory using cd
● Now execute this command ‚bin/logstash -e "input { stdin
{} } output { stdout {} }"
![Page 33: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/33.jpg)
![Page 34: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/34.jpg)
![Page 35: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/35.jpg)
![Page 36: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/36.jpg)
![Page 37: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/37.jpg)
![Page 38: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/38.jpg)
Installing KIBANA● Run the following command to import the Elasticsearch
public GPG key into apt
wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch
| sudo apt-key add -
● Create the Elasticsearch source list:
echo "deb https://artifacts.elastic.co/packages/5.x/apt
stable main" | sudo tee -a
/etc/apt/sources.list.d/elasticsearch-5.x.list
![Page 39: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/39.jpg)
Installing KIBANA (CONTD..)● Execute ‚apt-get update && apt-get install kibana‛
● Make changes in configuration at /etc/kibana/kibana.yml
as mentioned below
server.host:<ipaddress>
Server.name: <hostname>
Elasticsearch.url: <elasticsearchurl>
● Execute ‚service kibana start‛
● Test the kibana by accessing
http://<kibana_hostname>:5601
![Page 40: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/40.jpg)
![Page 41: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/41.jpg)
![Page 42: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/42.jpg)
![Page 43: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/43.jpg)
![Page 44: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/44.jpg)
![Page 45: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/45.jpg)
![Page 46: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/46.jpg)
![Page 47: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/47.jpg)
INSTALLING WINLOGBEAT● Download winlogbeat file from elastic site
● Extract zip file and change the following in
winlogbeat.yml
tags: ["us-east-1"]
fields:
globo_environment: production
Enable logstash configuration.
![Page 48: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/48.jpg)
INSTALLING WINLOGBEAT (CONTD..)● From powershell install winlogbeat template by using
following command
‚Invoke-WebRequest -Method PUT -InFile
.\winlogbeat.template.json -Uri
http://<elasticsearchserver>:9200/_template/winlogbeat‛
● From Powershell install winlogbeat service using
following command ‚.\install-service-winlogbeat.ps1‛
● Start service using start-service winlogbeat
![Page 49: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/49.jpg)
![Page 50: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/50.jpg)
Configure Logstash server for winlogbeat● Login into logstash server and navigate to
/etc/logstash/conf.d
● Create a file with name ‚beats.conf‛ and following
content and then start logstash
![Page 51: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/51.jpg)
![Page 52: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/52.jpg)
![Page 53: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/53.jpg)
![Page 54: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/54.jpg)
![Page 55: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/55.jpg)
Installing FIle BEAT● Create the ubuntu instance
Curl -L -O
https://artifacts.elastic.co/downloads/beats/filebeat/filebea
t-5.4.1-amd64.deb
● dpkg -i filebeat-5.4.1-amd64.deb
● We will now configure filebeat to read syslog from
/var/log/syslog
● Upload template by curl -XPUT
‘http://<elasticsearch>:9200/_template/filebeat’ -d
/etc/filebeat/filebeat.template.json
![Page 56: ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG](https://reader034.fdocuments.in/reader034/viewer/2022051320/5a9f94127f8b9a89178cec65/html5/thumbnails/56.jpg)
Installing File Beat (ContD..)● Configure Logstash from configuration @ https://s3-us-
west-2.amazonaws.com/qt-elastic-
softwares/Configuration/LinuxSyslogfilebeat/beats.conf
in /etc/logstash/conf.d and restart logstash
● Create visualization in kibana
For Rest of configurations
https://s3-us-west-2.amazonaws.com/qt-elastic-
softwares/Configuration/centralized-logging-elastic-stack.zip