Real-time data analysis using ELK
-
Upload
jettro-coenradie -
Category
Technology
-
view
908 -
download
0
Transcript of Real-time data analysis using ELK
![Page 1: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/1.jpg)
REAL TIME DATA ANALYSIS USING ELK
@jettroCoenradie
![Page 2: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/2.jpg)
Jettro Coenradie
http://amsterdam.luminis.eu
![Page 3: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/3.jpg)
Jettro Coenradie
http://amsterdam.luminis.eu
![Page 4: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/4.jpg)
Jettro Coenradie
http://amsterdam.luminis.eu
![Page 5: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/5.jpg)
Jettro Coenradie
http://amsterdam.luminis.eu
![Page 6: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/6.jpg)
Jettro Coenradie
http://amsterdam.luminis.eu
![Page 7: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/7.jpg)
REAL TIME DATA ANALYSISUSING ELK
Real time log analysis
Introduction of ELK components
Who is abusing my blog?
Lessons learned from IDMC project
Good to know
![Page 8: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/8.jpg)
REAL TIME DATA ANALYSISUSING ELK
Real time log analysis
Introduction of ELK components
Who is abusing my blog?
Lessons learned from IDMC project
Good to know
![Page 9: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/9.jpg)
REAL TIME DATA ANALYSISUSING ELK
Real time log analysis
Introduction of ELK components
Who is abusing my blog?
Lessons learned from IDMC project
Good to know
![Page 10: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/10.jpg)
REAL TIME DATA ANALYSISUSING ELK
Real time log analysis
Introduction of ELK components
Who is abusing my blog?
Lessons learned from IDMC project
Good to know
![Page 11: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/11.jpg)
REAL TIME DATA ANALYSISUSING ELK
Real time log analysis
Introduction of ELK components
Who is abusing my blog?
Lessons learned from IDMC project
Good to know
![Page 12: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/12.jpg)
REAL TIME LOG ANALYSIS
![Page 13: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/13.jpg)
indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/scripts-gzip/gzip.php?js=wp-includes%2Fjs%2Fjquery%2Fjquery.js%3Fver%3D1.11.1%2Cwp-includes%2Fjs%2Fjquery%2Fjquery-migrate.min.js%3Fver%3D1.2.1%2Cwp-content%2Fthemes%2Fspacious%2Fjs%2Fspacious-custom.js%3Fver%3D4.1.1 HTTP/1.1" 200 36442 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/akismet/_inc/form.js?ver=3.1.1 HTTP/1.1" 200 969 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/scripts-gzip/gzip.php?css=wp-content%2Fthemes%2Fspacious-child%2Fstyle.css%3Fver%3D4.1.1%2Cwp-content%2Fplugins%2Fjetpack%2F_inc%2Fgenericons%2Fgenericons%2Fgenericons.css%3Fver%3D3.1%2Cwp-content%2Fplugins%2Fjetpack%2Fcss%2Fjetpack.css%3Fver%3D3.4.3 HTTP/1.1" 200 36995 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/jetpack/modules/wpgroho.js?ver=4.1.1 HTTP/1.1" 200 1228 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/scripts/shBrushJScript.js?ver=3.0.9b HTTP/1.1" 200 2056 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/scripts/shCore.js?ver=3.0.9b HTTP/1.1" 200 24190 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-includes/js/comment-reply.min.js?ver=4.1.1 HTTP/1.1" 200 1026 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/themes/spacious/js/navigation.js?ver=4.1.1 HTTP/1.1" 200 1233 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/jetpack/modules/sharedaddy/sharing.js?ver=3.4.3 HTTP/1.1" 200 43124 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/styles/shCore.css?ver=3.0.9b HTTP/1.1" 200 7042 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/styles/shThemeDefault.css?ver=3.0.9b HTTP/1.1" 200 3257 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-admin/admin-ajax.php?action=wordfence_logHuman&hid=57D2A62DA174F5DA747EC194B32CBD32&r=0.9168877548072487 HTTP/1.1" 200 461 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/jetpack/_inc/genericons/genericons/svg HTTP/1.1" 404 32544 "http://www.gridshore.nl/wp-content/plugins/scripts-gzip/gzip.php?css=wp-content%2Fthemes%2Fspacious-child%2Fstyle.css%3Fver%3D4.1.1%2Cwp-content%2Fplugins%2Fjetpack%2F_inc%2Fgenericons%2Fgenericons%2Fgenericons.css%3Fver%3D3.1%2Cwp-content%2Fplugins%2Fjetpack%2Fcss%2Fjetpack.css%3Fver%3D3.4.3" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:36 +0200] "GET /favicon.ico HTTP/1.1" 200 2829 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"46.4.132.226 - - [21/Apr/2015:03:51:37 +0200] "GET /tag/united-states/feed/ HTTP/1.0" 200 15341 "http://www.gridshore.nl/tag/united-states/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"46.4.132.226 - - [21/Apr/2015:03:51:47 +0200] "GET /2008/11/04/threadsafe-applications-part-i/ HTTP/1.0" 200 57529 "http://www.gridshore.nl/2008/11/05/yes-we-canthe-day-after/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"46.4.132.226 - - [21/Apr/2015:03:52:01 +0200] "GET /2008/11/04/threadsafe-applications-part-i/feed/ HTTP/1.0" 200 7159 "http://www.gridshore.nl/2008/11/04/threadsafe-applications-part-i/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"54.197.215.119 - - [21/Apr/2015:03:52:10 +0200] "GET /favicon.ico HTTP/1.1" 200 2829 "-" "Pinterest/0.2 (+http://www.pinterest.com/)"54.197.215.119 - - [21/Apr/2015:03:52:11 +0200] "GET /apple-touch-icon.png HTTP/1.1" 404 32507 "-" "Pinterest/0.2 (+http://www.pinterest.com/)"46.4.132.226 - - [21/Apr/2015:03:52:12 +0200] "GET /tag/copyright/ HTTP/1.0" 200 37874 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"46.4.132.226 - - [21/Apr/2015:03:52:25 +0200] "GET /tag/copyright/feed/ HTTP/1.0" 200 17376 "http://www.gridshore.nl/tag/copyright/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"162.243.222.48 - - [21/Apr/2015:03:52:31 +0200] "GET /feed/atom/ HTTP/1.1" 200 70930 "-" "Feedbin - 1 subscribers"183.161.168.153 - - [21/Apr/2015:03:52:32 +0200] "GET /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 200 34606 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"183.161.168.153 - - [21/Apr/2015:03:52:34 +0200] "POST /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 302 790 "http://www.gridshore.nl/2014/08/15/creativity-inc-by-ed-catmull/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"183.161.168.153 - - [21/Apr/2015:03:52:36 +0200] "GET /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 200 34607 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; .NET CLR 2.0.50727 ; .NET CLR 4.0.30319)"46.4.132.226 - - [21/Apr/2015:03:52:38 +0200] "GET /tag/openness/ HTTP/1.0" 200 37864 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"66.249.65.32 - - [21/Apr/2015:03:52:47 +0200] "GET /feed HTTP/1.1" 301 451 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"66.249.65.32 - - [21/Apr/2015:03:52:48 +0200] "GET /feed/ HTTP/1.1" 304 243 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"46.4.132.226 - - [21/Apr/2015:03:52:52 +0200] "GET /2008/10/15/time-for-a-new-editor-and-other-vacation-musings/ HTTP/1.0" 200 46014 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"46.4.132.226 - - [21/Apr/2015:03:53:06 +0200] "GET /2008/10/19/one-liter-of-guice-during-spring-break/ HTTP/1.0" 200 72731 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"46.4.132.226 - - [21/Apr/2015:03:53:21 +0200] "GET /2008/10/19/one-liter-of-guice-during-spring-break/feed/ HTTP/1.0" 200 10136 "http://www.gridshore.nl/2008/10/19/one-liter-of-guice-during-spring-break/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"61.135.219.2 - - [21/Apr/2015:03:53:28 +0200] "GET /feed/ HTTP/1.1" 304 243 "-" "Mozilla/5.0 (compatible;YoudaoFeedFetcher/1.0;http://www.youdao.com/help/reader/faq/topic006/;1 subscribers;)"46.4.132.226 - - [21/Apr/2015:03:53:33 +0200] "GET /2008/09/23/when-good-guys-start-looking-like-bullies/ HTTP/1.0" 200 54917 "http://www.gridshore.nl/2008/09/27/does-professionalization-kill-open-source/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://
http://serverfault.com/questions/11028/do-you-have-any-useful-awk-and-grep-scripts-for-parsing-apache-logs
![Page 14: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/14.jpg)
indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/scripts-gzip/gzip.php?js=wp-includes%2Fjs%2Fjquery%2Fjquery.js%3Fver%3D1.11.1%2Cwp-includes%2Fjs%2Fjquery%2Fjquery-migrate.min.js%3Fver%3D1.2.1%2Cwp-content%2Fthemes%2Fspacious%2Fjs%2Fspacious-custom.js%3Fver%3D4.1.1 HTTP/1.1" 200 36442 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/akismet/_inc/form.js?ver=3.1.1 HTTP/1.1" 200 969 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/scripts-gzip/gzip.php?css=wp-content%2Fthemes%2Fspacious-child%2Fstyle.css%3Fver%3D4.1.1%2Cwp-content%2Fplugins%2Fjetpack%2F_inc%2Fgenericons%2Fgenericons%2Fgenericons.css%3Fver%3D3.1%2Cwp-content%2Fplugins%2Fjetpack%2Fcss%2Fjetpack.css%3Fver%3D3.4.3 HTTP/1.1" 200 36995 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/jetpack/modules/wpgroho.js?ver=4.1.1 HTTP/1.1" 200 1228 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/scripts/shBrushJScript.js?ver=3.0.9b HTTP/1.1" 200 2056 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/scripts/shCore.js?ver=3.0.9b HTTP/1.1" 200 24190 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-includes/js/comment-reply.min.js?ver=4.1.1 HTTP/1.1" 200 1026 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/themes/spacious/js/navigation.js?ver=4.1.1 HTTP/1.1" 200 1233 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/jetpack/modules/sharedaddy/sharing.js?ver=3.4.3 HTTP/1.1" 200 43124 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/styles/shCore.css?ver=3.0.9b HTTP/1.1" 200 7042 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/styles/shThemeDefault.css?ver=3.0.9b HTTP/1.1" 200 3257 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-admin/admin-ajax.php?action=wordfence_logHuman&hid=57D2A62DA174F5DA747EC194B32CBD32&r=0.9168877548072487 HTTP/1.1" 200 461 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/jetpack/_inc/genericons/genericons/svg HTTP/1.1" 404 32544 "http://www.gridshore.nl/wp-content/plugins/scripts-gzip/gzip.php?css=wp-content%2Fthemes%2Fspacious-child%2Fstyle.css%3Fver%3D4.1.1%2Cwp-content%2Fplugins%2Fjetpack%2F_inc%2Fgenericons%2Fgenericons%2Fgenericons.css%3Fver%3D3.1%2Cwp-content%2Fplugins%2Fjetpack%2Fcss%2Fjetpack.css%3Fver%3D3.4.3" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:36 +0200] "GET /favicon.ico HTTP/1.1" 200 2829 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"46.4.132.226 - - [21/Apr/2015:03:51:37 +0200] "GET /tag/united-states/feed/ HTTP/1.0" 200 15341 "http://www.gridshore.nl/tag/united-states/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"46.4.132.226 - - [21/Apr/2015:03:51:47 +0200] "GET /2008/11/04/threadsafe-applications-part-i/ HTTP/1.0" 200 57529 "http://www.gridshore.nl/2008/11/05/yes-we-canthe-day-after/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"46.4.132.226 - - [21/Apr/2015:03:52:01 +0200] "GET /2008/11/04/threadsafe-applications-part-i/feed/ HTTP/1.0" 200 7159 "http://www.gridshore.nl/2008/11/04/threadsafe-applications-part-i/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"54.197.215.119 - - [21/Apr/2015:03:52:10 +0200] "GET /favicon.ico HTTP/1.1" 200 2829 "-" "Pinterest/0.2 (+http://www.pinterest.com/)"54.197.215.119 - - [21/Apr/2015:03:52:11 +0200] "GET /apple-touch-icon.png HTTP/1.1" 404 32507 "-" "Pinterest/0.2 (+http://www.pinterest.com/)"46.4.132.226 - - [21/Apr/2015:03:52:12 +0200] "GET /tag/copyright/ HTTP/1.0" 200 37874 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"46.4.132.226 - - [21/Apr/2015:03:52:25 +0200] "GET /tag/copyright/feed/ HTTP/1.0" 200 17376 "http://www.gridshore.nl/tag/copyright/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"162.243.222.48 - - [21/Apr/2015:03:52:31 +0200] "GET /feed/atom/ HTTP/1.1" 200 70930 "-" "Feedbin - 1 subscribers"183.161.168.153 - - [21/Apr/2015:03:52:32 +0200] "GET /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 200 34606 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"183.161.168.153 - - [21/Apr/2015:03:52:34 +0200] "POST /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 302 790 "http://www.gridshore.nl/2014/08/15/creativity-inc-by-ed-catmull/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"183.161.168.153 - - [21/Apr/2015:03:52:36 +0200] "GET /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 200 34607 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; .NET CLR 2.0.50727 ; .NET CLR 4.0.30319)"46.4.132.226 - - [21/Apr/2015:03:52:38 +0200] "GET /tag/openness/ HTTP/1.0" 200 37864 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"66.249.65.32 - - [21/Apr/2015:03:52:47 +0200] "GET /feed HTTP/1.1" 301 451 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"66.249.65.32 - - [21/Apr/2015:03:52:48 +0200] "GET /feed/ HTTP/1.1" 304 243 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"46.4.132.226 - - [21/Apr/2015:03:52:52 +0200] "GET /2008/10/15/time-for-a-new-editor-and-other-vacation-musings/ HTTP/1.0" 200 46014 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"46.4.132.226 - - [21/Apr/2015:03:53:06 +0200] "GET /2008/10/19/one-liter-of-guice-during-spring-break/ HTTP/1.0" 200 72731 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"46.4.132.226 - - [21/Apr/2015:03:53:21 +0200] "GET /2008/10/19/one-liter-of-guice-during-spring-break/feed/ HTTP/1.0" 200 10136 "http://www.gridshore.nl/2008/10/19/one-liter-of-guice-during-spring-break/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"61.135.219.2 - - [21/Apr/2015:03:53:28 +0200] "GET /feed/ HTTP/1.1" 304 243 "-" "Mozilla/5.0 (compatible;YoudaoFeedFetcher/1.0;http://www.youdao.com/help/reader/faq/topic006/;1 subscribers;)"46.4.132.226 - - [21/Apr/2015:03:53:33 +0200] "GET /2008/09/23/when-good-guys-start-looking-like-bullies/ HTTP/1.0" 200 54917 "http://www.gridshore.nl/2008/09/27/does-professionalization-kill-open-source/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://
# tail -fn 100 access-log-2014-04-22
http://serverfault.com/questions/11028/do-you-have-any-useful-awk-and-grep-scripts-for-parsing-apache-logs
![Page 15: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/15.jpg)
indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/scripts-gzip/gzip.php?js=wp-includes%2Fjs%2Fjquery%2Fjquery.js%3Fver%3D1.11.1%2Cwp-includes%2Fjs%2Fjquery%2Fjquery-migrate.min.js%3Fver%3D1.2.1%2Cwp-content%2Fthemes%2Fspacious%2Fjs%2Fspacious-custom.js%3Fver%3D4.1.1 HTTP/1.1" 200 36442 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/akismet/_inc/form.js?ver=3.1.1 HTTP/1.1" 200 969 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/scripts-gzip/gzip.php?css=wp-content%2Fthemes%2Fspacious-child%2Fstyle.css%3Fver%3D4.1.1%2Cwp-content%2Fplugins%2Fjetpack%2F_inc%2Fgenericons%2Fgenericons%2Fgenericons.css%3Fver%3D3.1%2Cwp-content%2Fplugins%2Fjetpack%2Fcss%2Fjetpack.css%3Fver%3D3.4.3 HTTP/1.1" 200 36995 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/jetpack/modules/wpgroho.js?ver=4.1.1 HTTP/1.1" 200 1228 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/scripts/shBrushJScript.js?ver=3.0.9b HTTP/1.1" 200 2056 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/scripts/shCore.js?ver=3.0.9b HTTP/1.1" 200 24190 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-includes/js/comment-reply.min.js?ver=4.1.1 HTTP/1.1" 200 1026 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/themes/spacious/js/navigation.js?ver=4.1.1 HTTP/1.1" 200 1233 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/jetpack/modules/sharedaddy/sharing.js?ver=3.4.3 HTTP/1.1" 200 43124 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/styles/shCore.css?ver=3.0.9b HTTP/1.1" 200 7042 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/styles/shThemeDefault.css?ver=3.0.9b HTTP/1.1" 200 3257 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-admin/admin-ajax.php?action=wordfence_logHuman&hid=57D2A62DA174F5DA747EC194B32CBD32&r=0.9168877548072487 HTTP/1.1" 200 461 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/jetpack/_inc/genericons/genericons/svg HTTP/1.1" 404 32544 "http://www.gridshore.nl/wp-content/plugins/scripts-gzip/gzip.php?css=wp-content%2Fthemes%2Fspacious-child%2Fstyle.css%3Fver%3D4.1.1%2Cwp-content%2Fplugins%2Fjetpack%2F_inc%2Fgenericons%2Fgenericons%2Fgenericons.css%3Fver%3D3.1%2Cwp-content%2Fplugins%2Fjetpack%2Fcss%2Fjetpack.css%3Fver%3D3.4.3" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:36 +0200] "GET /favicon.ico HTTP/1.1" 200 2829 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"46.4.132.226 - - [21/Apr/2015:03:51:37 +0200] "GET /tag/united-states/feed/ HTTP/1.0" 200 15341 "http://www.gridshore.nl/tag/united-states/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"46.4.132.226 - - [21/Apr/2015:03:51:47 +0200] "GET /2008/11/04/threadsafe-applications-part-i/ HTTP/1.0" 200 57529 "http://www.gridshore.nl/2008/11/05/yes-we-canthe-day-after/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"46.4.132.226 - - [21/Apr/2015:03:52:01 +0200] "GET /2008/11/04/threadsafe-applications-part-i/feed/ HTTP/1.0" 200 7159 "http://www.gridshore.nl/2008/11/04/threadsafe-applications-part-i/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"54.197.215.119 - - [21/Apr/2015:03:52:10 +0200] "GET /favicon.ico HTTP/1.1" 200 2829 "-" "Pinterest/0.2 (+http://www.pinterest.com/)"54.197.215.119 - - [21/Apr/2015:03:52:11 +0200] "GET /apple-touch-icon.png HTTP/1.1" 404 32507 "-" "Pinterest/0.2 (+http://www.pinterest.com/)"46.4.132.226 - - [21/Apr/2015:03:52:12 +0200] "GET /tag/copyright/ HTTP/1.0" 200 37874 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"46.4.132.226 - - [21/Apr/2015:03:52:25 +0200] "GET /tag/copyright/feed/ HTTP/1.0" 200 17376 "http://www.gridshore.nl/tag/copyright/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"162.243.222.48 - - [21/Apr/2015:03:52:31 +0200] "GET /feed/atom/ HTTP/1.1" 200 70930 "-" "Feedbin - 1 subscribers"183.161.168.153 - - [21/Apr/2015:03:52:32 +0200] "GET /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 200 34606 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"183.161.168.153 - - [21/Apr/2015:03:52:34 +0200] "POST /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 302 790 "http://www.gridshore.nl/2014/08/15/creativity-inc-by-ed-catmull/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"183.161.168.153 - - [21/Apr/2015:03:52:36 +0200] "GET /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 200 34607 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; .NET CLR 2.0.50727 ; .NET CLR 4.0.30319)"46.4.132.226 - - [21/Apr/2015:03:52:38 +0200] "GET /tag/openness/ HTTP/1.0" 200 37864 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"66.249.65.32 - - [21/Apr/2015:03:52:47 +0200] "GET /feed HTTP/1.1" 301 451 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"66.249.65.32 - - [21/Apr/2015:03:52:48 +0200] "GET /feed/ HTTP/1.1" 304 243 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"46.4.132.226 - - [21/Apr/2015:03:52:52 +0200] "GET /2008/10/15/time-for-a-new-editor-and-other-vacation-musings/ HTTP/1.0" 200 46014 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"46.4.132.226 - - [21/Apr/2015:03:53:06 +0200] "GET /2008/10/19/one-liter-of-guice-during-spring-break/ HTTP/1.0" 200 72731 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"46.4.132.226 - - [21/Apr/2015:03:53:21 +0200] "GET /2008/10/19/one-liter-of-guice-during-spring-break/feed/ HTTP/1.0" 200 10136 "http://www.gridshore.nl/2008/10/19/one-liter-of-guice-during-spring-break/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"61.135.219.2 - - [21/Apr/2015:03:53:28 +0200] "GET /feed/ HTTP/1.1" 304 243 "-" "Mozilla/5.0 (compatible;YoudaoFeedFetcher/1.0;http://www.youdao.com/help/reader/faq/topic006/;1 subscribers;)"46.4.132.226 - - [21/Apr/2015:03:53:33 +0200] "GET /2008/09/23/when-good-guys-start-looking-like-bullies/ HTTP/1.0" 200 54917 "http://www.gridshore.nl/2008/09/27/does-professionalization-kill-open-source/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://
# tail -fn 100 access-log-2014-04-22
awk -F'[ "]+' '$7 == "/" { ipcount[$1]++ } END { for (i in ipcount) { printf "%15s - %d\n", i, ipcount[i] } }' access-log-2015-04-21
http://serverfault.com/questions/11028/do-you-have-any-useful-awk-and-grep-scripts-for-parsing-apache-logs
![Page 16: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/16.jpg)
EVERY NIGHT A BATCH USING WEBALIZER
![Page 17: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/17.jpg)
GOOGLE ANALYTICS
![Page 18: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/18.jpg)
GOOGLE ANALYTICS
![Page 19: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/19.jpg)
GOOGLE ANALYTICS
![Page 20: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/20.jpg)
WHAT IS REAL TIME?
![Page 21: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/21.jpg)
THERE IS ALWAYS A DELAY
![Page 22: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/22.jpg)
HOW MUCH DELAY CAN YOU ACCEPT?
![Page 23: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/23.jpg)
ARCHITECTURE OF DELAY
accesslogs shipper Queue Logstash
elasticsearch
Monitor Send Retrieve
Storeforwarderlogstashbeaver
RedisKafka
![Page 24: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/24.jpg)
DATA LIFECYCLE
![Page 25: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/25.jpg)
DATA LIFECYCLE
Obtain
![Page 26: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/26.jpg)
DATA LIFECYCLE
Obtain Transform
![Page 27: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/27.jpg)
DATA LIFECYCLE
Obtain Transform Store
![Page 28: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/28.jpg)
DATA LIFECYCLE
Obtain Transform Store Use
![Page 29: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/29.jpg)
DATA LIFECYCLE
Obtain Transform Store Use
Learn
![Page 30: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/30.jpg)
DATA LIFECYCLE: ELK
Obtain Transform Store Use
Learn
![Page 31: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/31.jpg)
DATA LIFECYCLE: ELK
Obtain Transform Store Use
Learn
Logstash
![Page 32: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/32.jpg)
DATA LIFECYCLE: ELK
Obtain Transform Store Use
Learn
Logstash Logstash
![Page 33: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/33.jpg)
DATA LIFECYCLE: ELK
Obtain Transform Store Use
Learn
Logstash Logstash Elasticsearch
![Page 34: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/34.jpg)
DATA LIFECYCLE: ELK
Obtain Transform Store Use
Learn
Logstash Logstash Elasticsearch Kibana
![Page 35: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/35.jpg)
DATA LIFECYCLE: ELK
Obtain Transform Store Use
Learn
Logstash Logstash Elasticsearch Kibana
YOU
![Page 36: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/36.jpg)
INTRODUCTION OF ELK COMPONENTS
![Page 37: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/37.jpg)
INTRODUCTION OF ELK COMPONENTS
![Page 38: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/38.jpg)
LOGSTASH: COMPONENTSfile
syslog
redis log4j
websockettwitter grok
mutate
drop
clone
geoipelasticsearch
file
graphite
statsd
![Page 39: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/39.jpg)
LOGSTASH: COMPONENTSfile
syslog
redis log4j
Input
websockettwitter
Filter
Output
grok
mutate
drop
clone
geoipelasticsearch
file
graphite
statsd
![Page 40: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/40.jpg)
LOGSTASH: COMPONENTSfile syslog redislog4jInput web
sockettwitter
Filter
Output
grok mutatedropclone geoip
elasticsearch filegraphite statsd
![Page 41: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/41.jpg)
ELASTICSEARCH
![Page 42: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/42.jpg)
ELASTICSEARCH
cluster
![Page 43: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/43.jpg)
ELASTICSEARCH
cluster
Node Node Node
![Page 44: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/44.jpg)
ELASTICSEARCH
cluster
Node Node NodeIn
dex
Inde
xIn
dex
Inde
xIn
dex
Inde
x
![Page 45: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/45.jpg)
ELASTICSEARCH
cluster
Node Node NodeIn
dex
Inde
xIn
dex
Inde
xIn
dex
Inde
x
shar
dsh
ard
shar
dsh
ard
shar
dsh
ard
shar
dsh
ard
shar
dsh
ard
shar
dsh
ard
![Page 46: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/46.jpg)
ELASTICSEARCH
cluster
Node Node NodeIn
dex
Inde
xIn
dex
Inde
xIn
dex
Inde
x
shar
dsh
ard
shar
dsh
ard
shar
dsh
ard
shar
dsh
ard
shar
dsh
ard
shar
dsh
ard
Mapping
![Page 47: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/47.jpg)
ELASTICSEARCH
cluster
Node Node NodeIn
dex
Inde
xIn
dex
Inde
xIn
dex
Inde
x
shar
dsh
ard
shar
dsh
ard
shar
dsh
ard
shar
dsh
ard
shar
dsh
ard
shar
dsh
ard
Mapping
Search API
![Page 48: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/48.jpg)
ELASTICSEARCH
cluster
Node Node NodeIn
dex
Inde
xIn
dex
Inde
xIn
dex
Inde
x
shar
dsh
ard
shar
dsh
ard
shar
dsh
ard
shar
dsh
ard
shar
dsh
ard
shar
dsh
ard
Mapping
Search API
Aggregations
![Page 49: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/49.jpg)
AGGREGATIONS
![Page 50: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/50.jpg)
AGGREGATIONS
27.159.213.10 - - [25/Feb/2015:03:35:57 +0100] "GET /2008/04/29/using-ehcache-and-verifying-that-it-works-with-jpa-and-springframework/ HTTP/1.1" 200 18720 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; .NET CLR 1.0.3705)"
![Page 51: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/51.jpg)
AGGREGATIONS
27.159.213.10 - - [25/Feb/2015:03:35:57 +0100] "GET /2008/04/29/using-ehcache-and-verifying-that-it-works-with-jpa-and-springframework/ HTTP/1.1" 200 18720 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; .NET CLR 1.0.3705)"
GET
![Page 52: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/52.jpg)
AGGREGATIONS
27.159.213.10 - - [25/Feb/2015:03:35:57 +0100] "GET /2008/04/29/using-ehcache-and-verifying-that-it-works-with-jpa-and-springframework/ HTTP/1.1" 200 18720 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; .NET CLR 1.0.3705)"
![Page 53: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/53.jpg)
AGGREGATIONS
78.5.169.90 - - [15/Apr/2015:03:08:17 +0200] "POST /wp-login.php HTTP/1.1" 403 3538 "http://www.gridshore.nl/wp-login.php" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML like Gecko) Chrome/24.0.1312.56 Safari/537.17"
![Page 54: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/54.jpg)
AGGREGATIONS
78.5.169.90 - - [15/Apr/2015:03:08:17 +0200] "POST /wp-login.php HTTP/1.1" 403 3538 "http://www.gridshore.nl/wp-login.php" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML like Gecko) Chrome/24.0.1312.56 Safari/537.17"
POST
![Page 55: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/55.jpg)
AGGREGATIONS
78.5.169.90 - - [15/Apr/2015:03:08:17 +0200] "POST /wp-login.php HTTP/1.1" 403 3538 "http://www.gridshore.nl/wp-login.php" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML like Gecko) Chrome/24.0.1312.56 Safari/537.17"
![Page 56: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/56.jpg)
AGGREGATIONS
175989 133343 2008 2POSTGET HEAD PUT
![Page 57: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/57.jpg)
AGGREGATIONS
Date histogram
Feb Mar Apr
311344 395654 157623
![Page 58: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/58.jpg)
AGGREGATIONS
Date histogram
Feb Mar Apr
311344 395654 157623
Cardinality [client ip] 11848 26152 9064
![Page 59: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/59.jpg)
GET /gridshore-logs-*/_search?search_type=count{ "aggs": { "byDate": { "date_histogram": { "field": "@timestamp", "interval": "month" }, "aggs": { "uniqueVisitors": { "cardinality": { "field": "clientip" } } } } }}
![Page 60: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/60.jpg)
KIBANA
Discover
![Page 61: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/61.jpg)
KIBANA
Discover
Visualise
![Page 62: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/62.jpg)
KIBANA
Discover
Visualise
Analyse
![Page 63: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/63.jpg)
Discover
![Page 64: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/64.jpg)
Visualise
![Page 65: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/65.jpg)
Analyse
![Page 66: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/66.jpg)
WHO IS ABUSING MY BLOG?
![Page 67: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/67.jpg)
OBTAINING LOGS
daily rolling file
![Page 68: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/68.jpg)
OBTAINING LOGS
daily rolling file
shell script ftp
![Page 69: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/69.jpg)
OBTAINING LOGS
daily rolling file
shell script ftp
logstash
![Page 70: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/70.jpg)
OBTAINING LOGS
daily rolling file
shell script ftp
logstash
elasticsearch
![Page 71: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/71.jpg)
1. input { 2. file { 3. path => "/access-log-*" 4. type => "apache" 5. start_position => "beginning" 6. } 7. }
OBTAIN
![Page 72: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/72.jpg)
1. input { 2. file { 3. path => "/access-log-*" 4. type => "apache" 5. start_position => "beginning" 6. } 7. }
OBTAIN
files to import
![Page 73: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/73.jpg)
1. input { 2. file { 3. path => "/access-log-*" 4. type => "apache" 5. start_position => "beginning" 6. } 7. }
OBTAIN
used for filtering
![Page 74: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/74.jpg)
1. input { 2. file { 3. path => "/access-log-*" 4. type => "apache" 5. start_position => "beginning" 6. } 7. }
OBTAIN
start reading from
![Page 75: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/75.jpg)
indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/scripts-gzip/gzip.php?js=wp-includes%2Fjs%2Fjquery%2Fjquery.js%3Fver%3D1.11.1%2Cwp-includes%2Fjs%2Fjquery%2Fjquery-migrate.min.js%3Fver%3D1.2.1%2Cwp-content%2Fthemes%2Fspacious%2Fjs%2Fspacious-custom.js%3Fver%3D4.1.1 HTTP/1.1" 200 36442 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/akismet/_inc/form.js?ver=3.1.1 HTTP/1.1" 200 969 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/scripts-gzip/gzip.php?css=wp-content%2Fthemes%2Fspacious-child%2Fstyle.css%3Fver%3D4.1.1%2Cwp-content%2Fplugins%2Fjetpack%2F_inc%2Fgenericons%2Fgenericons%2Fgenericons.css%3Fver%3D3.1%2Cwp-content%2Fplugins%2Fjetpack%2Fcss%2Fjetpack.css%3Fver%3D3.4.3 HTTP/1.1" 200 36995 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/jetpack/modules/wpgroho.js?ver=4.1.1 HTTP/1.1" 200 1228 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/scripts/shBrushJScript.js?ver=3.0.9b HTTP/1.1" 200 2056 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/scripts/shCore.js?ver=3.0.9b HTTP/1.1" 200 24190 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-includes/js/comment-reply.min.js?ver=4.1.1 HTTP/1.1" 200 1026 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/themes/spacious/js/navigation.js?ver=4.1.1 HTTP/1.1" 200 1233 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/jetpack/modules/sharedaddy/sharing.js?ver=3.4.3 HTTP/1.1" 200 43124 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/styles/shCore.css?ver=3.0.9b HTTP/1.1" 200 7042 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/styles/shThemeDefault.css?ver=3.0.9b HTTP/1.1" 200 3257 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-admin/admin-ajax.php?action=wordfence_logHuman&hid=57D2A62DA174F5DA747EC194B32CBD32&r=0.9168877548072487 HTTP/1.1" 200 461 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/jetpack/_inc/genericons/genericons/svg HTTP/1.1" 404 32544 "http://www.gridshore.nl/wp-content/plugins/scripts-gzip/gzip.php?css=wp-content%2Fthemes%2Fspacious-child%2Fstyle.css%3Fver%3D4.1.1%2Cwp-content%2Fplugins%2Fjetpack%2F_inc%2Fgenericons%2Fgenericons%2Fgenericons.css%3Fver%3D3.1%2Cwp-content%2Fplugins%2Fjetpack%2Fcss%2Fjetpack.css%3Fver%3D3.4.3" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:36 +0200] "GET /favicon.ico HTTP/1.1" 200 2829 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"46.4.132.226 - - [21/Apr/2015:03:51:37 +0200] "GET /tag/united-states/feed/ HTTP/1.0" 200 15341 "http://www.gridshore.nl/tag/united-states/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"46.4.132.226 - - [21/Apr/2015:03:51:47 +0200] "GET /2008/11/04/threadsafe-applications-part-i/ HTTP/1.0" 200 57529 "http://www.gridshore.nl/2008/11/05/yes-we-canthe-day-after/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"46.4.132.226 - - [21/Apr/2015:03:52:01 +0200] "GET /2008/11/04/threadsafe-applications-part-i/feed/ HTTP/1.0" 200 7159 "http://www.gridshore.nl/2008/11/04/threadsafe-applications-part-i/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"54.197.215.119 - - [21/Apr/2015:03:52:10 +0200] "GET /favicon.ico HTTP/1.1" 200 2829 "-" "Pinterest/0.2 (+http://www.pinterest.com/)"54.197.215.119 - - [21/Apr/2015:03:52:11 +0200] "GET /apple-touch-icon.png HTTP/1.1" 404 32507 "-" "Pinterest/0.2 (+http://www.pinterest.com/)"46.4.132.226 - - [21/Apr/2015:03:52:12 +0200] "GET /tag/copyright/ HTTP/1.0" 200 37874 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"46.4.132.226 - - [21/Apr/2015:03:52:25 +0200] "GET /tag/copyright/feed/ HTTP/1.0" 200 17376 "http://www.gridshore.nl/tag/copyright/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"162.243.222.48 - - [21/Apr/2015:03:52:31 +0200] "GET /feed/atom/ HTTP/1.1" 200 70930 "-" "Feedbin - 1 subscribers"183.161.168.153 - - [21/Apr/2015:03:52:32 +0200] "GET /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 200 34606 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"183.161.168.153 - - [21/Apr/2015:03:52:34 +0200] "POST /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 302 790 "http://www.gridshore.nl/2014/08/15/creativity-inc-by-ed-catmull/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"183.161.168.153 - - [21/Apr/2015:03:52:36 +0200] "GET /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 200 34607 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; .NET CLR 2.0.50727 ; .NET CLR 4.0.30319)"46.4.132.226 - - [21/Apr/2015:03:52:38 +0200] "GET /tag/openness/ HTTP/1.0" 200 37864 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"66.249.65.32 - - [21/Apr/2015:03:52:47 +0200] "GET /feed HTTP/1.1" 301 451 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"66.249.65.32 - - [21/Apr/2015:03:52:48 +0200] "GET /feed/ HTTP/1.1" 304 243 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"46.4.132.226 - - [21/Apr/2015:03:52:52 +0200] "GET /2008/10/15/time-for-a-new-editor-and-other-vacation-musings/ HTTP/1.0" 200 46014 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"46.4.132.226 - - [21/Apr/2015:03:53:06 +0200] "GET /2008/10/19/one-liter-of-guice-during-spring-break/ HTTP/1.0" 200 72731 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"46.4.132.226 - - [21/Apr/2015:03:53:21 +0200] "GET /2008/10/19/one-liter-of-guice-during-spring-break/feed/ HTTP/1.0" 200 10136 "http://www.gridshore.nl/2008/10/19/one-liter-of-guice-during-spring-break/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"61.135.219.2 - - [21/Apr/2015:03:53:28 +0200] "GET /feed/ HTTP/1.1" 304 243 "-" "Mozilla/5.0 (compatible;YoudaoFeedFetcher/1.0;http://www.youdao.com/help/reader/faq/topic006/;1 subscribers;)"46.4.132.226 - - [21/Apr/2015:03:53:33 +0200] "GET /2008/09/23/when-good-guys-start-looking-like-bullies/ HTTP/1.0" 200 54917 "http://www.gridshore.nl/2008/09/27/does-professionalization-kill-open-source/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://
![Page 76: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/76.jpg)
indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/scripts-gzip/gzip.php?js=wp-includes%2Fjs%2Fjquery%2Fjquery.js%3Fver%3D1.11.1%2Cwp-includes%2Fjs%2Fjquery%2Fjquery-migrate.min.js%3Fver%3D1.2.1%2Cwp-content%2Fthemes%2Fspacious%2Fjs%2Fspacious-custom.js%3Fver%3D4.1.1 HTTP/1.1" 200 36442 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/akismet/_inc/form.js?ver=3.1.1 HTTP/1.1" 200 969 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/scripts-gzip/gzip.php?css=wp-content%2Fthemes%2Fspacious-child%2Fstyle.css%3Fver%3D4.1.1%2Cwp-content%2Fplugins%2Fjetpack%2F_inc%2Fgenericons%2Fgenericons%2Fgenericons.css%3Fver%3D3.1%2Cwp-content%2Fplugins%2Fjetpack%2Fcss%2Fjetpack.css%3Fver%3D3.4.3 HTTP/1.1" 200 36995 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/jetpack/modules/wpgroho.js?ver=4.1.1 HTTP/1.1" 200 1228 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/scripts/shBrushJScript.js?ver=3.0.9b HTTP/1.1" 200 2056 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/scripts/shCore.js?ver=3.0.9b HTTP/1.1" 200 24190 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-includes/js/comment-reply.min.js?ver=4.1.1 HTTP/1.1" 200 1026 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/themes/spacious/js/navigation.js?ver=4.1.1 HTTP/1.1" 200 1233 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/jetpack/modules/sharedaddy/sharing.js?ver=3.4.3 HTTP/1.1" 200 43124 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/styles/shCore.css?ver=3.0.9b HTTP/1.1" 200 7042 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/syntaxhighlighter/syntaxhighlighter3/styles/shThemeDefault.css?ver=3.0.9b HTTP/1.1" 200 3257 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-admin/admin-ajax.php?action=wordfence_logHuman&hid=57D2A62DA174F5DA747EC194B32CBD32&r=0.9168877548072487 HTTP/1.1" 200 461 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:35 +0200] "GET /wp-content/plugins/jetpack/_inc/genericons/genericons/svg HTTP/1.1" 404 32544 "http://www.gridshore.nl/wp-content/plugins/scripts-gzip/gzip.php?css=wp-content%2Fthemes%2Fspacious-child%2Fstyle.css%3Fver%3D4.1.1%2Cwp-content%2Fplugins%2Fjetpack%2F_inc%2Fgenericons%2Fgenericons%2Fgenericons.css%3Fver%3D3.1%2Cwp-content%2Fplugins%2Fjetpack%2Fcss%2Fjetpack.css%3Fver%3D3.4.3" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"71.59.251.16 - - [21/Apr/2015:03:51:36 +0200] "GET /favicon.ico HTTP/1.1" 200 2829 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"46.4.132.226 - - [21/Apr/2015:03:51:37 +0200] "GET /tag/united-states/feed/ HTTP/1.0" 200 15341 "http://www.gridshore.nl/tag/united-states/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"46.4.132.226 - - [21/Apr/2015:03:51:47 +0200] "GET /2008/11/04/threadsafe-applications-part-i/ HTTP/1.0" 200 57529 "http://www.gridshore.nl/2008/11/05/yes-we-canthe-day-after/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"46.4.132.226 - - [21/Apr/2015:03:52:01 +0200] "GET /2008/11/04/threadsafe-applications-part-i/feed/ HTTP/1.0" 200 7159 "http://www.gridshore.nl/2008/11/04/threadsafe-applications-part-i/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"54.197.215.119 - - [21/Apr/2015:03:52:10 +0200] "GET /favicon.ico HTTP/1.1" 200 2829 "-" "Pinterest/0.2 (+http://www.pinterest.com/)"54.197.215.119 - - [21/Apr/2015:03:52:11 +0200] "GET /apple-touch-icon.png HTTP/1.1" 404 32507 "-" "Pinterest/0.2 (+http://www.pinterest.com/)"46.4.132.226 - - [21/Apr/2015:03:52:12 +0200] "GET /tag/copyright/ HTTP/1.0" 200 37874 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"46.4.132.226 - - [21/Apr/2015:03:52:25 +0200] "GET /tag/copyright/feed/ HTTP/1.0" 200 17376 "http://www.gridshore.nl/tag/copyright/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"162.243.222.48 - - [21/Apr/2015:03:52:31 +0200] "GET /feed/atom/ HTTP/1.1" 200 70930 "-" "Feedbin - 1 subscribers"183.161.168.153 - - [21/Apr/2015:03:52:32 +0200] "GET /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 200 34606 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"183.161.168.153 - - [21/Apr/2015:03:52:34 +0200] "POST /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 302 790 "http://www.gridshore.nl/2014/08/15/creativity-inc-by-ed-catmull/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"183.161.168.153 - - [21/Apr/2015:03:52:36 +0200] "GET /2014/08/15/creativity-inc-by-ed-catmull/ HTTP/1.1" 200 34607 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; .NET CLR 2.0.50727 ; .NET CLR 4.0.30319)"46.4.132.226 - - [21/Apr/2015:03:52:38 +0200] "GET /tag/openness/ HTTP/1.0" 200 37864 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"66.249.65.32 - - [21/Apr/2015:03:52:47 +0200] "GET /feed HTTP/1.1" 301 451 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"66.249.65.32 - - [21/Apr/2015:03:52:48 +0200] "GET /feed/ HTTP/1.1" 304 243 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"46.4.132.226 - - [21/Apr/2015:03:52:52 +0200] "GET /2008/10/15/time-for-a-new-editor-and-other-vacation-musings/ HTTP/1.0" 200 46014 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"46.4.132.226 - - [21/Apr/2015:03:53:06 +0200] "GET /2008/10/19/one-liter-of-guice-during-spring-break/ HTTP/1.0" 200 72731 "http://www.gridshore.nl/2008/10/15/redesigning-libraries-completely/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"46.4.132.226 - - [21/Apr/2015:03:53:21 +0200] "GET /2008/10/19/one-liter-of-guice-during-spring-break/feed/ HTTP/1.0" 200 10136 "http://www.gridshore.nl/2008/10/19/one-liter-of-guice-during-spring-break/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)"61.135.219.2 - - [21/Apr/2015:03:53:28 +0200] "GET /feed/ HTTP/1.1" 304 243 "-" "Mozilla/5.0 (compatible;YoudaoFeedFetcher/1.0;http://www.youdao.com/help/reader/faq/topic006/;1 subscribers;)"46.4.132.226 - - [21/Apr/2015:03:53:33 +0200] "GET /2008/09/23/when-good-guys-start-looking-like-bullies/ HTTP/1.0" 200 54917 "http://www.gridshore.nl/2008/09/27/does-professionalization-kill-open-source/" "Mozilla/5.0 (compatible; SEOkicks-Robot; +http://
71.59.251.16 - - [21/Apr/2015:03:51:34 +0200] "GET /wp-content/plugins/scripts-gzip/gzip.php?css=wp-content%2Fthemes%2Fspacious-child%2Fstyle.css%3Fver%3D4.1.1%2Cwp-content%2Fplugins%2Fjetpack%2F_inc%2Fgenericons%2Fgenericons%2Fgenericons.css%3Fver%3D3.1%2Cwp-content%2Fplugins%2Fjetpack%2Fcss%2Fjetpack.css%3Fver%3D3.4.3 HTTP/1.1" 200 36995 "http://www.gridshore.nl/2014/07/26/transform-the-input-before-indexing-in-elasticsearch/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
![Page 77: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/77.jpg)
![Page 78: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/78.jpg)
1. filter { 2. grok { 3. match => { "message" => "%{COMBINEDAPACHELOG}" } 4. remove_field => ["message"] 5. } 6. grok { 7. match => { "request" => "%{URIPATH:request_noparam}"} 8. } 9. geoip { 10. source => "clientip" 11. } 12. useragent { 13. source => "agent" 14. target => "useragent" 15. remove_field => ["agent"] 16. } 17. date { 18. match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] 19. } 20.}
TRANSFORM
![Page 79: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/79.jpg)
1. filter { 2. grok { 3. match => { "message" => "%{COMBINEDAPACHELOG}" } 4. remove_field => ["message"] 5. } 6. grok { 7. match => { "request" => "%{URIPATH:request_noparam}"} 8. } 9. geoip { 10. source => "clientip" 11. } 12. useragent { 13. source => "agent" 14. target => "useragent" 15. remove_field => ["agent"] 16. } 17. date { 18. match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] 19. } 20.}
remove parsed message
TRANSFORM
![Page 80: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/80.jpg)
1. filter { 2. grok { 3. match => { "message" => "%{COMBINEDAPACHELOG}" } 4. remove_field => ["message"] 5. } 6. grok { 7. match => { "request" => "%{URIPATH:request_noparam}"} 8. } 9. geoip { 10. source => "clientip" 11. } 12. useragent { 13. source => "agent" 14. target => "useragent" 15. remove_field => ["agent"] 16. } 17. date { 18. match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] 19. } 20.}
TRANSFORM
extra parse of request
![Page 81: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/81.jpg)
1. filter { 2. grok { 3. match => { "message" => "%{COMBINEDAPACHELOG}" } 4. remove_field => ["message"] 5. } 6. grok { 7. match => { "request" => "%{URIPATH:request_noparam}"} 8. } 9. geoip { 10. source => "clientip" 11. } 12. useragent { 13. source => "agent" 14. target => "useragent" 15. remove_field => ["agent"] 16. } 17. date { 18. match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] 19. } 20.}
TRANSFORM
request => /wp-content/plugins/scripts-gzip/gzip.php?css=wp-content%2Fthemes%2Fspacious-child%2Fstyle.css%3Fver%3D4.1.1%2Cwp-content%2Fplugins%2Fjetpack%2F_inc%2Fgenericons%2Fgenericons%2Fgenericons.css%3Fver%3D3.1%2Cwp-content%2Fplugins%2Fjetpack%2Fcss%2Fjetpack.css%3Fver%3D3.4.3
request_noparam => /wp-content/plugins/scripts-gzip/gzip.php
![Page 82: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/82.jpg)
1. filter { 2. grok { 3. match => { "message" => "%{COMBINEDAPACHELOG}" } 4. remove_field => ["message"] 5. } 6. grok { 7. match => { "request" => "%{URIPATH:request_noparam}"} 8. } 9. geoip { 10. source => "clientip" 11. } 12. useragent { 13. source => "agent" 14. target => "useragent" 15. remove_field => ["agent"] 16. } 17. date { 18. match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] 19. } 20.}
TRANSFORM
add geo information
![Page 83: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/83.jpg)
1. filter { 2. grok { 3. match => { "message" => "%{COMBINEDAPACHELOG}" } 4. remove_field => ["message"] 5. } 6. grok { 7. match => { "request" => "%{URIPATH:request_noparam}"} 8. } 9. geoip { 10. source => "clientip" 11. } 12. useragent { 13. source => "agent" 14. target => "useragent" 15. remove_field => ["agent"] 16. } 17. date { 18. match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] 19. } 20.}
TRANSFORM
parse useragent fields
![Page 84: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/84.jpg)
1. filter { 2. grok { 3. match => { "message" => "%{COMBINEDAPACHELOG}" } 4. remove_field => ["message"] 5. } 6. grok { 7. match => { "request" => "%{URIPATH:request_noparam}"} 8. } 9. geoip { 10. source => "clientip" 11. } 12. useragent { 13. source => "agent" 14. target => "useragent" 15. remove_field => ["agent"] 16. } 17. date { 18. match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] 19. } 20.}
TRANSFORM
agent => Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Safari/537.36
useragent => {"name": "Safari", "os": “Mac OS X 10.10.2”, "os_name": “Mac OS X”, "device": "Other", "major": "537", "minor": "36" }
![Page 85: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/85.jpg)
1. filter { 2. grok { 3. match => { "message" => "%{COMBINEDAPACHELOG}" } 4. remove_field => ["message"] 5. } 6. grok { 7. match => { "request" => "%{URIPATH:request_noparam}"} 8. } 9. geoip { 10. source => "clientip" 11. } 12. useragent { 13. source => "agent" 14. target => "useragent" 15. remove_field => ["agent"] 16. } 17. date { 18. match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] 19. } 20.}
TRANSFORM
take timestamp from log
![Page 86: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/86.jpg)
STORE1. output {2. if "_grokparsefailure" not in [tags] {3. elasticsearch {4. protocol => "transport"5. host => "localhost:9300"6. cluster => "jc-play"7. index => "gridshore-logs-%{+YYYY.MM}"8. manage_template => false9. template_name => "gridshore-logs"10. }11. }12.}
![Page 87: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/87.jpg)
STORE1. output {2. if "_grokparsefailure" not in [tags] {3. elasticsearch {4. protocol => "transport"5. host => "localhost:9300"6. cluster => "jc-play"7. index => "gridshore-logs-%{+YYYY.MM}"8. manage_template => false9. template_name => "gridshore-logs"10. }11. }12.}
in case of an error
![Page 88: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/88.jpg)
STORE1. output {2. if "_grokparsefailure" not in [tags] {3. elasticsearch {4. protocol => "transport"5. host => "localhost:9300"6. cluster => "jc-play"7. index => "gridshore-logs-%{+YYYY.MM}"8. manage_template => false9. template_name => "gridshore-logs"10. }11. }12.}
use faster binary protocol
![Page 89: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/89.jpg)
STORE1. output {2. if "_grokparsefailure" not in [tags] {3. elasticsearch {4. protocol => "transport"5. host => "localhost:9300"6. cluster => "jc-play"7. index => "gridshore-logs-%{+YYYY.MM}"8. manage_template => false9. template_name => "gridshore-logs"10. }11. }12.}
format of index to create:gridshore-logs-2015.02
![Page 90: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/90.jpg)
STORE1. output {2. if "_grokparsefailure" not in [tags] {3. elasticsearch {4. protocol => "transport"5. host => "localhost:9300"6. cluster => "jc-play"7. index => "gridshore-logs-%{+YYYY.MM}"8. manage_template => false9. template_name => "gridshore-logs"10. }11. }12.}
provide our own index template
![Page 91: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/91.jpg)
DEMO
![Page 92: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/92.jpg)
Integrated Disease Management Control
LESSONS LEARNED
![Page 93: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/93.jpg)
DATA ENHANCEMENT
![Page 94: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/94.jpg)
PROBLEM WITH DATES
![Page 95: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/95.jpg)
PROBLEM WITH DATES
![Page 96: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/96.jpg)
PROBLEM WITH DATES
![Page 97: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/97.jpg)
PROBLEM WITH DATES
![Page 98: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/98.jpg)
WHAT CANNOT BE DONE
![Page 99: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/99.jpg)
THINGS ABOUT AGE
![Page 100: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/100.jpg)
THINGS ABOUT AGE
![Page 101: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/101.jpg)
GOOD TO KNOW
![Page 102: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/102.jpg)
GETTING BIG
![Page 103: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/103.jpg)
SMAP - Soil Moisture Active Passive
http://smap.jpl.nasa.gov/mission/why-it-matters/
MonitorDrought
PredictFloods
AssistCrop
ProductivityWeatherForecasting
![Page 104: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/104.jpg)
![Page 105: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/105.jpg)
VERIZON
https://speakerdeck.com/bhaskarvk/elastic-on-15-500-billion-documents-and-counting
“We offer technology products and solutions that transform the way our customers connect, collaborate and innovate”
![Page 106: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/106.jpg)
VERIZON
https://speakerdeck.com/bhaskarvk/elastic-on-15-500-billion-documents-and-counting
“We offer technology products and solutions that transform the way our customers connect, collaborate and innovate”
Store massive logging data
Store in high rate
Query in acceptable rate
![Page 107: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/107.jpg)
VERIZON
128 Nodes
8 cores - 64 Gb RAM - 6 x 1TB disk
10+ Bilion documents a day
Over 500 Billion documents total
![Page 108: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/108.jpg)
SAVING YOUR DASHBOARDS
![Page 109: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/109.jpg)
![Page 110: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/110.jpg)
WHAT ABOUT SECURITY
Elastic shield
![Page 111: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/111.jpg)
FUTURE DIRECTIONS
![Page 112: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/112.jpg)
LOGSTASH
![Page 113: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/113.jpg)
LOGSTASH
• API for pipeline
![Page 114: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/114.jpg)
LOGSTASH
• API for pipeline
• Internal / persistent queues
![Page 115: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/115.jpg)
LOGSTASH
• API for pipeline
• Internal / persistent queues
• Clustered logstash
![Page 116: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/116.jpg)
ELASTICSEARCH
![Page 117: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/117.jpg)
ELASTICSEARCH
• Better error responses
![Page 118: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/118.jpg)
ELASTICSEARCH
• Better error responses
• Reindex API
![Page 119: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/119.jpg)
ELASTICSEARCH
• Better error responses
• Reindex API
• Changes API
![Page 120: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/120.jpg)
KIBANA
![Page 121: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/121.jpg)
KIBANA
• Formatting output: numbers, currency, urls, video
![Page 122: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/122.jpg)
KIBANA
• Formatting output: numbers, currency, urls, video
• Edit and save or pin filters
![Page 123: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/123.jpg)
KIBANA
• Formatting output: numbers, currency, urls, video
• Edit and save or pin filters
• Choose your own colours in charts
![Page 124: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/124.jpg)
KIBANA
• Formatting output: numbers, currency, urls, video
• Edit and save or pin filters
• Choose your own colours in charts
• Create API for custom plugins
![Page 125: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/125.jpg)
SUMMARISE
![Page 126: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/126.jpg)
SUMMARISE
• Real time data analysis
![Page 127: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/127.jpg)
SUMMARISE
• Real time data analysis
• Obtain and transform data using logstash
![Page 128: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/128.jpg)
SUMMARISE
• Real time data analysis
• Obtain and transform data using logstash
• Index data in elasticsearch
![Page 129: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/129.jpg)
SUMMARISE
• Real time data analysis
• Obtain and transform data using logstash
• Index data in elasticsearch
• Show data using Kibana
![Page 130: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/130.jpg)
SUMMARISE
• Real time data analysis
• Obtain and transform data using logstash
• Index data in elasticsearch
• Show data using Kibana
• What Kibana does well and what not
![Page 131: Real-time data analysis using ELK](https://reader030.fdocuments.in/reader030/viewer/2022012913/55c4ea3dbb61eb973f8b47cf/html5/thumbnails/131.jpg)
MORE INFORMATION
@jettroCoenradie
[email protected]://amsterdam.luminis.eu/news/
https://www.elastic.co/products