Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK...
Transcript of Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK...
![Page 1: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/1.jpg)
S
Finding the needle in the haystack with ELK
Elasticsearch for Incident Handlers and Forensic Analysts
![Page 2: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/2.jpg)
Whoami
S Working for the Belgian Government my own company S Incident Handling S Malware analysis
S Forensics (network + system)
S Open Source minded
S Creator of MISP – Malware Information Sharing Platform
S Creator of pystemon – pastebin monitoring tool
S Core organizer of the FOSDEM conference for many years
S Contact me: [email protected]
![Page 3: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/3.jpg)
S
Finding the needle in the haystack with ELK
Elasticsearch for Incident Handlers and Forensic Analysts
![Page 4: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/4.jpg)
image by James Lumb
![Page 5: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/5.jpg)
What tools do you use?
S Text logs
S notepad
S Grep
S awk / sed / cut
S MS Excel / OOo Calc
![Page 6: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/6.jpg)
image by velorichard.wordpress.com
![Page 7: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/7.jpg)
Optimizing
S grep -F log.txt
S zgrep -F log.txt
S zgrep -f patterns.txt -F log.txt
S find "$LOGS_DIR" -iname "*.gz" -print0 | parallel --gnu -0 -n1 -P8 zgrep -f patterns.txt –F > result-all.txt
S Fast for single search, however no column lookup !
![Page 8: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/8.jpg)
Optimizing
S MySQL / MS Access
S Splunk S free = 500MB/day
S ELSA – Enterprise Log Search and Archive S Limitation of the # of columns
S ${COMMERCIAL_TOOL}
![Page 9: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/9.jpg)
Trick for Splunk Addicts
S Limit is 500 MB /day
S 3 license violations allowed per month
S Set the date to 00:01 AM
S Index as much as possible 24h/day for 3 days (while loops are your friend)
S Enjoy searching
![Page 10: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/10.jpg)
logstash kibana
Trick for all = ELK
S Elasticsearch Logstash Kibana
S Index as much as you want
S No limit on volume, speed or position of the moon
S Open Source, Free to use, commercial support
![Page 11: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/11.jpg)
Configurations
S https://github.com/cvandeplas/ELK-forensics
S Repository with Logstash and Kibana configurations
S Mactime, BlueCoat, Mail IMSS, IWSVA, IIS, SuperTimeline, Plaso, …
S http://christophe.vandeplas.com/2014/06/setting-up-single-node-elk-in-20-minutes.html
S Our focus today: S Forensics and Incident Handling
S Batch-Import
![Page 12: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/12.jpg)
![Page 13: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/13.jpg)
![Page 14: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/14.jpg)
![Page 15: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/15.jpg)
![Page 16: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/16.jpg)
![Page 17: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/17.jpg)
![Page 18: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/18.jpg)
S
How does it work?
![Page 19: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/19.jpg)
logstash kibana
Trick for all = ELK
S Elasticsearch Logstash Kibana
S Index as much as you want
S No limit on volume, speed or position-of-the-moon-licensing
S Open Source, Free to use, commercial support
![Page 20: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/20.jpg)
Inputs
S Inputs & codecs S collectd, drupal_dblog, elasticsearch, eventlog, exec, file,
ganglia, gelf, gemfire, generator, graphite, heroku, imap, invalid_input, irc, jmx, log4j, lumberjack, pipe, puppet_facter, rabbitmq, rackspace, redis, relp, s3, snmptrap, sqlite, sqs, stdin, stomp, syslog, tcp, twitter, udp, unix, varnishlog, websocket, wmi, xmpp, zenoss, zeromq
S cloudtrail, collectd, compress_spooler, dots, edn, edn_lines, fluent, graphite, json, json_lines, json_spooler, line, msgpack, multiline, netflow, noop, oldlogstashjson, plain, rubydebug, spool
S Outputs
S Filters
![Page 21: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/21.jpg)
Input Example
S I usually don’t use “file” as input
S Keeps a reference to the position in the file
S TCP socket is the easiest for me
S ncat log01.lab.internal 18001 < logfile.log!
![Page 22: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/22.jpg)
Outputs
S Inputs & codecs
S Outputs S boundary, circonus, cloudwatch, csv, datadog,
datadog_metrics, elasticsearch, elasticsearch_http, elasticsearch_river, email, exec, file, ganglia, gelf, gemfire, google_bigquery, google_cloud_storage, graphite, graphtastic, hipchat, http, irc, jira, juggernaut, librato, loggly, lumberjack, metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty, pipe, rabbitmq, rackspace, redis, redmine, riak, riemann, s3, sns, solr_http, sqs, statsd, stdout, stomp, syslog, tcp, udp, websocket, xmpp, zabbix, zeromq
S Filters
![Page 23: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/23.jpg)
Output Example
![Page 24: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/24.jpg)
Filters
S Inputs & codecs
S Outputs
S Filters S advisor, alter, anonymize, checksum, cidr, cipher, clone,
collate, csv, date, dns, drop, elapsed, elasticsearch, environment, extractnumbers, fingerprint, gelfify, geoip, grep, grok, grokdiscovery, i18n, json, json_encode, kv, metaevent, metrics, multiline, mutate, noop, prune, punct, railsparallelrequest, range, ruby, sleep, split, sumnumbers, syslog_pri, throttle, translate, unique, urldecode, useragent, uuid, wms, wmts, xml, zeromq
![Page 25: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/25.jpg)
Filter Example
![Page 26: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/26.jpg)
Filter Example
![Page 27: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/27.jpg)
Grok
S Named regular expressions to match patterns/extract data.
S Logstash ships with lots of patterns ! https://github.com/elasticsearch/logstash/tree/master/patterns
S Test app: http://grokdebug.herokuapp.com
![Page 28: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/28.jpg)
Testing complex Groks
![Page 29: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/29.jpg)
Data Enrichment with Filters
S Extract fields: csv, grok, kv!
S Extract date!
S Modify using mutate!
S Enrich with S Geoip
S User-agent
S Urldecode
S Translate
S …
![Page 30: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/30.jpg)
Geoip
![Page 31: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/31.jpg)
Geoip
![Page 32: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/32.jpg)
User-Agent
![Page 33: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/33.jpg)
User-Agent
![Page 34: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/34.jpg)
Translate
![Page 35: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/35.jpg)
Translate
![Page 36: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/36.jpg)
Ruby as last resort
* There might be a better way to do this, but ruby and I are not really friends yet
![Page 37: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/37.jpg)
Data Enrichment with Filters
S Extract fields: csv, grok, kv!
S Extract date!
S Modify using mutate!
S Enrich with S Geoip
S User-agent
S Urldecode
S Translate
S …
![Page 38: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/38.jpg)
logstash kibana
Trick for all = ELK
S Elasticsearch Logstash Kibana
S Index as much as you want
S No limit on volume, speed or season-licensing
S Open Source, Free to use, commercial support
![Page 39: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/39.jpg)
Elasticsearch
S Wikipedia: Elasticsearch is a search server based on Lucene. It provides a distributed, multitenant-capable full-text search engine with a RESTful web interface and schema-free JSON documents. Elasticsearch is developed in Java and is released as open source under the terms of the Apache License.
S Very very fast
S Adding an node = easier than extremely easy
![Page 40: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/40.jpg)
Elasticsearch
S Be cautious
S No security by default
S Auto-discovery, auto-distribution if other node is present
S Elastic HQ plugin S cd /usr/share/elasticsearch/bin!S ./plugin -install royrusso/elasticsearch-HQ!
![Page 41: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/41.jpg)
logstash kibana
Trick for all = ELK
S Elasticsearch Logstash Kibana
S Index as much as you want
S No limit on volume, speed or horoscope-licensing
S Open Source, Free to use, commercial support
![Page 42: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/42.jpg)
Kibana
S Fancy GUI
S Extremely easy to build up a dashboard
S Gives good overview over data
S Powerful, but limited in capability
S For more: write a python script or use REST API
![Page 43: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/43.jpg)
DO NOT PRESS
THIS BUTTON
![Page 44: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/44.jpg)
Search syntax
S Apache Lucene Search syntax
S title:foo title:"foo bar”
S title:"foo bar” AND body:"quick fox”
S (title:"foo bar" AND body:"quick fox") OR title:fox
S title:foo -title:bar
S title:foo*bar
S time_taken:[10000 TO 999999999]
http://www.lucenetutorial.com/lucene-query-syntax.html
![Page 45: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/45.jpg)
Load dashboards
![Page 46: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/46.jpg)
Filter
![Page 47: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/47.jpg)
![Page 48: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/48.jpg)
S
Performance
![Page 49: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/49.jpg)
Performance goals
S Focus Incident Handling and Forensics
S Max speed of indexing
S Max speed of searching
S During indexation search may be slow
S No need for redundancy
S So don’t use this advice for operations-live-production
![Page 50: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/50.jpg)
Performance Logstash
S Memory setting: (/etc/default/elasticsearch) S LS_HEAP_SIZE="500m"!
S Command line flag: S -w or –filterworkers AMOUNT_OF_CORES (default: 1)!
S Each extra filter slows it down S Grok aka regex = slow
S Prefer csv, kv
S Use the least possible wildcards (* or +)!
S Geoip = slow but very practical
S User-agent = slow, often practical
![Page 51: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/51.jpg)
Performance Elasticsearch
S Memory setting (/etc/default/elasticsearch) S ES_HEAP_SIZE=12g => set to half of RAM (max 32 GB)
S Disable redundancy (/etc/elasticsearch/elasticsearch.yml)
S index.number_of_replicas: 0!
S Shards for number of nodes (/etc/elasticsearch/elasticsearch.yml) S index.number_of_shards: 1
S Increase memory buffer for search S indices.memory.index_buffer_size: 50%!
![Page 52: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/52.jpg)
Perf. Elasticsearch Indexes
S Open Index = memory usage + disk usage Closed Index = disk usage, so close index when not needed
S Per case new indexes Similar logs in the same index, but use a field “host” to differentiate investigations S system timelines: logstash-%{[case]}-%{[type]}
S mail logs: logstash-%{[case]}-%{[type]}-%{+YYYY.MM}
S proxy logs: logstash-%{[case]}-%{[type]}-%{+YYYY.MM.dd}
S curl -XPOST 'localhost:9200/logstash-${case}*/_close' curl -XPOST 'localhost:9200/logstash-${case}*/_open'!
![Page 53: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/53.jpg)
Performance Kibana
S Each block/graph is extra search
S So 10 graphs equals 10 simultaneous searches
1. First select small date/time window
2. Test your search on small data set
3. Add filters
4. Zoom out on date/time
5. Dig deeper
![Page 54: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/54.jpg)
Keep in mind
S Logstash is (relatively) SLOW
S Finished? Close the index, do NOT delete it
S Or save JSON to files (output plugin Logstash), re-index them later
S Node++ = Speed++
![Page 55: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/55.jpg)
S
Forensic analysis
![Page 56: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/56.jpg)
Plaso
S Plaso = the new log2timeline and more
S log2timeline.py win7-64-nfury-10.3.58.6.dump /path/to/disk/image
S psort.py -o elastic win7-64-nfury-10.3.58.6.dump
![Page 57: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/57.jpg)
ELK-forensics
S https://github.com/cvandeplas/ELK-forensics
S Logstash configs
S Kibana dashboards
S Mactime, Log2timeline csv, BlueCoat, Mail IMSS, IWSVA, IIS
S More to come
![Page 58: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/58.jpg)
Other interesting projects using Elasticsearch
S Moloch – Open Source large scale IPv4 full PCAP capturing, indexing and database system. https://github.com/aol/moloch
S Mozdef – PoC – automate IH process and facilitate real-time activities - https://github.com/jeffbryner/MozDef
S Suricata – Exports data in EVE format (JSON). Great to visualize malware activity from sandbox
![Page 59: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/59.jpg)
![Page 60: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/60.jpg)
![Page 61: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/61.jpg)
![Page 62: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/62.jpg)
![Page 63: Finding the needle in the haystack with ELK...Finding the needle in the haystack with ELK Elasticsearch for Incident Handlers and Forensic Analysts by Christophe@Vandeplas.com](https://reader034.fdocuments.in/reader034/viewer/2022042115/5e91f950aafbfb28c27c4429/html5/thumbnails/63.jpg)
S
Places to be? • https://github.com/cvandeplas/ELK-forensics • http://www.elasticsearch.org/overview/elkdownloads/ • http://logstash.net/ • https://groups.google.com/forum/#!forum/logstash-users