Log AnalysisLog Analysiswithwith

Presenter: Nathan HunstadMay 2015

Obligatory Disclaimer

● This talk represents my own work: I am not representing any employer, organization, or affiliated group, past, present, or future

● This talk is based on my experiences in my home lab network and not in an enterprise setting

● This is an overview only and is provided without warranty: do not rely on what you learn here for compliance or legal obligations!

What is Log Analysis?

Not this:

What is Log Analysis?

Or this:

What is Log Analysis, really?

● Forensics: Reconstructing events that have already happened

● Incident Response: Acting on logs in real-time to identify, contain, and remediate security incidents

● Troubleshooting: Evaluating systems for faults or unintended behavior and fixing as necessary

Handling Logs

● Help!

Splunk

● Splunk captures all kinds of machine data – app log files, syslog, text files, configuration files...basically any text data can be ingested

● Splunk provides a powerful search engine based on MapReduce for fast searching1

● Splunk has add-ins that allow for quickly setting up dashboards and reports for common log sources

● No, I do not work for Splunk

1 https://www.splunk.com/content/dam/splunk2/pdfs/technical-briefs/splunk-and-mapreduce.pdf

Splunk Licensing

● Splunk Enterprise: based on log volume

● Splunk Free: fewer features, 500MB/day

– Go over? You will lose search access!– But good enough for home use

Splunk Licensing

● Average Logging Volume

Adding Data to Splunk

Getting Data Into Splunk

● Splunk Forwarder

– Install on any system to read log files locally and forward to Splunk Indexer

– Versions available for Windows, MacOS, Linux, Solaris, HPUX, AIX, and FreeBSD

– Configure using GUI or edit .conf files

Getting Data Into Splunk

● Listen on port

– Splunk daemon binds to a port to listen for traffic (TCP or UDP)

– Typically used with syslog data

Getting Data Into Splunk

● Monitor Files/Directories

– Splunk daemon monitors individual files or an entire directory for new files/changes to files

– Computes CRC and bytes read on files to detect changes

– Can automatically decompress common formats like zip files

Getting Data Into Splunk

● Remote Hosts

– What if you can't install a forwarder on a remote host (for example, your shared web host?)

– My Solution: cron job + monitoring files

Splunk Basics

Indexes

● Indexes are the logical buckets into which data is stored● By default, all data gets stored in the main index, but

other indexes can be defined● A number of internal indexes exist for tracking Splunk

functionality and start with _, such as _internal and _audit

● Data retention and access control* is done on a per-index basis

Buckets

● Buckets are collections of index data and metadata

● Buckets age through several stages: Hot, Warm, Cold, and Frozen

● Not terribly important for home use, but managing retention becomes important for large data sets

My Environment

● Splunk server: located on server running CentOS

● Feeds from VMs, Windows desktops, EdgeOS router, managed switch, application logs, external website

My Environment

● Data is split up into multiple indexes for logical grouping

● Indexes for firewall, switch, Linux, Windows, website, and BOINC events, plus a throwaway index for testing

Windows Events

● Events from Security, Application, and System logs

Windows Events

● PerfMon performance monitoring events

Linux Events

● Syslog events

Website Logs

● Multiple access logs

Website Logs

● Apache access_combined

Firewall Logs

● Dropped and specific accepted connections

Switch Logs

● Connected devices

Application Logs

● BOINC (Berkeley Open Infrastructure for Network Computing) events

Basic Search Syntax

Search Syntax

● Basic search: just type in what you want to see

Search Syntax

● Limiting by fields

Search Syntax

● Counting events: stats count

Search Syntax

● Top events: top

Search Syntax

● Bucketing events and charting: timechart

Security Events

Brute Force Windows

● Using ncrack against RDP

Brute Force Windows

● Success!

Brute Force Linux

● Using Metasploitable ssh_login module and default root_userpass.txt

Port Scanning (External)

● Port Scanning: Same source IP, multiple destination ports

Port Scanning (Internal)

● Port Scanning: Same source IP, multiple destination ports

SQL Injection

● sqlmap against DVWA

● Apache logs sent to Splunk

Blind SQL Injection

● sqlmap/DVWA

XSS

● Persistent XSS on DVWA

Mimikatz

● Running mimikatz to dump hashes

● Nothing happens

Correlation

● Transactions: group events together that match a pattern

● Successful login following failed logins

Correlation

● Show attackers in a table

● index=linux | transaction host,rhost startswith="eventtype=sshd-login-failure" endswith="eventtype=ssh_open" | bucket _time span=30m | table _time, rhost

More Splunking

Field Extraction

● Splunk can handle some log types automatically pretty well, but adding rules for field extraction can help with searching and indexing

● A number of extractions come with Splunk ready for use, or you can add your own

● Uses regex for extraction

Lookups

● Uploading CSV files for extracting or expanding on data in logs

Lookups

● Previous timechart, now with names

Data Models

● Data Models are a powerful way of structuring data to generate specialized searches and visualizations

● Can be used to generate pivot tables and other complex objects

Pivot Tables

● Based on defined data models

● Display data in tabular format

Dashboards

● Bringing all your data to one spot, with user-selectable attributes

Visualizations

● Looking closer at the “Website Attacks” dashboard:

– Logarithmic Y-axis

– Daily Buckets

Visualizations

● Grouping Events

– Attacks: index=website joomlafailure sourcetype="php_error" | transaction IP maxpause=1h maxevents=5000| where eventcount>1 | table _time, IP, eventcount

– Port Scanning: index=firewall RuleName=WAN-*default-D | bucket _time span=30 | eventstats dc(DPT) AS PortsScanned by SRC, _time | where PortsScanned > 5 | dedup SRC, PortsScanned | table SRC, PortsScanned, _time

Visualizations

● Firewall Drops

Geolocation

● Splunk can Geolocate IP addresses

Geolocation

● Search: index=website joomlafailure sourcetype="php_error" | transaction IP maxpause=1h maxevents=5000 | where eventcount>1 | iplocation IP | geostats latfield=lat longfield=lon sum(eventcount)

Splunk TAs

● Splunk comes with TAs (Technology add-on) with pre-defined field extractions, transformations, and dashboards

Happy Logging!