Security Monitoring Thru Log AnalysisSam.NG@PISA

What the hacker did Startup script was modified, a

line is added to the rc.local ./usr/bin/t0x5mm (I can’t re

member the exact name) “ls” the file, nothing showe

d I tried to remove the file by “

rm”, “rm” ran without any error

Many months later, I knew that’s something call “rootkit”

Log was gone! Some of the log files are

truncated I wanted to know how the hacker

got in, and I redirected syslog to “/dev/lp0” and ran for a couple of weeks

Several inchesSeveral inches of paper printed, but there is more than I could handle

I can’t remember how, but finally I knew the hacker got in by exploiting an IMAP4 vulnerability

Lesson Learnt

You will need the log in sometime

And better still, store it in a safe place

Write once read many (WORM) is good, but searching capability is even more important

And the most important: you have to prepare it

beforehand

But how can I use the log for security monitoring?

The Challenge

“I don’t have the staff to handle 140,000 alerts. I don’t even have the staff to handle 12,000 alerts”

David MacLeod, Ph.D., CISSPThe Regence Group CISO

Reference:Counterpane’s Security Monitoring Service

Bruce Schneier

Block Diagram

Phase IPhase I Phase IIIPhase III Phase IVPhase IVPhase IIPhase II

LogCollection

NoiseReduction

DataMining

Alert/TicketManagement

Log Collectors Data Mining

Alert/TicketManagement

Noise Reduction

Log Collectors

Send your log to DATABASEDATABASE Can be direct (open database

connection from the client) or in-direct (e.g. post data to a web application)

Each different log may need a different log collector

Many logs have built-in support to store data in database

Unix syslog syslog can be redirected Can be redirected to printer! (“/dev/lp0” in

Linux) Can be redirected to remote syslog host But redirect to program (pipe) seems not

working!!?? syslog-ng support logging directly to database Win32 solutions to accept remote syslog

records and log to database Plain text format, easy to write a program to

watch the data and then send to database

Demo

A smiple “tail” base perl script to monitor new records in syslog and send it to remote database

Windows Eventlog

Microsoft logparser can log to database

WMI interface allows you to query remote eventlog

Many third party solutions support logging to database

Demo

A smiple C# program to monitor new records in Eventlog and send it to remote database

Quiz 1

A. Windows NTB. Windows 2000C. Windows 2003D. None of the above

Which of the following Windows Server will pass Microsoft Baseline Analyzer (MSBA) 2.0 auditing requirements with default install?

DO YOU KNOW? You can pass MSBA 2.0 if you enabled Success

and Failure auditing for the Account Logon Events

But indeed, it recommends you to enable Success and Failure auditing: Account Logon Events, Account Management, Policy Change, System EventsFailure auditing: Directory Service Access, Object Access

Data volume may be quite high, a server config in this way may generate a eventlog for every 2 seconds (actual volume depends on your server)

Microsoft IIS Web Server

Native support log to ODBC (but seems to be only available on server platform only)

Microsoft IIS Web Server (2) Microsoft does not recommend logging to

database if the IIS is busy (Q245243) But nowadays most web pages are generated

with SQL queries Test ODBC logging on your own environment PrepWebLog (Q296093) convert lIS log to SQL

insertion statements in plain text format, but still have to run these SQL insertion by some other means

Still, the best would be realtime logging to database

Writing a “tail -f” similar program in IIS is difficult

DO YOU KOWN? IIS log file is updated in 64k chunks.

On servers that do not have a high usage rate, the statistics will not be up-to-date because of the delayed write (Q142557)

When it flush, may be only first 20k of the chunk contains data, the rest 44k data is ‘\0’ and will be filled in next flush

Obviously a mmapped file, may be due to performance consideration… but

Suggestions to programmer In C/C++, STDERR, cerr are not buffered In fact, error log should be send out

immediately, should not be cached/buffered

# man stderr…………CONSIDERATIONS The stream stderr is unbuffered. The stream stdout is line- buffered when it points to a terminal. Partial lines will not appear until fflush(3) or exit(3) is called, or a newline is printed. This can produce unexpected results, especially with debugging output.…………

J2EE Application Server

Most application server support log4j or java.util.logging (JDK 1.4+)

log4j natively support logging to database thru JDBCAppender

My Experience I have a program developed since JDK1.3 At that time, Java don’t have java.util.logging And I don’t know much about log4j I wrote my own log handling routine (similar to s

yslog) to insert my own records into database But is not configurable/adaptable, and is not co

nsolidated with the Application Server’s log Since logging is widely spread all over the codes,

it is very difficult for me to change the program to use these new technology

Log architecture should be planned in DESIGN PDESIGN PHASEHASE

Snort IDS

Comes with database output plugin to send alerts to database

The packet analysis thread is also responsible for database insertion

My Experience

I have experience in using Snort to monitor a ~20M Internet link, with database output plugin, default rules

CPU is just about 30%, seems good When I changed to log to local file,

CPU sharp jump to 100% Barnyard seems to be a solution but it

does not support database!!

And many others

Checkpoint FW1: thru LEA enabled product http://www.opsec.com/

Microsoft ISA: default support ODBC logging (KB838710)

Microsoft Exchange: same as Eventlog Apache httpd: mod_log_sql

Noise Reduction Very important Noise will kill the systemNoise will kill the system And to improve overall performance Can be done on several layers1. ignore certain message at the log collection

sender2. ignore certain message at the log collection

receiver end3. delete (mark as ignore) at the database4. fine tune your IDS rule, firewall logging policy

etc..5. exception cases for data mining phase

Data Mining

“Data processing using sophisticated data search capabilities and statistical algorithms to discover patterns and correlations in large preexisting databases; a way to discover new meaning in data”From http://www.tfd.com

Data Mining (2) The heart of Security

Monitoring A board term, a gener

al concept Utilize database queri

es to get the information you want

Can be an external program, can be a SQL server schedule job

Data Mining Techniques

Data Mining

Attack Detection

Anomaly Detection

Attack Definition

Event Correlation

Normal Definition

Statistical Analysis

Event Correlation

Statistical Analysis

Attack Definition

GET /..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\GET /MSADC/root.exe?/c+dir GET /_mem_bin/..%255c../..%255c../..%255c../winnt/system32/cmd.exe?/c+dir GET /_vti_bin/.%252e/.%252e/.%252e/.%252e/winnt/system32/cmd.exe?/c+dir+c:\GET /_vti_bin/..%255c../..%255c../..%255c../winnt/system32/cmd.exe?/c+dir GET /_vti_bin/..%c0%af..%c0%af..%c0%af..%c0%af..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\ GET /_vti_bin/..%c0%af../..%c0%af../..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\GET /_vti_cnf/..%c0%af..%c0%af..%c0%af..%c0%af..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\ GET /adsamples/..%c0%af..%c0%af..%c0%af..%c0%af..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\ GET /cgi-bin/..%255c../..%255c../..%255c../winnt/system32/cmd.exe?/c+dir+c:/ GET /cgi-bin/..%c0%af..%c0%af..%c0%af..%c0%af..%c0%af../winnt/system32/cmd.exe?/c+dir+c:/ GET /cgi-bin/..%c0%af..%c0%af..%c0%af..%c0%af..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\ GET /cgi-bin/..%f0%80%80%af../winnt/system32/cmd.exe?/c+dir+c:/ GET /iisadmpwd/..%c0%af..%c0%af..%c0%af..%c0%af..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\ GET /iisadmpwd/..%c0%af../..%c0%af../..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\GET /msadc/.%252e/.%252e/.%252e/.%252e/winnt/system32/cmd.exe?/c+dir+c:\GET /msadc/..%c0%af../..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\GET /msdac/root.exe?/c+dir+c:\GET /scripts/..%e0%80%af../winnt/system32/cmd.exe?/c+dir+c:\ GET /scripts/..%f0%80%80%af../winnt/system32/cmd.exe?/c+dir+c:\

IIS Unicode Directory Traversal (cmd.exe) attack strings

Vulnerability characteristic ≠attack characteristic

CodeRed I [/default.idq?NNNNN……] CodeRed II [/default.idq?XXXXX……] In fact, the vulnerability can be exploited if the va

riable name is around 240 bytes Regular Expression [\/default\.idq\?.{240,}] Difficult to write an effective and accurate definiti

on And not all attack leave audit trail

Event Correlation

E.g. 1000 login failures followed by ONE successful login from the same IP

E.g.IF http_response_code = 500; THEN find_all_other_url_accessed();ENDIF

Anomaly Detection

Normal Definition

You define what is normal and then monitor it

E.g. Operators login time should be corresponding to their shift duty

E.g. All server services should not be restarted unless necessary (ignore service start within 3minutes of system startup)

Statistical Analysis E.g. on average a event

occurs 10 time a day, with a standard deviation of 2.3. But today we have 2000 records.

E.g. anything happens more than 200 times in the past 30 minutes

E.g. a event never seen in the past 7 days

E.g. “TOP 10” events/users/hosts, etc…

Alert/Ticket Management Works like Bug

Tracker Save the alert as a

“Ticket” A ticket is a

something like an outstanding job

Assign the ticket to a staff to follow up

Escalate it if remain unresolved for some time

And don’t forget People Management(Time and Skill)

Alert/Ticket Management (2) According to my

experience, at least 90% of the alerts generated by data mining are still FALSE ALARMS

People will get use to it and tend to think EVERY alerts are false alarms

If possible, fine tune the system to eliminate the false alarm from occurring again

Couterpane 2003 Results

DO YOU KNOW?

TSL providing Email to pager service for about HK$80/mth

How should I start? Do it step by step, phase by

phase, EventLog, syslog are easy to start with

A group brain storming section would give you at least 10 such data mining rules, and is a good starting point

Security Monitoring is a (long term) process, do not regard it as a single one-shot install and forget project

Contact: samngms@yahoo.com