How to capture who did what in your SQL

Server databases

Andy Warren

Free Tool SQL XEvent Profiler

SQL XEvent efficiency with SQL Profiler simplicity

View sessions running on servers and the SQL statements

they are executing

Group, sort and export events to find what you need and

share with others

Simple, clean design provides quick start and run in 2 clicks

Easily connect to both local and remote SQL Servers

www.idera.com

Goals For TodayGet you thinking about what you should be logging/auditing now, before you need it

Show you some places you can look for answers today

Discuss the options for auditing within SQL and mention some third party options

Look at the bigger picture of compliance and security

What Changed?It’s the first question we ask when something is wrong.

Who changed a database object or what did it look like before?

When did this row get updated?

When did the instance configuration change?

It’s not always an easy question to answer

Who is often less important than what – we want to fix the problem, deal with anything else later

But sometimes “who” matters a great deal!

It’s not always about change either, it can be “who saw these records”?

Logging Or AuditingLogging tends to be very verbose and unfiltered, auditing more targeted – but even logs can be set to various levels of verbosity

Audit often equals “legal” or “compliance”

For our purposes not much difference – we want to capture certain events and certain attributes of those events

Don’t get hung up on the difference. Just be sure you understand what you have to keep vs what you choose to keep

What Should I Audit?All changes to security (never go wrong doing this)

All DDL changes

Any change to your audit mechanism

Any access of credit card data (and I would suggest including ACH and SSN as well)

Can’t stress enough that you should validate what you’re capturing with compliance

Also – try to think separately about instance/DBA auditing and data/application logging

Before You Look in SQLAsk. Yes it’s low tech, but a quick shout across the cube farm and/or a quick email to the likely culprits can often get you the answer fast.

Change control – a quick query of tickets may be all you need

Source control – not easy for databases, but some do it, and even if not perfect, looking may give you what you need

Windows event logs – SQL writes some stuff there, but lots of other things get written there too

Monitoring tools/services – maybe you’re using SCOM, or Idera DM, or something similar

Already On – Places You Can Look NowFailed Logins

SQL Error Log

SQL Agent Log

Default Trace

Transaction Log

Existing Tables/Columns (User Data, Backup History, etc)

Already On – Failed LoginsThis defaults to on, so it should be on. If not, go fix it!

Already On – SQL Error Log Not a ton of detail, but convenient

Should schedule a job to cycle it (sp_cycle_errorlog) to manage size and simplify retention

Default is 6 logs, but you can change if you need more◦ EXEC xp_instance_regwrite N'HKEY_LOCAL_MACHINE',

N'Software\Microsoft\MSSQLServer\MSSQLServer', N'NumErrorLogs', REG_DWORD, 10

Rarely backed up/archived beyond that

Already On – SQL Agent LogNot usually interesting from a “who did that” perspective, more for debugging

One more place to look though

Path = MSSQL11.MSSQLSERVER\MSSQL\Log\SQLAGENT.OUT

Already On – Default TraceAlready running, unless you turned it off (rare)

How do you find it?◦ SELECT REPLACE(CONVERT(VARCHAR(1000),SERVERPROPERTY('ErrorLogFileName')),

'\ERRORLOG','\log.trc'); (Credit: Jason Strate)

Contains a limited but useful set of events

Marked as deprecated in 2012, move to extended events – but works now!

It roles over automatically and you only get 5 files at any one time

Decide if you want to retain them, or disable and put your own trace in place

Already On – Transaction LogCan read from log using fn_dblog or read from a log backup using fn_dump_dblog

Pro – it’s there and you’re probably backing it up too

Con – it’s not always easy to figure out (example in a minute)

SELECT [Begin Time], [Operation], [Description], SUSER_SNAME ([Transaction SID]) AS [User], [transaction id]

FROM fn_dblog (NULL, NULL) Log where [transaction name] like 'drop%'

C2It’s a trace

Overkill for 99.9% of cases!

Deprecated

Must restart to change location and location has to be the default data folder

Will stop the service if you run out of space

Volume can get crazy depending on environment

sp_configure 'c2 audit mode', 1 ;

ProfilerGreat for ad hoc investigating, not for permanent solution

Easy to use, but requires a logged in user

Can drop events

Can see basically everything, have to decide what (always the challenge) to capture

Cost is roughly proportional to what you capture

Don’t use the option to write directly to a table, expensive

Extended EventsSupersedes Profiler/Trace

More powerful, but perhaps a little harder to work with too

Event NotificationsMiddle ground between Trace and SQL Audit

Built on top of service broker

Similar audit options to those in SQL Audit, minus the nice GUI!

CREATE EVENT NOTIFICATION EventDDL_Table_events

ON DATABASE

FOR DDL_TABLE_EVENTS --covers create, alter, drop

TO SERVICE 'svcname here' , 'current database'

http://technet.microsoft.com/en-us/library/bb510452.aspx

Server TraceSyntax is a chore, unless you generate it using Profiler (the only way to go!)

Less costly than sending to Profiler, runs on server

Writes to disk with rollover files, have to manage those files or you’ll run out of space eventually

You can open in Profiler, or load into a table – which depends on how often you’re going to use it.

Like Profiler, sees all

SELECT IDENTITY(int, 1, 1) AS RowNumber, * INTO temp_trc FROM fn_trace_gettable('c:\temp\demo.trc', default);

TriggersDML, DDL, Server Triggers give you a range of options

Can not only log but change or stop an action (blurs the lines more)

Part of the transaction and make the transaction more expensive

Can even cause transaction to fail if the logging goes wrong

Don’t dismiss as evil, don’t use everywhere – be surgical

SQL AuditEnterprise edition feature

Elegant, built on extended events

Can write to a file, Security Log (best), or Application Log

Auditing SelectsIt’s easy enough to capture select statements

But…

The challenge is answering the question “what rows did they see”. Imagine your merchant bank calls and says “show me everyone who saw card number xxxx-xxxx-xxxx-1234”. That’s a level of detail we don’t want to infer, we need the detail!

No select trigger, it would be handy at times like this.

Best way is to do the logging within a procedure they have to call to see the data (to get a single row/record).

Also possible to capture the network traffic (TDS) and store that

Sound painful? Yes.

Recap of TechniquesThe “already on” items are probably not enough

Easiest is to build your own version of a default trace – works across all supported versions and editions

Most elegant is SQL Audit writing to event log

Event notifications not quite as elegant and require more work, but works on Standard edition

Use triggers for “tactical” logging like date stamping records and when you need to see the history of changes easily

Other OptionsGuardium sits between OS and SQL, can see all, but DBA (assuming not local admin) can’t change/tamper with logging settings

Imperva has an appliance based solution

GreenSQL acts as reverse proxy for SQL traffic

SQL Compliance Manager by Idera (sponsoring the webcast today!)

Buy Or Build?I wish I had an easy answer for you!

I’ve seen it built in a large enterprise and it was less work than you might think – define the target, provide a service and a bulk load approach, let every team go from there

It’s almost never all one or the other. I think a lot of value in buying a product that will do correlation and alert management, and a lot of value in using some very powerful options within SQL to give us a great view of what occurred.

Don’t get so focused on perfect that you fail to act. Better to have some than none.

Backup and RetentionI know you know, but it bears repeating – decide up front how long you’ll keep the audit data before you archive/purge (and be clear if you can ever purge)

For example, PCI requires 90 days online, 12 months available

You can run into legal situations where all deletes are on hold – have to keep forever

Back this stuff up too, you may end up needing to go back in history to find out when something started (data was corrupted by bad process or hacker first gained entry)

CorrelationAs much as we zoom in on what happened on an instance, that is just the start

Correlation is looking across instances, across servers, across applications to see if we can find patterns – think of a failed login across all instances within a few seconds. By itself it’s not notable. Combined it could mean someone sweeping the subnet for a toe hold.

You can roll up all your data into one place to aid this (data warehouse anyone?) and that also makes it easy to reduce the chances of tampering. It also makes it easy(easier) to manage retention.

Correlation isn’t easy and it’s not just data from SQL, you have to pull in lots (all) of the data to make it really go.

Can You Answer These Questions?Who changed the password on login ‘abc’?

When did the recovery model get changed to simple for database ‘xyz’?

Who made the foreign key untrusted?

How many failed login attempts did you have last week?

Who made the last change to the CLR assembly marked as unsafe/external?

What end users viewed the credit card record for John Doe?

Connect With AndyI blog on various topics at http://sqlandy.com

Please do connect with me on LinkedIn, http://www.linkedin.com/in/sqlandy

I’m on Twitter occasionally, but it is a good way to reach me: @sqlandy

I live in Orlando. If you’re in town let’s do lunch!

SummaryAuditing is simple and complex, embrace that paradox

Spend some thinking about the data points you want to have the next time something goes wrong, then see if you can afford to capture it (or have a way to)

Plan up front for archive/purging of audit data

Resourceshttp://www.sqlskills.com/blogs/paul/using-fn_dblog-fn_dump_dblog-and-restoring-with-stopbeforemark-to-an-lsn/

http://colleenmorrow.com/2013/04/15/event-notifications-101-intro-to-event-notifications/

http://www.mssqltips.com/sqlservertip/2121/event-notifications-in-sql-server-for-tracking-changes/

Book: Logging and Log Management: The Authoritative Guide to Understanding the Concepts Surrounding Logging and Log Management