Become a SQL Server Performance Detective

Danette Dineen RivielloMagellan HealthMarch 17, 2015

1

Game Plan

What triggers an Investigation? Emergent Performance Issues Chronic Performance Problems Solving the Case

2

Open a case

Increase in User ComplaintsApplication TimeoutsLong-running queriesOpen TransactionsChain of blocking

3

Look for Clues

Look at all running processes sp_who2 active

Look for one login or one database:

4

SELECT spid, [status],loginame [Login],hostname, blocked BlkBy,Db_name(dbid) DBName, cmd Command,cpu CPUTime,physical_io DiskIO,last_batch LastBatch,[program_name] ProgramName FROM master.dbo.sysprocesses where [status] not in ('sleeping')and loginame like '%login%'And Db_name(dbid) = 'DBName'ORDER BY dbname

Apprehend the Culprit

Look for the lead of a blocking chainSELECT spid,sp.STATUS ,loginame = SUBSTRING(loginame, 1, 12) ,hostname = SUBSTRING(hostname, 1, 12) ,blk = CONVERT(CHAR(3), blocked) ,open_tran ,dbname = SUBSTRING(DB_NAME(sp.dbid),1,10) ,cmd,waittype,program_name ,waittime ,last_batch ,SQLStatement = SUBSTRING ( qt.text, er.statement_start_offset/2, (CASE WHEN er.statement_end_offset = -1 THEN LEN(CONVERT(nvarchar(MAX), qt.text)) * 2 ELSE er.statement_end_offset END - er.statement_start_offset)/2 )FROM master.dbo.sysprocesses spLEFT JOIN sys.dm_exec_requests er ON er.session_id = sp.spidOUTER APPLY sys.dm_exec_sql_text(er.sql_handle) AS qtWHERE spid IN (SELECT blocked FROM master.dbo.sysprocesses)AND blocked = 0

5

Gather Evidence

Look at object locks

6

SELECT resource_type, object_name(resource_associated_entity_id), request_status, request_mode,request_session_id, resource_description FROM sys.dm_tran_locksWHERE resource_type = 'OBJECT'

Collect further clues

Look for open transactions

7

SELECT spid, [status],loginame [Login],hostname, blocked BlkBy,Db_name(dbid) DBName, cmd Command,cpu CPUTime,physical_io DiskIO,last_batch LastBatch,[program_name] ProgramName FROM master.dbo.sysprocesses WHERE open_tran>0ORDER BY spid

Search for the “Smoking Gun”

Find the line of code that is running:    SELECT [Spid] = session_Id      , ecid      , [Database] = DB_NAME(sp.dbid)      , [User] = nt_username      , [Status] = er.status      , [Wait] = wait_type      , [Individual Query] = SUBSTRING (qt.text,             er.statement_start_offset/2,      (CASE WHEN er.statement_end_offset = -1             THEN LEN(CONVERT(NVARCHAR(MAX), qt.text)) * 2            ELSE er.statement_end_offset END -                                er.statement_start_offset)/2)      ,[Parent Query] = qt.text      , Program = program_name      , Hostname      , nt_domain      , start_time    FROM sys.dm_exec_requests er    INNER JOIN sys.sysprocesses sp ON er.session_id = sp.spid    CROSS APPLY sys.dm_exec_sql_text(er.sql_handle) as qt    WHERE session_Id = @SPID          -- where @SPID is the one in Question    ORDER BY 1, 2

8

Investigate the environment What has changed? Look at default system trace:

9

Inspect System Trace Data

Look for recent changes

Look at the Log directory for prior files

10

Identify Chronic Offenders

To find most expensive stored procedures:

SELECT TOP 100 d.object_id, d.database_id, OBJECT_NAME(object_id, database_id) 'proc name', d.cached_time, d.last_execution_time, d.total_elapsed_time, d.total_elapsed_time/d.execution_count AS [avg_elapsed_time], d.last_elapsed_time, d.execution_countFROM sys.dm_exec_procedure_stats AS dORDER BY [total_worker_time] DESC;

11

Prioritize Worst Offenders Most expensive Stored procedure

runs

12

Set up a Profiler trace Job - Why

Less impact than an interactive trace Can load trace data on an alternate

server Can load trace data at a different

time of day Capture specific parameters passed Compare same time of day on

different days

13

Set up a Profiler trace Job - How

14

Set up a Profiler trace Job - How

15

Inspect the Profiler Trace File

Load the trace file to another serverselect * into dbo.tmp_loadtraceFile_ServerA_20150201_8FROM ::fn_trace_gettable('d:\trace_20150201_8.trc', 1)

Query trace file to find commands that are calling the suspected stored procedure select top 25 textdata, loginname, spid, duration, starttime, endtime, reads, cpuFrom dbo.tmp_loadtraceFile_ServerA_20150201_8Where textdata like '%offendingproc%'Order by duration desc

16

Inspect Profiler Trace Data

17

Retrieve the query plan

Use “Display Estimated Execution Plan”

Use “Include Actual Execution plan” Query to get query plans from DMV:

18

select top 25 st.text, qp.query_plan, qs.plan_handle from sys.dm_exec_query_stats qscross apply sys.dm_exec_sql_text(qs.sql_handle) stcross apply sys.dm_exec_query_plan(qs.plan_handle) qp

Examine long running procedures

Look at the query plan Missing index or wrong index

chosen? Look at the parameters sent in Check for other runs that perform

better Could it be a parameter sniffing

issue?

19

Parameter Sniffing - Defined Query plan developed based on the first

values passed to the procedure Pros:

Saves time: only one compile needed Cons:

Wrong query plan chosen

20

Parameter Sniffing - Detection

Look at Query plans If one procedure performs well in one

case and not others Do the index choices make sense?

21

Parameter Sniffing - Example

22

Parameter Sniffing – Solutions

Do Nothing Force Recompile each run

(expensive!) Query Hints (OPTIMIZE FOR) Break down stored procedures to

handle specific cases Education users on best parameter

choices

23

Rule out other culprits

Check for table scans caused by: Missing index Broad “where” clause

Check for improper join (many-to-many)

Check for too many tables in one join Use of a function in a large query

result set

24

Search for missing IndexesSELECT migs.avg_total_user_cost * (migs.avg_user_impact / 100.0) * (migs.user_seeks + migs.user_scans) AS improvement_measure, 'CREATE INDEX [missing_index_' + CONVERT (varchar, mig.index_group_handle) + '_' + CONVERT (varchar, mid.index_handle) + '_' + LEFT (PARSENAME(mid.statement, 1), 32) + ']' + ' ON ' + mid.statement + ' (' + ISNULL (mid.equality_columns,'') + CASE WHEN mid.equality_columns IS NOT NULL AND mid.inequality_columns IS NOT NULL THEN ',' ELSE '' END + ISNULL (mid.inequality_columns, '') + ')' + ISNULL (' INCLUDE (' + mid.included_columns + ')', '') AS create_index_statement, migs.*, mid.database_id, mid.[object_id]FROM sys.dm_db_missing_index_groups migINNER JOIN sys.dm_db_missing_index_group_stats migs ON migs.group_handle = mig.index_group_handleINNER JOIN sys.dm_db_missing_index_details mid ON mig.index_handle = mid.index_handleWHERE migs.avg_total_user_cost * (migs.avg_user_impact / 100.0) * (migs.user_seeks + migs.user_scans) > 10ORDER BY migs.avg_total_user_cost * migs.avg_user_impact * (migs.user_seeks + migs.user_scans) DESC

25

Solve the Case!

Solution may change over time Tables grow Statistics out of date Parameter Sniffing

Some problems result from multiple issues

Do least disruptive changes first: Add an index Close open connections

26

Questions?

Thank you for attending!

Further questions: ddriviello@magellanhealth.com

27