Performance Tuning - Using Waits & Queues ThomasDa

8/8/2019 Performance Tuning - Using Waits & Queues ThomasDa

http://slidepdf.com/reader/full/performance-tuning-using-waits-queues-thomasda 1/43

Real SQL Server Deployment

Performance Tuning ± Using Queuesand Waits

Gert Drapers & Tom Davidson

Customer Advisory TeamSQL Server Development



Agenda

Performance methodology Why waits & queues?

What impacts performance? App life cycle - before, during, after development

A Performance Tuning methodology Waits & Queues

Performance metrics & ratios

Other performance tools

Correlating Waits & Queues

Other sources of information

Conclusion

Q&A



Performance ± Why waits & queues?

Performance is a broad topic Art or Science?

Waits & Queues Methodology

Based on SQL Customer Lab

experiences Objective - analyze customer

app over 3-4 day period

Repeatable process

Useful in quickly identifyingsome bottlenecks

Performance, Architecture, Code

Not a replacement for good

design

??



DB Design

SQL

HardwareTuning

Test

Matrix

Production

workloads

App Design

Application Life Cycle



Impact on Performance

Application & Database Design and T-SQL have major impact on performance

µTypical¶ performance tuning plays smaller role

Performance monitoring can point out some deficiencies

(or opportunities for improvement!) throughoutapplication life cycle.

App

Design

DB Design SQL Hardware

tuning

Perf

Monitoringworkload

changes

25% 25% 20% 10% 20%



Performance Life Cycle

Performance monitoring plays a key rolethroughout the application life cycle, not

just at the end (prior to production)

Provides clues that identifies problemareas in an application including

SQL

App architecture

Database design

Configuration

Hardware



What can Performance Monitoring do?

Some typical bottlenecks

What are some of the typical problem areas in

application?

Cache management

Query plan reuse

Recompilation

Transaction management e.g. concurrency

Resource utilization

Doesn¶t replace need for good app design



Start Demo

SQLStressTest application

PerfMon counters

Track_waitstats



Waits & Queues

Complementary information Together, explains application performance

Most are familiar with queues ( ½ the story)

Both are essential for problem

determination

Queues

Perfmon counters - measure resource

utilization

Queues indicate unfulfilled resourcerequests

SQL Waits

Waits from an Application or User

connection perspective



Waits

SQL waits

Application sends SQL queries

SQL Server in turn issues resource requests ± Translates to subsystem IO, memory, CPU

± E.g. acquire locks, read | write data pages, sorts

50+ wait types for SQL2000

Anytime a user connection waits for results, SQLsets a wait type



Queues

PERFMON counters provide aview of system performance from

a resource standpoint

Queues measure resourceutilization

Requests that cannot be

immediately provided are queued

Key Perfmon counters (a few«)

Physical Disk (IO)

SQL Buffer Cache

SQL Statistics



Wait types

If a thread is not currently executing, a wait type or state

is set in sysprocesses

Sysprocesses contains

Lastwaittype

Waitype

Waittime Limitation: Transience of sysprocesses, spid history



Wait types DBCC sqlperf(waitstats,[clear])

Cumulative waittypes & waittimes

Limitation: relevance of types, what you can control?



Track_waitstats stored proc

captures SQL waitstats from DBCC

SQLPERF

ranks wait types in descending order

based on percentage of total waits

identifies greatest opportunites for

performance improvement

Can be graphed to see how waits varyover a workload

Limitation: long durations can ³hide´ or

smooth out waits



Track_waitstats code (1 of 2)CREATE proc track_waitstats (@num_samples int=10,@delaynum int=1,@delaytype nvarchar(10)='minutes')as-- T. Davidson

-- This stored procedure is provided "AS IS" with no warranties, and confers no rights.

-- Use of included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm-- @num_samples is the number of times to capture waitstats, default is 10 times. default delay interval is 1 minute-- delaynum is the delay interval. delaytype specifies whether the delay interval is minutes or seconds-- create waitstats table if it doesn't exist, otherwise truncateset nocount onif not exists (select 1 from sysobjects where name = 'waitstats')

create table waitstats ([wait type] varchar(80),requests numeric(20,1),[wait time] numeric (20,1),[signal wait time] numeric(20,1),now datetime default getdate())

else truncate table waitstatsdbcc sqlperf (waitstats,clear) -- clear out waitstatsdeclare @i int,@delay varchar(8),@dt varchar(3), @now datetime, @totalwait numeric(20,1)

,@endtime datetime,@begintime datetime

,@hr int, @min int, @sec intselect @i = 1select @dt = case lower(@delaytype)

when 'minutes' then 'm'when 'minute' then 'm'when 'min' then 'm'when 'mm' then 'm'when 'mi' then 'm'when 'm' then 'm'when 'seconds' then 's'when 'second' then 's'when 'sec' then 's'when 'ss' then 's'when 's' then 's'else @delaytype

end

if @dt not in ('s','m')begin

print 'please supply delay type e.g. seconds or minutes'return

endif @dt = 's'begin

select @sec = @delaynum % 60select @min = cast((@delaynum / 60) as int)select @hr = cast((@min / 60) as int)select @min = @min % 60

end



Track_waitstats code (2 of 2)if @dt = 'm'

begin

select @sec = 0select @min = @delaynum % 60

select @hr = cast((@delaynum / 60) as int)

end

select @delay= right('0'+ convert(varchar(2),@hr),2) + ':' +

+ right('0'+convert(varchar(2),@min),2) + ':' +

+ right('0'+convert(varchar(2),@sec),2)

if @hr > 23 or @min > 59 or @sec > 59

begin

select 'hh:mm:ss delay time cannot > 23:59:59'

select 'delay interval and type: ' + convert (varchar(10),@delaynum) + ',' + @delaytype + ' converts to ' + @delay

return

end

while (@i <= @num_samples)

begin

insert into waitstats ([wait type], requests, [wait time],[signal wait time])

exec ('dbcc sqlperf(waitstats)')

select @i = @i + 1

waitfor delay @delay

end

select @now=max(now),@begintime=min(now),@endtime=max(now)

from waitstats where [wait type] = 'Total'

--- subtract waitfor, sleep, and resource_queue from Total

select @totalwait = sum([wait time]) + 1 from waitstatswhere [wait type] not in ('WAITFOR','SLEEP','RESOURCE_QUEUE','Total', '***total***') and now = @now

-- insert adjusted totals, rank by percentage descending

insert into waitstats select '***total***',0,@totalwait,@totalwait,@now

select 'start, end, duration'='start: ' + convert(varchar(20),@begintime,20) + ' end: ' + convert(varchar(20),@endtime,20) + ' duration (minutes): ' + convert(varchar(10),

datediff(mi,@begintime,@endtime))

select 'waitstats samples'=@num_samples, 'delay, type'= convert (varchar(10),@delaynum) + ',' + @delaytype,'waitfor delay interval'=@delay

select [wait type],[wait time],percentage=cast (100*[wait time]/@totalwait as numeric(20,1))

from waitstats

where [wait type] not in ('WAITFOR','SLEEP','RESOURCE_QUEUE','Total')

and now = @now

order by percentage desc



Track_waitstats Sample output



Sample Wait Stats ± Bank (Linked Server)



Demo

Walk through perfmon, track_waitstats



Correlating Waits and Queues

Waits should be corroborated with Queues Look beyond the symptom

Symptom can mask the problem

Identifies the specific resource constraintalong with the associated wait types

Actual problem could be app, SQL, or DB

design, rather than resource limitations

Helpful to know performance profile whenthings are good

Now performance is bad - What changed?



SQL 2000 Waits & Queues * coming very soon on

MSDN



Correlating Waits & Queues: IO or Memory Pressure?

Waits Queues ExplanationIO_Completion

Async_IO_Completion

PageIOLatch_x

PageLatch_x

1. SQL Buffer Mgr

Avg Page Life

Expectancy

(seconds)

Checkpoint

pages/sec

Lazywrites/sec

2. Physical Disk

Avg disk sec/read

Avg disk sec/write

Disk queues

Waits indicate IO

issue

High Avg disk

secondsindicates IO

issue

HOWEVER Low

average page

life indicates

memory

pressure e.g.

cache flushing



Correlating Waits & Queues ± IO or DB Design?

Waits Queues Explanation1. IO_Completion

2. Async_IO_Completion

3. Writelog

1. SQL Buffer Mgr

Avg Page Life

Expectancy

(seconds)

Checkpointpages/sec

Lazywrites/sec

2. Physical Disk

Avg disk sec/read Avg disk sec/write

Disk queues

1. If Profiler shows:

Scan started

Reads

Writes

2. If Showplan shows

Table scans

Clustered

index range

scans Nonclustered

index range

scans

Sorts



Other sources of information

Virtual tables

Undocumented

DBCC options



Virtual System Tables

::Fn_virtualfilestats (dbid, [fileId | -1]) Provides breakdown of IO by file

Look for IostallMS

compare to Physical Disk reads/writes

Indentifies SQL component of IO

Syscacheobjects

Contains compiled objects e.g. query plans

Sysprocesses

Contains each SQL thread & wait types

Syslocks



Concurrency/CPU Issues: Cached Objects

Master..Syscacheobjects Procedure or batch name

Set options for plans

Ref counts, Use counts

Compiled plan

Single copy (serial and parallel)

Re-entrant and re-usable

Re-comps place **lock** on compile plan

Executable plan

Data structure for user context, not re-entrant

Look for plan reuse: usecounts > 1

Plan re-use of

Procs, Triggers, Views

Defaults, Check contraints, rules

adhoc SQL, sp_executesql



Syscacheobjects & Re-use



Plan re-use

SQL Batch requests/sec Compare to initial SQL Compilations/sec

SQL Compilations/sec

Includes initial compiles AND re-compiles

Eliminate re-compilations for initial compiles

Look for identical SQL statements with low

usecounts in syscacheobjects

SQL Re-compilations/sec Just re-compiles

Check profiler for sp:recompile event to

identify SQL statement.



Profiler events

Plan re-use (or lack of)

Compare batch requests to SQL compiles/sec

IO

Reads and writes

Recompilation

Cache hits, misses, discards

Index usage (or lack of)

Object access



Concurrency/CPU: Recompilation

Plan determination is CPU Intensive Recomp good if benefit of new plan > CPU cost

Profiler Lists recomp events and statements

Data column for reason: EventSubClass

locks on system tables Re-compiling stored procedure plans serialize other

users during high concurrency places lock on single compile plan

Re-compilation based on Rows changed thresholds (rowmodctr)

DDL placement, schema changes

Code practice & temp tables (P1 & P2)



EventSubClass: Reason for

recompilation1 Schema, bindings or permissions changed

between compile or execute.

2 Statistics changed.3 Object not found at compile time, deferred

check at run-time.

4 Set option changed in batch.

5 Temp table schema, binding or permission

changed.

6 Remote rowset schema, binding or

permission changed.



Uses of CPU resources

Sorts Distincts, order by,group by

Compare to index & IXoverhead

Joins Worktables, temp

tables

Query plans indicate

joins,types

Lack of planre-use SQL Compiles ±SQL Recompiles

syscacheobjects usecounts

Compare to batch requests

Look at parameterization

Recompilation SQL

Recompiles/sec

Compare to batch

requests

Manages IO Profiler Scans,

memory pressure,

DB design

Index utilization?

Page Life high enough?

UMS ± context

switching

Dbcc

sqlperf(umsstats)

Look for BIG

differences. Match big



When does more memory help?

Additional memory may help if«..

SQL Buffer Mgr:Page Life Expectancy is low

Average time in seconds a page stays in cache

SQL Server Memory Mgr: Memory GrantsPending is high

SQL won¶t execute a query if there isn¶t enough

memory for it

If after adding, Page Life Expectancy is

higher, more memory helps performance



SQL Server Scheduler ± understand it!

4 way CPU4 way CPU

UMSUMS

UMSUMS

UMSUMS

UMSUMS

1)1) Connections get assigned to UMSConnections get assigned to UMS

2)2) UMS schedules across ProcessorsUMS schedules across Processors

3)3) A connection stayed on UMS for life of threadA connection stayed on UMS for life of thread

4)4) Two heavy threads will fight on the same UMSTwo heavy threads will fight on the same UMS



SQL Server Scheduler ± example

5 BCP.exe jobs5 BCP.exe jobs

4 way CPU4 way CPU

UMSUMS

UMSUMS

UMSUMS

UMSUMS

1)1) Each load job takes 2 minutes to runEach load job takes 2 minutes to run

2)2) Task 2,3 and 4 each finished in 2 minutesTask 2,3 and 4 each finished in 2 minutes

3)3) Task 1 & 5 took closer to 4 minutesTask 1 & 5 took closer to 4 minutes

4)4) Using 4 threads instead of 5 would be aUsing 4 threads instead of 5 would be a

better solution for this example.better solution for this example.

5) Monitor using DBCC SQLPERF (UMSTATS)5) Monitor using DBCC SQLPERF (UMSTATS)

TaskTask11

33

44

55

22 1, 51, 5

22

33

44



Tracking UMS issues

Dbcc sqlperf(umsstats,clear)

Clears out UMS info

run workload

Dbcc sqlperf(umsstats)

Look for ***BIG*** differences in context

switching



Concurrency: Helpful Lock scripts

sp_blockinfo ± lists locking chain



Uses of IO resourcesScans Can flush cache,

cause memory

pressure

Look at Page life.

Data drive block

size=64K for large

IO intensive apps

Physical disk:split

IO/sec

Page splits SQL Access

Methods: Page

splits/sec

Order imposed by

index. New row

doesn¶t fit on page

Forwarded records SQL Access

Methods:Forwarded

records/sec

Small record

updated, doesn¶t fitback on page

Index maintenance Inserts, updates,

deletes

Balance with reads



IO issues Tranlog ± RAID10 is popular

Set controller or IO Subsystem cache for log as 100%write through (unless reading log with replication or alot of trigger activity)

Set Goal for 1-2ms / write (Avg Disk sec/write)

Test your SAN throughput prior to databaseactivity. Use IOMETER(shareware) or just copylarge files You should get ~ 120mb/sec throughput per channel

If heavy updated system then make cache more

write through vs read. make sure data is spread across as many drives as

possible. Individual disks are still the slowestcomponent for database activity.

RAID5 ± write penalty for write intensive apps



Design Issues There are design considerations resulting from

the Waits and Queues methodology.Observation Application issue Possible remedies

High IO waits 1. Database design

2. Memory Pressure

1. Bad query plans resulting from

improper indexing.

2. Add correct indexes to

minimize IO.3. Add more memory

High CPU Utilization 1. Memory pressure

2. Plan re-use

3. Recompilation

4. Parameterization

1. Check for correct plan re-use,

parameterization, re-

compilation , see

http://msdn.microsoft.com/libra

ry/default.asp?url=/library/en-us/dnsql2k/html/sql_queryreco

mpilation.asp

High Blocking / Low

concurrency

1. Transaction

management

1. Redo transaction management

2. use correct transaction

isolation levels



Summation

Application performance all about

waits and queues

Workloads may vary, shifting waits &

queues System performance is all about

identifying (& resolving) bottlenecks,

maximizing system capabilities

Figure out the problem, if it ain¶tbroke«.



Conclusions

SQL Server 2000 is a high performance

enterprise class database.

Performance Monitoring is a tool that should

be used throughout the application life cycle

Identify bottlenecks in application

database

resource utilization

Exploit all resources

CPU

I/O

Network

Data (concurrency vs. consistency)



References

³Inside SQL Server 2000´ by KalenDelaney

³SQL Server 2000 Performance Tuning´by Whalen, Garcia, DeLuca, Thompson

³SQL Server 2000 Recompilation´ athttp://msdn.microsoft.com

³ SQL Server 2000 Performance Tuning

using Waits & Queues Methodology´ at http://msdn.microsoft.com

SQL Server 2000 Waits & Queues (part 1)SQL Magazine (October 2003)

Performance Tuning - Using Waits & Queues ThomasDa

Documents

Transcript of Performance Tuning - Using Waits & Queues ThomasDa