SQLBits 2019

Copyright Heraflux Technologies. Do not redistribute or copy as your own. 1

© Heraflux Technologies®  2

0

10

20

30

40

50

60

00:00

00:30

01:00

01:30

02:00

02:30

03:00

03:30

04:00

04:30

05:00

05:30

06:00

06:30

07:00

07:30

08:00

08:30

09:00

09:30

10:00

10:30

11:00

11:30

12:00

12:30

13:00

13:30

14:00

14:30

15:00

15:30

16:00

16:30

17:00

17:30

18:00

18:30

19:00

19:30

20:00

20:30

21:00

21:30

22:00

22:30

23:00

23:30

% C

PU

Co

nsu

mp

tion

Time of Day (Avg)

CPU by Core ‐ Five Minute Avg

CPU00 CPU01 CPU02 CPU03 CPU04 CPU05 CPU06 CPU07

1

2

SQLBits 2019

Copyright Heraflux Technologies. Do not redistribute or copy as your own. 2

© Heraflux Technologies®  3

@kleegeek

davidklee.net

heraflux.com

davidaklee

Specialties / Focus Areas / Passions:

• Performance Tuning • Business Continuity• Virtualization & Cloud• Infrastructure Architecture• Health & Efficiency• Capacity Management Founder & Technical Exorcist

Completely subjective…

… until youmake it objective.

No more arguments.

Numbers drive the discussion.

© Heraflux Technologies®  4

3

4

SQLBits 2019

Copyright Heraflux Technologies. Do not redistribute or copy as your own. 3

© Heraflux Technologies®  5

VM Host

VM Host

SharedStorage

VM

VM

VM

VM

VM

VM

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

DB

Highest CPU consuming instance

Memory pressure

Storage throughput consumption

© Heraflux Technologies®  6

5

6

SQLBits 2019

Copyright Heraflux Technologies. Do not redistribute or copy as your own. 4

Impact of “noisy neighbors” CPU scheduling impact

Storage latency spike not originating in that server

Impact of system backups on the system

Correlated (seemingly) random events

© Heraflux Technologies®  7

© Heraflux Technologies®  8

7

8

SQLBits 2019

Copyright Heraflux Technologies. Do not redistribute or copy as your own. 5

Third‐Party Tools

Perfmon built‐in to Windows

Meaning, not just numbers

Show before and after

Granularity 

© Heraflux Technologies®  9

“I have 32 CPUs!”

6% CPU consumption average?

© Heraflux Technologies®  10

9

10

SQLBits 2019

Copyright Heraflux Technologies. Do not redistribute or copy as your own. 6

© Heraflux Technologies®  11

0

10

20

30

40

50

60

70

80

90

100

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

% C

PU

Co

nsu

mp

tion

Hour of Day (Avg)

CPU AVG (1H)

AvgCPUConsumption

0

10

20

30

40

50

60

70

80

90

100

00:00

00:40

01:20

02:00

02:40

03:20

04:00

04:40

05:20

06:00

06:40

07:20

08:00

08:40

09:20

10:00

10:40

11:20

12:00

12:40

13:20

14:00

14:40

15:20

16:00

16:40

17:20

18:00

18:40

19:20

20:00

20:40

21:20

22:00

22:40

23:20

% C

PU

Co

nsu

mp

tion

Time of Day (Avg)

CPU ‐ Five Minute Avg

SystemCPUConsumption UserCPUConsumption

© Heraflux Technologies®  12

0

10

20

30

40

50

60

70

80

90

100

00:00

00:40

01:20

02:00

02:40

03:20

04:00

04:40

05:20

06:00

06:40

07:20

08:00

08:40

09:20

10:00

10:40

11:20

12:00

12:40

13:20

14:00

14:40

15:20

16:00

16:40

17:20

18:00

18:40

19:20

20:00

20:40

21:20

22:00

22:40

23:20

% C

PU

Co

nsu

mp

tion

Time of Day (Avg)

CPU ‐ Five Minute Avg (BEFORE)

SystemCPUConsumption UserCPUConsumption

0

10

20

30

40

50

60

70

80

90

100

00:00

00:40

01:20

02:00

02:40

03:20

04:00

04:40

05:20

06:00

06:40

07:20

08:00

08:40

09:20

10:00

10:40

11:20

12:00

12:40

13:20

14:00

14:40

15:20

16:30

17:10

17:50

18:30

19:10

19:50

20:30

21:10

21:50

22:30

23:10

23:50

% C

PU

Co

nsu

mp

tion

Time of Day (Avg)

CPU ‐ Five Minute Avg (AFTER)

SystemCPUConsumption UserCPUConsumption

11

12

SQLBits 2019

Copyright Heraflux Technologies. Do not redistribute or copy as your own. 7

Collect raw data first

Perfmon how‐to guide Download at hfxte.ch/perfmon

No contact info required

30‐second collection interval

© Heraflux Technologies®  13

hfxte.ch/perfmon

© Heraflux Technologies®  14

13

14

SQLBits 2019

Copyright Heraflux Technologies. Do not redistribute or copy as your own. 8

© Heraflux Technologies®  15

Counter Set Counter Explanation

Processor % Privileged Time Windows OS layer CPU consumption

% User Time Application CPU consumption

% Processor Time % Privileged Time + % User Time

Process % Processor Time CPU consumption by process EXE

Processor Information % of Maximum Frequency < 100% = CPU core parking (bad)

System Context Switches / sec > 5000 x proc count is CPU scheduling pressure

Processor Queue Length > 5 x proc count is CPU scheduling pressure

© Heraflux Technologies®  16

Counter Set Counter Explanation

Memory Available MBytes Windows OS layer CPU consumption

Lazy Writes / sec Memory flushed out to disk

Memory Grants Pending Processes waiting for memory

Page Faults / sec Thought it was in RAM, but it was not

Page Reads / sec Memory successfully used to cache blocks of data

Page Writes / sec Memory successfully written to

Pages / sec Memory paged out of RAM to disk

SQL Server: Buffer Manager

Page Life Expectancy Average of lifetime of pages in buffer pool

Buffer Cache Hit Ratio % of pages found in BP without going to disk

Paging File % Usage When available RAM not enough

15

16

SQLBits 2019

Copyright Heraflux Technologies. Do not redistribute or copy as your own. 9

© Heraflux Technologies®  17

Counter Set Counter Explanation

Network Interface Bytes Received / sec Inbound data traffic rates (convert to MB/s)

Bytes Sent / sec Outbound data traffic rates (convert to MB/s)

Packets Outbound Discarded Thrown away blocks even though no errors occurred

Packets Outbound Errors Failed transmits

Packets Received Discarded Thrown away blocks even though no errors occurred

Packets Received Errors Failed transmits

© Heraflux Technologies®  18

Counter Set Explanation

PhysicalDisk % Idle Time Indication of load

Avg. Disk Bytes / Read Workload patterns

Avg. Disk Bytes / Write

Avg. Disk sec / Read Disk Latency (convert to milliseconds)

Avg. Disk sec / Write

Disk Read Bytes / sec Throughput (convert to MB/s)

Disk Write Bytes / sec

Disk Reads / sec IOs per second (IOPS)

Disk Writes / sec

17

18

SQLBits 2019

Copyright Heraflux Technologies. Do not redistribute or copy as your own. 10

© Heraflux Technologies®  19

Counter Set Counter Explanation

SQL Server:Buffer Manager Buffer Cache Hit Ratio % in cache without having to go to disk

Page Life Expectancy Duration (in sec) data held in buffer pool

SQL Server:SQL Statistics Batch Requests / sec # of statements executed per second

SQL Compilations / sec Statement compilations

SQL Re‐Compilations / sec Statement re‐compilations

SQL Server:Locks Lock Waits / sec Locks that had to wait

SQL Server:General Statistics Processes Block # of blocked processes

SQL Server:Database Application Database transactions / sec Transactions per second

PowerShell Loader for Perfmon Files

github.com/heraflux/BLGtoSQL

© Heraflux Technologies®  20

19

20

SQLBits 2019

Copyright Heraflux Technologies. Do not redistribute or copy as your own. 11

© Heraflux Technologies®  21

0

10

20

30

40

50

60

00:00

00:20

00:40

01:00

01:20

01:40

02:00

02:20

02:40

03:00

03:20

03:40

04:00

04:20

04:40

05:00

05:20

05:40

06:00

06:20

06:40

07:00

07:20

07:40

08:00

08:20

08:40

09:00

09:20

09:40

10:00

10:20

10:40

11:00

11:20

11:40

12:00

12:20

12:40

13:00

13:20

13:40

14:00

14:20

14:40

15:00

15:20

15:40

16:00

16:20

16:40

17:00

17:20

17:40

18:00

18:20

18:40

19:00

19:20

19:40

20:00

20:20

20:40

21:00

21:20

21:40

22:00

22:20

22:40

23:00

23:20

23:40

% C

PU

Co

nsu

mp

tion

Time of Day (Avg)

CPU by Core ‐ Five Minute Avg. (8 Core)

CPU00 CPU01 CPU02 CPU03 CPU04 CPU05 CPU06 CPU07

© Heraflux Technologies®  22

21

22

SQLBits 2019

Copyright Heraflux Technologies. Do not redistribute or copy as your own. 12

© Heraflux Technologies®  23

0

50

100

150

200

250

300

350

00:00

00:20

00:40

01:00

01:20

01:40

02:00

02:20

02:40

03:00

03:20

03:40

04:00

04:20

04:40

05:00

05:20

05:40

06:00

06:20

06:40

07:00

07:20

07:40

08:00

08:20

08:40

09:00

09:20

09:40

10:00

10:20

10:40

11:00

11:20

11:40

12:00

12:20

12:40

13:00

13:20

13:40

14:00

14:20

14:40

15:00

15:20

15:40

16:00

16:20

16:40

17:00

17:20

17:40

18:00

18:20

18:40

19:00

19:20

19:40

20:00

20:20

20:40

21:00

21:20

21:40

22:00

22:20

22:40

23:00

23:20

23:40

MB

pe

r se

c

Hour of Day (Avg)

Physical Disk Read MBps

MP0c MP10 MP11 MP12 MP13 MP14 MP15 MP16 MP17 MP18w MP19v MP1o MP20

MP21 MP22 MP23d MP24 MP25 MP2e MP3f MP4g MP5h MP6i MP7j MP8 MP9

© Heraflux Technologies®  24

0

500

1000

1500

2000

2500

3000

3500

00:00

00:20

00:40

01:00

01:20

01:40

02:00

02:20

02:40

03:00

03:20

03:40

04:00

04:20

04:40

05:00

05:20

05:40

06:00

06:20

06:40

07:00

07:20

07:40

08:00

08:20

08:40

09:00

09:20

09:40

10:00

10:20

10:40

11:00

11:20

11:40

12:00

12:20

12:40

13:00

13:20

13:40

14:00

14:20

14:40

15:00

15:20

15:40

16:00

16:20

16:40

17:00

17:20

17:40

18:00

18:20

18:40

19:00

19:20

19:40

20:00

20:20

20:40

21:00

21:20

21:40

22:00

22:20

22:40

23:00

23:20

23:40

IOP

s

Hour of Day (Avg)

Physical Disk Read IOPs

MP0c MP10 MP11 MP12 MP13 MP14 MP15 MP16 MP17 MP18w MP19v MP1o MP20

MP21 MP22 MP23d MP24 MP25 MP2e MP3f MP4g MP5h MP6i MP7j MP8 MP9

23

24

SQLBits 2019

Copyright Heraflux Technologies. Do not redistribute or copy as your own. 13

© Heraflux Technologies®  25

0

2

4

6

8

10

12

00:00

00:20

00:40

01:00

01:20

01:40

02:00

02:20

02:40

03:00

03:20

03:40

04:00

04:20

04:40

05:00

05:20

05:40

06:00

06:20

06:40

07:00

07:20

07:40

08:00

08:20

08:40

09:00

09:20

09:40

10:00

10:20

10:40

11:00

11:20

11:40

12:00

12:20

12:40

13:00

13:20

13:40

14:00

14:20

14:40

15:00

15:20

15:40

16:00

16:20

16:40

17:00

17:20

17:40

18:00

18:20

18:40

19:00

19:20

19:40

20:00

20:20

20:40

21:00

21:20

21:40

22:00

22:20

22:40

23:00

23:20

23:40

Mill

ise

con

ds

La

ten

cy

Hour of Day (Avg)

Physical Disk Read Latency

MP0c MP10 MP11 MP12 MP13 MP14 MP15 MP16 MP17 MP18w MP19v MP1o MP20

MP21 MP22 MP23d MP24 MP25 MP2e MP3f MP4g MP5h MP6i MP7j MP8 MP9

© Heraflux Technologies®  26

0

5000

10000

15000

20000

25000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Page Life Expectancy

Hour of Day

Page Life Expectancy

NUMA Node 1 NUMA Node 2

25

26

SQLBits 2019

Copyright Heraflux Technologies. Do not redistribute or copy as your own. 14

© Heraflux Technologies®  27

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

Va

lue

Time of Day (Avg)

Page Life Expectancy

ValMin Val25th ValAvg Val50th Val95th Val96th Val97th Val98th Val99th Val99_9th ValMax

© Heraflux Technologies®  28

27

28

SQLBits 2019

Copyright Heraflux Technologies. Do not redistribute or copy as your own. 15

© Heraflux Technologies®  29

0

50

100

150

200

250

300

350

400

450

500

0:00

0:20

0:40

1:00

1:20

1:40

2:00

2:20

2:40

3:00

3:20

3:40

4:00

4:20

4:40

5:00

5:20

5:40

6:00

6:20

6:40

7:00

7:20

7:40

8:00

8:20

8:40

9:00

9:20

9:40

10:00

10:20

10:40

11:00

11:20

11:40

12:00

12:20

12:40

13:00

13:20

13:40

14:00

14:20

14:40

15:00

15:20

15:40

16:00

16:20

16:40

17:00

17:20

17:40

18:00

18:20

18:40

19:00

19:20

19:40

20:00

20:20

20:40

21:00

21:20

21:40

22:00

22:20

22:40

23:00

23:20

23:40

Percentage

 CPU Consumed (2700 actual scale)

Aggregate CPU Consumption ‐ 27 Servers

On‐going sampling

Periodic analysis

Short‐term performance triage

Long‐term capacity trends

© Heraflux Technologies®  30

29

30

SQLBits 2019

Copyright Heraflux Technologies. Do not redistribute or copy as your own. 16

© Heraflux Technologies®  31

@kleegeekdavidklee.netheraflux.comdavidaklee

31

32