201403 Tampa Performance3

8/10/2019 201403 Tampa Performance3

1/99

RED HAT CONFIDENTIAL | JEREMY EDER1

Performance Tuning and

Anal!i! of Red Ha"

En"er#ri!e Linu$ % and &

Jeremy Eder

Principal Software EngineerMarch 13, 2014


2/99


Agenda' Performance Anal!i! of RHEL%(&

Performance Engineering O)er)ie* Performance Anal!i! +"ili"ie!

Tuned

N+MA Tuning

Po*er Managemen"


3/99


Performance EngineeringO)er)ie*

Micro,-enc.mar/!A##lica"ion!(-enc.mar/!


4/99RED HAT CONFIDENTIAL | JEREMY EDER4

Red Ha" Confiden"ial

Red Ha" Performance Engineering

Benchmark ! code path co"erage #P$ ! linpack, lm%ench

Memory ! lm%ench, Mc#alpin Stream

&ik '( ! 'o)one, aiotre ! ci, *#, iS#S'

*ileytem ! '()one, potmark! e+t34, +f- gf2,gl.ter

/etwork ! /etperf ! 10 %it, 40 %it 'B, P#'3

Bare Metal, E5 67M

8hite %o+ 9M&'ntel, with o.r (EM partner


5/99RED HAT CONFIDENTIAL | JEREMY EDER:

Performance Pro0ec"! ( Tooling

E-: ;n.mad< ;t.na


6/99

RED HAT CONFIDENTIAL | JEREMY EDER

Performance O#"imi1a"ion

Ou",of,".e,2o$


7/99


Performance Tuning Au"oma"ion

9.tomatic =.ning

=.ned

=ranparent .gepage

n.mad

irF%alance

Man.al =.ning

/9

Static .gepage

/$M9 Pinning n.mactlC

'> Pinning t.naC

E5

n.maG%alancing


8/99

RED HAT CONFIDENTIAL | JEREMY EDERH

-u"333*.a" if *e .a)e a #ro2lem 4

9.tomatic not eno.gh---


9/99


-u"333*.a" if *e .a)e a #ro2lem 4

9.tomatic not eno.gh--- /eed to eek o.t the lat I percent


10/99


-u"333*.a" if *e .a)e a #ro2lem 4


/eed &eterminim


11/99


-u"333*.a" if *e .a)e a #ro2lem 4


/eed &eterminim

(6---here? where it get? f.n


12/99


O)er)ie* of PerformanceAnal!i! +"ili"ie!

#erf

f


13/99


#erf

$erpace tool to read #P$

co.nter and kernel tracepoint

f li


14/99

RED HAT CONFIDENTIAL | JEREMY EDER1:

#erf li!"

it co.ntertracepoint a"aila%le

on yo.r ytem

f li "


15/99


#erf li!"

grep for omething intereting,

may%e to ee what n.ma%alance idoing K

---more on n.ma%alance later---

f "


16/99


#erf "o#

Sytem@wide ?top? "iew of %.y

f.nction


17/99

#erf record


18/99

RED HAT CONFIDENTIAL | JEREMY EDER1H

#erf record

ecord ytem@wide @aC

9 ingle command

#erf record


19/99


#erf record

ecord ytem@wide @aC

9 ingle command9n e+iting proce @pC

#erf record


20/99


#erf record

ecord ytem@wide @aC


9dd call@chain recording @gC

#erf record


21/99


#erf record

ecord ytem@wide @aC


9dd call@chain recording @gC(nly pecific e"ent @eC

#erf record


22/99


#erf record

ecord ytem@wide @aC


9dd call@chain recording @gC(nly pecific e"ent @eC

#erf re#or"


23/99


#erf re#or"

de")ero

#erf re#or"


24/99


#erf re#or"

de")ero

oflagLdirect

#erf diff


25/99


#erf diff

#ompare 2 perf recording

f 2 5d i i 6


26/99


#erf #ro2e 5dnamic "race#oin"!6

'nert a tracepoint on any f.nction---

=ry ?perf pro%e @*? to lit poi%ilitie

Cool7 *.a" el!e a go" 4


27/99


Cool7 *.a" el!e a go" 4


28/99


O)er)ie* of PerformanceAnal!i! +"ili"ie!

Performance Co,Pilo" 5PCP6


29/99

#mc.ar" 8 gra#.ical me"ric #lo""ing "ool


30/99


#mc.ar" gra#.ical me"ric #lo""ing "ool

#an plot myriad performance tatitic



31/99




ecording mode allow for replay

i-e- on a different ytem

ecord in $', then

# pmafm $recording.folio



32/99




ecording mode allow for replay

i-e- on a different ytem

ecord in $', then

# pmafm $recording.folio

Ship with many pre@cooked ;"iew


33/99


9.a" do ".e!e )ie*! loo/ li/e 4

Performance Co,Pilo" Demo :cri#"


34/99


Performance Co Pilo" Demo :cri#"

# CPU

/root/pig -s 5

# DISKdd if=/dev/zero of=/root/2GB count=2!" s=$% of&'g=direct

# ()*+,K

netperf -. &' -& 5

# %)%,0

/root/pig -1 $3"! -& s&eep -s 5

=iny cript to e+ercie 4 food gro.p---


35/99


#P$

oad 9"g

'(PS

/etwork

Memory9llocated

collec"l mode


36/99


#P$


37/99

collec"l mode


38/99


'(PS

/E=

#P$

collec"l mode


39/99


'(PS

MEM

/E=

#P$

a"o# mode


40/99


a"o# mode

;ue!"ion! !o far 4


41/99


;ue!"ion! !o far 4


42/99


N+MA Tuning

Di!co)er


43/99


# g #

ow can ' "i.ali)e my ytem? /$M9topology in ed at Enterprie in.+K

httpDacce-redhat-comiteol.tion25H

/$M9 /ode 0


44/99




/$M9 /ode 0 /$M9 /ode 1


45/99




P#' &e"ice

/$M9 /ode 0 /$M9 /ode 1


46/99


# lscpu

Architecture: x86_64...CPUs!: "6nline CPUs! list: %"&

'hreads! per core: "Cores! per soc(et: 8)oc(ets!: *+U,A nodes!: *...

+U,A node% CPUs!: %-+U,A node" CPUs!: 8"&

ogical #ore=


47/99


# lscpu

Architecture: x86_64...CPUs!: "6nline CPUs! list: %"&

'hreads! per core: "Cores! per soc(et: 8)oc(ets!: *+U,A nodes!: *...

+U,A node% CPUs!: %-+U,A node" CPUs!: 8"&

ogical #ore=

/$M9 /ode#ore Map

N+MA To#olog and PCI -u!


48/99


'ntall adapter ;cloe< to the #P$ that will r.n theperformance critical application-

8hen B'(S report locality, irF%alance handle/$M9'> affinity a.tomatically-

# lstoponographics egrep /+U,Aeth4/ +U,A+ode 0#% P#% "4412!

+U,A+ode 0#" P#" "4412!

+et 0#"% 3eth43

RHEL N+MA :c.eduler


49/99

RED HAT CONFIDENTIAL | JEREMY EDER:0

E

n.mactl, n.matat enhancement

n.mad ! .ermode tool, dynamically monitor, [email protected]

E5 ! n.ma%alance Ena%le &ia%le

yctl kernel-n.ma%alancingLN0,1O

Ho* "o manage N+MA manuall , C.ec/li!"


50/99


#hecklit

eearch =opology

Make a reo.rce plan

#onider '(

7irt.ali)ation

=ool

ltopolcp.

cgro.p, n.mactl

irF%alanceP#' B.

n.mat.nen.mad



51/99


#hecklit

eearch =opology

Make a reo.rce plan

#onider '(

7irt.ali)ation

=ool

ltopolcp.

cgro.p, n.mactl

irF%alanceP#' B.

n.mat.nen.mad



52/99


#hecklit

eearch =opology

Make a reo.rce plan

#onider '(

7irt.ali)ation

=ool

ltopolcp.

cgro.p, n.mactl

irF%alanceP#' B.

n.mat.nen.mad


53/99



54/99

RED HAT CONFIDENTIAL | JEREMY EDER::

#hecklit

eearch =opology

Make a reo.rce plan

#onider '(

7irt.ali)ation

=ool

ltopolcp.

cgro.p, n.mactl

irF%alanceP#' B.

n.mat.nen.mad

N+MA Tool!' numa!"a"


55/99

RED HAT CONFIDENTIAL | JEREMY EDER:

#ompletely rewritten for E-4

Per@node procmeminfoBackward compati%le

N+MA Tool!' numa!"a"


56/99


#ompletely rewritten for E-4

Per@node procmeminfoBackward compati%le

numa!"a"' com#a"i2ili" mode 5old6


57/99


# numastat

node% node"

numa_hit --&8--5 ""55%%4*numa_miss % %numa_foreign % %

interleae_hit %*&4 %%55local_node 65%*-"% "*5&""6%other_node 8*8&%*5 *4-868*

numa!"a"' #er,node meminfo 5ne*6# numastat -mczs

d 0


58/99

RED HAT CONFIDENTIAL | JEREMY EDER:H

Node 0------

MemTotal 65491MemFree 60366MemUsed 5124Actie 2650File!a"es 2021

Actie$ile% 16&6Actieanon% 964Anon!a"es 964'nactie 341'nactie$ile% 340

(la) 3&0(*eclaima)le 20&(Unreclaim 1+3Anon,u"e!a"es 134

numa!"a"' #er,node meminfo 5ne*6# numastat -mczs

N d 0 N d 1 T t l


59/99


Node 0 Node 1 Total ------ ------ ------

MemTotal 65491 65536 13102+MemFree 60366 59+33 120099MemUsed 5124 5&03 1092+Actie 2650 2&2+ 54++File!a"es 2021 3216 523&

Actie$ile% 16&6 22++ 3963Actieanon% 964 551 1515Anon!a"es 964 550 1514'nactie 341 946 12&+'nactie$ile% 340 946 12&6

(la) 3&0 43& &1&(*eclaima)le 20& 20+ 415(Unreclaim 1+3 230 403Anon,u"e!a"es 134 236 3+0


60/99


N+MA Tuning

numad

N+MA' Proce!! :c.eduler -e.a)ior

S h d l di t i% t l d l ll


61/99


Sched.ler ditri%.te load e"enly acro all core

Maintain reponi"ene

(ptimi)ing for #P$ .tili)ation

=rie to .e idle #P$, regardle of where procememory i located

B$=

$ing remote memory degrade

performance



62/99


Short =erm E-4C

$erpace ol.tion

n.mad

ong =erm E5C

ed at ha taken 2 approache for c.tomerD



63/99


Short =erm E-4C

$erpace ol.tion

n.mad

ong =erm E5C

'n@6ernel Sol.tion

n.ma%alance

ed at ha taken 2 approache for c.tomerD

9.a" doe! i" loo/ li/e 4


64/99


Effec" of Au"oma"ic N+MA -alancing :of"*are


65/99


# numastat c pig default scheduler 7 nonoptimal!

P9 +ode % +ode "*&-8 pig! *"* ""8-8*&-5 pig! "588 "*%"*&8% pig! "4%%% "*&8" pig! "58" "*%*%

Before n.mad

Effec" of Au"oma"ic N+MA -alancing :of"*are


66/99


# numastat c pig default scheduler 7 nonoptimal!

P9 +ode % +ode "*&-8 pig! *"* ""8-8*&-5 pig! "588 "*%"*&8% pig! "4%%% "*&8" pig! "58" "*%*%

P9 +ode % +ode "*&-8 pig! "4%%% %*&-5 pig! % "4%%%*&8% pig! "4%%% %

*&8" pig! % "4%%%

Before n.mad

9fter n.mad

Effec" of numad(numa2alance

9.tomatic /$M9 Balancing @ /$M9&


67/99


1 2 3 4 : 5 H 10 11 12 13 14 1: 1

0

2000

4000

000

000

10000

12000

14000

/(&E@0@MB /(&E@1@MB

=ime econdC

Mega%ytePer/$M9

/ode

n.mad %egin

n.mad done

;ue!"ion! on N+MA 4


68/99



69/99


"uned

9.a" i! ="uned> 4


70/99


=.ning profile deli"ery mechanim

ed at hip tuned profilesthatimpro"e performance for manyworkload---hopef.lly yo.r

Ye!7 2u" *. do I care 4


71/99


Tuned' :"orage Performance -oo!"


72/99


arger i %etter

Tuned' Ne"*or/ La"enc Performance -oo!"


73/99


0

:0

100

1:0

200

2:0

Ma+

# #3 #1 #0

3atencyBMicroecondC

#@tate lock impro"e determinim, red.ce itter

=ime 1@ec inter"alC

9.a" do ".e gu"! loo/ li/e 4


74/99



75/99

9.a"?! .a##ening "o Tuned for RHEL& 4


76/99


Tuned' +#da"e! for RHEL&


77/99


'ntalled %y defa.lt



78/99


'ntalled %y defa.lt

Profile a.tomatically et %aed on intall typeD

&ektop8orktationD %alanced

Ser"erP#D thro.ghp.t@performance


i f i i %ili d i%ili


79/99


e@written for maintaina%ility and e+teni%ility-


itt f i t i %ilit d t i%ilit


80/99



#onfig.ration conolidated to inglet.ned-conf file




81/99



#onfig.ration i now conolidated a inglet.ned-conf file

(ptional hookcallo.t capa%ility




82/99





9dd concept of 'nheritance .t likehttpd-confC


e written for maintaina%ility and e+teni%ility


83/99





9dd concept of 'nheritance .t likehttpd-confC

Profile .pdated for E5 feat.re andcharacteritic

Tuned' Profile In.eri"ance


84/99


#hildren

Parent

[email protected]@performance

network@[email protected]

"irt.al@hot

"[email protected]

%alanced

dektop

Tuned' Profile In.eri"ance


85/99


#hildren

Parent

[email protected]@performance

network@[email protected]

"irt.al@hot

"[email protected]

%alanced

dektop

Qo.r@&BQo.r@8e% Qo.r@Middleware

9.a" do ".e gu"! loo/ li/e 4


86/99


Tuna2le +ni"! -alanced ".roug.#u",#erformance ne"*or/,".roug.#u"

'nherit *rom/ote thro.ghp.t@performance

Tuned' T.roug.#u" Profile! , RHEL&


87/99


chedGminG gran.larityGn nanoecond a.to@caling 10000000

chedGwake.pGgran.larityGn nanoecond 3000000 1:000000

dirtyGratio Percent 20 40

dirtyG%ackgro.ndGratio Percent 10 10

wappine 8eight 1@100 0 10

'( Sched.ler Ele"atorC deadline

*ileytem Barrier Boolean Ena%led

#P$ o"ernor ondemand performance

&ik ead@ahead 6B 12 40H

&ia%le =P Boolean Ena%led

Energy Perf Bia normal performance

kernel-chedGmigrationGcotGn nanoecond :00000

minGperfGpct intelGptate onlyC Percent a.to@caling 100

tcpGrmem Byte a.to@caling Ma+L155521

tcpGwmem Byte a.to@caling Ma+L155521

.dpGmem Page a.to@caling Ma+L155521

Tunable +ni"! -alanced la"enc,#erformance ne"*or/,la"enc

'nherit *rom/ote la"enc,#erformance

chedGminG gran.larityGn nanoecond a.to@caling 10000000

Tuned' La"enc Profile! , RHEL&


88/99


chedGwake.pGgran.larityGn nanoecond 3000000 10000000

dirtyGratio percent 20 10

dirtyG%ackgro.ndGratio percent 10 3

wappine 8eight 1@100 0 10

'( Sched.ler Ele"atorC deadline

*ileytem Barrier Boolean Ena%led

#P$ o"ernor ondemand performance

&ia%le =P Boolean /9 No Qe

#P$ #@State /9 ocked R 1

Energy Perf Bia normal performance

kernel-chedGmigrationGcotGn nanoecond /9 :000000

minGperfGpct intelGptate onlyC percent 100

net-core-%.yGread microecond :0

net-core-%.yGpoll microecond :0

net-ip"4-tcpGfatopen Boolean Ena%led

kernel-n.maG%alancing Boolean &ia%led

Tunable +ni"! ".roug.#u",#erformance )ir"ual,.o!" )ir"ual,gue!"

'nherit *rom/ote thro.ghp.t@performance

thro.ghp.t@performance

Tuned'


89/99

RED HAT CONFIDENTIAL | JEREMY EDERH0

p p

chedGminG gran.larityGn nanoecond 10000000

chedGwake.pGgran.larityGn nanoecond 1:000000

dirtyGratio percent 40 30

dirtyG%ackgro.ndGratio percent 10 : 30

wappine 8eight 1@100 10

'( Sched.ler Ele"atorC

*ileytem Barrier Boolean

#P$ o"ernor performance

&ik ead@ahead Byte 40H

Energy Perf Bia performance

kernel-chedGmigrationGcotGn nanoecond :000000

minGperfGpct intelGptate onlyC percent 100

RHEL ="uned> #ac/age

Aailale profiles:


90/99


p alanced

des(top latenc;performance m;profile net


91/99


Po*er Managemen"

CP+ Tuning' C,!"a"e! 5idle !"a"e!6

P S i P f


92/99


0

:0

100

1:0

200

2:0

Ma+

# #3 #1 #0

#@tate 'mpact on Jitter

3atencyBMicro

econdC

=ime 1@ec inter"alC

Power Sa"ing Performant


93/99

Tur2o!"a" !.o*! P(C,!"a"e! on In"el CP+!

t.r%otat in E-4, cp.power.til package or kernel@tool in *edoraE5


94/99

RED HAT CONFIDENTIAL | JEREMY EDERH:

Def'u&t

p4 cor CPU c G.z *SC c$ c3 c c

62! 2673 26"" 562 $632 6 7262

$ $ 265! 363 26"" 36$3 6$5 6 7!6$"

2 2 2627 36" 26"" $6! 6 6 7625

3 3 $65 $65 26"" $62$ 6! 6$2 76!!

&'tenc8-perfor1'nce

p4 cor CPU c G.z *SC c$ c3 c c

6 363 267 $6 6 6 6

$ $ 6 363 267 $6 6 6 6

2 2 6 363 267 $6 6 6 6

3 3 6 363 267 $6 6 6 6

Profiling c#uidle and c#ufreB

*i+ed .ptream cp.idle regreion in J.ne


95/99


*i+ed .ptream cp.idle regreion in J.ne

*.t.re #P$ .ch a awell add more #@tate,#,#H,#10

=.r%otat diplay a %it awkward for 40U core

&irection in f.t.re i to hrink the e+it@latency %etween#@tate

httpDwww-%reakage-org201211proceor@ma+Gctate@intelGidle@ma+Gctate@and@de"cp.GdmaGlatency


96/99


Ta/e,a*a!

Hel#ful +"ili"ie!

S.pporta%ility

/$M9

hwloc

/etworking

dropwatch
http://www.breakage.org/2012/11/processor-max_cstate-intel_idle-max_cstate-and-devcpu_dma_latency/http://www.breakage.org/2012/11/processor-max_cstate-intel_idle-max_cstate-and-devcpu_dma_latency/


97/99


[email protected]@tool

o

kd.mp

perf

pmic trace

ytat

ytemtap

trace@cmd

.til@lin.+@ng

'ntel P#M n.mactl

n.mad

n.matop 01-orgC

Power=.ning

cp.power.til C

kernel@tool 5C

powertop

t.na

t.ned

p

ethtool netniff@ng EPEC

tcpd.mp

wireharkthark

Storage

%lktrace

iotop

iotat

Hel#ful Lin/!

ed at ow atency Performance =.ning .ide


98/99

RED HAT CONFIDENTIAL | JEREMY EDERHH

y g

(ptimi)ing E Performance %y =.ning '> 9ffinity

ed at Performance =.ning .ide

ed at 7irt.ali)ation =.ning .ide

S=9# /etwork '( S' *inteligent ow atency =.ning w67M

Perf

BlogD httpDwww-%reakage-orgor Reremyeder
https://access.redhat.com/knowledge/articles/221153


99/99


;ue!"ion!
http://www.breakage.org/https://perf.wiki.kernel.org/http://finteligent.net/pg/file/onxenterprise/read/51036http://www.stacresearch.com/niohttps://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/Virtualization_Tuning_and_Optimization_Guide/index.htmlhttps://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/Performance_Tuning_Guide/index.htmlhttps://access.redhat.com/knowledge/articles/216733https://access.redhat.com/knowledge/articles/221153

201403 Tampa Performance3

Documents

Transcript of 201403 Tampa Performance3