© 2017 ISC

ISC's Performance Lab

DNS-OARC Workshop 26 - Madrid

1

© 2017 ISC

Goals of the Project

▪ Provide a framework for automated performance testing of BIND– Test experimental code– Look at long-term trends to avoid

regressions

2

© 2017 ISC

System Features #1

▪Multiple configurations– git branch or tag– ./configure settings– named.conf options– named command line options– etc

3

© 2017 ISC

System Features #2

▪Multiple authoritative zone configurations– root zone– 1M delegations (small TLD) – 1M small zones (web hoster)– 1 zone with 1M A+AAAA

▪ dnsperf query sets for the above▪ (Some) Recursive support4

© 2017 ISC

System Features #3

▪ HTML5 Web UI– real time updates over WebSockets

▪ Round-robin scheduler–with ad-hoc priority bump

5

© 2017 ISC

System Features #4

▪ Graphs▪ CSV Output▪ A/B comparison with Student's t-test▪ Extensible for other servers▪ NSD 4, Knot 2 already supported▪ Dummy 'echo' server support, too

6

© 2017 ISC

Technology

▪MongoDB ▪ NodeJS (ES6)–Promises– Express

▪ HTML5–Bootstrap–Angular JS (ES5)–WebSockets

7

© 2017 ISC

QUICK DEMO…

© 2017 ISC

Logical Architecture

DNS Server

Web Client

dnsperfREST API

Server

Test Agent

MongoDB

9

© 2017 ISC

Physical Architecture

Server under Test

Traffic Generator Backend

10 GE

Gig-E Management LAN

Apache Reverse Proxy

10

© 2017 ISC

Test Methodology

▪ Build and start server▪ Run dnsperf 30 times▪ 30 seconds each time▪ first run ignored for statistic (allows

for cache and buffer warmup)

11

© 2017 ISC

Challenge - Test Variability

▪ The results from 30x dnsperf tests against a single run of BIND are generally consistent

▪ Testing a new run of BIND often shifts the mean significantly, by several standard deviations

12

© 2017 ISC

Challenge - Test Variability

13

Master Branch Test Variance Detail

610

615

620

625

630

635

640

kQPS

© 2017 ISC

(Some) Mitigation

▪ Tune for stability, not peak performance:–Disable Hyperthreading– Lock CPU clock rate•Disable Intel SpeedStep•Disable Turbo mode

– Lock Client and Server CPU core affinity– Lock NIC RX/TX queue CPU core affinity– Set NIC queue flow hashing to use

deterministic ports14

© 2017 ISC

Challenge - Test Variability #2

15

Dual Core Variance Detail

660

680

700

720

740

760

780

800

kQPS

●

●

●

●●

●●

●

●

●●●

●

●

●●

●

●

●●●

●

●●●

●●

●

●●

●

●

●●

●

●●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●●

●

●

●●

●●

●●

●●

●

●

●

●●

●

●

●

●

●

●

●

●●

●●

●

●●

●●

●●

●●

●

●

●●

●●

●

●●

●

●

●

●●●

●

●

●

●

●

●●●

●

●

●

●●●

●

●●

●

●

●

●

●

●

●

●

●

●●●●

●

●

●

●

●

●

●●

●●

●

●●

●●

●

●●

●

●

●

●●●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●●●●

●●

●●

●

●

●

●

●●

●

●

●

●●●

●

●●●●●

echo server two threads locked to CPU#0 and #1

© 2017 ISC

Challenge - Test Variability #2

16

Dual Core Variance Detail

660

680

700

720

740

760

780

800

kQPS

●

●

●

●●

●●

●

●

●●●

●

●

●●

●

●

●●●

●

●●●

●●

●

●●

●

●

●●

●

●●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●●

●

●

●●

●●

●●

●●

●

●

●

●●

●

●

●

●

●

●

●

●●

●●

●

●●

●●

●●

●●

●

●

●●

●●

●

●●

●

●

●

●●●

●

●

●

●

●

●●●

●

●

●

●●●

●

●●

●

●

●

●

●

●

●

●

●

●●●●

●

●

●

●

●

●

●●

●●

●

●●

●●

●

●●

●

●

●

●●●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●●●●

●●

●●

●

●

●

●

●●

●

●

●

●●●

●

●●●●●

echo server two threads locked to CPU#0 and #1

Huh?!

© 2017 ISC

Coming Soon

▪ In 9.11.2 and back-ported to 9.10– optimised name compression– optimised owner case preservation

▪ In 9.12– "minimal-responses" on by default– "glue cache" replaces "acache"• improves delegation performance

17

© 2017 ISC

2012 2013 2014 2015 2016 2017 2018

0

200

400

600

800

BIND Performance − 1M Delegations

12−core Intel Xeon E5−2680 v3 @ 2.50GHz

Release Date

kQPS

9.9 9.10 9.11 dev

●●

● ● ● ●

● ● ●

● ●

●

● ●●

●

●

● ●●

●

18

© 2017 ISC

2012 2013 2014 2015 2016 2017 2018

0

200

400

600

800

BIND Performance − 1M Delegations

12−core Intel Xeon E5−2680 v3 @ 2.50GHz

Release Date

kQPS

9.9 9.10 9.11 dev

●●

● ● ● ●

● ● ●

● ●

●

● ●●

●

●

● ●●

●

19

Change to default "-U" setting

© 2017 ISC20

2012 2013 2014 2015 2016 2017 2018

0

100

200

300

400

BIND Performance − Root Zone

12−core Intel Xeon E5−2680 v3 @ 2.50GHz

Release Date

kQPS

● ● ●●

● ●

● ● ●

● ●

● ● ● ●

● ●● ●

9.9 9.10 9.11

© 2017 ISC

2012 2013 2014 2015 2016 2017 2018

0

100

200

300

400

BIND Performance − Root Zone

12−core Intel Xeon E5−2680 v3 @ 2.50GHz

Release Date

kQPS

9.9 9.10 9.11 dev

● ● ●●

● ●

● ● ●

● ●

● ● ● ●

● ●● ●●

●

21

6x faster!

© 2017 ISC

How Do I Get It?

https://github.com/isc-projects/perflab▪ This is not supported software▪ It's not turn-key - installation is likely

to require lots of fiddling▪ Pull requests welcomed…

22

© 2017 ISC

QUESTIONS?