Operating System Noise: Linux vs....

Operating System Noise:Linux vs. Microkernel

Stefan Wächtler1, Michal Sojka1,2

1Technische Universität DresdenOperating-Systems Group

2Czech Technical University in PragueFaculty of Electrical Engineering

14th Real-Time Linux WorkshopOctober 18, Chapel Hill, NC, USA

S. Wächtler, M. Sojka OS Noise: Linux vs. Microkernel RTLWS14 1 / 16

Introduction

I Computers running Linux are often used as a tool tomeasure time in various experiments.

I Benchmarking etc.

How does the operating system influence the results ofbenchmarks?

Expectation

I Microkernels should give results with less variance.I Less interference from the OS.


This work

I Statistical comparison of benchmark results.I The same set benchmarks run on different OSes:

I Linux, Debian kernelI Linux, minimal kernelI NOVA microkernel


NOVA microkernel

I Microkernel-based hypervisor.I Extremely small (9 kLoC) and fast.I Almost nothing runs behind application’s back.


OS Noise in High Performance Computing

I Timer interrupts etc.I Scalability problem


Benchmarks

Benchmarks

I Measure time of certainoperations

I In total 46 experiments:I cache/memory bandwidth

(8 experiments),I cache-related preemption

delay (32 experiments),I cache-related migration

delay (6 experiments)

Method

I Treat benchmarks as blackboxes

I Look at the variance ofresults

Testbed

I Dell Precision T7500, 2×Intel Xeon X5650 (6 coreseach), NUMA architecture


First Benchmark – Memory Access Times

●

● ● ●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●

2

51020

50100200

500

2 8 32 128 512 2048 8192 32768WSS [KiB]

cycl

es p

er a

cces

s

Kernel ●● ● ●NOVA Linux minconfig Linux Debian


Observations

1. Steps on the NOVA curve are much sharper.2. The differences between the mean values for the two Linux

kernels is very small.3. The differences at the left end of the graph is caused by

inaccuracy of our measurement method.4. There are big differences in the observed noise (min/max

values).5. Peaks on the NOVA curve: perhaps caused SMIs.


Second Benchmark – Memory Access TimesModified cache lines

●

● ●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●

●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

50

100

200

500

1,000

2 8 32 128 512 2048 819232768WSS [KiB]

cycl

es p

er a

cces

s



How to compare “the noise”?

Several options:I Standard deviation σI Variance σ2

I Coefficient of variation cv = σ/µ

I Variance to mean ratio (index of dispersion) D = σ2/µ


Variance of the Second Benchmark

I Benchmark executed 1024 times

●●

●●●●●●●●●●●●

●●●●●●●●

●●

●●●●●●●●

●

●●

●

●●

●

●●●●●●●●●●

●

●●

●

●●●

●

●

●

●●

●●

●●

●

●

●●●

●●

●●●●●●●●●

●●●●●●●●●●

●

●●●●●●●●●

●●●●●

100,000

1,000,000

10,000,000

100,000,000

1,000,000,000

10,000,000,000

100,000,000,000

2 8 32 128 5122048819232768WSS [KiB]

varia

nce

[cyc

les2 ]



Variance to mean ratio

●

●

●●●

●●●●●●●

●●●●●●

●

●●●

●●

●●●●●●

●

●

●

●●

●

●●

●

●●●●●●●●●●

●

●●

●

●●●

●

●

●

●●

●●

●●

●

●

●●●

●●

●●

●

●●●

●●●●●●

●●●●

●

●

●

●

●●●●

●

●●●●

●●●●●

2

51020

50100200

5001,000

2 8 32 128 512 2048 819232768WSS [KiB]

varia

nce/

mea

n [c

ycle

s]



First benchmark – Aggregated results

I Split to 4 WSS areas according to cache sizesI Median calculation

●

● ● ●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●●●●●●●●●

1020

50100200

50010002000

5000

2 8 32 128 512 2048 8192 32768WSS [KiB]

Wei

ghte

d m

ean

+ s

tdde

v[c

ycle

s/K

iB]


●●

● ●●

●●●

●●●●●●●●●●●

●●●

●●

●●

●

●●●

●

●

●●●●

●

●

●●●

●

●

●●●

●●●

●●●●●●

●

●●●●

●

●

●●●

●

●

●

●

●

●

●

●●

●

●●●●●●●●●●●●●●

●

●●

●●●

●●

●

●

●●●●●●●

1e−03

1e−01

1e+01

1e+03

2 8 32 128 512 2048 8192 32768WSS [KiB]

varia

nce/

mea

n [c

ycle

s]


1e−02

1e+00

1e+02

L1 L2 L3 MEMWSS region

varia

nce/

mea

n [c

ycle

s]

NOVA Linuxmin

LinuxDebian

RO, warm, local mem


Aggregated results of all 46 experiments

1e−02

1e+00

1e+02

L1 L2 L3 MEM

RO, warm, local mem

1e−02

1e+00

1e+02

L1 L2 L3 MEM

RO, warm, remote mem

1e−02

1e+00

1e+02

L1 L2 L3 MEM

RW, warm, local mem

1e−02

1e+00

1e+02

L1 L2 L3 MEM

RW, warm, remote mem

1

3

10

30

100

L1 L2 L3 MEM

RO, modif, local mem

1

3

10

30

L1 L2 L3 MEM

RO, modif, remote mem

1

3

10

30

L1 L2 L3 MEM

RW, modif, local mem

1

3

10

30

L1 L2 L3 MEM

RW, modif, remote mem

1

100

L1 L2 L3 MEM

intra−pkg. migration 0−3

10

1000

L1 L2 L3 MEM

inter−pkg. migration 0−6

10

1000

L1 L2 L3 MEM


10

1000

L1 L2 L3 MEM

inter−pkg. migration 6−0

1

100

L1 L2 L3 MEM


1

10

100

L1 L2 L3 MEM


1e−01

1e+01

1e+03

L1 L2 L3 MEM

Preemption 1ms, local mem

1

100

L1 L2 L3 MEM


1

100

L1 L2 L3 MEM


1

100

L1 L2 L3 MEM


1

100

L1 L2 L3 MEM


1

100

L1 L2 L3 MEM


1

100

L1 L2 L3 MEM


1

100

L1 L2 L3 MEM


1

100

L1 L2 L3 MEM


1

100

L1 L2 L3 MEM


1e−01

1e+01

1e+03

L1 L2 L3 MEM


1

100

L1 L2 L3 MEM


1

100

L1 L2 L3 MEM


1e−01

1e+01

1e+03

L1 L2 L3 MEM


1

100

L1 L2 L3 MEM


1e−01

1e+01

1e+03

L1 L2 L3 MEM

Preemption 0−20ms, local mem

1e−02

1e+00

1e+02

L1 L2 L3 MEM

Preemption 1ms, remote mem

1e−01

1e+01

1e+03

L1 L2 L3 MEM


1

100

L1 L2 L3 MEM


1

100

L1 L2 L3 MEM


1e−01

1e+01

1e+03

L1 L2 L3 MEM


1e−01

1e+01

1e+03

L1 L2 L3 MEM


0.1

10.0

L1 L2 L3 MEM


1e−01

1e+01

1e+03

L1 L2 L3 MEM


0.1

10.0

L1 L2 L3 MEM


1e−01

1e+01

1e+03

L1 L2 L3 MEM


1

100

L1 L2 L3 MEM


1e−01

1e+01

1e+03

L1 L2 L3 MEM


1

100

L1 L2 L3 MEM


0.1

10.0

L1 L2 L3 MEM


0.1

10.0

L1 L2 L3 MEM


0.1

10.0

L1 L2 L3 MEM

Preemption 0−20ms, remote mem


Summarized results

highest

middle

lowest

0 50 100 150Frequency

Noi

se le

vel Kernel

NOVA

Linux minconfig

Linux Debian

OS lowest middle highestNOVA 108 (59%) 35 (19%) 41 (22%)Linux min. 37 (20%) 70 (38%) 77 (42%)Linux Debian 39 (21%) 79 (43%) 66 (36%)


Conclusions

I In most cases NOVA microkernel produced lowest noise.I In several other cases NOVA produced highest noise.I System management interrupts(?) are really a big source

of noise.I Operating system is not the only dominant source of noise.

Off-chip access increases the noise.I Configuration of Linux kernel has significant impact on the

observed noise.


Operating System Noise: Linux vs....

Documents

Transcript of Operating System Noise: Linux vs....