dCache Locality Performance Testing

Sarah WilliamsIndiana University

2013-09-13

Test Cases

• Downloading the files via xrdcp– 5 MB and 2 GB files dummy files, full of 0’s

• Directly access files via readDirect– The test file is NTUP_SMWZ.01120689._000089.root.1,

a fairly typical user input file of size 4.6 GB.– The tests read 10%-90% of the file, in 10% increments– readDirect does not allow us control the bytes read,

only number of events read. I ran a series of tests to find the number of events read which would correspond to 10-90% of bytes read.

Test Conditions

• The test clients are worker nodes temporarily removed from the cluster. They are running only the OS (Scientific Linux 6.4), Condor in an offline mode, and the tests themselves.

• The tests were run serial and single-threaded.• The servers are our production servers. The tests are

spread across 10 servers.• The cluster was running a normal production load. The

LAN network and WAN connections showed typical usage. Tests from times of non-typical conditions were discarded.

Network conditions

Test WorkerNodes LAN Storage

Remote Storage

Remote Storage

Remote Storage

Clients ran at IU, with local storage at IU and remote storage at UC. Note this route does not use LHCOne.Near-future network improvements are not shown.

[root@iut2-c200 ~]# traceroute uct2-s20.uchicago.edutraceroute to uct2-s20.uchicago.edu (128.135.158.170), 30 hops max, 60 byte packets 1 149.165.225.254 (149.165.225.254) 20.328 ms 20.325 ms 9.372 ms 2 et-10-0-0.2012.rtr.ictc.indiana.gigapop.net (149.165.254.249) 0.448 ms 0.491 ms 0.484 ms 3 149.165.227.22 (149.165.227.22) 5.087 ms 5.196 ms 5.273 ms 4 10.4.247.230 (10.4.247.230) 5.229 ms 5.396 ms 5.547 ms 5 uct2-s20.uchicago.edu (128.135.158.170) 5.323 ms 5.347 ms 5.220 ms

Test Code• FAX End-user tutorial code base:

– https://github.com/ivukotic/Tutorial/– readDirect opens a ROOT file and reads the specified percentage of events from

it.• Locality-caching specific code:

– https://github.com/DHTC-Tools/ATLAS/tree/master/Performance%20Tests/Locality%20Caching

– test_local.sh is run directly on the worker node, and runs the local disk test with memory flushing. The line that executes ‘releaseFileCache’ can be commented out to do non-flushed tests.

– test_direct.sh, test_hit.sh and test_miss.sh are run on the management host and ssh to the test hosts to run the tests.

– ref_points.txt contains the set of percentage of events read needed to get the correct percentage of bytes read for readDirect

– releaseFileCache removes a specific file from the Linux page cache– pooltest*.txt contain the basenames of the files used for the tests. The

directories are hardcoded in the scripts. The filename is also the name of the data server the file resides on, ex. ‘uct2-16_1’

– testshosts.txt contains the hostnames of the worker nodes to use for testing.

Small file downloadsIn these small files tests, the optimum strategy is no caching. You can see in the chart below that cache hit downloads take about 700 ms longer than non-cached downloads. This is caused by the small overhead caching imparts on the central dCache manager, which must look up what data servers a client is permitted to read and check if the requested file is on one of those servers.

0 10 20 30 40 50 60 700

0.1

0.2

0.3

0.4

0.5

0.6

0.7

5 MB file download times

cache hitcache missno caching

Iteration

Dow

nloa

d tim

e in

seco

nds

-0.090.010.110.210.310.410.51

5MB file download times

Cache hit

5 MB file

Dow

nloa

d tim

e in

seco

nds

Larger files show the advantages of caching in an environment where files will be reused. Note that the time to download a file on a cache miss is roughly the sum of an non-cached download and a cache hit download. Cached performance is much more consistent than non-cached.

Large file downloads

0 5 10 15 20 25 30 350

20

40

60

80

100

120

140

2 GB file download times

cache hitcache missno caching

Iteration

Dow

nloa

d tim

e in

seco

nds

0255075

100125150175

2GB file download times

Cache hitLinear (Cache hit)

2 GB fileDow

nloa

d tim

e in

seco

nds

The test file was pre-loaded to local disk, and then the tests were run on it sequentially. Due to memory caching in the Linux kernel, there was no actual disk IO during these tests. This is not necessarily a realistic test case. In production there would be N jobs competing for memory, and it would be impossible for them all to keep their input files in memory. Amore realistic case would be to removethe file from memory between tests.

Local disk (memory cached) tests

Read speed in MB/s X5660 E5440 AMD 2350 AMD2218Average 85.89 74.85 43.09 53.13Std dev 2.94 2.40 2.61 4.03

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%0

20

40

60

80

100

120

Times for Local Memory-Cached Tests

E5440Linear (E5440)AMD 2218

Percent of file read

Wal

ltim

e in

seco

nds

The test file was pre-loaded to local disk, and then the tests were run on it sequentially. Between each test, the releaseFileCache utility was used to remove the test file from memory. The tests were narrowed down to the two faster processors, the X5660 and E5440, to simplify testing.

Local disk (memory flushed) tests

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%0

20

40

60

80

100

120

Times for Local Disk (memory flushed) Tests

X5660Linear (X5660)E5440Linear (E5440)

Percent of file read

Wal

ltim

e in

seco

nds

Read speed in MB/s

X5660 E5440

Average 50.54 42.41

Std dev 0.99 0.82

WAN reads (caching disabled)WAN tests are vulnerable to network conditions on inter-campus links, making them more variable than LAN tests

Read speed in MB/s

X5660 E5440

Average 22.58 22.63

Std dev 4.15 1.65

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%0

25

50

75

100

125

150

175

200

225

Times for Remote Read tests (caching off)

X5660Linear (X5660)E5440

Percent of file read

Wal

ltim

e in

seco

nds

Cache HitsReads over the LAN show more consistency thanWAN tests.

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%0

10

20

30

40

50

60

70

80

90

100

Times for Cache Hit Tests

X5660Linear (X5660)E5440Linear (E5440)

Percent of file read

Wal

ltim

e in

seco

nds

Read speed in MB/s

X5660 E5440

Average 47.45 44.77

Std dev 2.02 1.41

Cache Misses

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%0

20

40

60

80

100

120

140

160

180

200

Times for Cache Miss Tests

X5660Linear (X5660)E5440Linear (E5440)

Percent of file read

Wal

ltim

e in

seco

nds

Cache miss tests results are equivalent to cache hit results plus 90s. We can infer thatit takes 90s to transfer the file from remote storage into local storage, at about 52 MB/s. Standard deviations are higher due to the inclusion of the inter-site link, the remote storage server and the local storage server, all of which are also serving the active cluster.

Read speed in MB/s

X5660 E5440

Average 16.24 15.41

Std dev 6.38 5.87

Direct comparison of caching strategies

No caching is the optimum strategy when less than 75% of the file is read. When more than 75% is read, caching becomes optimal.

0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00% 100.00%0

50

100

150

200

250

Comparison of Caching Strategies on X5660

Cache hitLinear (Cache hit)Local diskLinear (Local disk)

Percent of file read

Wal

ltim

e in

seco

nds

0

10

20

30

40

50

60

Comparison of Download Speed of Caching Strategies, on X5660

Spee

d in

MB/

s

Conclusions

• Caching is less effective for very small files and for jobs that read only part of the file

• Cache is most effective medium to large files, when the entire file is downloaded, or when the file is reused

• Next steps:– Turn on locality caching for production nodes and

monitor. Monday 9/16?