State of Gluster PerformanceManoj Pillai - mpillai@redhat.comRed Hat Perfomance and Scale Engineering

Agenda● Motivation● Gluster performance by workload class● EC for video storage use-cases● Work in Progress

Motivation● Understand gluster performance characteristics, for

different workload classes and workload parameters● Identify areas for improvement

Configuration● Systems:

○ 12 drives per server

○ Replica-2 results with single 12-disk RAID-6 device on each server

○ EC results with brick on each drive○ Upto 6 servers, 4 clients in these tests○ 1 10GbE link each

● Software versions○ Glusterfs-3.8.2-1.el7.x86_64, RHEL-7

● Standard tuning○ lookup-optimize, client-io-threads: on○ event-threads=4

Sequential I/O: Replica Volume

● With sufficient number of requests concurrently in flight, close to network line speed on both writes and reads.

Sequential I/O: Disperse Volume

● Similar to dist-rep, but larger gap in throughput between low thread count and high thread count.

Seq I/O: Comparison of Volume Types

● Replica-2 performs better for low thread count, especially on writes.

● Disperse excels for writes with higher no. of threads.

Seq Write: Comparison with kNFS

● kNFS better on single-thread tests, gluster does well with higher number of threads.

Seq Read: Comparison with kNFS

● kNFS dominates on single-thread test, hits line speed

Random Read: Comparison with kNFS

● “1-cached” is single thread test where server-side cache has not been dropped.○ Good test of latency in the I/O path

Random Write: Comparison with kNFS

● All of these tests bottleneck on disk○ RAID-6 not a good choice for random writes.

Random Read: Comparison of Volume Types

Random Write: Comparison of Volume Types

● Replica-2 suffers possibly because of RAID-6 poor performance on random writes.

Small File Read: Comparison of Volume Types

● Disperse at a clear disadvantage in these tests

Small File Create: Comparison of Volume Types

● Small file performance still under investigation

EC for Video Storage Use-Cases

● Lot of interest seen in this area● Significantly lower cost for EC, good performance

Work In Progress

● Rebalance Performance○ Crawl, migration and throttle settings

● Small-file Performance○ Performance degradation with increasing number of

files

● Tiering performance for small files○ Small file performance with EC cold tier

Theme

Thanks!!