Efficient interdomain transmission ofperformance data

John Elsejohn.else@citrix.com

XAPI projectXenAPIXAPIXCP/XenServer toolstack

XAPI projectHigh level OO API around XenHTTP interfaceCollects performance data from dom0/guests

Performance dataNumericChanges regularly - requires samplingIf we sometimes lose a sample - that's OK

Interdomain performance dataCurrently one metric per domU: free memoryTransmitted via xenstoreEven this has been enough to cause performance problems!

Improvements so farData collection caused laggy API response from XAPI-> XAPI disaggregation: rrdd, networkd and xenopsd

Improvements so farHigh xenstore load reduced bootstorm performance

-> Watch xenstore instead of polling

What next?Take xenstore out of the data pathCreate a generic API for:

Driver domain monitoringApplication monitoring...

rrdd plugins - dom0

rrdd plugins - interdomain

rrdd plugins - protocol v1header 10 bytesdata length (ASCII) 8 bytesmd5sum (ASCII) 32 bytesjson data inc. timestamp ? bytes

rrdd plugins - protocol v1DATASOURCES <- header00000a79 <- data length6cc0472a94896d245dcea04a37c26474 <- data md5sum{ "timestamp": 1407715758, "datasources": { "io_throughput_read_9d0e83ec": { "description": "Data read from the SR, in MiB/s", "owner": "host", "value": "0.55", "value_type": "float", "type": "absolute_to_rate", "units": "MiB/s", "min": "0.00", "max": "inf" } }}

rrdd plugins - protocol v1DATASOURCES <- header00000a79 <- data length6cc0472a94896d245dcea04a37c26474 <- data md5sum{ "timestamp": 1407715758, "datasources": { "io_throughput_read_9d0e83ec": { "description": "Data read from the SR, in MiB/s", "owner": "host", "value": "0.55", "value_type": "float", "type": "absolute_to_rate", "units": "MiB/s", "min": "0.00", "max": "inf" } }}

rrdd plugins - protocol v2Separate data from metadataPack as much as possible as binary

rrdd plugins - protocol v2header 10 bytesdata crc32 4 bytesmetadata crc32 4 bytesdatasource count (int32) 4 bytestimestamp (int64) 8 bytesdatasource values n * 8 bytesmetadata length (int32) 4 bytesjson metadata ? bytes

rrdd plugins - protocol v2header 10 bytesdata crc32 4 bytesmetadata crc32 4 bytesdatasource count (int32) 4 bytestimestamp (int64) 8 bytesdatasource values n * 8 bytesmetadata length (int32) 4 bytesjson metadata ? bytes

rrdd plugins - protocol v2DATASOURCES <- header.... <- data CRC.... <- metadata CRC.... <- datasource count........ <- timestamp........ <- datasource value.... <- metadata length{ "datasources": { "io_throughput_read_9d0e83ec": { "description": "Data read from the SR, in MiB/s", "owner": "host", "value_type": "float", "type": "absolute_to_rate", "units": "MiB/s", "min": "0.00", "max": "inf" } }}

rrdd plugins - protocol v2DATASOURCES <- header.... <- data CRC.... <- metadata CRC.... <- datasource count........ <- timestamp........ <- datasource value.... <- metadata length{ "datasources": { "io_throughput_read_9d0e83ec": { "description": "Data read from the SR, in MiB/s", "owner": "host", "value_type": "float", "type": "absolute_to_rate", "units": "MiB/s", "min": "0.00", "max": "inf" } }}

rrdd plugins - protocol v2V2 protocol gives about a 10x speedup in the "usual" case, i.e. set ofexported datasources doesn't change.

Demo

Further infoProtocol:Frontend:Slides:IRC: freenode #xen-apiEmail: john.else@citrix.com

github.com/xapi-project/rrd-transportgithub.com/xapi-project/ocaml-rrdd-pluginjohnelse.github.io/xen-summit-2014