Distributed systems-radiology

Modern Radiology forDistributed Systems

Dietrich Featherston@d2fn

Thursday, October 11, 12

This is a talk about monitoring


But not just any kind of monitoring

Non-invasive monitoring


non-invasive monitoring

measures taken to describe the state of a system with minimal changes to the system being monitored


Insight

Invasiveness

Radiographic Imagery


preventative care

measures taken to prevent diseases or injuries rather than curing them or treating their symptoms


Non-invasive monitoring techniques focus primarily on host-based metrics

Why is this a problem?


Because applications are distributed


Information emittedabout nodes in the network

n Information emittedabout edges

in the network

n²Network size


We analyze cell-structure because we can’t envision

the whole organism

We react to disease and injury because we lack

preventative care


We lack preventative care for applications because our non-invasive monitoring techniques are growing less and less meaningful


Radiology is useful in illuminating non-invasive monitoring of distributed systems


Context iseverything


How do we use context?


Context

Your BigDumb Data

!!!


Human brain

+med school

Radiographic Imagery

Diagnoses


Signal Processing

VLA Output

E.T.


NetworkData

ApplicationBehavior

Application TopologySignal ProcessingExpert Brain


dimensions (11)epoch secondsepoch minutesepoch hoursnode idsource ipsource portdest ipdest portinterfacecountrynetwork/asn

measurements (8)egress packetsegress octetsingress packetsingress octetsretransmitserrorsapp-rtthandshake-rtt


Case Study #1

GC-Death of a distributed JVM application


Case Study #2

Symptoms:- Latent Riak handoff- Cluster throughput bottoming out


busy_dist_port


+zdbbl 8192


Case Study #3

Bringing a dead riak node back online


Case Study #4

Retransmits 10% of total network throughput


var put: HttpPut = nulltry { // ... put data}catch { case e: Exception => // ... handle exception}finally { if(put != null) { put.abort() }}


abortpublic void abort()Description copied from interface: HttpUriRequestAborts execution of the request.

Source: http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/client/methods/HttpRequestBase.html#abort()

THANKS


http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/client/methods/HttpUriRequest.html#abort()

http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/client/methods/HttpUriRequest.html#abort()

http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/client/methods/HttpRequestBase.html#abort()

http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/client/methods/HttpRequestBase.html#abort()

129 public void abort() {130 ClientConnectionRequest localRequest;131 ConnectionReleaseTrigger localTrigger;132 133 this.abortLock.lock();134 try {135 if (this.aborted) {136 return;137 } 138 this.aborted = true;139 140 localRequest = connRequest;141 localTrigger = releaseTrigger;142 } finally {143 this.abortLock.unlock();144 } 145146 // Trigger the callbacks outside of the lock, to prevent147 // deadlocks in the scenario where the callbacks have148 // their own locks that may be used while calling149 // setReleaseTrigger or setConnectionRequest.150 if (localRequest != null) {151 localRequest.abortRequest();152 }153 if (localTrigger != null) {154 try {155 localTrigger.abortConnection();156 } catch (IOException ex) {157 // ignore158 }159 }160 }


http://grepcode.com/file/repo1.maven.org/maven2/org.apache.httpcomponents/httpclient/4.0/org/apache/http/client/methods/HttpRequestBase.java#129


http://grepcode.com/file/repo1.maven.org/maven2/org.apache.httpcomponents/httpclient/4.0/org/apache/http/client/methods/HttpUriRequest.java#HttpUriRequest.abort%28%29

http://grepcode.com/file/repo1.maven.org/maven2/org.apache.httpcomponents/httpclient/4.0/org/apache/http/client/methods/HttpUriRequest.java#HttpUriRequest.abort%28%29



http://grepcode.com/file/repo1.maven.org/maven2/org.apache.httpcomponents/httpclient/4.0/org/apache/http/conn/ClientConnectionRequest.java#ClientConnectionRequest

http://grepcode.com/file/repo1.maven.org/maven2/org.apache.httpcomponents/httpclient/4.0/org/apache/http/conn/ClientConnectionRequest.java#ClientConnectionRequest



http://grepcode.com/file/repo1.maven.org/maven2/org.apache.httpcomponents/httpclient/4.0/org/apache/http/conn/ConnectionReleaseTrigger.java#ConnectionReleaseTrigger

http://grepcode.com/file/repo1.maven.org/maven2/org.apache.httpcomponents/httpclient/4.0/org/apache/http/conn/ConnectionReleaseTrigger.java#ConnectionReleaseTrigger





http://grepcode.com/file/repo1.maven.org/maven2/org.apache.httpcomponents/httpclient/4.0/org/apache/http/client/methods/HttpRequestBase.java#HttpRequestBase.0abortLock


http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/util/concurrent/locks/Lock.java#Lock.lock%28%29

http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/util/concurrent/locks/Lock.java#Lock.lock%28%29





http://grepcode.com/file/repo1.maven.org/maven2/org.apache.httpcomponents/httpclient/4.0/org/apache/http/client/methods/HttpRequestBase.java#HttpRequestBase.0aborted














http://grepcode.com/file/repo1.maven.org/maven2/org.apache.httpcomponents/httpclient/4.0/org/apache/http/client/methods/HttpRequestBase.java#HttpRequestBase.0connRequest

http://grepcode.com/file/repo1.maven.org/maven2/org.apache.httpcomponents/httpclient/4.0/org/apache/http/client/methods/HttpRequestBase.java#HttpRequestBase.0connRequest



http://grepcode.com/file/repo1.maven.org/maven2/org.apache.httpcomponents/httpclient/4.0/org/apache/http/client/methods/HttpRequestBase.java#HttpRequestBase.0releaseTrigger

http://grepcode.com/file/repo1.maven.org/maven2/org.apache.httpcomponents/httpclient/4.0/org/apache/http/client/methods/HttpRequestBase.java#HttpRequestBase.0releaseTrigger







http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/util/concurrent/locks/Lock.java#Lock.unlock%28%29

http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/util/concurrent/locks/Lock.java#Lock.unlock%28%29

















http://grepcode.com/file/repo1.maven.org/maven2/org.apache.httpcomponents/httpclient/4.0/org/apache/http/conn/ClientConnectionRequest.java#ClientConnectionRequest.abortRequest%28%29

http://grepcode.com/file/repo1.maven.org/maven2/org.apache.httpcomponents/httpclient/4.0/org/apache/http/conn/ClientConnectionRequest.java#ClientConnectionRequest.abortRequest%28%29









http://grepcode.com/file/repo1.maven.org/maven2/org.apache.httpcomponents/httpclient/4.0/org/apache/http/conn/ConnectionReleaseTrigger.java#ConnectionReleaseTrigger.abortConnection%28%29

http://grepcode.com/file/repo1.maven.org/maven2/org.apache.httpcomponents/httpclient/4.0/org/apache/http/conn/ConnectionReleaseTrigger.java#ConnectionReleaseTrigger.abortConnection%28%29



http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/io/IOException.java#IOException

http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/io/IOException.java#IOException









augmented intelligence precedesartificial intelligence


1895

Wilhelm Röntgen discovers X-RaysFirst medical use of x-rays in human imaging takes place one month later


1895


1905

First English text on chest radiography


1895


1920

1905

First English text on chest radiography

Society of Radiographers formed


Recognition of radiology as a formal medical discipline was a cultural problem, not

a technology problem

http://www.bshr.org.uk/page13.htmlThursday, October 11, 12

http://www.bshr.org.uk/page13.html

http://www.bshr.org.uk/page13.html

If you want to talk to me about the query language used to ask questions of the network data we collect at Boundary talk to me after or hit me up on twitter.

@d2fngithub.com/dietrichf


Find 45 minutes of total traffic

seen on meters 1, 2, 226, & 301

starting 18 hours ago broken

down by peer ip retain top 10 by

the ratio of retransmits to

packets

get volume_1s_meter_ip [ meter in {1, 2, 226, 301}; epochMillis from -18h for 45m;]categorize sum(ingress) as ingress, sum(egress) as egress, sum(ingressPackets + egressPackets) as packets, sum(retransmits) as retransmits, mean(appRttUsec/1000) as appRttMsby epochMillis, ipretain top 10 per epochMillis on retransmits/packets


Distributed systems-radiology

Technology

Transcript of Distributed systems-radiology