Pivot Tracing: Dynamic Causal Monitoring for...
Transcript of Pivot Tracing: Dynamic Causal Monitoring for...
Pivot Tracing: Dynamic Causal Monitoring for Distributed Systems
Student: Hunter Ingle
1
Original Paper
Jonathon Mace, Ryan Roelke, and Rodrigo Fonseca. “Pivot Tracing: Dynamic Causal Monitoring for Distributed Systems.” In ACM Transactions on Computer Systems, Vol. 35, Issue 4, pp 11:1-11:28. December 2018.
2
Distributed Systems
• Several machines (nodes) working together to perform a certain task
• Great for large scale data processing and parallel processing• However, some problems and issues exist
§ Data encryption and transmission§ Fault detectionoMonitoring and troubleshooting functionality
3
Main Problem
• Monitoring and troubleshooting distributed systems is both hard and time-consuming§ Hardware and software failures§ Misconfigurations across the system§ Unrealistic expectations
• Current tools:§ Logs, counters, and metrics
• Limitations:§ Recorded at deployment (a priori)oMay not always contain necessary information
§ Captured via components or machinesoDifficult to correlate between them
4
Solution: Pivot Tracing
• Combines dynamic instrumentation with causal tracing• Provides metrics at any one point of a system• Selects, filters, and groups events for other points• Allows crossing component and machine boundaries, as
mentioned before
5
Four Contributions
• “Happened-before join”• Query optimization for the join at runtime by combining
dynamic instrumentation and causal tracing• Prototype implementation of Pivot Tracing
§ Applied to Hadoop distributed system framework• Evaluation based on diagnosing problems at runtime
6
Overview
• Requirements in a system:§ Dynamic code injection§ Causal metadata propagation
• Based on tracepoints, where PT can insert instrumentation§ Instructions based on the location and methods needed for
changing the system
7
Overview Cont.
• Queries (1) – sent into DS via dataset (2) and utilize a vocab defined by tracepoints
• Queries are compiled into advice (3) – instruction set that processes queries
• Advice is mapped to code that PT injects to tracepoints(4). Each time execution reaches the tracepoint the advice is called as well.
8
Overview Cont.
• The happened-before join is based on advice in a tracepoint delivering information through the execution path to advice in other tracepoints. This uses causal metadata propagation known as baggage (5).
• Advice can also carry tuples (6) that are aggregated and sent to the client via a message bus (7) and (8)
9
Design
• Tracepoints§ Act as an entry point to the system for PT queries§ Based on some eventoRequests, I/O operation completion, etc.
§ Only reference the locations for entry points, so not defined or limited by a priori modifications
§ Only compiled and installed at runtime whenever a query is sent
§ When request reaches the tracepoint, instrumentation at that point is called, exporting some variables needed for a tupleoHost, timestamp, process ID, process name, and
definition
10
Tracepoint Example
11
Design – Happened-Before Joins
• Allows tuples from different PT queries to be joined based on the “happened before” relation. § If a and b are events on the same process and a happens
first, then a -> b and vice versao Same if the event a was the cause of event b
§ Based on Lambert’s study on timing in systems (see references slide)
• The join is based on queries and their effects on events• Using the join between two queries ( ) results in the
tuples t1 and t2 where all t1 in Q1 happened before t2 in Q2 (t1->t2)
• Provides insight into the relationships between events being monitored in a system
12
Design – Advice
• Intermediate representation of PT queries• Determines the operations to be performed at tracepoints
and provides monitoring code to be installed at those points• Based on an advice API that can
13
Design – Advice Cont.
14
Optimization - Baggage
• Optimizes happened-before joins during request execution• Container for an instance of a tuple
§ Propagated with a request through thread, application, and machine boundaries
§ Also observes the happened-before events of the request
15
Attached to Hadoop
• Previously mentioned design was attached to multiple aspects of the Hadoop framework§ HBase – non-relational database that runs atop the HDFS
• Extended Hadoop functionality and protocols to allow baggage and tracepoints§ Tracepoints were implemented inoDataNode’s DataTransferProtocoloNameNode’s ClientProtocol
16
Evaluation
• Experiment led to finding a bug in the HDFS for uneven load
distribution for replicated data
§ 8 DataNode and 1 NameNode cluster on HDFS
§ Used 96 stress test clients to determine high load levels on
two hosts and almost zero loads on the rest for replication
factor of 3
§ Replica Selection Bug (HDFS-6268) has since been fixed in
subsequent Hadoop release
17
Evaluation Cont.
18
Evaluation Cont.
• Based on experiments with the PT-infused Hadoop framework, results show:§ It is dynamic and extensible§ It is scalable with low overhead§ It allows cross-boundary analysis§ It uses event causality for diagnosis of errors§ It provides analysis even with minimal tracepoints
• However, it is not meant to replace all functionality of logs§ Security auditing§ Forensics§ Debugging
19
Conclusion
• Pivot Tracing is the first monitoring system of its kind§ i.e. combining dynamic instrumentation and causal tracing§ Provided a happened-before join to boost efficiency of both
• Low overhead for cross-boundary analysis§ Extremely effective method for error diagnosis within
distributed systems
20
References
• Leslie Lamport. 1978. Time, clocks, and the ordering of events in a distributed system. Commun. ACM 21, 7 (1978), 558–565.
21