Dapper, a Large-Scale Distributed System Tracing Infrastructure
Distributed Tracing
-
Upload
soasme -
Category
Engineering
-
view
432 -
download
2
Transcript of Distributed Tracing
![Page 1: Distributed Tracing](https://reader031.fdocuments.in/reader031/viewer/2022030316/587604d51a28ab4a508b660d/html5/thumbnails/1.jpg)
Distributed Tracing
![Page 4: Distributed Tracing](https://reader031.fdocuments.in/reader031/viewer/2022030316/587604d51a28ab4a508b660d/html5/thumbnails/4.jpg)
What's this?
![Page 5: Distributed Tracing](https://reader031.fdocuments.in/reader031/viewer/2022030316/587604d51a28ab4a508b660d/html5/thumbnails/5.jpg)
– wikipedia.org `Tracing(software)`
“In software engineering, tracing involves a specialized use of logging to record information
about a program's execution. ”
![Page 6: Distributed Tracing](https://reader031.fdocuments.in/reader031/viewer/2022030316/587604d51a28ab4a508b660d/html5/thumbnails/6.jpg)
What is slow?
![Page 7: Distributed Tracing](https://reader031.fdocuments.in/reader031/viewer/2022030316/587604d51a28ab4a508b660d/html5/thumbnails/7.jpg)
Who did wrong?
![Page 8: Distributed Tracing](https://reader031.fdocuments.in/reader031/viewer/2022030316/587604d51a28ab4a508b660d/html5/thumbnails/8.jpg)
How?
![Page 9: Distributed Tracing](https://reader031.fdocuments.in/reader031/viewer/2022030316/587604d51a28ab4a508b660d/html5/thumbnails/9.jpg)
Possible solution• Blackbox
• application-agnostic
• low correctness
• Unique identifier
• application-intrusive
• instrumentation for various transports
![Page 10: Distributed Tracing](https://reader031.fdocuments.in/reader031/viewer/2022030316/587604d51a28ab4a508b660d/html5/thumbnails/10.jpg)
![Page 11: Distributed Tracing](https://reader031.fdocuments.in/reader031/viewer/2022030316/587604d51a28ab4a508b660d/html5/thumbnails/11.jpg)
Pre-Order Traversal
• Log parent entering
• Log all children
• Log parent leaving
![Page 12: Distributed Tracing](https://reader031.fdocuments.in/reader031/viewer/2022030316/587604d51a28ab4a508b660d/html5/thumbnails/12.jpg)
Zipkin Example
![Page 13: Distributed Tracing](https://reader031.fdocuments.in/reader031/viewer/2022030316/587604d51a28ab4a508b660d/html5/thumbnails/13.jpg)
Some infrastructure
• HTTP: X-Header
• Thrift: add tracer
• RPC: god-mode
• ...
![Page 14: Distributed Tracing](https://reader031.fdocuments.in/reader031/viewer/2022030316/587604d51a28ab4a508b660d/html5/thumbnails/14.jpg)
Clock skewcentral problem of distributed system
![Page 15: Distributed Tracing](https://reader031.fdocuments.in/reader031/viewer/2022030316/587604d51a28ab4a508b660d/html5/thumbnails/15.jpg)
Things worth mentioned
• sampling
• reporting
• aggregate analysis
![Page 16: Distributed Tracing](https://reader031.fdocuments.in/reader031/viewer/2022030316/587604d51a28ab4a508b660d/html5/thumbnails/16.jpg)
Zipkin
![Page 17: Distributed Tracing](https://reader031.fdocuments.in/reader031/viewer/2022030316/587604d51a28ab4a508b660d/html5/thumbnails/17.jpg)
Annotationstring data associated with particular timestamp,
service, host, ...
![Page 18: Distributed Tracing](https://reader031.fdocuments.in/reader031/viewer/2022030316/587604d51a28ab4a508b660d/html5/thumbnails/18.jpg)
Annotation Types• cs: client start
• sr: server receive
• ss: server send
• cr: client receive
• other: further insight of service
![Page 19: Distributed Tracing](https://reader031.fdocuments.in/reader031/viewer/2022030316/587604d51a28ab4a508b660d/html5/thumbnails/19.jpg)
SpanA set of annotations, represents one specific method call.
Has name, id.
![Page 20: Distributed Tracing](https://reader031.fdocuments.in/reader031/viewer/2022030316/587604d51a28ab4a508b660d/html5/thumbnails/20.jpg)
TraceA set of span all associated to the same request
![Page 21: Distributed Tracing](https://reader031.fdocuments.in/reader031/viewer/2022030316/587604d51a28ab4a508b660d/html5/thumbnails/21.jpg)
Identifiers
• Trace ID
• Span ID
• Parent ID
![Page 22: Distributed Tracing](https://reader031.fdocuments.in/reader031/viewer/2022030316/587604d51a28ab4a508b660d/html5/thumbnails/22.jpg)
Big Brother Bird• X-B3-TraceId: 64 encoeded bits
• X-B3-SpanId: 64 encoded bits
• X-B3-ParentSpanId: 64 encoded bits
• X-B3-Sampled: Boolean (1/0
• X-B3-Flags: a Long
![Page 23: Distributed Tracing](https://reader031.fdocuments.in/reader031/viewer/2022030316/587604d51a28ab4a508b660d/html5/thumbnails/23.jpg)
Architecture
![Page 24: Distributed Tracing](https://reader031.fdocuments.in/reader031/viewer/2022030316/587604d51a28ab4a508b660d/html5/thumbnails/24.jpg)
Some adjustment
• scribe -> fluentd + fluentd-plugin-scribe
• fiangle -> http-based intrakit
• serialization & deserialization annotations
![Page 25: Distributed Tracing](https://reader031.fdocuments.in/reader031/viewer/2022030316/587604d51a28ab4a508b660d/html5/thumbnails/25.jpg)
Q&A
![Page 26: Distributed Tracing](https://reader031.fdocuments.in/reader031/viewer/2022030316/587604d51a28ab4a508b660d/html5/thumbnails/26.jpg)
Thanks@soasme