Bridging the gap between hardware and software tracing
-
Upload
christian-babeux -
Category
Technology
-
view
646 -
download
1
description
Transcript of Bridging the gap between hardware and software tracing
![Page 1: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/1.jpg)
1
Bridging the gap between hardware and software tracing
Embedded Linux Conference Europe 2013
[email protected] @c_bab
![Page 2: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/2.jpg)
2Source: arm.com [1]
![Page 3: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/3.jpg)
3
ARM Cortex A9
Source: arm.com [1]
![Page 4: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/4.jpg)
4
ARM Cortex A9
PTM
Program Trace Macrocell
Source: arm.com [1]
![Page 5: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/5.jpg)
5
whoami
Christian Babeux, Software Developer, EfficiOS,
Background in embedded and ASIC tools,
Active contributor to the LTTng projects:● lttng-tools, lttng-ust, babeltrace,● CI infra, Website, Twitter.
AUR package maintainer for Arch Linux.
![Page 6: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/6.jpg)
6
Content
What is hardware tracing?,
Why is it useful?,
ARM Coresight & ETM,
Freescale QorIQ & Nexus tracing,
LTTng & hardware tracing.
![Page 7: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/7.jpg)
7
What is hardware tracing?What is hardware tracing?
![Page 8: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/8.jpg)
8
What is hardware tracing?
Hardware component(s) used to traces instructions and data movement of a processing device
Real-time observation
Low intrusiveness
![Page 9: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/9.jpg)
9
What is hardware tracing? Cont.
● External trace
![Page 10: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/10.jpg)
10
What is hardware tracing? Cont.● Pros
– Can accommodate high data bandwidth
– Minimal impact on system performance
● Cons– Trace port not always available
– Custom hardware needed
![Page 11: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/11.jpg)
11
What is hardware tracing? Cont.
● Self-hosted trace
![Page 12: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/12.jpg)
12
What is hardware tracing? Cont.● Pros
– Self-contained, facilities can be used by host OS
– No need for special hardware
● Cons– Limited internal buffer space
– Might impact system performance
![Page 13: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/13.jpg)
13
Hardware tracing support● ARM
– Embedded Trace Macrocell (ETM),
– Program Trace Flow Macrocell (PTM),
– System Trace Macrocell (STM)
● PowerPC– Freescale QorIQ with Nexus tracing
– Branch History Rolling Buffer (BHRB)
● Intel– Intel Processor Tracing (PT)
– Last Branch Record (LBR), Branch Trace Store (BTS)
![Page 14: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/14.jpg)
14
Hardware tracing vs. software tracing● Software tracing:
– Static instrumentation or dynamic code patching
– Can be intrusive
– Can be slow
– Tracepoint level granularity
● Hardware tracing:– Tracing done on hardware
– Instrumentation not required
– Instruction level granularity
![Page 15: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/15.jpg)
15
Why hardware tracing is usefulWhy hardware tracing is useful
![Page 16: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/16.jpg)
16
Why hardware tracing is useful
Profiling– Very fine granularity profiling
Performance measurement
Code coverage
Monitoring– Statistics on application currently running?
![Page 17: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/17.jpg)
17
Why hardware tracing is useful cont.
Snapshot on crash/anomaly– Trace overwrites old data until anomaly detected
Event trigger trace
Hardware-assisted software tracing– Use hardware facility (ringbuffer)
![Page 18: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/18.jpg)
18
ARM Coresight & ETM ARM Coresight & ETM
![Page 19: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/19.jpg)
19
ARM Coresight● Coresight
– Collection of hardware components
– Trace and debug a complete SoC
– Open architecture
● Trace source– Processing elements (CPU, DSP, etc.)
– Buses
– System trace (generated from software)
![Page 20: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/20.jpg)
20
ARM ETM● ETM
– Monitor the core internal bus
– Instructions + data trace
– Hardware filters and triggers
– Trace stream compression
– Traces can be saved in internal buffer (ETB) or shared system memory
![Page 21: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/21.jpg)
21
ARM Coresight
Source: arm.com [2]
![Page 22: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/22.jpg)
22
State of Coresight & ETM in Linux● ETM tracer implementation available in Linux
– Seems to work only on specific hardware
● Coresight support status– Framework patchset proposed by Pratik Pratel [3]
● Trace decoder availability?
![Page 23: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/23.jpg)
23
Interested about Coresight/ETM?
“Hardware Trace in the Kernel” BoF by Pawel Moll from ARM
Today, 4:30 PM in Pentland
![Page 24: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/24.jpg)
24
Freescale QorIQ & Nexus tracingFreescale QorIQ & Nexus tracing
![Page 25: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/25.jpg)
25
Freescale QorIQ● PowerPC based platform targeted for high-performance
communications usage– Multiple e500mc processors,
– DPAA support (packet processing offloading),
– Support the Nexus debugging & tracing standard.
Source: radiolocman [4]
![Page 26: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/26.jpg)
26
Nexus standard● ISO standard for debugging embedded systems (IEEE-
ISTO-5001-2003),
● Designed for low pin count, standard set of connectors (JTAG or Debug port),
● Multiple level of Nexus “compliance”.
![Page 27: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/27.jpg)
27
Nexus standard level● Level 1:
– Run time control only (run, stop, breakpoints, etc.)
– Tracing not supported
● Level 2:– Ownership and program trace
● Level 3:– Data write trace & memory read/write on the fly
● Level 4:– Memory substitution
– Trace triggering via a watchpoint
![Page 28: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/28.jpg)
28
Nexus standard format● Packet based output format,
– Standard defines public messages, vendors can define extensions (TCODES)
– Fixed packet size per message, last packet can be of variable length
– Message can have an optional timestamp
![Page 29: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/29.jpg)
29
Example decoded Nexus message
Message # 328TCODE : 2 Ownership Trace MessageSRC ID : 0 Core0 / CPU0 (Clst0:Core0:Thread0)PID INDEX : 0x02 - Sync PIDPID VALUE : 0x01e01bf59c - LPIDR, MSR[GS], PID/NPIDR (ref:DC1[OTS]), MSR[PR]TIMESTAMP : 1140061 (0x11655d)
Message # 329TCODE : 27 Resource Full MessageSRC ID : 0 Core0 / CPU0 (Clst0:Core0:Thread0)RCODE : 0x8 - Timestamp counterRDATA : 0x00ffffffTIMESTAMP : 0 (0x0)
![Page 30: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/30.jpg)
30
State of Nexus in Linux● Nexus qoriq-debug kernel module,
– Available in Freescale QorIQ SDK Yocto Layer
– Implements a debugfs with memory mapped access to Nexus control register
– cat /sys/kernel/debug/npc/trace_buffer > trace
![Page 31: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/31.jpg)
31
Nexus debugfsroot@p3041ds:~# ls -al /sys/kernel/debug/qoriq-dbg[...]drwxr-xr-x 2 root root 0 Aug 6 20:22 cpu0drwxr-xr-x 2 root root 0 Aug 6 20:22 cpu1drwxr-xr-x 2 root root 0 Aug 6 20:22 cpu2drwxr-xr-x 2 root root 0 Aug 6 20:22 cpu3drwxr-xr-x 2 root root 0 Aug 6 20:22 ddr1drwxr-xr-x 2 root root 0 Aug 6 20:22 dpaa[...]drwxr-xr-x 2 root root 0 Aug 6 20:22 npcnpcdrwxr-xr-x 2 root root 0 Aug 6 20:22 nxc[...]
root@p3041ds:~# ls -al /sys/kernel/debug/qoriq-dbg/cpu0[...]-rw-rw-rw- 1 root root 0 Aug 6 20:22 dc1-rw-rw-rw- 1 root root 0 Aug 6 20:22 dc2-rw-rw-rw- 1 root root 0 Aug 6 20:22 dc4--w--w--w- 1 root root 0 Aug 6 20:22 ddam[...]
![Page 32: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/32.jpg)
32
State of Nexus in Linux● Nexus decoder availability
– Released as part of the babeltrace project
● Open-source software to reconstruct program flow not yet developed
– Integrate such functionality in an IDE ? perf?
![Page 33: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/33.jpg)
33
LTTng & Hardware tracingLTTng & Hardware tracing
![Page 34: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/34.jpg)
34
Project overview
![Page 35: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/35.jpg)
35
Tracers
● lttng-modules: kernel tracer module, compatible with kernels from 2.6.38* to 3.11,
● lttng-ust: user-space tracer, in-process library.
* Kernel tracing is now possible on 2.6.32 to 2.6.37 by backport of 3 Linux Kernel patches [1].
![Page 36: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/36.jpg)
36
Utilities
● lttng-tools: cli utilities and daemons for trace control,
– lttng: cli utility for tracing control,
– lttng-sessiond: tracing registry daemon,
– lttng-consumerd: consume trace data,
– lttng-relayd: network streaming daemon.
![Page 37: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/37.jpg)
37
Viewers
● babeltrace: cli text viewer, trace converter, plugin system,
● lttngtop: ncurse top-like viewer,● Eclipse lttng plugin: front-end for lttng, collect,
visualize and analyze traces, highly extensible.
![Page 38: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/38.jpg)
38
Hardware tracing support
● Initial attempt to support hardware tracing:– Babeltrace Nexus to CTF converter [5]
– Goal: Leverage existing traces visualizer
● Encountered issues:– Traces are not self-contained, need sideband
information
– Internal trace buffer size limitation
– Synchronization with other traces can be tricky
![Page 39: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/39.jpg)
39
Hardware tracing support demo
DEMO
![Page 40: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/40.jpg)
40
Future work
● Decoder/Converter for ARM ETM?
● Control hardware tracing facilities with the lttng-tools command-line?
● Custom views for hardware traces in Eclipse plugin
![Page 41: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/41.jpg)
41
Conclusion
● Availability and usefulness of hardware tracing
● Initial support for self-hosted hardware tracing
● Common abstraction for hardware tracing in the Linux kernel?
![Page 42: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/42.jpg)
42
Questions ?
? lttng.org
@lttng_project
www.efficios.com
![Page 43: Bridging the gap between hardware and software tracing](https://reader034.fdocuments.in/reader034/viewer/2022042714/5549e8c3b4c90512488b4d52/html5/thumbnails/43.jpg)
43
References
[1] - http://www.arm.com/images/processor/Cortex-A9-osprey.jpg
[2] - http://www.arm.com/images/CoreSight_Diagram_Tiny.jpg
[3] - https://lkml.org/lkml/2012/12/19/331
[4] - http://www.rlocman.ru/i/Image/2010/06/24/2.jpg
[5] - https://github.com/cbab/babeltrace/tree/nexus