Embedded Linux Debugging with JTAG ICE

http://www.kmckk.co.jp/

1

Embedded Linux Debugging with JTAG ICE

Kyoto Microcomputer Co., Ltd.http://www.kmckk.co.jp/eng

[Translated by ikoma]




http://www.kmckk.co.jp/eng


2

Linux Debugging





3

Limitation of GDB(1) ptrace() system call in kernel is used for debugging.

Debugging daemon （ gdbserver) is one of user applications.

gdbserver Target ApplicationFor Debugging

Kernel/Loadable Modules

ptrace Driver

Kernel Space

User Space





4

Limitation of GDB (1) As kernel is one program, ptrace() can not run

when breakpoint reached in device driver.



ptrace Driver

Kernel Space

User Space

Breakpoint reached

Kernel stops





5

Limitation of GDB (1) While stopped in driver, it is not possible to peek into

memory or to set break points on debug target app.



ptrace Driver

Kernel Space

User Space

Breakpoint reached

Kernel stops

Kernel stops andgdbserver cannot work





6

Issue in GDB （２） Issue of debugging more than one processes at

the same timegdbserverkernel Process A Process B

User SpaceKernel Space

Break requestsignal

Break request

signal

executionstops

executionstops

Process Bmay run





7

Debugging Targets on Linux and Their Features

Type of Program Memory Space Address Paging Support for Debugging

Boot Loader NonMMU Space Fixed Address Nothing Same as Ordinary Embedded Debugging

Linux KernelNon-multilplexed Kernel Space on

MMUFixed Address Nothing （ *

１）Same as Ordinary Embedded Debugging

Loadable Module

Non-multilplexed Kernel Space on

MMURelocatable On Demand

Paging

Requires Resolving Relocation and Support for Paging

Shared Library

Logical Multiplexed

Virtual Space on MMU

Relocatable On Demand Paging

Requires Resolving Relocation, Logical Multiplexed Virtual Space, and Support for Paging

ApplicationLogical

Multiplexed Virtual Space on

MMUFixed On Demand

Paging

Requires Logical Multiplexed Virtual Space and Support for Paging

PARTNER-Jet Supports All Above*1 Depending on CPU, Initialization Section may be “Paged Out”





8

Technical Challenges Relocatable

Actual addresses are unknown until kernel loads image onto memory.

On Demand Paging When setting breakpoints or referring variables, it is not

assured that the page is on memory. Multiplexed Virtual Spaces on MMU

An address may designate different memory locations if the process IDs are different

Support of hardware breakpoint Support of trace to process (ETM/AUD etc.)





9

Resolving Relocation （ Loadable Modules ）

Tricks for Debugging on Loadable ModulesInserting software break before module initialization

At insmod, execution of software break transfers control onto ICE. With that address, .text is resolved. .data and .bss are resolved using informoation embedded around SBP.

As this method does not depend on Linux, the same method has also been applied for other systems using relocatable object.

#define module_init(x) \int init_module(void) __attribute__((alias(#x))); \static inline __init_module_func_t __init_module_inline(void) \{ return x; }

#define module_init(x) \ int init_module(void) \ { __kmc_module_debug_start(); return x(); } \ static inline __init_module_func_t __init_module_inline(void) \ { return x; }

asm(" .text"); asm(" .long 0x4c434d81"); asm("__kmc_driver_start:"); asm(" .long 0xdeeedeee"); asm(" .long 0x4c434d80"); asm(" .long __kmc_driver_name"); asm(" .long __kmc_driver_tmp");





10

Resolving Relocation （ Shared Library ） PARNTER Obtains Info from Kernel

Referring task_struct file_struct mm_struct

of the process, /proc/(pid)/maps-like info is obtained.With the info obtained, .text, .data and .bss are

relocated.





11

Resolving On Demand Paging

Send program from ICE to cause read access at desired memory address whose page is not on memory

Kernel loads the page User does not have to care about and

PARTNER-Jet automatically does this operation PARTNER-Jet analyzes various info in kernel to

make sure to access memory





12

Nikkei Electronics 2006/1/2 IssueKMC”Virtualizing Technologies to Support Debugging on Linux and

Multicore Environment” P121Kernel Debugger Engine UserLogical Address

Space on MMUKernel loads pages of exec code

Debugger engine stops execution of process

User specify a break point on source code

Debugger engine let execute the code calling code with specified break point

Page fault occurs and kernel loads page of new exec code Set the break point at the

logical addr corresponding to user specified location

User orders to resume processDebugger engine

restarts the processKernel executes the process to the breakpoint Debugger engine stops the

process at the breakpoint

Fig.5 Intentional Page Fault

Linux kernel does not load all execution code onto main memory at once. “Page fault”, detecting that code to execute has not been loaded on memory,causes the execution code to be read into. In order to make it possible to set breakpoint at point kernel has not yet loaded on memory, we have intentionally made page fault occur. We have implemented the function to call beforehand execution code, which correspondis to the user specified breakpoint.





13

Resolving Multiplexed Memory Spaces ICE Manages Virtual Spaces

While debugging process, tracks the management structure of the process（ task_struct ）

When breakpoint reached, keeps the state of the process at that timing Minimal Kernel Patches

Essential to dispatch process of real time trace （ sched.c ） . And some others.void __kmc_schedule(prev,next)

struct task_struct *prev, *next;{ int index_next;

index_next=__kmc_schedules_index & (KMC_MAX_SHEDULE_LIST-1); ++__kmc_schedules_index; __kmc_schedules_list_pid[index_next]=next->pid;}

・・・・・・

#ifdef CONFIG_KMC_TRACE_EXT __kmc_schedule_call(prev,next);#endif /* * This just switches the register state and the * stack. */ switch_to(prev, next, prev); __schedule_tail(prev);





14

Demonstration





15

Tricks for Linux Attaching to Running Process

Possible at any timing, while running, while blocking in kernel, or while bloking in process

ICE saves PC which has been saved as context, and modifies it to address in attach area. When rescheduled, attach area is executed and breaks.

Automatic Loading of Debugging Information Loading automattically all debugging information of shared libraries which the

process is using, and relocating automatically Analyzing the control structure of the process

Function Trace for Process Even when stopped in kerenl space, backtrace of the specified thread can be

referred ICE refers to the PC, SP, FP and stacks saved as context and replays

Application Mode （ See following slides ） Debugging while keeping other processes than target process alive





16

ICE Dedicated to Process Can debug only processes on virtual spaces with

external ICE Memory swapping is fully supported

Debugging of Target Process Only Debuggable with other processes than target and

kernel to keep running Effective Use of ICE Functions

Hardware breakpoints, traces

Application Mode- Virtual ICE Technology -





17

ターゲットシステム

Application Mode- Virtual ICE Technology -

Operating System

CPU

ICE PARTNER-Jet

PARTNER Debug Control DLL

Process1

DebuggerPARTNER/Win

VirtualICEDebugger

PARTNER/Win

Process2 Process3

VirtualICE VirtualICE

Debuggingof Process1



Debugging of kernel & modules

DebuggerPARTNER/Win

DebuggerPARTNER/Win

DeviceDrivers

JTAGTarget System

Debugging Host PC





18

Profiling Features for Linux Execute Profile from JTAG ICE

No target memory used to keep trace data (for memory-tight embedded systems)

Implemented with Minimal Patches to Kernel 4 patches in 3 source file for ARM11 Efficient communication by using ARM DCC

static inline unsigned int jtag_com_stat11(void){ unsigned int stat; __asm__("mrc p14,0,%0,c0,c1":"=r"(stat)); return stat;}

static inline void jtag_com_put11(unsigned int data){ __asm__("mcr p14,0,%0,c0,c5"::"r"(data)); return;}





19

Profiling for Linux Key Features

Profilable integrating both kernel and processes At 100usec sampling, load on target is 1/100 or less （ in case

of ARM9 ） Get transitions of generation/distruction/switching of processes

and threads, displaying with process IDs/process names Get execution time ratio per process Get execution time ratio of each function in processes/kernel Get execution time ratio of processes/kernel Referring on debugger, as well as saving data in CSV format to

analyze yourself withEXCEL etc.





20

Being Open Source





21

Efficient Debugging on Embedded Linux Fully Enjoy Being Open Source

Debugging environment becomes more powerful with combination of ICE + kernel modification, than with ICE only

It is easier for us, tool vendor, to develop tools for Linux because of open source

PARNTER-Jet is monitoring major data structures in Linux kernel （ task_struct,mm_struct,file_struct etc. ）

Matching of trace of ETM/AUD etc. and souce code enables to investigate in detail of Linux dynamics





22

Interaction of Linux and Tools

Transfers trace and logging data from kernel and processes through JTAG

Memory freeze, trace freeze, and analysis functions at specified location in kernel

App A App B App C

KernelDriversJTAGICE

Analyzing App





23

ContactHeadquarter

2-44 Ooe-nakayama-cho, Nishikyo-ku, Kyoto, 610-1104, JapanTEL +81-75-335-1050 FAX +81-75-335-1051

Tokyo OfficeR Bldg 5F, 2-14-4 Shimbashi, Minato-ku, Tokyo, 105-0004, JapanTEL +81-3-5157-4530 FAX +81-3-5157-4531

URL:http://www.kmckk.co.jp/[email protected]@kmck.co.jp




http://www.kmckk.co.jp/eng

mailto:[email protected]

mailto:[email protected]

Embedded Linux Debugging with JTAG ICE

Documents

Transcript of Embedded Linux Debugging with JTAG ICE