© 2011 InHand Electronics, Inc. – Troubleshooting Real-Time Software Issues Using a Logic...

© 2011 InHand Electronics, Inc. – www.inhand.com

Troubleshooting Real-Time

Software Issues

Using a Logic Analyzer

Dave Stewart

Director of Software Engineering

InHand Electronics

[email protected]

www.inhand.com

Embedded Systems Conference Chicago 2011 Class ESC 217

Dave Stewart, ESC Chicago – June 2011

Troubleshooting Real-Time Issues using a Logic Analyzer © 2011 InHand Electronics, Inc. – www.inhand.com 2

What are The Tough Bugsto Debug in Real-Time Systems?

Glitches Timing and Synchronization Problems Driver Errors Misbehaving Interrupts Memory Corruption Priority Inversion Performance Issues Hardware Errors



Limitations of Traditional Debugging

Print Statements There is no console, so Print Statements don’t work Print Statements are too slow thus provide insufficient information Print function significantly affects real-time performance Writing debug output to a serial port changes the timing too much Adding print statements changes program behavior Can’t measure performance at a fine granularity Max 50 to 100 print statements per second The code crashes, but there is insufficient feedback as to where

Symbolic Debuggers (e.g. IDEs, via JTAG, ActiveSync, or other comm link) A symbolic debugger or emulator is not available Stepping through the code makes the program behave differently Breakpoints will “break” real-time performance There is real I/O, it doesn’t work Debugger doesn’t deal with interrupts properly There might be a race condition or other synchronization problem



Solution:Use a Logic Analyzer for Real-Time Issues

Most detailed view of your code Microsecond view of software Easily 50,000 debug data points per second Info is time-stamped for timing assessment

Real-Time Can use it for interrupts and I/O drivers Impact on real-time execution is negligible Identify temporal relationships among tasks Monitor interrupts and how they may affect execution “Watch” variables without changing execution time

Anomalies Spot anomalies and different patterns of execution Obtain sufficient proof that a problem is hardware, not software Fine-grain timing measurements to identify performance culprits



Logic Analyzer for DebuggingShould I use it all the time?

Using a Logic Analyzer is NOT easy Use Print Statements and Symbolic Debuggers to solve

easy and non-real-time problems first Add to your repertoire of available tools to solve hard

problemsSolve functional problems using print statements.

If necessary, run the functions on the desktop, and debug them there. Only move to embedded environment when it is working well.

Use Symbolic Debuggers mainly for Tracing through code that fails in a consistent manner Post-Mortem debugging of crashes, to view all variables



Sample Logic Analyzer OutputTiming Diagram

This output format will be used for most examples in this class.

More details interpreting this diagram to follow.



Sample Logic Analyzer OutputState Listing

count trig mode16 mode data time-relative time-absolute



Logic Analyzer Features

Preferably at least 8 channels 16 to 24 channels is nice to have, to monitor many additional signals under 7 channels still useable, but it prevents use of some techniques

Large memory depth in M-Samples, not K-Samples

Multiple Views Timing Diagrams State Listings Decoding of Serial Protocols Useful features: Search, Filtering, and Triggering

High-Speed Interface to a PC Built-in PC, USB Memory Stick, Ethernet all OK

Common logic analyzer features that are NOT needed: High Speed. Don’t need GHz. Very slow analyzers, e.g. 10MHz, are OK. Many channels. Don’t need 64 or 128 channel analyzers.

Low cost USB Logic Analyzer Pods are OK A $25,000 logic analyzer is nice to have and will have many extras that could be

useful• Examples in this presentation use the Tektronix TLA700

What is discussed in this talk can be accomplished with a $400 USB pod.• Examples in this presentation use the Intronix LogicPort



Target Setup: Interconnections

Logic

Analyzer

(Assuming

8-bit

channels)

System

Under Test

D0-D7Ch.1

RX/TX

Ch.2I2C

SPI

Keys

This is an example setup given 16 bits.

With fewer bits, connect whatever you can.



An AsidePrint Statement Debugging

Forms the basis for logic analyzer methods



An AsidePrint Statement Debugging

myfunc() {code hereprintf(“I got here\n”);more code hereprintf(“Going to call yourfunc()\n”);result = yourfunc();printf(“My result is %d\n”,result);etc

}

I got here

Going to call yourfunc()

My result is 384



Print Statement Debugging: Problems

A Lot of typing Minimal information per statement Hard to separate between debug and normal print

statements Prone to errors Cannot easily disable them Ultimately very inefficient



Debug Macros instead of Print Statements DEBUG_WHERE()

#define DEBUG_PRINT fprintf(stderr,

#define DEBUG_WHERE() \DEBUG_PRINT \“[%s:%u-%s]\n” , \__FILE__,__LINE__,__FUNCTION__)

Complete .h file for all macros given in this class are in the paper

that accompanies this presentation on the conference website or CD.

Note: Not all compilers support __FUNCTION__.



Print Statement Debugging

myfunc() {code hereDEBUG_WHERE();more code hereDEBUG_WHERE();result = yourfunc();DEBUG_INT(result);etc

}

[myfile.c:3-myfunc]

[myfile.c:5-myfunc]

[myfile.c:7-myfunc] result=384



Debug Macros instead of Print StatementsDEBUG_INT()

#define DEBUG_INT(_var) \DEBUG_PRINT \“[%s:%u-%s] %s=%d\n” , \

__FILE__,__LINE__,__FUNCTION__ \#_var,_var)

result = 384

DEBUG_INT(result)

[myfile.c:7-myfunc] result=384



Debug Macros instead of Print StatementsDEBUG_HEX8()

#define DEBUG_HEX8(_var) \DEBUG_PRINT \“[%s:%u-%s] %s=0x%02X\n” , \ __FILE__,__LINE__,__FUNCTION__ \#_var,_var)

result = 0xA4

DEBUG_HEX8(result)

[myfile.c:7-myfunc] result=0xA4



Debug Macros instead of Print StatementsDEBUG_HEX16()

#define DEBUG_HEX16(_var) \DEBUG_PRINT \“[%s:%u-%s] %s=0x%04X\n” , \

__FILE__,__LINE__,__FUNCTION__ \#_var,_var)

result = 0x52A4

DEBUG_HEX16(result)

[myfile.c:7-myfunc] result=0x52A4



Debug Macros forLogic Analyzers

Similar to debugging with Print Statements, Except: (disadvantage) Much more difficult to use

• Learning curve could be days, not minutes

(but its worth it for the “hard” bugs that could otherwise take weeks to debug)

(advantage) 1000 times more information• Print Statement: 50 Lines/Data Points per second• Logic Analyzer: 50000 Data Points per second is easy

(advantage) Real-time view of the system• Data points are time-stamped with microsecond resolution



Basics of Logic Analyzer Debugging

bart = 0x0063;

lisa = 0x1018;

homer= 0xF04E;

marge= 0x00E5;

PDEBUG_HEX8(0x40);

PDEBUG_HEX16(bart);

PDEBUG_HEX8(0x41);

PDEBUG_HEX16(lisa);

PDEBUG_HEX8(0x42);

PDEBUG_HEX16(homer);

PDEBUG_HEX8(0x42);

PDEBUG_HEX16(marge);

See next pages for zoom of

this timing diagram.

This example assumes only a single

8-bit port is available.




PDEBUG_HEX8(0x40); bart = 0x0063;

PDEBUG_HEX16(bart);

Timing Diagram of each bit.

We’ll see later the value of

these for creating code

patterns.




Each code represents a different variable. Correlate with source code.

PDEBUG_HEX8(0x40);

PDEBUG_HEX16(bart);

PDEBUG_HEX8(0x41);

PDEBUG_HEX16(lisa);

PDEBUG_HEX8(0x42);

PDEBUG_HEX16(homer);

PDEBUG_HEX8(0x43);

PDEBUG_HEX16(marge);

bart lisa homer marge40 41 42 43




Timescale is Microseconds.

Easily add debug statements with minimal intrusion on real-time code.

In contrast, each print statement takes 2+ milliseconds.

1 usec



PDEBUG_HEX8() MacroInitialization (e.g. MSP430)

// Example, init MSP430 Port 4// Simple memory-mapped port

uint8_t *pdebug_dout8 = (uint8_t *) 0x001D;uint8_t *pdebug_dir8 = (uint8_t *) 0x001E;*pdebug_dir8 = 0xFF;// Initialize to output



PDEBUG_HEX8() Macro(E.g. MSP430)

#define PDEBUG_HEX8(_val) \*pdebug_dout8 = (uint8_t)(_val)

More details and more macros are provided in the paper

that accompanies this presentation on the conference website or CD.

bart = 0x0063;

PDEBUG_HEX8(0x40);

PDEBUG_HEX16(bart);



PDEBUG_HEX16() MacroIf only 8 bits available

#define PDEBUG_HEX16(_val) \PDEBUG_HEX8( (_val) >> 8); \PDEBUG_HEX8( (_val) )

bart = 0x0063;

PDEBUG_HEX8(0x40);

PDEBUG_HEX16(bart);



PDEBUG_VAR16() Shortcut

bart = 0x0063;

PDEBUG_HEX8(0x40);

PDEBUG_HEX16(bart);

bart = 0x0063;

PDEBUG_VAR16(0x40,bart);

#define PDEBUG_VAR16(_code,_val) \

PDEBUG_HEX8( _code); \

PDEBUG_HEX16(_val )



Test your MacrosExample to test PDEBUG_HEX8()

Always test your macros before using them.

PDEBUG_HEX8(0xFF);

PDEBUG_HEX8(0x01);

PDEBUG_HEX8(0x02);

PDEBUG_HEX8(0x04);

PDEBUG_HEX8(0x08);

PDEBUG_HEX8(0x10);

PDEBUG_HEX8(0x20);

PDEBUG_HEX8(0x40);

PDEBUG_HEX8(0x80);

PDEBUG_HEX8(0x55);

PDEBUG_HEX8(0xAA);



Test your MacrosExample to test PDEBUG_HEX8()

Allows you to test logic analyzer setup too.

If color analyzer, adjust colors to your liking too.



PDEBUG_HEX32() MacroIf only 8 bits available

#define PDEBUG_HEX32(_val) \PDEBUG_HEX8( (_val) >> 24); \PDEBUG_HEX8( (_val) >> 16); \PDEBUG_HEX8( (_val) >> 8); \PDEBUG_HEX8( (_val) )



PDEBUG_HEX32() Macro

mickey = 0x0009B900;

donald = 0x0000CFB8;

PDEBUG_HEX8(0x44);

PDEBUG_HEX32(mickey);

PDEBUG_HEX8(0x45);

PDEBUG_HEX32(donald);



PDEBUG_HEX8() Macro(PXA270)

// Example, PXA270, use LCD data bits LDD6..LDD13// 32-bit architecture, uses GPIO2// Separate registers to set/clear bits

uint32_t *pdebug_set8 = (uint32_t *) 0x40E00020;uint32_t *pdebug_clr8 = (uint32_t *) 0x40E0002C;uint32_t *pdebug_dir8 = (uint32_t *) 0x40E00014;*pdebug_dir8 |= 0x000000FF; // Init to output



PDEBUG_HEX8() MacroInitialization (PXA270)

#define PDEBUG_HEX8(_hex8) \{ uint32_t h32 = (uint32_t) hex8; \

uint32_t sg2,cg2; \\

sg2= ( h32 & 0x000000FF); // set bits \

cg2= ((~h32) & 0x000000FF); // clear bits\ *_gpsr2 = sg2; \

*_gpcr2 = cg2; }

Note race condition between set and clear operations; if logic analyzer samples between the two, then it will capture invalid data that needs to be ignored.

Option, with an extra bit, pulse it once valid data is written, and only look at data when the extra bit indicates it is valid.



Logic Analyzer Debug Macros

There are many possibilities for defining macros

Use macros to enable a common interface across platforms

For each new platform, define the macros in a hardware-dependent manner, and place in a .h file

Simply include a different .h file for each platformAdditional macro examples

Paper that accompanies this presentation includes additional macro examples not covered here.



Tracing Drivers, Low-Level Code, and Real-time Code

Recall prior example using Print Statements

myfunc() {

code here

DEBUG_WHERE();

more code here

DEBUG_WHERE();

result = yourfunc();

DEBUG_INT(result);

etc

}

[myfile.c:3-myfunc]

[myfile.c:5-myfunc]

[myfile.c:7-myfunc] result = 384




#define PDEBUG_WHERE() \PDEBUG_HEX8(0x11); \PDEBUG_HEX8( (__LINE__) >> 8); \PDEBUG_HEX8( (__LINE__) )

#define PDEBUG_INT(code,_val) \PDEBUG_WHERE(); \PDEBUG_HEX16(_val);




myfunc() {

code here

PDEBUG_WHERE();

more code here

PDEBUG_WHERE();

result = yourfunc();

PDEBUG_INT(result);

etc

}

<11><00><03><11><00><05><11><00><07><01><80>

384 = 0x180

Line Numbers in Hex

<11> just a code to spot line numbers

The output is cryptic compared to using print statements. Hence,

continue using print statements if you can.

But use the logic analyzer to trace code

when print statements don’t work.

(Some analyzers will print decimal!)



Fine-Grained Code Optimization

ALWAYS measure execution time of a code segment before and after optimization. If an optimization breaks a pattern that the compiler

recognizes, it could in fact make things take longer. Not all optimizations are obvious.

• What is faster, pointers or index arrays?• What is faster, case statements or if-then else?• What is faster, for loops or while loops?• Measure to find out.

Use PDEBUG Macros to measure code segments PDEBUG_HEX8(x) before the code segment PDEBUG_HEX8(x+1) after the code segment X is unique in the code, to enable measuring multiple code

segments at once



Performance Anomalies

Quite often, performance is not as expected Why Not?

Scatter PDEBUG_HEX8() statements throughout code Start and stop of each function and thread is a good

starting point. Often, just keep all the prior debug statements in place, they

may provide the data needed.

Run the code, and monitor on the logic analyzer. Look for patterns of behavior. Identify patterns that use too much CPU. Verify what is happening when analyzer shows periods of

inactivity. Monitor execution time of interrupts.

Some examples on next few slides.




Green is commmunication.

We can clearly see communication stopped for a period of time.

Why? Is that normal?




See the Pattern?

Four short communication packets, one long one, then none for what seems like two or three system cycles.




What line of code executes immediately after PDEBUG_HEX8(0x9F)?

Whatever it is, it takes a LONG time.

If there is a performance issue, focus on that code first.




This is an example of code executed with PDEBUG_WHERE() plus a few other PDEBUG statements between every two

lines of code.

What is happening on Line 51 (0x33)?



Using Additional Signals

Create second PDEBUG_HEX8-like macro E.g. call second one PDEBUG_MODULE() Upon entry or exit to any function in a module, call this one. Use PDEBUG_HEX8 to trace and print variable data. Makes it easier to correlate logic analyzer output to source

code

Reserve some bits as triggers A lot easier to setup logic analyzer to trigger on an

individual bit, rather than a sequence of bytes.



Additional Signals Example Keys and Scope

Monitor key presses They often serve as trigger points Ensure that if debouncing is needed, it is properly

handled

Oscilloscope Some logic analyzers have integrated

oscilloscopes Use for instantaneous analog measurements that

are correlated with the PDEBUG outputs Aids in software design and troubleshooting of

potential hardware issues



Additional Signals ExampleLogic Analyzer Screen Captures Follow

Next few pages show a logic analyzer setup with: PDEBUG_HEX8(): 8-bit debug port (yellow) PDEBUG_MODE(): 3-bit [cpu_speed/100MHz] (green) PDEBUG_TRIG(): 1-bit trigger (magenta) 2 keys on keypad (cyan) serial TX/RX (grey) instantaneous current draw (green analog, lower means

more current) input voltage (red).

Each page is a zoom of the prior page. The zoom area is shown by the magenta rectangle.



Setup ExtrasExample

zoom



Setup ExtrasExample



Setup ExtrasMonitoring Serial Transmissions

Serial Communication Protocol Analyzers exist to monitor integrity of data.

• Built-in to some logic analyzers In many cases, you don’t need anything that complex. Rather, you just want to know how many bytes, and when. Use PDEBUG_HEX8() macros to print the bytes being

transmitted.



Serial CommunicationE.g. I2C

CLK

Data

The code that

immediately follows the

I2C transfer can then be

traced line by line.

Some Analyzers will convert clock/data to HEX values and

validate bus protocols. (Example later)



Serial CommunicationE.g. I2C

Actual Signals. Verify:

- Clock Rates

- Glitches and Noise

- Sequence

- Timing between edges of different signals

Protocol. Verify:

Number of Bytes Transmitted –

Acks and NAcks –

Start and Stop Bits –

Value of each byte –



Serial CommunicationReal-Time Gotchas

Code will “write” register, but serial transmission is usually NOT done when code continues, as operations continue in parallel.

For many hardware platforms, then the code in fact is pre-loading the NEXT byte, while previous byte is being transmitted, but then it waits on a status flag of the current byte to be completed.• This could get confusing when creating the code or

debugging the low-level read and write functions.• Use logic analyzer coupled with PDEBUG to view precisely

what is happening in the code.



Serial CommunicationMiscellaneous Notes

Use built-in interpreters provided by logic analyzer software to analyze the protocol• I2C, SPI, RS232

For more complex protocols, use a protocol analyzer, not a logic analyzer.• E.g. USB or Ethernet• Use a oscilloscope to verify signal integrity• Use a protocol analyzer to validate the protocol

Collect data in parallel with PDEBUG information, to see relationship between code execution and serial transmission



Summary

Logic Analyzer provides a Window into Embedded Code A window with micro-second resolution

Use it in addition to other methods It is complimentary to using print statements and emulators

Defining the right macros requires some creativity Many possibilities. This class only scratches the surface.

Look for Anomalies and anything that just doesn’t look right These are usually the clues to where the problems lie

Use it to troubleshoot bugs or measure performance Wide variety of tricks and techniques using the logic

analyzerGood Luck!

This is a tool to tackle the toughest embedded system bugs. Hopefully weeks will become days, and days will become

hours

© 2011 InHand Electronics, Inc. – Troubleshooting Real-Time Software Issues Using a Logic...

Documents

Transcript of © 2011 InHand Electronics, Inc. – Troubleshooting Real-Time Software Issues Using a Logic...