Hands-on Workshop on How To Debug Codes at the Institute•Available: linux and unix systems •...
Transcript of Hands-on Workshop on How To Debug Codes at the Institute•Available: linux and unix systems •...
-
1
Hands-on Workshop on How ToDebug Codes at the Institute
H. Birali Runesha, Shuxia Zhang and Ben Lynch
(612) 626 0802 (help)[email protected]
October 13, 2005
Outline
• Debuggers at the Institute
• Totalview– Starting TotalView
– The TotalView GUI and basic windows
– Data management and visualization
– Breakpoints and Action points
– Parallel debugging: MPI, OpenMP, etc.
-
2
Debuggers at the Institute• Write statements• dbx• pdbx• Xpdbx• xldb• Pedb• GNU gdb
• CVD• Intel IDB• DDD• Totalview• Other tools: Compiler
options, profilers,hardware counters
• dbx is a command line debugger for serial andparallel Fortran and C programs respectively
• Most useful Commands:stop at linenumber # set a breakpoint at the given linenumberstop in myroutine # set a breakpoint at the given routinerun < infile # run your program, reading from the file infilenext # to execute the next step of your codeprint var # see the value of a given variablewhatis var # determine the data type of a given variablecont # continue execution from where it stopped
• Compile your program with –g option• Start dbx with or without the executable as
an argument.• More info: man dbx, on dbx prompt use help• Available: unix systems: IBM, SGI and SUN
dbx
-
3
• Command-line parallel debugger built on dbx• IBM debugger for parallel programs on AIX systems and
accepts all flags supported by POE• Supports C, C++, Fortran 77, Fortran 90 programs• Compile the code with -g flag• Setup POE environment variables, e.g.:
setenv MP_BUFFER_MEM 28MB setenv MP_EAGER_LIMIT 4090kb• Run pdbx:
pdbx a.out [program_options] [poe options] or
pdbx -a poe process id [limited poe options]• Available: IBM Power3 and Power4
• More information: man pdbx
pdbx
• Parallel debugger with graphical interface• Load executable after starting xpdbx. Use
the File menu• Use X resources to customize the xpdbx
window– The default resources for xpdbx are listed in
/usr/lib/X11/app-defaults/Xpdbx• Needs the –rmpool option on the IBM
SP/Power4 for an interactive run• Can set break/trace points
xpdbx
-
4
• Serial and multi-thread debugger withgraphical interface
• xldb a.out– Window pops up with source, etc
• Group of blue bars at the top right– Click on bar to open window– To minimize window, click on bar at top to get
menu, click on “minimize”• To set breakpoint, click on source line• To navigate, see “commands” window
xldb
• Graphical user interface based parallel debuggerfor Fortran and C programs
• Same as xpdbx• pedb a.out
– Window pops up with source, etc
• To set breakpoint, double-click on source line. Todelete breakpoint, right-click icon in left margin,select “ delete”
• To navigate, use buttons below source listingStep over step into step return Continue halt play stop
• “tasks”(processes for MPI) are chosen usingbuttons in “task” window
• Use “Find” to find a specific variable
pedb
-
5
• gdb is a command line debugger for serialprograms
• Support C, C++, and Fortran77 programs• Available: linux and unix systems• Compile the code :
g77 –g my.f or gcc –g my.c• Start debugging:
gdb ./a.outor
gdb ./a.out core # to include the coreor
gdb ./a.out 1234 # to attach GDB to process 1234
GNU gdb
• Most useful Commands:quit: Exit from GDB.
break [file:]function: Set a breakpoint at function in file. run: Start your program bt: Backtrace: display the program stack. print expr: Display the value of an expression. c: Continue running your program (after stopping)
stop:set a breakpoint at line number or routine
• Getting help:– (gdb) help– http://sources.redhat.com/gdb/onlinedocs/gdb
_toc.html– man gdb
-
6
• CVD - Invokes the WorkShop Debugger• a source-level debugging tool with GUI• set various types of breakpoints, watchpoints, and so on.• view variables, expressions, structures, arrays, call tree• debug Ada, C, C++, Fortran 77, and Fortran 90 programs.
• Available on SGI system running Irix Operating systems(SGI workstations and Origin)
• To invoke the Debugger for a Fortran 77 program:$ f77 -g -o myprog myprog.f$ cvd myprog # serial job
or $ cvd myprog core # serial job with core dumpor $ cvd mpirun -args -np 2 a.out # For an mpi job
• More information:• Manpage: man cvd• http://techpubs.sgi.com/library/tpl/cgi-bin/init.cgi
CVD
CVD and Other Workshop Tools:• Build Analyzer (cvbuild) - Invokes the Build Analyzer that
provides a graphical display of build dependency information.• Parallel Analyzer (cvpav) - Invokes the Parallel Analyzer that
presents information about MP Fortran programs.• Performance Analyzer (cvperf) - Invokes the Performance
Analyzer that lets you specify the name of a performance taskto enable proper data collection for your experiment.
• Static Analyzer (cvstatic) - Invokes the Static Analyzer thatlets you display source code information about your code andfacilitates the setting of breakpoints.
• Tester (cvxcov) - Invokes the test coverage tool that lets youperform dynamic test coverage over any set of tests.
-
7
• Intel debugger are part of the compiler installation• Support C, C++, Fortran 77/90• Both dbx (default) and gdb like interfaces• Documentation:
– http://www.intel.com/software/products/compilers/docs/linux/idb_manual_l.html
• Debugging of optimized code generated by intel compilersdoes not work
• Does not support multithreaded/multiprocess applications.claims do support mpich programs (untested)
• Available on the SGI Altix and systems with Intel compilers
Intel IDB
• Graphical debugger• Part of Redhat distribution• Works best with gdb• Can hook in idb via: ddd --debugger “idb” -- dbx ./a.out• Multiple panes for
– Data/array browsing– Source code– Assembler code– Command line interface
• Use help menu for details
DDD
-
8
Other Debuggers
• Portland Group PGDBG graphical debugger• LF95: Fujitsu debugger FDB
• Write statements!• Editors
Totalview
-
9
Introduction• Debugger for Unix/Linux platform• Supports Fortran, C/C++ programs• Serial, multi-threaded, parallel programs
employing multiple processors• MPI, MPICH, OpenMP, PVM, SHMEM, Global
arrays, and UPC• User interface and debugging style remain
uniform and consistent across platforms andprogramming models.
Starting TotalView
• Compile with –g option• Starting Totalview on MSI machines:
module load totalview• On a new process: % totalview a.out -a • On a core file:
% totalview a.out core
• To attach to a running process:% totalview
-
10
o• File > New Program dialog Box• Short cut: ctrl-N
• F1 function keytells Totalview todisplay help infoabout the window
TotalView Windows
ProcessWindow
RootWindow
UnattachedProcessesWindow
DataWindows
•Menu driven•Alternative keyboard shortcuts for those who prefer them•All 3 mouse buttons are used
Test drive:
-
11
Root WindowProcess name
Number of threads
Expandlist
Process/thread status
Process Window
StackTrace pane
Sourcepane
Threadpane
Localvariablesfor theselectedframe
ActionPointspane
-
12
Current function and source fileGridded box isa possible sitefor a breakpoint
Selectto
setoneCurrent
point ofexecution
• Dive on a source word to get more information
• Select a line to use Run to selection command
• Select or dive on a line number to set an action point
Source Pane
Stack Trace Pane
Stack framevalue
• Select a line tochoose a new stackframe
• Just dive on aroutine andTotalView showsthe routine in theSource Pane andits variables in theStack Frame Pane.
-
13
Stack Frame Pane
Names ofarguments
Names offunction’slocal variables
Values of variablesStack Frame pane in the processwindow contains current call stack
Dive on a variable to bring up adata pane (and follow a pointer)
Starting, Stopping, & RestartingYour Program
• Method 1– Set a breakpoint– Select Go on the icon bar– The program starts executing and stop when it reaches
the breakpoint• Method 2
– Select step– Totalview starts and then stops it immediately before
the first statement in your main function• To stop a running program, select halt• To restart a program, select the Group > Restart
command
-
14
• Stepping through a Program– Use the Step and Next commands. Step and Next both
execute the current line. The difference between themis that if the line has a function call, Step dives into thefunction while Next executes it.
– If you want to get to a line without individually steppingeach intervening line, select the line, and then selectRun To.
• Stepping Out of a Function– If you've stepped into a function and want to pop out to
the statement that called it, select the Out command.
Data Management
-
15
Seeing a value of a variable• Method 1 Dive on a variable displayed in the
Source Pane. The variable appears in a VariableWindow.
• Method 2 Scroll to the variable in the StackFrame Pane. If you want to display a compoundvariable like an array or structure, dive on it.TotalView responds by displaying information in aVariable Window.
• Method 3 Use the View > Lookup Variablecommand. The variable can be global or local. Ifyou've defined the same variable in more than oneroutine, TotalView displays the variable containedin the current stack frame.
Displaying Data
Array Data Window• Array has an
actual type aswell as adeclaration type
• Arrays have aslice field thatyou can edit tospecify thedimensions todisplay
• You can updatedata by Selecting(left mousebutton) the valueand editing it
Variable Window
-
16
Array Slice
• An array sliceindicates whichportion of an arrayyou want to see. I.e.:see elements 3 to 8
• Fortran example-TotalView displays aone-dimensional arrayby using (:). It uses(:,:) if it is displayinga two-dimensionalarray, and so on
Data Array FilteringGo to theFilter line andtype a filterexpression
You canspecify theelements of anarray you’reinterested in
This examplerequests allelements withvalues >0.8
-
17
Data Array Filtering (cont.)
• Filter: 5:998 will display elements between5 and 998
• Filter:== $nan• Filter:== $inf
Visualization• In an array data pane select
Visualize from the menu
• The Visualizer is automaticallystarted
• Arrays of rank up to two may bevisualized
• Unlike data panes visualizationsare not automatically updated
• $visualize in EVAL points foranimated display
-
18
Action Points
Action Point Types“Action point” is a generic name for severaltypes of breakpoints:
– Breakpoint stops execution of processes andthreads that reach it
– Barrier breakpoint holds each process thatreaches it until all processes in the groupreach it
– Conditional breakpoint stops code if conditionis met
– Evaluation point is point at which user-definedcode fragments get executed
– Data watchpoint
-
19
Setting a Breakpoint at a Function• Method 1: If you can see the function, click on a
line number within the function.• Method 2: Use the View > Lookup Function
command to find the function. After the functiondisplays, select a line number.
• Method 3: Select Action Points > At Locationand then type the function's name in the dialogbox.
Setting a Data Watchpoint• A data watchpoint is a type of breakpoint that
monitors a variable's value. Unlike other actionpoints, it isn't set on a source line. Instead, it'sset on the memory location where your programstores the variable. When the contents of thismemory location change, TotalView stopsexecution and displays the source line thatchanged it. This means you should only setwatchpoints on global variables or on malloc'dmemory.
• Set a watchpoint by first diving on the variable todisplay its Variable Window, and then select theTools > Watchpoint command.
-
20
Printing Something at a Breakpoint• After creating a
breakpoint, right-click onthe sign, and then selectProperties from the popupmenu.
• In the Properties dialogbox, select Evaluate andthen type a printstatement. For example:
printf("The value of foodivided by 20 is:%d\n",foo/20);
• When code is associatedwith a breakpoint, thebreakpoint is called aneval point.
Stopping Your Program Using a Condition• Create an eval point (this was discussed in the
previous section), entering a condition instead ofa statement. For example, here's a condition thatstops execution when the value of the variable i inthe eval point's line is equal to 30:
if (i == 30) $stop• One of the big differences between an eval point
and a breakpoint is that execution doesn't stopunless you tell it to. That is, if you don't add a$stop to your eval point, TotalView just continuesexecuting.
•
-
21
Stopping Your Program Every "x" TimesTotalView Executes a Line
• Create an eval pointthat uses the $countstatement to tellTotalView how manytimes the line shouldexecute. For example:
$count 20• $count is another of
TotalView's built-infunctions. $stop isimplied when you usethis statement.
Saving Your Action Points• You usually want this to happen
automatically and, unless someone haschanged things, TotalView automaticallysaves them. If it isn't saving them, go tothe Options Tab in the File > Preferencesdialog box and select Save preferencesfile on exit.
• While you're looking at preferences, youmight want to look at the other things youcan set.
-
22
Action Point List PaneEnabledbreakpoint
Disabledbreakpoint
Breakpointlocation
• Select on action point icon enables/disables it• Dive on action point refocuses source pane to
location; this behaves like a bookmark
Parallelism in TotalView
-
23
Supported Parallelism
TotalView supports all major parallelprogramming paradigms– MPI– OpenMP– Pthreads– PVM– HPF– Shmem
TotalView Multi-Process Functionality• TotalView performs actions on groups of
processes– Start / stop– Share breakpoints across executables– Take action on multiple processes when at
breakpoint– Attach to multiple running processes– Single step groups of processes– Set barrier breakpoints
• Easily view and manipulate variables processes– Laminated variables for threads and processes– Distributed HPF arrays
-
24
Debugging MPI Programs on the IBM SPwith Totalview
% totalview poe -a a.out
[program_args][PE_args]
• Automatically attaches to all of theMPI processes
• TotalView can display MPI message
queues
Displaying a variable in all processes or threads• Need to view or update the
value of a variable in all ofthe processes or threads atonce
1. First display an instance of thevariable in a Variable Window
2. View > Laminate > Process, orView > Laminate > Thread
3. View > Laminate > None,delaminates the window
• Example on the right is foran OpenMP
-
25
Other tools
• Profiler: Xprofiler:– GUI based IBM performance profiling tool– Graphical function call tree– Flat report: line by line CPU time– Compile and link your program with –pg– Run your code => produce gmon.out– Invoke xprofiler xprofiler executable [-s] gmon.out.
Other Tools
-
26
• Hardware counter: Hpm on the IBM systems– Hardware performance counters
• Cycles• Instructions• Float point and fixed point operations• Loads/stores• Cache misses• TLB misses• Etc
– Supports MPI, OpenMP and threaded applications– Multiple instrumentations and derived metrics
• MSI– www.msi.umn.edu– Help: help@,msi.umn.edu 612 626 0802
• Totalview– Visit www.etnus.com– Online documentation at
www.etnus.com/Support/docs/index.html