DDT: Distributed Debugging Tool
DDT from Allinea Software is a parallel debugger that can be used for scalar, multi-threaded and large-scale parallel applications. The Allinea DDT web page and users guide is a good resource for learning more about some of the advanced DDT features. Helpful videos and blogs are available from the Allinea website.
How to use DDT
Since DDT is GUI-based and does not provide command line interface X11 forwarding must be enabled for your login session. This can be done by passing -Y flag to ssh:
> ssh -Y bw.ncsa.illinois.edu
NOTE: for memory debugging load the memory debugging module for DDT BEFORE linking
> module load ddt-memdebug
Add -g flag to enable the generation of debugging information used by DDT, then (re)compile your program:
> ftn -g test.f90 -o test
> cc -g test.c -o test
Starting a debug session with DDT
- submit a job through DDT
- manually launch a program with DDT
- attach DDT to a running program
- start a debug session from inside an interactive job
The first three ways begin by loading the ddt module and starting DDT:
> module unload altd ; module unload xalt > module load ddt-memdebug # (note the use of the ddt-memdebug module from above) > export DDT_NODE_SCAN_TIMEOUT=90 > export DDT_NO_TIMEOUT=1 > export DDT_PROCESS_TIMEOUT=0 > ddt
Submit a job through DDT
Submits a job, waits until the job is scheduled and starts a debug session.
Click on. A new window with expandable tabs will appear, click on to expand a tab.
Application tab is used to select a program binary, working directory, arguments and input file.
MPI, OpenMP, CUDA and Memory Debugging tabs are used to allow respective features and set parameters (e.g., number of nodes and processes per node for an MPI program).
Queue Submission Parameters tab is used to change job parameters such as wall clock time, target queue, etc.
Clicking onbutton will submit a job to the scheduler, DDT will wait for the job to start.
DDT will start a debug session automatically as soon as the job starts.
Manually launching a program
Manual launch allows debugging multi-process and multi-executable programs.
To launch a program manually click onbutton.
Select how many processes you want to debug and click on. At this point start a program or programs using the following command:
> ddt-client <path-to-program-binary>
Note, ddt-client command must be issues for each process selected at the previous window. The above command can also be used in a job submission script.
DDT will automatically start debugging session once all requested programs have been launched manually.
Attaching to a running program
To attach to a program that is already running click onbutton.
(With nodes=256 or more, start ddt from the command line with: DDT_NODE_SCAN_TIMEOUT=90 ddt )
DDT will start scanning each of 64 mom nodes to locate active jobs owned by you. If there are more than one active job DDT will find all of them. Once DDT finds the desired job select it and click onbutton.
start a debug session from inside an interactive job
Starting a debug session from inside an interactive job
To start DDT from an interactive job X11 forwarding must be enabled (-X flag):
> qsub -I -X
Once the job has started load the ddt module and start DDT with -noqueue flag:
> module load ddt-memdebug > ddt -noqueue
Click on. A new window with expandable tabs will appear. Tabs Application, MPI, OpenMP, CUDA and Memory Debugging are the same as described above.
Click onbutton to start a debug session.