Skip to Content

DDT: Distributed Debugging Tool

Description

DDT from Allinea Software is a parallel debugger that can be used for scalar, multi-threaded and large-scale parallel applications. The Allinea DDT web page and users guide is a good resource for learning more about some of the advanced DDT features. Helpful videos and blogs are available from the Allinea website. 

How to use DDT

Prerequisites

Since DDT is GUI-based and does not provide command line interface X11 forwarding must be enabled for your login session. This can be done by passing -Y flag to ssh:

> ssh -Y bw.ncsa.illinois.edu

NOTE: for memory debugging load the memory debugging module for DDT BEFORE linking

> module load ddt-memdebug

Compilation

Add -g flag to enable the generation of debugging information used by DDT, then (re)compile your program:

  Fortran example

> ftn -g test.f90 -o test

  C example:

> cc -g test.c -o test

 

Starting a debug session with DDT

  • submit a job through DDT
  • manually launch a program with DDT
  • attach DDT to a running program
  • start a debug session from inside an interactive job

The first three ways begin by loading the ddt module and starting DDT:

> module unload altd ; module unload xalt
> module load ddt-memdebug # (note the use of the ddt-memdebug module from above)
> export DDT_NODE_SCAN_TIMEOUT=90
> export DDT_NO_TIMEOUT=1
> export DDT_PROCESS_TIMEOUT=0
> ddt

DDT - Welcome

 

Submit a job through DDT

Submits a job, waits until the job is scheduled and starts a debug session.

Click on  Run and Debug a Program . A new window with expandable tabs will appear, click on  Details...  to expand a tab.

DDT - Run

 

Application tab is used to select a program binary, working directory, arguments and input file.

MPI, OpenMP, CUDA and Memory Debugging tabs are used to allow respective features and set parameters (e.g., number of nodes and processes per node for an MPI program).

Queue Submission Parameters tab is used to change job parameters such as wall clock time, target queue, etc.

Clicking on  Submit  button will submit a job to the scheduler, DDT will wait for the job to start.

DDT - Job Submitted

 

DDT will start a debug session automatically as soon as the job starts.

DDT Debug Session

 

Manually launching a program

Manual launch allows debugging multi-process and multi-executable programs.

To launch a program manually click on  Manually Launch a Program  button.

 

Select how many processes you want to debug and click on  Listen . At this point start a program or programs using the following command:

> ddt-client <path-to-program-binary>

Note, ddt-client command must be issues for each process selected at the previous window. The above command can also be used in a job submission script.

DDT - Connecting

DDT - Connected

 

DDT will automatically start debugging session once all requested programs have been launched manually.

 

Attaching to a running program

To attach to a program that is already running click on  Attach to a Running Program  button.

(With nodes=256 or more, start ddt from the command line with:  DDT_NODE_SCAN_TIMEOUT=90 ddt )

DDT - Attach

 

DDT will start scanning each of 64 mom nodes to locate active jobs owned by you. If there are more than one active job DDT will find all of them. Once DDT finds the desired job select it and click on  Attach to listed processes  button.

DDT - Attach - Found

 

start a debug session from inside an interactive job

 

Starting a debug session from inside an interactive job

To start DDT from an interactive job X11 forwarding must be enabled (-X flag):

> qsub -I -X

Once the job has started load the ddt module and start DDT with -noqueue flag:

> module load ddt-memdebug
> ddt -noqueue

Click on  Run and Debug a Program . A new window with expandable tabs will appear. Tabs Application, MPI, OpenMP, CUDA and Memory Debugging are the same as described above.

DDT Run No Queue

 

Click on  Run  button to start a debug session.