DDT Offline debugging for batch jobs that crash or hang

Using DDT in offline node is fairly straightforward. DDT will generate a backtrace and also show the local variables for the current routine. Follow these steps in your job script to get DDTs offline mode to work with a program that crashes (as an alternative to Cray's ATP). You could also use this technique with a hanging job by sending it a SIGABRT or SIGSEGV via qsig or apkill .

Scenario 1: ddt -offline with a MPI cpu-only code

steps to use ddt -offline
#!/bin/bash #PBS -lnodes=2:ppn=32,walltime=00:10:00 . /opt/modules/default/init/bash # enable module commands # unload conflicting modules, disable Cray ATP module unload darshan export ATP_ENABLED=0 # load the ddt module and optionally firefox for viewing filename.html from a login node module load ddt-memdebug module load firefox # on login node # replace aprun with "ddt -offline filename -noqueue ... " use filename.txt for a text report # pass extra aprun arguments with the -mpiargs flag and double quotes export DDT_NODE_SCAN_TIMEOUT=90 export DDT_NO_TIMEOUT=1 export DDT_PROCESS_TIMEOUT=0 cd ${PBS_O_WORKDIR} # the binary was in the directory where job was submitted ddt -offline myfile.html -noqueue -n 32 -mpiargs "-N 16 -d 1" ./hello_world_bug

After the program runs, you can copy the html file back to your local machine or view it on a login node with firefox. The equivalent text file report is also shown:

equivalent output with myfile.txt
arnoldg@nid00014 13:57 ~/c ___--Cray- $ cat myfile.txt message (0-31): Launching program /mnt/abc/u/staff/arnoldg/c/hello_world_bug message (0-31): at Thu Aug 14 13:54:49 2014 message (0-31): Executable modified on Thu Aug 14 08:22:48 2014 message (0-31): Startup complete. message (n/a): Select process group All message: Current Stack message: #1 __libc_start_main (main=0x401ac0 <main>, argc=1, ubp_av=0x7fffffff0a78, init=0x5728e0 <__libc_csu_init>, fini=0x5728a0 <__libc_csu_fini>, rtld_fini=0, stack_end=0x7fffffff0a68) at /usr/src/packages/BUILD/glibc-2.11.3/csu/libc-start.c:201 (at 0x0000000000572391) message: #0 __libc_csu_init (argc=1, argv=0x7fffffff0a78, envp=0x7fffffff0a88) at /usr/src/packages/BUILD/glibc-2.11.3/csu/elf-init.c:123 message (n/a): /mnt/abc/u/staff/arnoldg/c/hello_world_bug.c is not tracked by a version control system message: Stacks message: Processes Function message: 0-31 __libc_start_main (libc-start.c:201) message: 0-31 __libc_csu_init (elf-init.c:123) message: Locals message: argc: 1 argv: 0x7fffffff0a78 envp: 0x7fffffff0a88 i: 16 message (0-31): Play message (3): Process stopped in main (hello_world_bug.c:25) with signal SIGSEGV (Segmentation fault). message (3): Reason/Origin: address not mapped to object (attempt to access invalid address) message (3): Your program will probably be terminated if you continue. message (3): You can use the stack controls to see what the process was doing at the time. message: Stacks message: Processes Function message: 3 main (hello_world_bug.c:25) message (n/a): Select process 3 message: Current Stack message: #0 main (argc=1, argv=0x7fffffff0a78) at /mnt/abc/u/staff/arnoldg/c/hello_world_bug.c:25 (at 0x0000000000401bc9) message: Locals message: a: argc: 1 argv: 0x7fffffff0a78 core: 1077483333 i: 7404 len: -933688543 name: "\253\252\252\252\252\252\032@" rank: 1431655765 size: 1077548610 message (3): Play error (Other): _pmiu_daemon(SIGCHLD): [NID 00002] [c0-0c0s1n0] [Thu Aug 14 13:55:09 2014] PE RANK 3 exit signal Segmentation fault error (Other): [NID 00002] 2014-08-14 13:55:10 Apid 238591: initiated application termination output (Other): rank 1 of 32 on nid00002 core 1 output (Other): rank 0 of 32 on nid00002 core 0 output (Other): rank 12 of 32 on nid00002 core 12 output (Other): rank 13 of 32 on nid00002 core 13 output (Other): rank 8 of 32 on nid00002 core 8 output (Other): rank 9 of 32 on nid00002 core 9 output (Other): rank 4 of 32 on nid00002 core 4 output (Other): rank 15 of 32 on nid00002 core 15 output (Other): rank 7 of 32 on nid00002 core 7 output (Other): rank 6 of 32 on nid00002 core 6 output (Other): rank 3 of 32 on nid00002 core 3 output (Other): rank 5 of 32 on nid00002 core 5 output (Other): rank 2 of 32 on nid00002 core 2 output (Other): rank 14 of 32 on nid00002 core 14 output (Other): rank 10 of 32 on nid00002 core 10 output (Other): rank 11 of 32 on nid00002 core 11 output (Other): rank 25 of 32 on nid00003 core 9 output (Other): rank 24 of 32 on nid00003 core 8 output (Other): rank 17 of 32 on nid00003 core 1 output (Other): rank 21 of 32 on nid00003 core 5 output (Other): rank 23 of 32 on nid00003 core 7 output (Other): rank 28 of 32 on nid00003 core 12 output (Other): rank 29 of 32 on nid00003 core 13 output (Other): rank 20 of 32 on nid00003 core 4 output (Other): rank 22 of 32 on nid00003 core 6 output (Other): rank 18 of 32 on nid00003 core 2 output (Other): rank 19 of 32 on nid00003 core 3 output (Other): rank 16 of 32 on nid00003 core 0 output (Other): rank 31 of 32 on nid00003 core 15 output (Other): rank 30 of 32 on nid00003 core 14 output (Other): rank 26 of 32 on nid00003 core 10 output (Other): rank 27 of 32 on nid00003 core 11 message (n/a): Every process in your program has terminated. output (aprun): Application 238591 exit codes: 139 output (aprun): Application 238591 resources: utime ~2s, stime ~21s, Rss ~6592, inblocks ~12876, outblocks ~28782 message (n/a): Select process 0 arnoldg@nid00014 13:57 ~/c ___--Cray- $

Scenario 2: ddt -offline with MPI cpu+gpu code

With ddt version 4.2.2 or later, the -offline option supports gpu debugging. The following launch command was used in the batch script to trace a kernel invocation and variable on the gpu. A snapshot of the resulting .html output follows.

ddt -offline test1.html -noqueue -n 1 \

-trace-at simpleMPI.cu:42,output[tid] \

./simpleMPI