The Blue Waters batch system provides an interactive mode for interactive debugging and/or optimization activities. Long running jobs or production runs should not be run from interactive sessions. To start an interactive session, use the following command
The -I option indicates to the batch system to start an interactive job. You must also specify the number of nodes as shown in this example (this would be for a 1-node interactive job) and the node type (xe for CPU nodes, xk for GPU nodes). Also, you must specify the default number of processes per node (ppn), which is limited to 32 on an XE node or 16 on an XK node. Other options accepted by qsub can be used in the interactive session.
For example, suppose you want access to 2 nodes for testing. Run the following command on a login node:
The job will go into the queue. When the command returns, you'll be in a shell running on a "MOM" node. A MOM node is a shared service resource that manages job execution. Use
Following is an example of testing application performance by trying different
where job.inp is the application input data, and job16.out is the application output. Note that the value of -N can be smaller than the value requested in the qsub ppn parameter, but it cannot exceed the requested ppn.
After this job finishes, type in
Also try other aprun options. See "man aprun" for details. Determine which options give the best performance and then use those in the production run. The benefit of trying several runs from the same interactive job is to get the job executed each time on the same nodeset, thus minimizing the performance variability due to non-reproducible job placement. Note that such sequence of aprun commands can be programmed into the batch script. The advantage of an interactive session is the flexibility of experimenting without the need to wait for a new session to start.
After the requested time has passed (1 hour in this example), the connection will be closed. Logging out of the MOM node will terminate the job early.
If the intended use of the interactive session is to work directly on a compute node (for example: to access the GPU in an XK node), consider using CCM mode, which provides the opportunity to do computations interactively on a dedicated compute node rather than on a shared MOM node.
Scheduling interactive jobs is a subject of the standard job queue policies. Submit the interactive job into debug or high queue in order to boost the job priority over that provided by normal queue if the wait time in the normal queue becomes excessive due to high machine load.
If your PBS script involves elaborate computation and I/O operations it might need to be corrected before being used on Blue Waters. Seek help at email@example.com if needed.