Login Node Policies

The login nodes are shared resources intended for editing and managing files, submitting and monitoring jobs, as well as compiling and linking. Use of scp, sftp, or rsync for large file transfers should be avoided. 

Details on acceptable use of Blue Waters login nodes and methods to enforce these policies, are discussed below. Accpetable useages are those which do not violate the terms of use agreement and do not impact the login experience for other Blue Waters users. Use of the Blue Waters login nodes that does not fit within the terms of use or impacts other users will be cause for review by the Blue Waters Project Office.

Long running processes

To regulate the use of the login nodes (h2ologin[1-4]) a process watcher has been implemented. The process watcher monitors processes running on the login nodes recording the CPU utilization per task per user per login node. 

The policy for long running user tasks are:

  1. Terminate user process over 100 hours of cputime which is our max user process hard limit.
  2. Terminate user process over 4 hours of cputime with an average cpu usage over 20%.
  3. For each process killed, email with details will be sent to the process owner.
  4. Exemptions by approved request via a support ticket help+bw@ncsa.illinois.edu.
    1. Current exemptions include most processes used in building code, such as the compilers and things like "make".  We always want people to be able to build code on the logins.  If you're running a build and something gets killed, please submit a ticket and let us know.
  5. Limits exist on certain I/O-intensive processes that affect the user experience of other users but tend not to hit the CPU limits.  These "targeted" processes will be limited to 1 hour wall time regardless of level of CPU use.  Processes in this class generally include anything that can recursively walk the file system, which puts a lot of load on the meta-data server.  If you need to run any of these processes longer than a few minutes, please consider running them in a job with aprun. The list of processes under these restrictions is dynamic, but includes rsync, cp, scp, diff, cat, find, and tar.
  6. Memory limit for non-exempt processes using over 400GB of system memory.
  7. On 10/22/2020, a policy limiting processes on the login nodes to a 30-day elapsed wall clock time was enabled to ensure login node stability. 

Idle Session Limits

Unattended/idle sessions have an ssh session timeout of 4 hours.  Screen sessions may persist longer, but left unattended/idle, will timeout and be terminated after 3 days.  These policies exist for both security and resource management purposes.

[Update 5/14/2020]

  • Beginning 5/21/2020, the 3-day login node idle timeouts imposed by the shell will be eliminated (ssh 4 hour timeouts will remain). However, for node maintenance and health purposes, sessions still may be terminated with a 24 hour notice to the terminal session, but this is a rare occurrence.  

Rsync

The use of rsync can impact the experience of all users of a login node if used improperly.  Rsync is recommended for transfer of source code and small amounts of data between Blue Waters and other systems. Large data transfers should use the recommended method of Globus Online.

To regulate usage, user rsync processes utilizing Blue Waters login nodes [h2ologin1-4] as a source or destination are restricted to one instance per user per login node. An error message will be displayed with our policy details for rsync processes originating from Blue Waters login nodes. External rsync processes above our limit of one will be denied connection with an error message "protocol version mismatch -- is your shell clean? (see the rsync man page for an explanation) rsync error: protocol incompatibility (code 2) at compat.c(171) [sender=3.0.4]"
 

Crontab

The use of crontab tool is not allowed on Blue Waters. This policy is implemented to prevent the potential interference of user processes with maintenance work, and to improve the transparency of processes running on the service nodes. Upon execution of crontab tool, a message will be printed "You (username) are not allowed to use this program (crontab)".