Getting Started Guide

This brief guide is intended to give fairly experienced users the basic information they need to get on the system, set up their environment, compile applications, and run batch jobs.

Contents

Obtaining an Allocation

Logging In

Level of Expertise Expected of Blue Waters Users

Transfering Files and Data

Setting Up Your Environment

Building Executables

Running Batch Jobs

Quick Hardware Summary

Obtaining an Allocation

There are several pathways available for researchers and educators to apply to use Blue Waters for their work. Visit the Allocations page.

Logging In

You may connect to Blue Waters via the external login hosts at bw.ncsa.illinois.edu using ssh with your RSA PIN+token. This two factor authenication scheme is an effective one time password method (OTP). The bw.ncsa.illinois.edu address is a DNS roundrobin alias for h2ologin[1-4]. If you find that connecting to bw.ncsa.illinois.edu is not successful please try specifying a particular login host.

You can also use GSI with the two-factor authentication:

$ myproxy-logon -s tfca.ncsa.illinois.edu -p 7512
$ gsissh|ssh -o PreferredAuthentications=keyboard-interactive h2ologin.ncsa.illinois.edu
or
$ gsissh|ssh -o GSSAPIAuthentication=yes h2ologin.ncsa.illinois.edu  # for Mac

You must use the RSA token that was sent to you from NCSA when your account was created. After you have set your PIN (or before you have set your PIN) it is recommended that you visit the RSA Self-service console to set security questions that will allow you to reset and change your RSA PIN. Note that there are cases where browser session data gets cached, confusing the RSA self-service website, that requires either clearing the session data or exiting the browser and restarting.

Helpful Hints:

  • For the Authentication Method there is a dropdown for either a password or passcode – select passcode.)
  • The PIN is required to be exactly 8 characters of both numbers and letters.
  • The passcode refers to the [PIN][tokencode] combo. If your PIN is not set then you use only the token code.
  • After setting your PIN, you will be asked to set security questions.  Those questions will allow you to reset your PIN (or create a new one in case you forget it) and to disable your OTP token if it is temporarily lost. 
  • The RSA website can be finicky at times, and it will time out if you take too long to complete the process.  If it times out, use the URL in this email again as the back button will not work.

Science team members can obtain tokens from the Blue Waters help desk after the project PI has started the account creation process.

Level of Expertise Expected for Blue Waters Users

Most users of systems like Blue Waters have experience with other large high-performance computer systems.  The instructions on this portal generally assume that the reader knows how to use a Unix-style command line, edit files, run (and modify) Makefiles to build code, write scripts, and submit jobs to a batch queue system.  There are some things that work slightly differently on the Cray XE system than other systems; the portal documentation covers those in detail, but we assume that you know the basics already.

If you're not at that level yet (if you're unfamiliar with things like ssh, emacs, vi, jpico, qsub, make, top) then you'll need to gain some knowledge before you can use Blue Waters effectively.  Here are a few links to resources that will teach you some of the basics about Unix command line tools and working on a high-performance computing system:

Transfering Files and Data

It is recommended that Globus Online (GO) is used for file transfers to and from Blue Waters. Blue Waters has dedicated import/export resources to provide superior I/O access to the filesystems. Most HPC centers provide GO endpoints and GO provides clients for desktop and laptop transfers.

Please see the Data Transfer section of the User Guide.

Setting Up Your Environment

The default shell is /bin/bash.  You can change it by sending a request via email to help+bw@ncsa.illinois.edu.

The user environment is controlled using the modules environment management system. Modules may be loaded, unloaded, or swapped either on a command line or in your $HOME/.bashrc (.cshrc for csh ) shell startup file.

The command "module avail" will display the avail modules on the system.

Please see the Programming Environment section of the User Guide.

Building Executables

The compilers are defined in the PrgEnv module for each family of compilers.  Invoke the ftn or cc commands with the appropriate programming environment, and the underlying compilers will be employed.  To see the options specific to a compiler (pgi vs. cray) consult the man pages for the vendor's compiler (man pgf90 or man crayftn).  The correct xt-libsci providing the lapack libs for your programming environment is automatically included in the PrgEnv module you choose.

The default programming environment (module PrgEnv-cray ) invokes the Cray compilers. PGI compilers are available via the PrgEnv-pgi module. Gnu compilers are available via the PrgEnv-gnu module.

OpenACC support is provided by both the Cray and PGI compilers.

Please see the Compiling section of the User Guide for more information.

Running Batch Jobs

The batch environment is Torque/MOAB from Adaptive Computing which talk to the Cray's Application Level Placement Scheduler (ALPS) to obtain resource information.

The aprun utility is used to start MPI programs instead of the usual mpirun or mpiexec as found on many commodity clusters.  aprun supports options that are unique to the Cray.  There are flags to set process affinity by NUMA node, control thread placement, and set memory policy between NUMA nodes in a job.  See the aprun man page for more details.  Some common flags are highlighted and documented in the sample batch scripts.

Example scripts are located in /sw/userdoc/samplescripts/

Following is a minimalistic PBS script to start a job app.exe running under GNU environment on two XE nodes

#!/bin/bash
#PBS -l nodes=2:ppn=32
#PBS -l walltime=00:30:00
#PBS -N testjob

. /opt/modules/default/init/bash
module swap PrgEnv-cray PrgEnv-gnu

cd $PBS_O_WORKDIR

aprun -n 64 ./app.exe < input.dat > output.out

Please see the Running Your Jobs section of the User Guide for more information.

Quick Hardware Summary

The Blue Waters system is a Cray XE/XK hybrid machine composed of AMD 6276 "Interlagos" processors and NVIDIA GK110 "Kepler" accelerators all connected by the Cray Gemini torus interconnect. 

System Totals

Cabinets 288
Peak Performance (XE+XK) 13.24 PF
System Memory (XE+XK) 1.476 PB
Compute Nodes (XE+XK) 26,864
Bulldozer Cores (XE+XK) 405,248
Kepler Accelerators 4,224
Torus Dimensions 24x24x24
Usable Disk Storage 26.4 PB

 

Please see the detailed Hardware Summary for more information.