Module Improvements (by NCSA)

Introduction

The Blue Waters team has created a shell function package that improves the module command on Blue Waters and adds some Cray-specific features.  We call it the "Modules Improvements" package, or "ModImp".  Modimp are loaded by default for all users who use bash.  (FYI: Modimp only works in bash because it depends on specific features on in the bash shell.)

The modimp package has four separate options that can be turned on or off indepedently.  The four options are explained in detail below.

If you have any questions about modimp, or comments, or something doesn't work, please submit a ticket at help+bw@ncsa.illinois.edu or e-mail csteffen@ncsa.illinois.edu.

Improvements turned on by default

Improvements 1 and 2 below are turned on for everyone by default. You can take advantage of those options in your default environment, but unless you do something explicitly, you'll likely never notice the difference. Improvements 3 and 4 do effect how your shell looks and works, so they are off by default, and require the user to turn them on. How to do so is explained below.

Turning off all modimp options

Module improvement options 1 and 2 are on by default, and should not change how your shell works.  If for some reason you still want those features turned off, put the following lines in the .profile file in your home directory:

modimp_module_to_stdout_off
modimp_tabcomplete_off

System initialization will load the functions in your shell, but these statements will turn them off silently.

Module Improvement Option 1

This changes the definition of the "module" shell function so that it outputs to "standard out" rather than "standard error" so you can pipe the output to other commands.  With this option enabled, you can (for example) run "module avail | grep papi" to see all the versions of the papi module. 

Turn this option on with the command modimp_module_to_stdout_on and off with modimp_module_to_stdout_off.  (This option is enabled by default.)

Module Improvement Option 2

(TAB here means pushing the "TAB" key) The biggest module improvement is context-sensitive tab-completion for module commands.  With this turned on, you can type "module load hugeTAB" and the shell will autocomplete out of the available hugepages modules.  When you type "module unload TAB" will autocomplete from the list of currently loaded modules, since those are the only ones that can be unloaded.  Module swap is also context-sensitive.  If you type "module swap PrgTAB" it lists out your PrgEnv-* module that you currently have loaded, and if you hit TAB again it will list modules of the same beginning name that you can swap to.  (It knows that swapping between build environment flavors is common.)

To turn on tab completion, run modimp_tabcomplete_on, likewise modimp_tabcomplete_off to turn it off.  (This option is enabled by default.)

Module Improvement Option 3 (currently disabled)

Option 3 is a set of dynamic prompts in bash that make your prompt string sensitive to shell state, particularly if you have certain modules loaded. There are a couple of options to try it out.  One is run "modimp_prompt_setup PrgEnv" for a minimal context-sensitive prompt that only has one option turned on, the indicator of what programming environment you're in (which is a very useful feature on a Cray system).  Another is run "modimp_prompt_setup" (with no argument) to get a full example context-sensitive prompt.  The prompt has a *lot* of the options enabled and is probably too busy for normal use, but it gives you an idea of what's possible. 

Because it changes how your prompt looks, ModImp option 3 is NOT enabled by default.  You can enable temporarily as above; to install it in your shell, use the modimp_prompt_commit command as explained below.

This is a whole set of settings that use the bash intrinsic variable PROMPT_COMMAND to make your prompt sensitive to what modules are loaded.  The easiest way to describe it is to show a transcript of it being used when loading and unloading modules. 

user@bluewaters ~ $ _D_-___-Cray-_ modimp_tabcomplete_on
user@bluewaters ~ $ _D_-___-Cray-_ module swap PrgEnv-cray PrgEnv-gnu
user@bluewaters ~ $ _D_-___-Gnu-_ module swap PrgEnv-gnu PrgEnv-pgi
user@bluewaters ~ $ _D_-___-PGI-_

Note the part of the prompt that changes depending on the PrgEnv module that's loaded.

user@bluewaters ~ $ _D_-___-PGI-_ module unload darshan
user@bluewaters ~ $ ___-___-PGI-_ # "D" indicates that darshan is loaded
user@bluewaters ~ $ ___-___-PGI-_ module load perftools
user@bluewaters ~ $ P__-___-PGI-_ # "P" indicates that perftools is loaded
user@bluewaters ~ $ P__-___-PGI-_ module load cudatoolkit
user@bluewaters ~ $ P_T-___-PGI-_ # "T" indicates that cuda toolkit is loaded
user@bluewaters ~ $ P_T-___-PGI-_ module load craype-accel-nvidia35
user@bluewaters ~ $ P_5-___-PGI-_

There's a three-character section that tells you what the stripe count is in the current directory:

user@bluewaters ~ $ P_T-___-PGI-_ cd ~/stripe_test/stripe_3/
user@bluewaters ~/stripe_test/stripe_3 $ P_T- 3-PGI-_ cd ../stripe_0/
user@bluewaters ~/stripe_test/stripe_0 $ P_T-___-PGI-_ cd /tmp
user@bluewaters /tmp $ P_T-XXX-PGI-_ cd

And finally, the last dynamic character is an "M" when there's a Makefile in the directory, and "_" when not. This could be extended to indicating the presence of "configure" or anything else of note.

user@bluewaters ~ $ P_T-___-PGI-_ cd specfem3d/SF3DG_csteffen
user@bluewaters ~/specfem3d/SF3DG_csteffen $ P_T-___-PGI-M cd
user@bluewaters ~ $ P_T-___-PGI-_

How to configure the ModImp dynamic prompt:

You can set in an initial module-sensitive shell by running modimp_prompt_setup.  You tweak the shell by modifying the three environment variables.  Run modimp_prompt_help for details.  When you get the prompt the way you like it, run modimp_prompt_commit which will take the current configuration and save it to your .profile.

Here is a complete (as of May 2015) list of context-sensitive prompt functions:

pmt_init    Must always be the first function in the PROMPT_COMMAND variable, otherwise BAD WEIRD things will happen. 

pmt_finalize    Must always be the last function in the PROMPT_COMMAND variable, otherwise BAD WEIRD things will happen.

pmt_pipe    Static character "|"  Can be used as a separator.)

pmt_dash   Static character "-"  Can be used as a separator.)

pmt_cmpS     Indicates currently loaded programming environment.  The "S" indicates "short": The output is one character.  "C"=cray, "P"=PGI, "N"=Gnu.

pmt_cmpL       Currently loaded programming environment as a full world (thus "L" for long).  Output is "Cray", "PGI", or "Gnu". 

pmt_stripe       In a Lustre file system, this lists the stripe count of the current directory as a three-character number.  Blank if there's no set stripe count.  "XXX" if the current directory isn't Lustre. 

pmt_dar          "D" if the darshan module is loaded, "_" if not.

pmt_cpat        "P" if Cray perftools module is loaded, "_" if not.

pmt_ctkc35       "T" if the module cudatoolkit is loaded, "5" if craype-accel-nvidia35 (which depends on cudatoolkit), "_" if neither is loaded

pmt_space       static space; for formatting and clarity

pmt_und          static underscore "_" for formatting and clarity

pmt_make        "M" if the current directory contains a file called "Makefile", "_" otherwise

Module Improvement Option 4

The fourth option changes how successive presses of TAB work with tab completion. It changes tab-completion for all commands, not just that pertaining to the module command. When this is turned on, instead of bash's default tab-completion behavior, it just prints out all possibilities right away.  The author thinks this setting is great, but other people might not agree, so it's separated out into its own separate command to turn on and off.  Use modimp_show_all_if_ambiguous_on and modimp_show_all_if_ambiguous_off.