Nearline

Accessing Nearline

Globus Online

Globus Online should be used for all data transfers and some simple file system commands.  Please see the Globus sections of the Data Transfer portal page here.

UberFTP

As of April 2014, Globus Online only supports the following file management commands: ls, mkdir, rename (similar to mv), rm, and delete (advanced rm).  Additionally, Globus command line interface (CLI) interactive sessions are rather cumbersome as all commands require the endpoint to be explicitly prepended to all paths (e.g., ncsa#Nearline/<path>).

UberFTP, a GridFTP client developed at NCSA, can be used for some of the functionality not provided by Globus.  For example, UberFTP allows permission changes via chmod.  In addition, UberFTP interactive sessions take place on the Blue Waters data mover nodes, which means paths don't require endpoints.  Supported commands include cat, cd, chgrp, chmod, ls, mkdir, pwd, rm, rmdir, and others.  For a full list of available commands and other documentation, see the UberFTP page here.

Just like with the Globus CLI, UberFTP allows both interactive sessions and the issuing of remote commands, which can be used for scripting.  However, UberFTP commands can only be executed on the login nodes (they are unavailable on the MOM and compute nodes).

Note that data transfers can also be initiated with UberFTP.  However, we strongly recommend against transferring data with UberFTP at this time.  Globus transfers automatically take advantage of all 50 data mover nodes by spreading out the individual file transfers.  Without additional configuration, an UberFTP transfer will send all files in a transfer request through a single mover node, greatly decreasing the potential parallelism.

Using UberFTP

Using UberFTP first requires a credential for authenticating to the data mover nodes (please note that this credential is separate from your Globus credentials).  To get one, use myproxy-logon.  When asked for a MyProxy pass phrase, enter the usual Blue Waters [PIN][token code] combination.

h2ologin1:~$ myproxy-logon -s tfca.ncsa.illinois.edu -l $USER
Enter MyProxy pass phrase: [enter BW PIN+token_code]

Next, choose a data mover between 2 and 50.  The host names are hpss-md[02-50].ncsa.illinois.edu.  If the mover you choose is down, simply choose a different one.

You may either start an interactive session:

h2ologin1:~$ uberftp hpss-md02.ncsa.illinois.edu

Or you may issue commands from the login nodes in a manner similar to ssh:

h2ologin1:~$ uberftp hpss-md02.ncsa.illinois.edu <command>

For example:

h2ologin1:~$ uberftp hpss-md02.ncsa.illinois.edu "ls ~/bw_backup"

As you might expect, "~" above refers to your Nearline home directory.