Data Transfer for Education and Training Allocations

The security policies for the Education and Training accounts require a different way to do data movement. As the accounts do not have RSA tokens, the accounts are not able to use the Blue Waters Globus Online public endpoints, nor can they use scp or sftp to "push" data to Blue Waters from a remote host. Education and Training do not have access to the Blue Waters Nearline storage system by default due to the absence of the RSA token.

Below are some suggestions on how to transfer files and data to Blue Waters for Education and Training accounts.

Please contact help+bw@ncsa.illinois.edu if there are questions or issues.

Use of login nodes

Transfers must be initiated from the login nodes h2ologin[1-4]. Educational allocations do not have access to the published Blue Waters Globus Online endpoints.

Small files ( less than 1 GB or 10 GB total for directories )

  • Use scp or similar from your login session on Blue Waters to push or pull files with a remote server running sshd or httpd service(s).

    • scp , sftp , rsync ( with sshd servers )

    • wget , curl ( with httpd servers )

    • mailx -s mysubject -a sync.c my.email@somewhere.edu < /dev/null  # files smaller than a few meg as email attachments

  • If you do not have a dedicated server for use with wget, we have found that you can host files on file hosting services like box.com (see using FTP with box) and SeedMe that work with wget or curl. Let us know at help+bw@ncsa.illinois.edu if you are able to transfer files from other file hosting services. 

Larger files

  • For large files we recommend using Globus Online by way of Globus Connect. Start a globus connect client from your login session on Blue Waters, and use Globus Online (GO) to transfer data between a login node and off-site GO endpoints. Globus Connect setup is documented on the Blue Waters portal at https://bluewaters.ncsa.illinois.edu/setting-up-globus-connect.  

  • You will need to create a GO account if you do not already have one at globus.org .  Supporting software is available here (along with steps to get your Globus Connect endpoint going) by staring with a copy of the Globus Connect software (check with GO and download the latest if a newer version is available).

   
$ cd $HOME
$ tar xzvf /sw/userdoc/gofiles/globusconnect-latest.tgz
$ cd globusconnect*
$ ./globusconnectpersonal -setup <your_setup_key>
$ ./globusconnectpersonal -start -debug

Now start a browser somewhere and use your new globus connect endpoint on h2ologinN

Caveats

  • Interactive limits will apply on login nodes; long running transfers (hours) will require special arrangements
  • Use no more than 2 login nodes for moving files.  If load or interactive response becomes an issue, the admins will terminate data transfers without warning.
    • Interactive response for compiling and building is the top priority for login nodes