Data Transfer for Education and Training Allocations
The security policies for the Education and Training accounts require a different way to do data movement. As the accounts do not have RSA tokens, the accounts are not able to use the Blue Waters Globus Online public endpoints, nor can they use scp or sftp to "push" data to Blue Waters from a remote host. Education and Training do not have access to the Blue Waters Nearline storage system by default due to the absence of the RSA token.
Below are some suggestions on how to transfer files and data to Blue Waters for Education and Training accounts.
Please contact email@example.com if there are questions or issues.
Use of login nodes
Educational allocations do not have access to the published Blue Waters Globus Online endpoints.
Small files ( less than 1 GB or 10 GB total for directories )
You can use rsync to copy filles off the Blue Waters file system, but need to make sure that all data is piped through the bwbay host like so
rsync -e 'ssh traXXX@bwbay.ncsa.illinois.edu ssh' h2ologin.ncsa.illinois.edu:yourfile-on-bw your-file-on-local-machine
If you are using WinSCP or the Unix sftp command then you must set the "SFTP server" setting in the advanced settings or via the
mkdir ~/bw sshfs -o ssh_command="ssh traXXX@bwbay.ncsa.illinois.edu ssh" h2ologin:/ ~/bw
cd $HOME tar xzvf /sw/userdoc/gofiles/globusconnect-latest.tgz cd globusconnect* ./globusconnectpersonal -setup <your_setup_key> ./globusconnectpersonal -start -debug
Now start a browser somewhere and use your new globus connect endpoint on h2ologinN