Data Transfer

Go to https://app.globus.org/ and authenticate by your preferred method to use Globus Online via the web GUI. You will need to acticate Blue Waters endpoints using your NCSA password and Duo two-factor login.  Please see https://bluewaters.ncsa.illinois.edu/data-transfer-doc for details on using CILogin to get access to endpoints.  (The Blue Waters branded Globus Online landing page is no longer available.)

Moving Data

NCSA recommends using Globus Online for data transfer. Currently, the Blue Waters endpoint is named: ncsa#BlueWaters. The ncsa#BlueWaters endpoint transfers data to the Blue Waters Lustre file systems. The Globus Online environment will choose the "least busy" server for each transfer to distribute the data movement load so as to move the data as efficiently as possible. Use the ncsa#BlueWaters endpoint to transfer data into and out of your home and project spaces or even between the different Blue Waters file systems. In addition to the Blue Waters endpoint NCSA also provides Globus connectors to AWS and Google drive to facilitate data transfers between Blue Waters and those services.

Helpful Tips

  • Do not use scp/sftp/globus-url-copy on the login nodes to transfer data.  Performance will be far below that of Globus Online and the network load will impact other users.
  • Use Globus Connect if you need to transfer files with your local office machine.
  • Globus Online provides an interactive web interface as well as a command line interface. For more information please refer to the User Documentation or contact help+bw@ncsa.illinois.edu .

Refer to the documentation https://bluewaters.ncsa.illinois.edu/data-transfer-doc for additional information on data transfer.

Sharing Data 

Blue Waters partners are able to share data using Globus Online from their projects /projects/sciteam/psn/share directory where psn is your project serial number. Please visit the Data Sharing with Globus for more information. 

Contact help+bw@ncsa.illinois.edu for any concerns or questions.

Import/Export (IE) System

The Blue Waters system environment provides 28 dedicated Import/Export (IE) servers to efficiently manage and move data via Globus Online. The Import/Export (IE) servers are provisioned to offload the data movement transactions away from the external login nodes. Data transfer rates through the IE environment have been measured at greater than 100GB/s aggregate for read and write. The IE servers (Dell R720) are configured with dual socket 16 core processors, 192 GB of memory, FDR InfiniBand (IB) card, and a 40 Gigabit Ethernet card dedicated for data transfer. The following schematic shows the relationship of the Blue Waters Lustre filesystem, IE servers and the local and wide area networks (LAN ad WAN respectively).