Please note: Globus Online is the supported, preferred method of moving files or groups of files of any significant size around the Blue Waters system or between Blue Waters and other facilties. Basically if it's not an operation that can be completed by the 'cp' command, use Globus Online for the transfer. GO has parallel access to all the data moving hardware on the system and can typically move data very quickly and efficiently.
The Blue Waters prototype data sharing service is now available. This service allows allocated Blue Waters partners to share datasets generated on Blue Waters with members of their scientific or engineering community with two methods: web services for small files and Globus Online for large files. Please see the Data Sharing Service for more information.
Particularly, do NOT do large transfers with programs like scp, sftp, or rsync. They clog up the login nodes and cause other user interactive shells to slow down. There are specific limits for how long single processes can be run on the Blue Waters login nodes, and programs violating those limits will be killed automatically.
Globus Online is a hosted service that automates the tasks associated with moving files between sites. Users queue file transfers which are then done asynchronously in the background.
Globus Online transfers files between "endpoints" which are systems that are registered with the G.O. service. Blue Waters provides an endpoint to its filesystems (ncsa#BlueWaters) served by multiple Import/Export nodes optimized for performance with Globus Online (the equivalent archive storage endpoint is ncsa#Nearline). If you need to transfer files to or from a system that isn't yet registered as an endpoint (like your desktop or laptop), you register that system using Globus Connect.
How to use Globus Online
Sign in to the Blue Waters Data Movement portal to transfer data between sites by using the web interface. To get there, go to the Blue Waters portal (bluewaters.ncsa.illinois.edu), mouse over the "Using Blue Waters" tab and click on the "Data" button to go to the Data Transfer page. The "Globus Online" button on the top of that page takes you to the Globus Online user interface:
If you don't have a Globus Online account, see our page about how to create one. If you have a Globus Online account and it's linked to your Blue Waters portal account, you will see the following screen. Click on the "Proceed" button to authenticate with your Blue Waters credentials.
On the next screen, enter your username and then passcode (PIN plus token code) just like when logging into the portal or into the system:
Once you've logged into Globus Online, you see the "Transfer Files" screen:
which allows you to transfer files between Globus Online endpoints.
If you need to transfer files from a system that isn't yet a G.O. endpoint, you set it up to be one with the Globus Connect software:
Only transfer complete files
Please only initiate transfers within Globus Online sourcing from files that have been written to disk and closed by the application. If GO is trying to transfer a file from Lustre that is being written to by a user application, then it gets confused about where it is in the file and tries to transfer the file multiple times. This uses up a lot of extra resources and doesn't gain anything for the user. It's faster to transfer files once they're done being written.
Previously, when a file was transferred to a Blue Waters Lustre file system (home, scratch, projects) using GO, the file would use the system default stripe settings (stripe count 1, stripe size 1 MB). As of October 2013, this has been patched. Incoming files now inherit the stripe count, stripe size, and stripe offset settings of the directory in which they are placed by GO. The stripe count for a particular file will be revised higher, if necessary, for very large files.
Command Line Interface (CLI)
The ssh-based CLI will be deprecated August 1, 2018. We are packaging the globus-cli on Blue Waters as guided by Globus. Please check back for updates.
Self-deployment of Python-based Globus CLI
To self-deploy the new Python-based Globus CLI, follow these 7 steps:
You are now logged into Globus and can intiate queries and transfers. For example find the Blue Waters endpoint:
You can then activate an endpoint and list content:
Cut-n-paste the URL into a browser and authenticate against NCSA two-factor. Now you can list the endpoint.
Please see https://docs.globus.org/cli/ for more commands and examples.
THE FOLLOWING WILL BE DEPRECATED in August, 2018.
The Command Line Interface requires ssh key setup on the Blue Waters login node. You'll generate a key to use with cli.globusonline.org and then copy/paste it into the GO web interface.
Once that's completed (and you'll only need to do it once for Blue Waters), you may ssh to the GO command line interface and initiate transfers from there:
Note that you may manipulate (check status, cancel, view events) any transfer Task ID from either the command line or the web interface's "View Transfer Activity" page.
More on CLI
Files can be moved on Nearline using the
When using the Command Line Interface (CLI), there are three parameters a user can manually set that can potentially improve performance. Taken from the Globus Online transfer man page:
These options are for advanced users who want to override performance auto tuning. By default, Globus Online will select tuning options appropriate for the number and sizes of files used in the task.
Set parallelism level (Data streams per control channel). Allowed values are 1-16.
Set concurrency level (Number of control channels). Allowed values are 1-16.
Set pipeline depth (Files in flight per control channel). Allowed values are 1-32. This is useful for small files and/or high latency connections.
The default values used by Globus Online for Blue Waters are the same, regardless of transfer size and destination, so there is sometimes room for significant transfer rate improvement (but see the important notes below).
Note that these parameters cannot be set through the web interface.
For transfers of a significant number (dozens) of large files (tens of GB) between the Blue Waters /scratch file system and the Nearline storage system, we have found it best to set
Here's an example of such a CLI command that transfers an entire directory recursively (hence the -r at the end). Note that this command performs the ssh into cli.globusonline.org in addition to executing the transfer command (as opposed to ssh-ing in first and then initiating the transfer as shown in the Examples section above).
This is an example script demonstrating a method to transfer only files of a given pattern. NOTE, for N files, the script would generate N transfers.
Important CLI Notes
Deleting a File, Folder, or Directory
To delete files using Globus Online from a endpoint, you first need to login into Globus Online and select the enpoint you wish to delete from, by first following the proceedure for transfereing a file.
Next, you need to select the object to delete, click on the action "wheel" triangle and select "delete selected files".
You will then be asked to confirm the process and provide a label for future reference (optional). Click on "Yes" when ready.
The object to be deleted will then get set with a line across it as is shown in the next image.
If you have the option set to receive an email for every transaction you should see the following notice appear in your email once the action is completed.
Blue Waters' Nealine system will not able to retrieve hundreds of files from the storage tapes as the files will have been scattered over many tapes, each of which needs to be rewound to the correct location for every single file. Instead you must aggregate tiny files into chunks of approximately 10s of GBs in size.
First create a list of directories with files to archive:
h2ologin$ cat >files-to-archive.txt <<EOF
then obtain an interactive session and run tar
h2ologin$ qsub -I -l nodes=1:xe:ppn=32 -l walltime=2:0:0
which will use up to 32 cores on a compute node to create up to 32 tar files in parallel producing tar files simulation1.tar.gz simulation2.tar.gz etc.
Similarly you can use the same list of files to extract the resultant tar files in parallel
h2ologin$ qsub -I -l nodes=1:xe:ppn=32 -l walltime=2:0:0
These can be encapsulated in a convenient shell script:
which can be used like so
bash aggregate.sh simulation1 simulation2 input-data code
Data Compression and transfer discussion
There are a variety of factors to consider when deciding whether or not to compress data. For data that compress at high ratios (ascii text, highly structured binary data ) the space savings alone may warrant compression. In other cases, it may help to do some analysis before moving a large amount of data in order to accomplish the move in the least time.
To help with your decision, it's useful to take a few measurements. The examples and model below were developed with gzip for compression (other utilities like zip, bzip, compress are available on many unix systems) and globus-online for the transfer mechanism. See the detail log for specifics.
1) The bandwidth between your site and NCSA may be determined by a tool like perfsonar. For many Xsede resource providers, a table is available at: http://psarch.psc.xsede.org/maddash-webui/ . Alternatively, you may run your own bandwidth test to our endpoint from a browser by loading: http://ps.ncsa.xsede.org , or you may use one of the well known bandwidth testing applications available online to get your baseline bandwidth to the internet. We'll call this bandwidth : xfer_bandwidth (MB/s unless otherwise indicated).
2) When compressing your data, make a note of how long it takes (use the time command on a unix system). You may then calculate the compression bandwidth as : (raw_size / compression_time). We'll name the compression bandwidth : compression_bandwidth=(raw_size/compression_time). It may be useful to compress a smaller representative file to get this number if the workflow will be moving extremely large files.
3) The compression ratio for your data will be the size of the raw data divided by the size of the compressed data (this should be a number >= 1 ). This variable will be : compression_ratio=(raw_size/compressed_size).
With the measurments above, we can develop a simple procedure to help us determine when compression will save time with respect to data transfers.
The goal is to compare these 2 expressions:
With compression, the effective bandwidth is amplified and becomes compression_ratio*xfer_bandwidth and the second term represents time spent compressing data.
The smaller quantity in seconds will move your data in the least time. You may still opt to compress if saving space is the overriding concern. Here we look at a couple of example scenarios:
Scenario 1: gigabit ethernet enabled laptop moving a 1 gigabyte file to Blue Waters
1015MB / (112MB/s) vs. 1015MB / [(1015MB/775MB )*112MB/s] + 1015MB / [1015MB/118 s]
9s vs. ( 7 + 118 )s
Rule of thumb: With high bandwidth end-to-end networking (gigabit ethernet and better), compression will add time to your transfers. Until compression algorithms can run at much higher bandwidths, this will continue to hold.
Scenario 2: wifi enabled laptop moving an approx. 1 gigabyte file to Blue Waters
1015MB / (3.75MB/s) vs. 1015MB / [(1015MB/775MB )*3.75MB/s] + 1015MB / [1015MB/118 s]
270s vs. ( 206 + 118 )s
Rule of thumb: With end-to-end bandwidth near 100baseT speeds of 100Mbit/s, compression is more of a toss-up with respect to overall transfer speed. Other factors may come into play like the consistency of the xfer_bandwidth. The compression_ratio becomes more important and may be the dominant factor.
Scenario 3: broadband cable connection working from home and transfer of small source code directory tree
6.8MB / (0.11MB/s) vs. 6.8MB / [(6.8MB/2.4MB])*.11MB/s] + 6.8MB / [6.8MB/0.5s ]
62s vs (22 + 0.5 )s
Rule of thumb: With low bandwidth links (home broadband, hotel lobby...) , the compression_bandwidth is typically higher than the xfer_bandwidth and compressing saves both time and space: win-win. This is changing as commodity networks are upgaded, and watch out for asymmetric bandwidths. On the test setup for this scenario, the download bandwidth was 3x the upload bandwidth so compression may save time in one direction and cost time in the other.
The simplified model and discussion ignores a few factors:
We recommend users to transfer fewer but larger files. If a transfer attempts to move tens to hundreds of thousands of small (<1GB) files, performance will suffer due to file staging overhead. In such cases, It is best to use tar to form larger files of size tens or hundreds GB before starting a GO transfer.
Globus Transfers to AWS S3
Any user desiring to make use of the Globus AWS S3 connector needs to have a file created in their home directory that specifies the S3 credentials associated with their local user account. The configuration for the AWS S3 Connector is stored in $HOME/.globus/s3. This file provides a mapping of the current user’s ID to S3 access keys. This file can be created and populated by the user using the following two commands:
The “S3_ACCESS_KEY_ID” and “S3_SECRET_ACCESS_KEY” correspond to the Access Key ID and Secret Access Key for the user’s S3 credentials that have been granted access to the S3 bucket(s) the user intends to access. These must be provided by the user. When complete, the file should contain 3 fields, separated by semi-colons, that look something like:
See https://docs.globus.org/premium-storage-connectors/aws-s3#s3-credentials-anchor for more information on identity credentials.
The name of the S3 enabled endpoint is ncsa#BlueWatersAWS.
Additional Information / References