NICS provides several ways for transferring files to/from all of our HPC resources, that can benefit from the high speed network connections with the XSEDE network and Internet2.
The following table shows systems vs. data transfer methods.
|Resource||GridFTP||HSI/HTAR||SCP, SFTP, rsync||XSEDE File Manager||BBCP||Globus||Supermover|
The GridFTP protocol is the preferred method to transfer files. All XSEDE resources have GridFTP clients installed. Check for local availability at other locations. This method can be used on the command line (see our dedicated GridFTP page) or through the Globus web interface (see below).
For smaller data transfers, you can use the standard transfer utilities, SCP, SFTP and rsync, which are usually available on all Linux/Unix machines. There are command line and graphical clients available. These utilities are ill suited for transferring large amounts of data. More information on these utilities can be found on the transfer utilities page.
This is another tool to securely and quickly copy data from source to target. The BBCP utility is capable of breaking up your transfer into multiple simultaneously transferring streams, thereby transferring data faster than single-streaming utilities such as SCP and SFTP. Check for local availability, but if it is not available the source code can be obtained from its homepage. Several examples about how to use it, can be found at its dedicated BBCP page.
These tools are the only way to transfer files to/from the HPSS archival system. In order to archive files, it is required to login with the RSA tokens. HSI can be used to transfer single files, whereas HTAR offers a convenient way to bundle and archive multiple files in one step. More information can be found on the HSI page.
The XSEDE File Manager is a java applet available through the XUP. It enables access to desktop files, remote XSEDE systems, cloud storage systems (specifically Eucalyptus and Amazon's S3), and external systems defined by users. It supports third-party transfers, bookmarking, and remote searching. The TGUP File Manager was designed to be a common, intuitive interface for both novice and expert users. Visit XSEDE File Manager for more information.
NICS users can use the web Globus tool to perform data transfers to/from NICS supported resources. The visual interface makes it quite easy to move, back up or restore relevant data. To get you started, visit the Globus website and consult the Quick Start guide.
Globus endpoints to access NICS resources
XSEDE endpoints (i.e. using XSEDE authentication):
NICS endpoints (i.e. using NICS OTP authentication):
One of the latest features of Globus is Globus Connect Personal. Globus Connect Personal turns your personal computer into a Globus endpoint so you can share and transfer files to/from a local machine—campus server, desktop computer or laptop.
Figuring out the semantics of doing data movement using a variety of different protocols can be difficult. The supermover command attempts to solve this by acting as a wrapper around several data movement tools. Simply, one just needs to provide the source and destination locations of a file in a URL format. Visit Supermover for more information.