The National Institute for Computational Sciences

Data Transfer

  Data Transfer

The ACF provides several ways for transferring files to/from the home directories, project space and scratch space Lustre files systems from the Data Transfer Nodes, called DTNs. The DTN can be accessed at dtn-rsa-acf.nics.utk.edu. Authentication will be with Globus Security Infrastructure (GSI) with myproxy certificates or with ACF username and RSA one-time password.

To connect use a Secure Shell client like this: ssh {username}@dtn-rsa-acf.nics.utk.edu

New DNS addresses will be added as they are made available (in acf.utk.edu and acf.tennessee.edu domains).

DTNs and the ACF support team supports file transfers with SCP, SFTP, GSISCP, bbcp and GridFTP.

Performance note:SCP and SFTP utilities are available for transferring files but will usually perform slower than GSISCP and GridFTP. GridFTP transfers will usually be faster than any other method. Currently, SCP and SFTP will only work with an ACF RSA. Access via NetID and Duo is coming soon.

SCP, SFTP, GSISCP

The DTNs support file transfer with OpenSSH file transfer utilities SCP and SFTP. SCP and SFTP are installed and available on most Linux/Unix machines. The below shows an example of performing a file transfer from the DTN to the Newton login node with SCP.

The DTNs also support file transfer with GSI-OpenSSH file transfer utility GSISCP. GSI-OpenSSH is a variant of OpenSSH developed by the Globus project that supports high performance file transfer and authentication with Grid Security Infrastructure (GSI). GSISCP will be installed on Newton and ACF DTNs and the software is available from the Globus software repository. GSISCP will work with GSI user certificate authentication (XSEDE MyProxy certificates) and with ACF RSA. GSISCP will attempt GSI MyProxy authentication first and then RSA. If you need to install the GSI-OpenSSH software follow the XSEDE gsi-openssh install instructions. GSISCP just like SCP uses encryption to protect both the authentication and the data being transferred. With GSISCP from the GSI-OpenSSH with high performance networking an new cipher exists for use with GSISCP called "NONE". This cipher turns off data payload encryption and can lead to significant performance speedups over regular SCP. See how to use it in the example below. Note that both ends have to have the GSI-OpenSSH with HPN modifications to use the NONE cipher.

BBCP

The DTNs provide another tool to securely and quickly copy data from source to target. The BBCP utility is capable of breaking up file transfer into multiple simultaneously transferring streams, thereby transferring data faster than single-streaming utilities such as SCP and SFTP. Check for local availability, but if it is not available the source code can be obtained from its homepage. Several examples about how to use it, can be found at its dedicated BBCP page.

GridFTP

GridFTP, a server component of Globus Online, is the highest performance file transfer method ACF supports. ACF DTNs provide GridFTP services with the GridFTP client commands. Many other computing centers support GridFTP for file transfer. This method can be used using a web interface or at the command line. For more detailed information about how to use GridFTP see our dedicated GridFTP page.

Access GridFTP from the ACF DTN nodes using DNS entry datamover8.nics.utk.edu or dtn-rsa-acf.nics.utk.edu.

Globus Online

ACF users can use the web Globus Online interface to perform data transfers to/from ACF supported resources. The visual interface makes it quite easy to move, back up or restore relevant data. To get you started, visit the Globus website and consult the Getting Started guide.

Globus endpoints to access ACF resources

XSEDE endpoints (i.e. using XSEDE authentication):

  • XSEDE beacon
  • nics#beacon-myproxy

One of the latest features of Globus is Globus Connect Personal. Globus Connect Personal turns your personal computer into a Globus endpoint so you can share and transfer files to/from a local machine—campus server, desktop computer or laptop.


Transfer files between local computer and ACF resources