Overview
The HSI utility allows automatic authentication and provides a user-friendly command line and interactive interface to HPSS.
If you are archiving a large directory, or collection of files, please tar the files before using hsi
, or use htar which combines tar
and hsi
functionalities into a single command. HSI and HTAR are the preferred methods for accessing HPSS.
Please note: Large long-running data transfers should be submitted to the hpss queue rather than running hsi or htar interactively.
About HSI
- Password security: passwords are not transmitted in clear text over the network.
- Usability in pipelines and shell scripts and in batch jobs.
- Support for command stacking (multiple commands per line).
- Support for interactive or one-liner (i.e., command-line-only) modes.
- Support for abbreviations for most commands and keywords.
- Support for recursion for many common commands (
GET, PUT, REMOVE, LIST
). - Extensive online full-screen help.
Note: HSI can only be used with the OTP token. HSI can be used in a batch job only if it is submitted to the HPSS queue from the OTP login node.
Using HSI
Issuing the command
hsi
will start HSI in interactive mode. Alternatively, you can use
hsi [options] command(s)
to execute a set of HSI commands and then return. Note that you may need to add /opt/public/bin
to your search path to find the HSI executable.
The two most used options are put
and get
. Put transfers a file from the local file system to HPSS. Get transfers a file from HPSS to the local file system. However, unlike ftp
, the ordering of arguments does not change. The syntax is:
hsi {put | get} local_file:hpss_file
where, local_file
is the location and name of the local file and hpss_file
is the location and name of the file on HPSS.
Data written to HPSS is automatically stored using a method appropriate to the file size, however, HPSS is not intended for storing many small files. If you want to upload several small files (less than 1 GB per file), you should tar
the files before storing them in HPSS, or use htar. In general, htar
should take about as long as hsi
to transfer a given set of files.
Following is an example of storing a group of small files using tar
and HSI. HSI can read from standard input and write to standard output.
tar cvf - . | hsi put - : <filename.tar> hsi get - : <filename.tar> | tar xvf -
Users will be prompted for their PASSCODE when running HSI. See the reference manual for command line options and other startup information.
Please note:
- The ideal archive file size is in the range of 8 GB to 256 GB.
- Please do not run more than one
hsi
process simultaneously. Files are initially uploaded to a disk cache before being migrated to tape, this cache may be filled up with too many simultaneous transfers. Also, since transfers compete for bandwidth, additionalhsi
transfers may not be beneficial.
HSI Documentation
More information on HSI may be found from the NICS systems through the command
hsi help
Direct Transfers between HPSS and Remote Systems (e.g., User Workstations)
Because HSI is a third-party package, clients may be available for your system. However, the NICS currently supports access to the HPSS only through HSI clients on the HPC systems. To transfer data directly to or from the NICS HPSS, you will need to use a NICS resource as a staging system. For example, to transfer data from your directory on HPSS to a system outside the NICS, you will need to copy the data in reasonable chunks to a NICS system using the HSI utility. Once a portion of the data is on a NICS system, you can use a utility such as BBCP or other transfer utilities to move the data to the system outside the NICS.
Using HTAR
The htar
command combines tar
and hsi
functionalities into a single command. The syntax is:
htar -cvf filename.tar
The reverse operation is supported which is equivalent to an hsi get
followed by tar -xvf
.
htar -xvf filename.tar
Here, 'filename.tar
' is the name of the tar file on HPSS. Any further arguments list files to be extracted from the archive, that is "htar -xvf filename.tar .
" will probably not extract anything because there will be no file named ".
" in the archive. Also, it is only possible to extract files with htar
that have been created with htar
and have an idx
file. This is an index to the tar file, so if the archive itself resides on tape, it is still possible to list contents and extract specific files without migrating the whole archive to disk.
htar limits:
- Maximum number of files by default: 1,000,000
- Maximum number of files with special flags (-M 5000000): 5,000,000
- Maximum file size for a component file: 68 GB
- Maximum total file size for archive (limited by HPSS, not HTAR): 2^64 bytes
htar Manual and User Guide