HSI

Overview

The HSI utility allows automatic authentication and provides a user-friendly command line and interactive interface to HPSS.
If you are archiving a large directory, or collection of files, please tar the files before using hsi, or use htar which combines tar and hsi functionalities into a single command. HSI and HTAR are the preferred methods for accessing HPSS.

Please note: Large long-running data transfers should be submitted to the hpss queue rather than running hsi or htar interactively.

---

About HSI

• Password security: passwords are not transmitted in clear text over the network.
• Usability in pipelines and shell scripts and in batch jobs.
• Support for command stacking (multiple commands per line).
• Support for interactive or one-liner (i.e., command-line-only) modes.
• Support for abbreviations for most commands and keywords.
• Support for recursion for many common commands (GET, PUT, REMOVE, LIST).
• Extensive online full-screen help.

Using HSI

Issuing the command

hsi

will start HSI in interactive mode. Alternatively, you can use

hsi [options] command(s)

to execute a set of HSI commands and then return. Note that you may need to add /opt/public/bin to your search path to find the HSI executable.

The two most used options are put and get. Put transfers a file from the local file system to HPSS. Get transfers a file from HPSS to the local file system. However, unlike ftp, the ordering of arguments does not change. The syntax is:

hsi {put | get} local_file:hpss_file

where, local_file is the location and name of the local file and hpss_file is the location and name of the file on HPSS.

Data written to HPSS is automatically stored using a method appropriate to the file size, however, HPSS is not intended for storing many small files. If you want to upload several small files (less than 1 GB per file), you should tar the files before storing them in HPSS, or use htar. In general, htar should take about as long as hsi to transfer a given set of files.

Following is an example of storing a group of small files using tar and HSI. HSI can read from standard input and write to standard output.

  tar cvf - . | hsi put - : <filename.tar>

  hsi get - : <filename.tar> | tar xvf -

Users will be prompted for their PASSCODE when running HSI. See the reference manual for command line options and other startup information.

Please note:

• The ideal archive file size is in the range of 8 GB to 256 GB.
• Please do not run more than one hsi process simultaneously. Files are initially uploaded to a disk cache before being migrated to tape, this cache may be filled up with too many simultaneous transfers. Also, since transfers compete for bandwidth, additional hsi transfers may not be beneficial.

HSI Documentation

More information on HSI may be found from the NICS systems through the command

hsi help

---

Direct Transfers between HPSS and Remote Systems (e.g., User Workstations)

Because HSI is a third-party package, clients may be available for your system. However, the NICS currently supports access to the HPSS only through HSI clients on the HPC systems. To transfer data directly to or from the NICS HPSS, you will need to use a NICS resource as a staging system. For example, to transfer data from your directory on HPSS to a system outside the NICS, you will need to copy the data in reasonable chunks to a NICS system using the HSI utility. Once a portion of the data is on a NICS system, you can use a utility such as BBCP or other transfer utilities to move the data to the system outside the NICS.

---

Using HTAR

The htar command combines tar and hsi functionalities into a single command. The syntax is:

htar -cvf filename.tar

The reverse operation is supported which is equivalent to an hsi get followed by tar -xvf.

htar -xvf filename.tar

Here, 'filename.tar' is the name of the tar file on HPSS.   Any further arguments list files to be extracted from the archive, that is "htar -xvf filename.tar ." will probably not extract anything because there will be no file named "." in the archive.   Also, it is only possible to extract files with htar that have been created with htar and have an idx file.   This is an index to the tar file, so if the archive itself resides on tape, it is still possible to list contents and extract specific files without migrating the whole archive to disk.

   htar limits:

• Maximum number of files by default: 1,000,000
• Maximum number of files with special flags (-M 5000000): 5,000,000
• Maximum file size for a component file: 68 GB
• Maximum total file size for archive (limited by HPSS, not HTAR): 2^64 bytes

   htar Manual and User Guide