The National Institute for Computational Sciences


HPSS (Darter and Nautilus): Splitting a HPSS archive into multiple files

You can use the split command to split an archive into multiple files. Please follow the steps and examples provided below.

"Cd" into your /lustre/medusa/ directory where your data is temporary stored and run the following command. Make sure the file striping ( in the directory is appropriate for what is being done.

HPSS (Darter and Nautilus): How do I check my usage on HPSS?

In order to check one's usage on HPSS, enter the "hsi" command. Then, use the HPSS "du" command in the most top level directory, or "du -s" (summary for the entire directly only) option.

For example:

O:[/home/username]: du
2137614 4 /home/std00/
305920049 1 /home/std00/directory1/ 86648223 1 /home/std00/directory2/ 211942420 1 /home/std00/directory3/ 156677661 47 /home/std00/direcotry4/ 6455083743 1 /home/std00/directory5/ 0 0 /home/std00/ ----------------------- 7218409709 total 512-byte blocks, 55 Files (3,695,825,770,765 bytes)

HPSS (Darter and Nautilus): Is it possible for files to be overwritten in HPSS?

The ~ is appended if the user has "autobackup=on" in their .hsirc file. Otherwise, the file is simply overwritten. Another option is to use "hsi cput" instead of "hsi put". Using cput will cause hsi to give a warning message if the file exists. The file that the user is attempting to store won't be written to HPSS, but the old one won't be overwritten. (The user also needs to pay careful attention to the output from hsi so that they'll notice the file wasn't stored.)

Darter: Why do I get 'out of space' error when transferring files from HPSS to Darter?

Your file transfer has caused a Lustre storage server (OST) to become full, resulting in an error like:

ead_cond_timedwait() return error 22, errno=0 OUT OF SPACE condition detected while writing local file

This usually happens because the stripe count is too small (often 1). To solve this issue, remove the partially transferred file and change the stripe count of the directory before transferring the file. To change the stripe count of the directory, first cd to that directory. Second, type the following command:

HPSS (Darter and Nautilus): How do I share my files on HPSS with other members of my research team ?

To find out what groups you are a member of on HPSS use the groups command.

K:[/home/username]: groups
K:HPSS Group List:
  1045: nsf008       1928: nsf008q4s 

This shows the user is a member of groups nsf008 and nsf008q4s.

If other members of your team are listed in the same group you can simply log into HPSS using HSI and change the group and permissions to share the files or directories.

For example, if both you and other members are all in nsf008q4s you will simply need to do a chgrp.

HPSS (Darter and Nautilus): How do I verify the contents of an archive during creation?

HTAR provides the “-Hverify=option[,option...]” command line option, which causes HTAR to first create the archive file normally, and then to go back and check its work by performing a series of checks on the archive file. You choose the types of checks to be performed by specifying one or more comma-separated options. The options can be either individual items, or the keyword “all”, or a numeric level between 0, 1 or 2. Each numeric level includes all of the checks for lower-valued levels and adds additional checks. The verification options are:

HPSS (Darter and Nautilus): How do I retrieve a single file from HPSS?

Use hsi -ls to show the tar file in HPSS

>hsi ls -l file.tar
-rw-------   1 username     username          12800 Oct  2  2008 file.tar
Use "htar" to list the contents of the tar file:
> htar -tvf file.tar
HTAR: drwxr-xr-x  username/nicsstaff          0 2008-10-02 10:47  dir2/
HTAR: -rw-r--r--  username/nicsstaff       1492 2008-10-02 10:47  dir2/data.pbs
HTAR: -rw-r--r--  username/nicsstaff       1924 2008-10-02 10:47  dir2/mpi.pbs
Use "htar" to extract a single file (name must match what is listed by the above command):

HPSS (Darter and Nautilus): Is the HPSS system able to be accessed by more than one process at a time?

There is nothing that should prevent you from running a script that creates multiple simultaneous connections to HPSS. The HPSS system administrator recommends that you should not create more than 1 or 2 connections at a time. Every time you introduce a new instance, the performance of the overall system is degraded.

HPSS (Darter and Nautilus): What is the best way to transfer a large number of small files to HPSS?

HPSS performance is greatly improved when the transfer size is between 8 GB and 256 GB. For that reason, users with large numbers of relatively small files should combine those files into one or a few 8 GB to 256 GB files and then transfer the larger files. The files can be combined with tar on the HPC system, or they can be created on the fly with a command similar to tar cvf some_dir -|hsi put - : somedir.tar. This command will tar all files in the some_dir subdirectory into a file named somedir.tar on HPSS.


Subscribe to RSS - HPSS