Network File System (NFS)
The NFS server contains user's home directories, project directories, and software directories. Home and project directories can be used to store frequently used items such as source code, binaries, and scripts. NFS is accessible from all NICS resources except Kraken compute nodes.
On Kraken, the compute nodes can not access NFS—they can only access Lustre. Programs launched with the aprun command run on the compute nodes, therefore your executables and any input or output files must reside in Lustre scratch space.
A user's home directory is found at:
The environment variable $HOME contains this path. For home directories, each user receives a default storage limit of 2 GB. To request an increase in this limit please sumbit a request to email@example.com. Home directories are regularly backed up to tape.
Lustre File System
The scratch disk space for NICS resources utilizes Lustre, a massively parallel distributed file system. The following Lustre scratch areas are available:
|/lustre/medusa/$USER||Nautilus, Beacon, Keeneland||1.3 PB|
Note: /lustre/medusa is also cross-mounted on Kraken.
For convenience, the environment variable $SCRATCHDIR points to the user's corresponding Lustre directory for each machine.
Important Points for Users
The Lustre filesystem is scratch space intended for production work and not for long term storage. Files in scratch directories are not backed up and are subject to being purged after 30 days. It is the user's responsibility to back up all important data to HPSS or other storage resources.
The Lustre find command can be used to locate any of your files that have been around for 30 days or longer:
> lfs find /lustre/scratch/$USER -mtime +30 -type f
Striping is an important concept with Lustre—basically, the ability to break files into chunks and spread them across multiple storage targets (called OSTs). The striping defaults set up for NICS resources are usually fine but may need to be altered in certain use cases, like when dealing with very large files. Please see our Lustre Striping Guide for details.
Beware of using normal Linux commands for inspecting and managing your files and directories in Lustre scratch space. Using ls -l can cause undue load and may hang because it necessitates access to all OSTs holding your files. Make sure that your ls is not aliased to ls -l.
Use lfs quota to see your total usage on the Lustre system. You must specify your username and the Lustre path with this command, for example:
> lfs quota -u <username> /lustre/scratch
This will recursively list all regular files in your Lustre scratch area that are in danger of being purged.
For more detailed information regarding Lustre usage, see the following pages:
For sharing data among a research group, NICS provides project directories on NFS or Lustre. For information on requesting and managing a project directory see here.
High Performance Storage System (HPSS)
HPSS is the archival mass storage resource jointly operated by NICS and the Oak Ridge Leadership Computing Facility (OLCF). For long-term storage, user's can transfer data to HPSS from Kraken or Nautilus. Please see the HPSS documentation on using the HSI and HTAR utilities for storage in HPSS.