The National Institute for Computational Sciences

File Systems


Introduction


Two filesystems are available to ACF users for storing user files: the Network File System (NFS) and Lustre Haven. NFS contains and is used for home directories and Lustre Haven contains and is used for project and scratch directories. Table 1.1 summarizes the available filesystems.

Table 1.1 - File System Summary
File SystemPathQuotaPurged
NFS Home Directory /nics/[a,b,c,d]/home/<username> 10GB Not Purged
Lustre Scratch Directory /lustre/haven/user/<username> No Quota Purged
Lustre Project Directory /lustre/haven/proj/<project> 1 TB default
more by request
Not Purged

Please note that while both NFS and Lustre are reliable filesystems, errors and corruptions can still occur. It is each user's responsibility to back up your data. To learn about data transfer to/from the ACF, please review the Data Transfer document.

For more information on the Lustre file system, please refer to the Lustre User Guide.

Home Directories


On the ACF, fifteen terabytes of storage space is available via NFS for home directories. Home directories on NFS are periodically backed up for disaster recovery. Each new account on the ACF receives a home directory on NFS. This is each account's personal storage location for a small amount of files. Here you can store job scripts, virtual environments, and other types of files and data. In a Linux environment, you can refer to your home directory with the environment variable $HOME or with the tilde (~) character.

By default, your home directory is limited to 10GB of storage space. It is not intended to store large amounts of project or job-related data. For job-related data, please use your scratch directory. For project data that you do not want to be purged request and use project space.

To determine how much storage space you have consumed in your home directory, execute the quota -s command. Figure 2.1 shows the possible output of this command. Of interest are the first “space,” “quota,” and “limit” fields. The “space” field shows how much storage space is currently in use. The “quota” field displays the soft quota placed on your home directory. The “limit” field defines the hard quota, which is the absolute maximum storage space you can consume. When you exceed the quota, you will start a grace period that gives you time to reduce your storage space usage. If you do not reduce your storage space usage during this period, the soft limit defined by the “quota” field will be enforced.

Disk quotas for user user-x (uid 00001): 
     Filesystem   space   quota   limit   grace   files   quota   limit   grace
nfs.nics.utk.edu:/nfs/b
                  3144M  10240M  10752M           58432       0       0
Figure 2.1 - Output of quota -s

Project Directories


All users are initially put into one of the ACF institutional projects: ACF-UTK0011 for UTK users and ACF-UTHSC0001 for UTHSC users. These projects do not have a project space. All courses and research projects requested by a University faculty are created with a default 1 terabyte project space. For graduate student Master's Thesis or Dissertations it is suggested that a project should be requested with the graduate student's advisor as the project principal investigator to obtain a project directory. If a 1 terabyte project space is not sufficient please submit a project directory request on the portal to request more project storage space. Specify all the information requested on the form in order to process the request. Lustre project directories will be accessible under the /lustre/haven/courses and /lustre/haven/proj/ paths.

Lustre file systems are protected by RAID and are generally reliable. However, if requested Lustre project directories can be backed up for a fee. Submit a service request if you want this service.

Quotas on Lustre project directories will be set to the total storage space requested (default 1 terabyte) by the PI of the project.

Note: Lustre project directories are NOT purged.

Scratch Directories


Scratch directories are available for all users on the Lustre Haven filesystem. Approximately 2.7 petabytes of Lustre storage space is available on /lustre/haven which is shared with scratch directories and project directories.

Important: Lustre scratch directories are NOT backed up.

Important Purging Notice: Lustre Haven scratch space is purged monthly on approximately the 3rd Monday of each month. Files in Lustre Haven scratch directories are deleted by the purging process if they have not been accessed or modified within 180 days. Over the course of 2020, the purging timeperiod will be gradually reduced to 90 days. See the purging schedule below. In general, users have many temporary files that are no longer needed once a job completes and results are returned. Many times these files and other orphaned and unneeded files are not deleted by end users and they accumulate in scratch directories and can fill the file system which is detrimental to all users. Purging email notices are sent to all active users at least once at the beginning of each month to remind users that purging of scratch space will take place on the 3rd Monday of each month. This notice explains the purging process and how to request a temporary purge exemption or the process to request a project space (project directories are exempt from purging). To request a temporary purge exemption submit a service request with "temporary purge exemption request" in the subject. Project requests along with their corresponding project directory can be made at the portal project request page.

Table 2 - Lustre Haven Purge Schedule 2020
Purge Estimated DatePathFiles Purge if not accessed or modified in
Jul 20, 2020 /lustre/haven/user/<username> <180 days
Aug 17, 2020 /lustre/haven/user/<username> <180 days
Sep 21, 2020 /lustre/haven/user/<username> <180 days
Oct 19, 2020 /lustre/haven/user/<username> <180 days
Nov 16, 2020 /lustre/haven/user/<username> <180 days
Dec 21, 2020 /lustre/haven/user/<username> <180 days

To transfer data out of your scratch space see the ACF Data Transfer documenation.

Each user has access to a scratch directory in Lustre which is located at /lustre/haven/user/<username>. For convenience, use the $SCRATCHDIR environment variable to refer to your Lustre scratch directory.

If you wish to determine which files are eligible to be purged from Lustre space, execute the lfs find $SCRATCHDIR -mtime +180 -type f command. Files that will be purged from Lustre space are those that are not modified or accessed for 180 days. If you wish to view your total usage of Lustre space, execute the lfs quota -u <user> /lustre/haven command.

Any attempts to circumvent purging, such as using the touch command on all files in a users scratch directory, will be considered a violation of the ACF acceptable use policy. Instead of taking the time to circumvent purging, why not request a project with corresponding project space. As we are all Tennessee Volunteers, our research community will be improved with positive user actions and behaviors, such as cleaning up unneeded files or requesting a project, instead of circumventing ACF file purging policy. This will result in less wasted staff support time.


Return to Top


Last Updated: 01 / 29 / 2020