Each Darter node has 16 cores and 32 Gbytes of memory: about 2 GB per core if all cores are used. Sometimes it is necessary to leave some cores idle to make more memory available per core. For example, if you use 8 cores per node, each core has access to about 4 Gbytes of memory.
Your file transfer has caused a Lustre storage server (OST) to become full, resulting in an error like:
ead_cond_timedwait() return error 22, errno=0 OUT OF SPACE condition detected while writing local file
This usually happens because the stripe count is too small (often 1). To solve this issue, remove the partially transferred file and change the stripe count of the directory before transferring the file. To change the stripe count of the directory, first
cd to that directory. Second, type the following command:
In order to enable the creation of a coredump file when a program crashes in the compute node of a CRAY system like Darter, the following command should be added to the job script before the aprun call:
|Bourne shell||ulimit -c unlimited|
|C shell||limit coredumpsize unlimited|
For example if using a Bourne like job scrip, the script will look like:
- Replace all compiler commands (
pgf90, etc) with the following:
- Remove all references to MPI libraries and environment variables related to third-party libraries within the makefile.
The MPI_IN_PLACE option causes communication on an intra-communicator to happen in place, rather than being copied into buffers. This reduces the required number of operations (it is only possible within a node, not between nodes).
In order to use this option with
MPI_Alltoall, you need to disable Cray's optimization for that call:
Cray's MPICH has a number of settings (changed using environment variables) that affect what algorithms are used, buffer space, etc. For a list of these variables and their default settings, you can set the following prior to calling
This causes rank 0 to display all MPICH environment variables and their current settings at MPI initialization time. If two or more nodes are used, MPICH/GNI environment settings are also included in the listing.
Users may access HPSS from any NICS high-performance computing (HPC) system with the Hierarchical Storage Interface (HSI) utility. An OTP token is required upon entry. Access to HPSS is enabled by typing the command
hsi in your linux environment. To exit, simply type
Some sites recommend using the
.modulerc file to set your default modules. Do not do so on NICS systems
For Darter, consult the “Cray online documentation” (http://docs.cray.com).
For C, search for the Cray “C and C++ Reference Manual” and for Fortran, consult the “Cray Fortran Compiler Commands and Directives Reference Manual”.