The National Institute for Computational Sciences

Darter: Why is my job being rejected by the scheduler?

Each Darter node has 16 cores and 32 Gbytes of memory: about 2 GB per core if all cores are used. Sometimes it is necessary to leave some cores idle to make more memory available per core. For example, if you use 8 cores per node, each core has access to about 4 Gbytes of memory.

#PBS -l walltime=01:00:00,size=1500

aprun -n 1500 -S 4 $PBS_O_WORKDIR/a.out

The above aprun command won't work. The nodes on Darter have 2 sockets and each socket has 8 cores. That makes a total of 16 cores per node. Your size should be a multiple of 16. To make a long story short, use the following formula to get close to a multiple of 16 with what you want to do.

cores per socket on Darter * size / mpi processes per socket

8*(1500)/4 = 3000

The next number that is a multiple of 16 is 3008. Change your size = to 3008 in your pbs option and you should be fine.