If you need to run many instances of a serial code (as in a typical parameter sweep study for instance), we highly recommend using Eden. Eden is a simple script-based master-worker framework for running multiple serial jobs within a single PBS job.
Python's multiprocessing module is similar to threading, so you should use the following in your Darter batch script to launch the python script on a single node:
module load python aprun -d 16 python script.pyThis will make all 16 cores on the node available to the Python script. Please note: whether or not the cores are fully utilized is up to the programming of the script.
If the python script is parallelized using MPI (e.g. with mpi4py which is available on Darter), then it should be run just like any other MPI program using the following syntax in your batch script:
module load python aprun -n numproc python parallel_script.pyIf there is no MPI in the python script, use the following syntax in your batch script:
module load python aprun python serial.py
Listed below are the limits on the compute nodes of NICS operated resources. Here are the results from some basic tests that were run on our resources to check the real maximum values for allocatable memory and open files:
System | MaxMem | MaxOpenFiles ------------------------------ Darter | 31.1GB | 1015 Nautilus | 32.1GB | 48 Keeneland | 32.1GB | >2048
Request more time for interactive jobs by providing a specific number of hours/minutes/seconds using
qsub –I –l walltime=hh:mm:ss
Note that 24 hours is the maximum that can be requested. If you need an extension, send an email to email@example.com along with any job ids that need to run for more than 24 hours.
They should be stored at $TMPDIR/mic# and they need to be copied to either the home or Lustre filesystem before the submitted job is complete
If a user wants to use:
#PBS -l size=192 ### Assuming you want to use 24 MPI tasks
aprun -n 24 -N 2 -S 1
Here's what the above aprun command means. You are asking for 24 MPI tasks, 2 MPI tasks per node, and 1 MPI task per socket.
Each Darter node has 16 cores and 32 Gbytes of memory: about 2 GB per core if all cores are used. Sometimes it is necessary to leave some cores idle to make more memory available per core. For example, if you use 8 cores per node, each core has access to about 4 Gbytes of memory.
Array jobs are not supported on NICS systems. The submission filter will reject jobs which make use of job arrays (i.e. #PBS -t or qsub -t). These jobs (if submitted) will not run and should be deleted.
The shell initiation line in PBS scripts is not guaranteed to be used to determine the interpreting shell. The default behavior is to use the user's default login shell or the value of the PBS option -S (i.e. #PBS -S /bin/bash or qsub -S /bin/bash). If you are using a shell for a PBS script which is different than your default shell, please use the PBS -S option.