All non-interactive jobs must be submitted on Kraken through a job script via the qsub command. All job scripts start with a series of #PBS declarations that describe requirements of the job to the scheduler. The rest is a series of shell commands of your choice, which will usually include the aprun command that launches your executable.
The aprun command is used to launch one or more parrallel jobs on compute nodes.
Aprun accepts the following common options:
| -n | Total number of MPI processes (default: 1) |
| -N | Number of MPI processes per node. ( XT5: 1–12.) |
| -S | Number of MPI processes per socket (XT5 1–6) |
| -d | Specifies number of cores per MPI process (for use with OpenMP, XT5: 1–12) |
The following example shows a typical job script that includes the minimal requirements to submit a parallel job that executes ./a.out on 192 cores, charged to the fictitious account UT-NTNL0121 with a wall clock limit of one hour and 35 minutes:
#!/bin/bash #PBS -A UT-NTNL0121 #PBS -l size=192,walltime=01:35:00 cd $PBS_O_WORKDIR aprun -n 192 ./a.out
Jobs should be submitted from within a directory in the Lustre file system. A best practice technique is to always execute cd $PBS_O_WORKDIR as the first command. The $PBS_O_WORKDIR environment variable is internally defined by the scheduler at run time to point to the directory from which the qsub was issued. If the `cd $PBS_O_WORKDIR` command is used to make sure that you are in the Lustre file system, then the qsub command must be made from a location within Lustre.
Jobs may not reserve a portion of the (12) cores on a node. Jobs must schedule entire nodes for a run. Because there are 12 cores per node, the number of cores for the reservation should be a multiple of 12, even if a job requires fewer cores.
There is online documentation that describes the many PBS options that can be used for more complex job scripts.
Unless otherwise specified your default shell interpreter will be used to execute shell commands in job scripts. In some cases it may even try to guess what interpreter to use. If the job script should use a different interpreter, then specify the correct interpreter using:
#PBS –S /bin/XXXX
