The National Institute for Computational Sciences

Parallel Execution on Nautilus

Nautilus supports two main modes of parallel execution, message passing using MPI and shared memory multi-threading using OpenMP or pthreads. Executing programs using these two modes require using different mechanisms.

Batch script examples and common PBS information is available on the Batch Scripts page.

MPI


MPI programs must be started using the mpiexec command. By default, mpiexec automatically determines the number of MPI processes to run based on the number of CPUs requested from the batch environment. You may run your MPI code on a smaller subset of the CPUs requested by explicitly specifying the number with the -n or -np option. The mpiexec command only works in batch and interactive MPI jobs.

Examples (interactive)

Using one MPI process per CPU requested:

> qsub -I -l ncpus=64
[job starts]
> mpiexec ./my_program

Using fewer MPI processes than CPUs requested:

> qsub -I -l ncpus=64
[job starts]
> mpiexec -n 42 ./my_program
> # --OR--
> mpiexec -np 42 ./my_program

Running an MPI program with command line arguments:

> mpiexec ./mpi_prog_with_args -arg1 -arg2 -arg3

OpenMP



OpenMP programs must also be run in either batch or interactive jobs. The number of threads spawned by an OpenMP program is controlled by the environment variable OMP_NUM_THREADS.

Example

> qsub -I ncpus=16
[job starts]
> export OMP_NUM_THREADS=16
> # equivalent csh/tcsh syntax is "setenv OMP_NUM_THREADS 16"
> ./omphw

For programs that use pthreads, use the OMP_NUM_THREADS environment variable in the same way to control the number of threads.