Compiling

You need to do "module load rca". This will most likely affect anyone building NAMD or anyone using CHARM++ for development work.

• Replace all compiler commands (mpicc, mpif90, icc, ifort, pgCC, pgf90, etc) with the following: cc (C), CC (C++) or ftn (Fortran).
• Remove all references to MPI libraries within the makefile.
• Any references to libraries BLAS, LAPACK, BLACS, and ScalaPACK should be removed from your makefiles. The system will automatically link with the most highly optimized versions of these libraries. (For a complete list of libraries, enter: man libsci)
• References to MKL can often be removed because their function is replaced by libsci.

Before you compile your code, load any relevant modules for third-party libraries. For example:

module load hdf5-parallel

The documentation will tell you how to use environment variables in your makefile. In the hdf5 example, this is documented in HDF5.

cc -o hdf5example.x hdf5example.c ${HDF5_CLIB}

There are two advantages to using the module with the environment variable instead of the pathname:

1. If you change versions of hdf5, you only need to load a different module. The makefile does not have to be modified.
2. If you change to a different compiler and then reload the hdf5 module, the system will load a version of hdf5 that is compiled with the other compiler.

For a list of libraries and other software available for Kraken and Athena, see NICS Software.

For Kraken consult the “Cray online documentation” (http://docs.cray.com).

For C, search for the Cray “C and C++ Reference Manual” and for Fortran, consult the “Cray Fortran Compiler Commands and Directives Reference Manual”.

The Cray XT4 and XT5 are little-endian. There is a compiler switch -Mbyteswapio that makes the default Fortran unformatted I/O big-endian (read and write.)

Note that this little-endian-to-big-endian conversion feature is intended for Fortran unformatted I/O operations. It enables the development and processing of files with big-endian data organization. The feature also enables processing of the files developed on processors that generate big-endian data (such as IBM, Cray X1, Sun).

This error message indicates that the node is running Out Of Memory. This could be the result of a bug in the code, or memory requirements for the given input. Note that due to optimistic memory allocation, you probably will not get a null pointer, even if you are out of memory. The program should be killed at the point the memory is used.

One quick solution might be to run with only four MPI processes per socket so each process gets a larger share of the memory on the node:

aprun -n <XXXX> -S 4 ./a.out

Where <XXXX> is the total number of MPI processes. The above solution uses 4 out of 6 cores on each socket, so naively, each MPI task should get 50% more memory (6/4). If this is not enough memory, it is possible to reduce the number of tasks per core further (-S 2). The best solution may be to identify the memory requirements in the code and make any necessary changes there, in terms of memory parameters, domain decomposition, etc.

There is a bug that "affects" all previous PGI compilers that supported the C99 standard. The problem discovered, is related with the processing of a new array type added in C99 called "Variable Length Array", or VLA for short.

#include is the Standard C++ way to include header files. The 'iostream' is an identifier that maps to the file iostream.h. In older C++ versions you had to specify the file name of the header file, hence #include . Older compilers may not recognize the modern method but newer compilers will accept both methods even though the old method is obsolete.

The following error message:

#error "SEEK_SET is #defined but must not be for the C++ binding of MPI" 

Is the result of a name conflict between stdio.h and the MPI C++ binding. Users should place the mpi include before the stdio.h and iostream includes.

Users may also see the following error messages as a result of including stdio or iostream before mpi:

#error "SEEK_CUR is #defined but must not be for the C++ binding of MPI" 
#error "SEEK_END is #defined but must not be for the C++ binding of MPI"

When profiling with TAU, you may get this message regardless of the order. In this case, you can add -DMPICH_IGNORE_CXX_SEEK to the compile line to remove the error (this fix should work generally).

Under the 1.5 programming environments used under Catamount, ftn linked in libC.a. Under the 2. programming environments used under CNL, ftn does not link in libC.a. Fortran codes that link in libraries that contain C++ objects will need to add -lC to the link line.

Internally, the compilers use several variables/macros even if theyre not specified on the command line. These include F90FLAGS, FFLAGS, CFLAGS, and others. If your make file defines these variables with flags not intended for the link step, the link may fail. For example, if they contain the -c flag, which tells the compiler to skip the link step, the link will fail.

Use the pgf90 compiler (via the ftn wrapper) to link and provide the -Mnomain option.

If you are receiving the “multiple definition of main” error, you probably have a C program that calls Fortran, and you are linking with the Portland Group Fortran compiler. The Fortran compiler has its own default “main,” and now there is a second main from the C source. You may need to add the -Mnomain flag during link time to fix this.

In the other case, you may get undefined reference to `main', another option is to use the C/C++ compilers to link. Now, 'main' may be defined manually: -Wl tells pgcc/pgCC to pass the following comma-deliminated list to the linker, --defsym defines a list of symbols. Thus, the following should allow your Fortran-with-C program to compile and link (assuming the PGI modules are loaded)

cc -Wl,--defsym,main=MAIN_ ...

The “relocation truncated” error message occurs when an object file or executable is too large for the memory model. The default memory model for the PGI compilers is the “small” model. This requires that the object be smaller than 2 GB in size. The PGI compilers support the “medium” memory model, which allows objects to be larger than 2 GB. Unfortunately, for a code to use the medium memory model, all objects and static libraries must be compiled under the medium memory model. Several system libraries are not, so in general, executables on Kraken must use the small memory model.

To work around this error, you should reduce the static memory usage for your code. Common ways to do this include the following:

• Remove (either by deleting or via compiler directives) subroutines that are not used on the XT platform.
• Remove static variables (especially large arrays) that are not used on the XT platform.
• Use allocatable arrays instead of static arrays. Because the memory model applies to only static size, allocatable arrays can be larger than 2 GB with the small memory model.

This error message occurs when using the mpi* compiler wrappers (mpicc, mpif90, etc.). These are intermediate wrappers that should not be called directly by users. Instead, users should compile with either ftn, cc, or CC. The ftn, cc, and CC scripts will do the necessary setup and then automatically call the appropriate intermediate scripts and ultimately the compilers.

We support PGI, GNU, Intel, and Cray compilers. These should be more than sufficient; it is unlikely that we will add other compilers such as Borland. More information on compiling can be found at Compiling and see Modules for assistance with modifying your environment and changing compilers. When compiling code for the compute nodes, do not use the compilers directly, instead use the Cray wrappers (cc, CC, ftn).

We are investigating some new profiling features for standard MPI programs and are unlikely to purchase other compilers unless there is a strong demand for them. Please let us know if you would like to request a compiler (or any other software) by sending us an email at help@xsede.org