This guide gives an overview of how to use the Kraken system. If you need more information, see the full user guide.
- Kraken System
- Getting an Allocation
- Connecting
- File Systems and Storage
- Data Transfer
- User Environment
- Compiling
- Running Jobs
- Debugging and Optimization
- Other Documentation
Kraken System
Kraken is a Cray XT5 system with 8,256 compute nodes. Each node has:
- two 2.6 GHz six-core AMD Opteron processors (Istanbul)
- 12 cores (system total is 99,072 compute cores)
- 16 GB of memory (system total is 129 TB of compute memory)
Compute nodes run Cray Linux Environment (CLE) 2.2. Each node is connected to a Cray SeaStar router through HyperTransport, and the SeaStars are all interconnected in a 3-D-torus topology.
Getting an Allocation
Details on the eligibility requirements for principal investigators, the application process and the proposal deadlines are located on the TeraGrid Allocations page.
Connecting
If you have mailed in the notarized form and received email confirmation that your RSA One Time Password (OTP) token has been activated, use ssh to connect to kraken.nics.tennessee.edu. Otherwise, you can use GSI authentication to log in to kraken-gsi.nics.tennessee.edu.
File Systems and Storage
Home directories have a 2 GB quota. These directories are on a Network File System (NFS), and are generally available even if Kraken is down. However, home directories are not accessible from the compute nodes. In a batch job, the current directory, input and output files must be on Lustre. Otherwise, the job will fail with an error message saying no such file or directory.
Each user has a scratch directory in /lustre/scratch/$USER which is subject to purging based on the time of last access. Lustre is a highly scalable cluster file system that allows you to increase striping width to improve I/O performance. Programs launched with the aprun command can only access Lustre, not home directories or other directories.
Files may be stored on the mass storage system (HPSS) using the hsi command. You may only access HPSS if you have activated your RSA OTP token. You may access HPSS within a batch job, but only within jobs in the "hpss" queue that are submitted from the RSA SecurID login nodes (kraken.nics.tennessee.edu).
Data Transfer
If you are moving large amounts of data (>1 Gbyte) from another TeraGrid system, use GridFTP. For smaller amounts of data, use sftp.
User Environment
Kraken uses the module command (softenv is not available) to select libraries, compilers and other software that are not already in the default user environment.
Compiling
By default, Kraken uses the Portland Group (PGI) compilers. GNU and PathScale compilers are available via the module command. To compile your MPI program, use the cc, CC, and ftn commands for C, C++ and Fortran programs. These commands create executables to run on the compute nodes. mpicc, mpicxx, mpif77 and mpi90 are not available on Kraken. The Cray scientific libraries (BLAS, LAPACK, ScaLAPACK, BLACS, and others) are in the default path. Use the module command for access to libraries such as HDF, netCDF, FFTW, and PETSc.
The login nodes are not intended for computing, but if you need to run short pre- or post-processing programs on login nodes, you can compile them with the PGI compiler commands pgcc, pgCC, pgf90, etc.
Running Jobs
Kraken uses PBS/Torque (qsub, qdel, qstat, etc) for batch jobs. To specify the number cores allocated to a batch job, use "#PBS -l size=cores" in the batch script (where cores is the number of cores). Since the batch system allocates in units of 12-core nodes, the number of cores must be a multiple of 12.
To specify an account to charge, use "#PBS -A account". The showusage command lists your valid accounts/projects.
Parallel programs may only be run within a batch job. mpirun is not available on Kraken. Instead, use "aprun -n nprocs executable" command to start parallel programs on the compute nodes. By default, it runs 12 MPI processes on each dual-socket node (one per core). To run with fewer MPI processes per socket, use the -S option. For example, "aprun -n nprocs -S 4 executable" runs with 4 MPI processes per socket (8 per node).
A sample batch script is available. Use "qsub script" to submit a batch job. Use qsub -I for interactive tasks such as debugging.
Debugging and Optimization
Kraken has a license for running TotalView on up to 64 processors. CrayPAT (Cray Performance Analysis Tools) is useful for profiling and collecting hardware performance data to determine bottlenecks in performance.
