• National Institute for Computational Sciences is a UT/ORNL Partnership
Home

CrayPAT


Overview of CrayPAT Tools

CrayPAT is a package for instrumenting and tracing code. It may be used to selectively trace specific functions, or an entire application. The latter is highly discouraged for even moderately large applications. Trace files are inherently very large, and the overhead for running these applications can also be large. Also, it should be noted that CrayPAT attempts to account for profiling overhead, however, this is necessarily imperfect, and may result in false hot-spots. This can be mitigated by inlining small functions.

Profiling with the Cray tools requires multiple steps. Unlike the X1E it does require you to recompile your code. CrayPAT requires the xt-craypat (and probably apprentice2) modules to be loaded. Code must recompile your code with ftn or cc (the Cray wrappers) to link in the appropriate Cray performance tools/libraries. Additional resources may be found at the following man pages: pat, pat_build, pat_report, and pat_hwpc. Similar information may be available with the interactive help, invoked by pat_help. Note that the Fortran application programming interface (API) is similar to the C API, with an additional argument for the status of the call (which in C is provided as the return value).

Presentations on using CrayPAT may be found in our Workshop Archive.

Note: CrayPAT is incompatible with modules which set -B compile options. When pat_build is run, it will fail with the following error. Type env | grep "\-B" to determine if this is an issue. 

ERROR: Missing required ELF section 'link information' from the program. 
Load the correct 'craypat' module and rebuild the program.

Profiling and Hardware Performance Counter data

Automatic Profiling Analysis

Use the following steps to automatically profile your code:

  1. Remove all object files and any other user libraries that you want profiled. For example, do a make clean.
  2. Issue the commands 
      
  module load xt-craypat
  module load apprentice2
  3. Build your code with the following guidelines (you may not need to do anything different).
    • Use the cc, CC, or ftn compiler wrapper, which will automatically link against the proper libraries
    • CrayPAT relies on object files. If these are removed or located in different directories, CrayPAT may not work.
    • You must rebuild to ensure that proper symbols are in your code. Some makefiles, for example, contain strip commands which must be commented out.
  4. Run the pat_build command to build an instrumented executable: pat_build -O apa a.out, where a.out is the executable file you are instrumenting. This should create a file a.out+pat. The apa option will give an instrumentation overview, with which you can find more detailed profiles of "hot spots." If you already know the functions you would like to concentrate on, see the next section.
  5. Run a.out+pat (see Running Jobs). If you are using a PBS batch file, remember to modify it to run the instrumented version. This should create one or more .xf files.
  6. If there is one .xf file, use pat_report <file.xf>. For multiple (related) .xf files, instead give the directory containing the files. This sends a simple text version of the report to stdout. If you wish to save this, keep >> a.out.log.
  7. There should be a .apa file, which specifies functions to be traced. You may edit this file to your satisfaction, and re-instrument you code with pat_build -O <file>.apa. If no files have been moved, the .apa file should remember the executable.
  8. This should have created a file a.out+apa. Again, run this, and use pat_report on the resulting .xf file(s) (note that the more recent files have +apa rather than +pat in the filename). The resulting .ap2 may be viewed with Apprentice2 (GUI) or pat_build (CLI).
    • This output from this includes both a profile and hardware performance data (using group 1.)
    • To get different hardware stats, set the PAT_RT_HWPC environment variable to correspond with the desired hardware performance counter and then run the a.out+apa code again. See man hwpc for a list of options after loading the xt-craypat module.

Run-Time Library

To obtain very specific profile data, you can explicitly insert CrayPAT calls in your code. For example:

program test_module_kronuse pat_apiinteger ierr 
! Begin region of interest 

call PAT_region_begin ( 1, 'kron_matmul_kernel', ierr ) ! # and name must be unique to each region 

call kron_matmulL(
) 

! End region of interest 

call PAT_region_end   ( 1, ierr ) 

end program
  1. Compile with ftn wrapper. (Make sure the xt-craypat module is loaded.)
  2. Relink with pat_build -w a.out.
  3. Run and produce a report as outlined in previous sections.

Viewing Data

Profiles may be stored in a .xf or .ap2 format. The former requires access to the original executable file, whereas the latter is portable. Either format may be viewed with pat_report (as long as the .xf file has access to the executable). To convert either format to a vanilla text document, use pat_report -f txt <.xf or .ap2 file>. For more information, see man pat_report.

To use the GUI viewer, make sure X11 forwarding is enabled (echo $DISPLAY should be defined). Load the Apprentice2 module (module load apprentice2), and type app2 <file.ap2> &. The ampersand causes the process to run in the background so you don't lose your shell. Note that additional information on a given process can be obtained by mousing over or clicking on that process.