PBS tools

Copyright 2005-2009, Ohio Supercomputer Center

Copyright 2008-2009, University of Tennessee

Contained herein are several utilities that have been developed at NICS, OSC, and elsewhere to aid in the administration and management of PBS variants (including OpenPBS, PBS Pro, and TORQUE). They have been developed primarily on Linux clusters, but most of them should be applicable to virtually any system running PBS.

All programs in PBS Tools are licensed under the GNU GPL version 2.

User tools

• bin/ja -- Job accounting within a PBS job; similar to ja in NQE.
• bin/pbsdcp -- Distributed copy command within a PBS job. This can optionally include an MPI-based scatter functionality that should scale better than the default rcp-based approach.
• bin/qps -- ps for PBS jobs; relies on the all command.
• bin/qpeek -- Allows users to look at spooled output of running jobs.
• bin/qexec -- Workalike for the SGE qlogin, qrsh, and qsh commands.
• bin/parallel-command-processor -- An MPI program for running an array of serial jobs in the context of a single parallel job (a.k.a. the poor man's job arrays).
• bin/supermover -- A highly configurable wrapper around other data transfer utilities such as scp, globus-url-copy, and hsi.
• bin/dmsub -- A tool for submitting data movement jobs. It understands data transfer descriptions in the formats of DMOVER, RFT, and Stork; it can also use several different data movement tools, including supermover.
• bin/dagsub -- A workalike for condor_submit_dag. This allows the submission of large, complex sets of dependent jobs using a relatively simple syntax. It relies on dmsub for data movement.

Admin tools

• sbin/dezombify -- Requeue a "running" PBS job that's not actually running because it had JOIN JOB failures. Relies on the all command; requires root privilege.
• sbin/qtracejob.pbs-server -- Wrapper around the tracejob command to give a time-sorted list of log messages from all hosts involved. Must run on host where pbs_server resides; requires root privilege.
• sbin/reaver -- Find (and optionally clean up) processes on a host which have not been allocated jobs on that host; requires root privilege.
• sbin/showscript.pbs-server -- Show the script associated with a jobid. Must run on host where pbs_server resides; requires root privilege.
• bin/js -- DB-based equivalent to showscript; does not require root access but should probably be ACL-limited to admin and support staff.
• bin/jobinfo -- Command line lookup of usage data in the accounting database; like bin/js, does not require root access but should probably be ACL-limited to admin and support staff.

Reporting tools

• sbin/jobstats -- Generate an HTML report summarizing PBS job statistics over a period of months. Relies on gnuplot; requires root privilege.
• sbin/find-outlyers -- Extract statistical outlyers from the jobdist.dat file generated by jobstats.

Database/web tools

Note: these all assume a MySQL DB back end running on the same server as pbs_server and PHP-enabled httpd.

• etc/create-tables.sql -- Create "pbsacct" database, table, and two user accounts (pbsacct and webapp) used by the other DB tools. To initialize, run mysql -u root -p passwd < create-tables.sql.
• sbin/job-db-update -- Parses PBS job accounting records and inserts them into DB.
• sbin/jobscript-to-db -- Inserts one or more job scripts into the job DB.
• sbin/spool-jobscripts -- Spools newly submitted jobscripts to /tmp and then invokes jobscript-to-db on them.
• web/*.php -- A number of web-based reports. These rely on PEAR::DB, JPGraph, Spreadsheet_WriteExcel, and ODS-PHP.
• examples/db-example -- An example of putting job data from multiple PBS instances into a single job DB.

Installation

Basic Tools

# ksh/bash syntax
export PREFIX=/usr/local
make install PREFIX=$PREFIX

MPI-Based Tools

# ksh/bash syntax
export PREFIX=/usr/local
export MPICC=mpicc
make mpitools PREFIX=$PREFIX MPICC=$MPICC

Perl-Based Reporting Tools

# ksh/bash syntax
export PREFIX=/usr/local
make statstools PREFIX=$PREFIX

PHP/MySQL-Based Reporting Tools

# ksh/bash syntax
export PREFIX=/usr/local
export WEBPREFIX=/var/www/html/pbsacct
export DBSERVER=localhost
make dbtools PREFIX=$PREFIX WEBPREFIX=$WEBPREFIX DBSERVER=$DBSERVER

Further details can be found in the INSTALL file.

Bug List

To-Do List

---

Suggestions

Comments and suggestions on this software are welcomed; please send them to Troy Baer or Doug Johnson.