The National Institute for Computational Sciences

NICS' Betro Discusses Performance of GYRO on Cray Supercomputers

By Scott Gibson

Vince Betro, computational scientist at the National Institute for Computational Sciences (NICS), presented a talk on June 7 to staff members of the University of Tennessee’s (UT) Innovative Computing Laboratory at the UT Claxton Education building on the performance of the fusion code GYRO on four generations of Cray Computers. The aim of the talk was to explain what developers need in their products relative to portability and functionality, as well as provide insight into the Intel® MKL (Math Kernel Library) and the Intel® Xeon Phi™.

GYRO is a code used for the direct numerical simulation of plasma microturbulence. It has been ported to a variety of modern MPP (massively parallel processor) platforms, including several modern commodity clusters, as well as IBM and Cray supercomputers.

Betro briefly described the mathematical structure of the equations, the data layout and the discretization scheme. While the performance and scaling of GYRO on many of these systems has been shown before, he presented the comparative performance and scaling on three generations of Cray supercomputers simultaneously, including the newest additions, the Cray CS300-AC™ Cluster Supercomputer and the Cray XC30.

He also explained that the recently added hybrid OpenMP/MPI implementation holds a great deal of promise not only on traditional high-performance computing systems that use fast CPUs and proprietary interconnects but also on a coprocessor architecture in a cluster environment in native mode.

Four machines of varying sizes were used in the experiment, all of which are located at NICS and Oak Ridge National Laboratory. Darter is a 23,936-core Cray XC30 with Intel E5-2670 CPUs, a Dragonfly network, and a Sonexion lustre file system; Kraken is a 112,896-core Cray XT5 with AMD Opteron™ Istanbul CPUs; Mars is an 896-core Cray XE6 with AMD Opteron Interlagos CPUs; and Beacon is a 768-core Cray CS300-AC™ Cluster Supercomputer with Intel Xeon® E5-2670 (Sandybridge) CPUs and Intel Xeon Phi coprocessors.

Betro discussed the advantages, limitations and performance of each system, as well as the direction of future optimizations. In addition, he showed the results from porting several applications codes to the UT Beacon machine and the Intel MIC architecture, all done as part of the NSF-funded Beacon Project. Some of the main conclusions were that proprietary interconnects give better scaling performance and that the Xeon Phi can scale well to many threads if the proper paradigm is used to take advantage of the coprocessor.

Relatedly, Betro and NICS co-author, Mark Fahey, won the International Supercomputing Conference 2013 Poster Award for their poster entitled, “Performance of the Fusion Code GYRO on Three Generations of Cray Computers.” Betro and Fahey will travel to Leipzig, Germany, to present their work and receive the award during a ceremony at the conference on June 17.

About NICS: The National Institute for Computational Sciences (NICS) operates the University of Tennessee supercomputing center, funded in part by the National Science Foundation. NICS is a major partner in NSF’s Extreme Science and Engineering Discovery Environment, known as XSEDE. The Remote Data Analysis and Visualization Center (RDAV) is a part of NICS.

UT's Innovative Computing Laboratory