The National Institute for Computational Sciences

Driving Blood-Flow Research at The Petascale

Simulations of arteries in 3D help scientists identify red blood cell properties

by Eric Gedenk

Choosing directions during a road trip used to consist of passengers holding a map with arms fully extended and noses to the page trying to decipher exactly where to turn or what exit to take. Today global positioning systems lay out the route for travelers, anticipating the curves of the road, when they have to turn, and where traffic jams and inclement weather await.

A group of Brown University researchers is attempting to create its own three-dimensional innovation in navigation, but instead of mapping interstates and city blocks, the team is charting the vascular highway of the human body—the arterial tree.

Specifically, the team, led by mathematician George Karniadakis, aims to develop comprehensive three-dimensional models of the arterial tree of the brain and other organs, such as the coronary tree in the heart, to improve predictive capabilities related to the progression of vascular and hematological pathologies. The National Institutes of Health and National Science Foundation are funding the researchers’ multiscale modeling of the vascular system at various scales.

“High-performance computing makes realistic modeling of vascular and hematological diseases, from a whole organ down to protein-level representation of red blood cells, a possibility,” said Karniadakis, discussing the research conducted on the Cray XT5 system Kraken, the world’s fastest academic supercomputer, located at the National Institute for Computational Sciences, a National Science Foundation–funded supercomputing center managed by the University of Tennessee and located at Oak Ridge National Laboratory. Karniadakis’s goal is to enhance the ability to chart the behavior of individual red blood cells and improve predictive capabilities in medical procedures. The team is also developing software and algorithms pertaining to blood flow for petascale supercomputers, or computers capable of at least a thousand trillion calculations per second.

The team is currently running simulations on approximately 30,000 of Kraken’s 99,074 computing cores and developing software capable of crossing the 100,000-core threshold. Computational research is attempting to couple the NEKTAR code for flow dynamics and a modified version of the LAMMPS code for particle dynamics to get a complete picture of blood’s properties as it travels in vessels.

A recent breakthrough made by the team is the prediction of human blood viscosity stemming from large-scale simulations on Kraken. The apparent increase of viscosity in some diseases such as diabetes and HIV provides an accurate indication of the state of the disease.

In addition, being able to chart blood flow more accurately could benefit surgeons attempting to figure out the best course of action in delicate operations, guide pharmaceutical manufacturers to more effectively deliver medications, and help pathologists predict conditions such as aneurysms and hydrocephalus. Children’s Hospital in Boston and Rhode Island Hospital in Providence are collaborating with the Brown group on the project.

I can see clearly now

Arteries are elastic by nature, and accounting for how elasticity affects blood flow’s velocity and interaction with surrounding tissues can only now be accurately predicted through petascale computing. Blood-flow simulations in one dimension are able to measure elasticity but unable to predict the interactions it has with blood’s movement through arteries or how blood vessels’ contact with bone or muscle affects flow.

The team found that one- and three-dimensional simulations yield similar results until elasticity is taken into account. Understanding the properties of elasticity is essential for some of the pathogenic and pharmaceutical research performed in these simulations. Mathematics researcher Leopold Grinberg believes that the current simulations are “sparking even more research to go deeper and understand the dynamics of arterial walls’ motion.”

Modeling blood vessels in three dimensions allows researchers to observe individual red blood cells travelling through an artery and their properties under different conditions. For instance, science knows that red blood cells become stiffer when a human is infected with malaria. The Brown team has taken this finding and is attempting to understand why it happens, how it can affect other cells and plasma moving through a blood vessel, and how cells interact with the artery walls.

Caption: A simulation of malaria-infected blood traveling through an artery of the brain. Infected blood cells are blue; healthy ones are red. The team coupled the NEKTAR flow dynamic code with the LAMMPS particle dynamic code to observe velocity and plasmas’ interaction with the cells. Image credit: Joseph A. Insley and Michael E. Papka, Argonne National Laboratory

Some conditions are still far more mysterious to scientists, though. Aneurysms are unexpected and can be life-threatening, and the Brown team is simulating the impact of stents and clotting inside the aneurysm to see how to better combat this condition. Simulations on Kraken over the past 2 years led to the discovery that certain types of aneurysms produce audible sounds in the range of 100 Hertz that can be used as a diagnostic tool for the affliction. These sounds are due to a flow instability induced by the secondary motion of blood inside the aneurysm sack.

Medical professionals could also use computer simulations to decide on the best method and location for insertion of a stent into an artery. Researchers would be able to more accurately observe the interaction between these devices and blood vessels and predict whether the chances of a blood clot are increased by inserting a stent in certain situations.

Wide-open road

According to Grinberg, completing the validation process for practically employing these simulations in the medical community is still several years away, but as computational power and algorithms improve, new findings in prediction, diagnosis, and computer modeling are always possible. Many of the algorithms being developed are nearing the point at which their complexity will surpass the processing ability of today’s fastest supercomputers, so computation and medicine will have to align before these simulations can be put to use.

Even as validation continues, though, the research has led to advancements in education by showing next-generation computational scientists real-life examples of the field in which they will be working. Grinberg has used simulations from Kraken in his classroom at both Brown and Tufts University, allowing high-performance computing students the opportunity to see something tangible within the field and work hands on rerunning simulations and creating their own applications—some of which were able to run on 10,000 cores—to see the entire process at work. Considering the volume of research that can be done on this topic, some of these students may be working on aspects of this project after graduation.

Many aspects of the circulatory system have still been virtually untouched by research performed using high-performance computing, and the volume of unknowns in this field has Brown researchers eagerly hoping to remain working on Kraken into the future. Despite the fact that computation speed must increase to advance the field in certain directions, Grinberg said that “two years ago [the team] wouldn’t have been able to think about these things at all,” referring to the research happening today. “We try to look into the future, not just tomorrow, but the day after tomorrow.”