The Greenland and Antarctic ice sheets will make a dominant contribution to 21st century sea-level rise if current climate trends continue. However, predicting the expected loss of ice sheet mass is difficult due to the complexity of modeling ice sheet behavior.

To better understand this loss, a team of Sandia National Laboratories researchers has been improving the reliability and efficiency of computational models that describe ice sheet behavior and dynamics. The team includes researchers Irina Demeshko, Mike Eldred, John Jakeman, Mauro Perego, Andy Salinger, Irina Tezaur and Ray Tuminaro.

This research is part of a five-year project called Predicting Ice Sheet and Climate Evolution at Extreme Scales (PISCEES), funded by the U.S. Department of Energy's (DOE) Scientific Discovery through Advanced Computing (SciDAC) program. PISCEES is a multi-lab, multi-university endeavor that includes researchers from Sandia, Los Alamos, Lawrence Berkeley and Oak Ridge national laboratories, the Massachusetts Institute of Technology, Florida State University, the University of Bristol, the University of Texas Austin, the University of South Carolina and New York University.

Sandia's biggest contribution to PISCEES has been an analysis tool, a land-ice solver called Albany/FELIX (Finite Elements for Land Ice eXperiments). The tool is based on equations that simulate ice flow over the Greenland and Antarctic ice sheets and is being coupled to Earth models through the Accelerated Climate for Energy (ACME) project.

"One of the goals of PISCEES is to create a land-ice solver that is scalable, fast and robust on continental scales," said computational scientist Irina Tezaur, a lead developer of Albany/FELIX. Not only did the new solver need to be reliable and efficient, but it was critical that the team develop a solver equipped with next-generation and advanced analysis capabilities.

Tezaur said the team next needs to run the solver on new and emerging computers. They also need to be able to calibrate models and quantify uncertainties in expected sea-level rise.

"The data we get from climate scientists are usually measurements from the top surface of the ice," she said. "To initialize an ice sheet simulation, we need information about what is happening inside and at the bottom of the ice. Determining interior and bedrock ice properties is what we call model calibration, and requires the solution of an inverse problem. A lot of our work has been in developing and implementing optimization algorithms that are able to solve these inverse problems robustly and efficiently."

Tezaur stresses that the success of PISCEES is due in large part to strong collaborations between glaciologists, climate modelers, computational scientists and mathematicians.

"Glaciologists and climate scientist collaborators on PISCEES provide us with data sets to go into our model, while computational scientists come up with the right algorithms to use and implement them efficiently," she explains.

Improving computational modeling
The PISCEES project began in 2012 because there was no robust land-ice model as a building block for earth system models that calculate sea-level rise predictions to support the DOE's climate missions.

Sandia's Albany/FELIX solver includes advanced capabilities and provides increased robustness, scalability and speed, Tezaur said.

"In just three years of work, we have created a next-generation land-ice model that is verified, scalable and robust and portable to new and emerging architecture machines," Tezaur said. "These models are equipped with advanced analysis capabilities."

The Albany/FELIX solver was written using the so-called "component-based" software development strategy, an approach devised by Sandia computational scientists in which new application codes are written using mature modular libraries. New solvers created using this approach are "born" scalable: fast, robust and capable of advanced analysis since they are based on a collection of algorithms developed and tested by domain experts.

The components comprising Albany/FELIX are the Trilinos libraries, a collection of open-source packages developed by Sandia. In addition to recommending and executing the component-based code development strategy, Sandia researchers have developed approaches for improving the robustness of the nonlinear solver.

The Albany/FELIX code has demonstrated scalability up to 1 billion unknowns and tens of thousands of cores thanks to parallel scalable iterative linear solvers and newly developed preconditioning methods by Tuminaro. Adjoint-based deterministic inversion algorithms and software developed and implemented by Perego have enabled rigorous model calibration.

In collaboration with experts from the QUEST SciDAC institute, Eldred and Jakeman, a framework for forward and inverse uncertainty quantification (UQ) has been developed. Finally, Albany/FELIX has been made portable to new architecture machines thanks in large part to the efforts of computer scientist, Irina Demeshko.

Verification and validation are important
While code performance is critical to the success of Albany/FELIX, equally important are verification and validation, two procedures for evaluating a model and its code. Verification ascertains that a code is bug-free. In contrast, validation seeks to check that the physical process described by a model is consistent with what is seen in the real world.

Sandia has done a thorough verification of the Albany/FELIX solver using the method of manufactured solutions, code-to-code comparisons on canonical land-ice benchmarks and by performing convergence studies on realistic Greenland and Antarctica landscapes with real data, Tezaur said.

Tezaur explains that, in general, validation is much harder to do than verification. The degree to which ice sheet models have been validated by observations is fairly limited, due in part to the limited duration of the satellite observation era and the long adjustment time scales of ice sheets.

The PISCEES team has checked that its solver predicts ice sheet quantities such as surface velocities, surface mass balance and that these quantities are consistent with past and recent observations.

"We are in the process of doing a validation study for the Greenland Ice Sheet for the period 1991-2012," Tezaur said. "The ice sheet model output will be compared to ice surface elevation and ice sheet mass change observations from ICESat and IceSat 2, the Ice, Cloud and land Elevation Satellite."

Early results show promise for assessing the performance of different model configurations. A verification and validation test suite, known as the Land Ice Validation and Verification Kit, is being developed by PISCEES collaborators at Oak Ridge, and goes hand-in-hand with the efforts at Sandia.

In recent months, climate scientists Steve Price and Matt Hoffman at Los Alamos have begun to use Albany/FELIX for production runs on Edison and Cori, two of the newest Cray supercomputers at the National Energy Research Scientific Computing Center. During the next two years, calculations provided by the team will be used to estimate expected sea level rise during the 21st century.