kalypsso: A performance portable platform for compressible hydrodynamics simulations using adaptive mesh refinement
Description
We introduce kalypsso (a Kokkos Applicative LaYer for Parallel and Scalable Solvers on Octrees): a new octree-based block-structured adaptive mesh refinement (AMR) framework using the C++ kokkos library for designing performance portable applications in computational fluid dynamics (CFD). Mesh management in distributed memory is implemented with the help of the p4est library, which provides a MPI parallel CPU implementation of the forest of octrees AMR algorithms. All heavyweight application data are allocated on a computing device, either a CPU or a GPU, and managed directly by kalypsso. One of the key design choice of kalypsso architecture is to use a lightweight hash-table-based (or dictionary) data structure for exchanging mesh geometry information between p4est, running on the host CPU, and the computational kernels executed on the accelerated device. Several finite volume methods for compressible monofluid and bifluid hydrodynamics, as well as magnetohydrodynamics are implemented using the kokkos programing model for exploiting shared memory parallelism on most existing CPU and GPU-based architecture. Node-level performance metrics for a second-order MUSCL-Hancock finite volume solver are measured to evaluate the impact of the size of the grid of cells attached onto octree leaves. A single Nvidia GH200 GPU can perform about 1.4 billions cell-updates per second. The performance portability on a cluster of CPU and GPU is demonstrated; a node-to-node weak scaling efficiency of ∼ 80% is obtained on a cluster of 512 Nvidia GH200 GPUs. Using comparable hardware resources and considering the Euler equation solver in kalypsso with AMR activated, a consistent × 5 CPU to GPU time to solution speed-up is obtained.