Prospects for Improving Computational Efficiency of Hydrodynamic Simulations on Supercomputers by Increasing the Number of GPUs per Compute Node

Sergei S. Khrapov; Ekaterina O. Agafonnikova; Alexander V. Khoperskov

doi:10.14529/jsfi250204

Authors

Sergei S. Khrapov Volgograd State University, Volgograd, Russian Federation https://orcid.org/0000-0003-2660-2491
Ekaterina O. Agafonnikova Volgograd State University, Volgograd, Russian Federation https://orcid.org/0000-0002-2862-4531
Alexander V. Khoperskov Volgograd State University, Volgograd, Russian Federation https://orcid.org/0000-0003-0149-7947

DOI:

https://doi.org/10.14529/jsfi250204

Keywords:

computational fluid dynamics, GPUs, efficiency, scalability, data transfer

Abstract

Hydrodynamic models for studying surface water dynamics on realistic topography place special demands on computational performance. Such simulations must cover large areas to ensure hydrological connectivity of the territory due to the influence of catchment areas. On the other hand, small topographic inhomogeneities on the scale of a meter are often the determining factors of fluid dynamics. Our analysis is based on a model of surface water and sediment dynamics for a large mountainous area of the Krasnodar region under rainfall/runoff conditions. The results of such large-scale models can be provided by parallel OpenMP-CUDA codes for computing systems with multi-GPU. We focus on different ways of transferring data between GPUs using both GPUDirect and HostCopy technologies on computing systems with one to eight GPUs. The parallel code with HostCopy is on average several times slower and less efficient compared to the GPUDirect approach. We propose to use auxiliary characteristics to analyze the efficiency of parallel implementation of a numerical algorithm. These values are calculated based on the average processing time of one computational cell and allow us to determine the optimal grid resolution in terms of performance.

References

Aamodt, T.M., Fung, W.W.L., Rogers, T.G.: General-Purpose Graphics Processor Architectures. Springer Cham (2018)

Belikov, V., Borisova, N., Vasileva, E., et al.: A Numerical Hydrodynamic Model of a Long Segment of the Ural River and Its Application to Assessing the Inundation Risk of Residential Areas by Floods and Breakthrough Waves. Water Resources 51(5), 654–665 (2024). https://doi.org/10.1134/S0097807824701008

Bocharov, A., Evstigneev, N., Petrovskiy, V., et al.: Implicit method for the solution of supersonic and hypersonic 3D flow problems with Lower-Upper Symmetric-Gauss-Seidel preconditioner on multiple graphics processing units. Journal of Computational Physics 406, 109189 (2020). https://doi.org/10.1016/j.jcp.2019.109189

Diaz, M.C., Fernandez-Nieto, E., Ferreiro, A., et al.: Two-dimensional sediment transport models in shallow water equations. a second order finite volume approach on unstructured meshes. Computer Methods in Applied Mechanics and Engineering 198(33-36), 2520–2538 (2009). https://doi.org/10.1016/j.cma.2009.03.001

Dominguez, J.M., Fourtakas, G., Altomare, C., et al.: DualSPHysics: from fluid dynamics to multiphysics problems. Computational Particle Mechanics 9, 867–895 (2022). https://doi.org/10.1007/s40571-021-00404-2

Dominguez, J., Crespo, A., Valdez-Balderas, D., et al.: New multi-GPU implementation for smoothed particle hydrodynamics on heterogeneous clusters. Computer Physics Communications 184(8), 1848–1860 (2013)

Dong, B., Huang, B., Tan, C., et al.: Multi-GPU parallelization of shallow water modelling on unstructured meshes. Journal of Hydrology 657, 133105 (2025). https://doi.org/10.1016/j.jhydrol.2025.133105

Dyakonova, T., Khoperskov, A., Khrapov, S.: Numerical Model of Shallow Water: The Use of NVIDIA CUDA Graphics Processors. Communications in Computer and Information Science 687, 132–145 (2016). https://doi.org/10.1007/978-3-319-55669-7_11

Gorobets, A., Bakhvalov, P.: Heterogeneous CPU+GPU parallelization for high-accuracy scale-resolving simulations of compressible turbulent flows on hybrid supercomputers. Computer Physics Communications 271, 108231 (2022). https://doi.org/10.1016/j.cpc.2021.108231

Hafiyyan, Q., Harlan, D., Adityawan, M.B., et al.: 2D Numerical Model of Sediment Transport Under Dam-break Flow Using Finite Element. International Journal on Advanced Science Engineering and Information Technology 11(6), 2476–2481 (2021). https://doi.org/10.18517/ijaseit.11.6.14484

Jayaratne, R., Takayama, Y., Shibayama, T.: Applicability of suspended sediment concentration formulae to large-scale beach morphological changes. In: Lynett, P., McKee Smith, J.e. (eds.) Coastal Engineering Proceedings, vol. 1 (33), pp. 1–15. Coastal Engineering Research Council (2012). https://doi.org/10.9753/icce.v33.sediment.57

Khoperskov, A., Khrapov, S., Klikunova, A., et al.: Efficiency of Using GPUs for Reconstructing the Hydraulic Resistance in River Systems Based on Combination of High Performance Hydrodynamic Simulation and Machine Learning. Lobachevskii Journal of Mathematics 45(7), 3085–3096 (2024). https://doi.org/10.1134/S199508022460376X

Khrapov, S.: Numerical modeling of two-dimensional gas-dynamic flows in multicomponent nonequilibrium media. Mathematical Physics and Computer Simulation 28(1), 60–88 (2025)

Khrapov, S., Pisarev, A., Kobelev, I., et al.: The numerical simulation of shallow water: Estimation of the roughness coefficient on the flood stage. Advances in Mechanical Engineering 2013, 787016 (2013). https://doi.org/10.1155/2013/787016

Khrapov, S.: Numerical modeling of hydrodynamic accidents: Erosion of dams and flooding of territories. Vestnik of the St. Petersburg University: Mathematics 56(2), 261–272 (2023). https://doi.org/10.1134/s1063454123020085

Khrapov, S., Khoperskov, A.: Application of graphics processing units for self-consistent modelling of shallow water dynamics and sediment transport. Lobachevskii Journal of Mathematics 41(8), 1475–1484 (2020). https://doi.org/10.1134/S1995080220080089

Khrapov, S., Khoperskov, A.: Study of the Effectiveness of Parallel Algorithms for Modeling the Dynamics of Collisionless Galactic Systems on GPUs. Supercomputing Frontiers and Innovations 11(3), 27–44 (2024). https://doi.org/10.14529/jsfi240302

Klikunova, A., Polyakov, M., Khrapov, S., et al.: Problem of building high-quality predictive model of river hydrology: the combined use of hydrodynamic simulations and intelligent computing. Communications in Computer and Information Science 1909, 191–205 (2023). https://doi.org/10.1007/978-3-031-44615-3_13

Kotlyakov, V.M., Desinov, L.V., Dolgov, S.V., et al.: Flooding of July 6-7, 2012, in the town of Krymsk. Regional Research of Russia 3, 32–39 (2013). https://doi.org/10.1134/S2079970513010061

Li, W., Hu, P., Pahtz, T., et al.: Limitations of empirical sediment transport formulas for shallow water and their consequences for swash zone modelling. Journal of Hydraulic Research 55(1), 114–120 (2017). https://doi.org/10.1080/00221686.2016.1212942

Liu, T., Trim, S.J., Ko, S.B., et al.: The multi-GPUWetland DEM Ponding Model. Computers & Geosciences 199, 105912 (2025). https://doi.org/10.1016/j.cageo.2025.105912

de Luna, T.M., Diaz, M.J.C., Madronal, C.P.: On a sediment transport model in shallow water equations with gravity effects. In: Kreiss, G., Lotstedt, P., Malqvist, A., Neytcheva, M. (eds.) Numerical Mathematics and Advanced Applications, pp. 655–661. Springer, Berlin, Heidelberg (2010). https://doi.org/10.1007/978-3-642-11795-4_70

Macca, E., Avgerinos, S., Castro-Diaz, M.J., et al.: A semi-implicit finite volume method for the Exner model of sediment transport. Journal of Computational Physics 499, 112714 (2024). https://doi.org/10.1016/j.jcp.2023.112714

McKevitt, J., Vorobyov, E.I., Kulikov, I.: Accelerating Fortran codes: A method for integrating Coarray Fortran with CUDA Fortran and OpenMP. Journal of Parallel and Distributed Computing 195, 104977 (2025). https://doi.org/10.1016/j.jpdc.2024.104977

Mignot, E., Paquier, A., Haider, S.: Modeling floods in a dense urban area using 2D shallow water equations. Journal of Hydrology 327(1-2), 186–199 (2006). https://doi.org/10.1016/j.jhydrol.2005.11.026

Narasiman, V., Shebanow, M., Lee, C.J., et al.: Improving GPU performance via large warps and two-level warp scheduling. In: 2011 44th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), Porto Alegre, Brazil, pp. 308–317. IEEE (2011)

Ndengna, A.R.N., Njifenjou, A.: A well-balanced PCCU-AENO scheme for a sediment transport model. Ocean Systems Engineering 12(3), 359–384 (2022). https://doi.org/10.12989/ose.2022.12.3.359

Shao, X., Wang, X.: Introduction to River Dynamics. Tsinghua University Press Co., Beijing, China (2005)

Simonov, A.S., Semenov, A.S., Shcherbak, A.N., et al.: The High Performance Interconnect Architecture for Supercomputers. Supercomputing Frontiers and Innovations 10(2), 127–136 (2023). https://doi.org/10.14529/jsfi230208

Siviglia, A., Vanzo, D., Toro, E.: A splitting scheme for the coupled Saint-Venant-Exner model. Advances in Water Resources 159, 104062 (2022). https://doi.org/10.1016/j.advwatres.2021.104062

Sukhinov, A.I., Protsenko, E.A., Protsenko, S.V.: WAVEWATCH III Hybrid Parallelization for Azov Sea Wave Modeling. Supercomputing Frontiers and Innovations 11(1), 81–96 (2024). https://doi.org/10.14529/jsfi240104

Taccone, F., Antoine, G., Delestre, O., et al.: A new criterion for the evaluation of the velocity field for rainfall-runoff modelling using a shallow-water model. Advances in Water Resources 140, 103581 (2020). https://doi.org/10.1016/j.advwatres.2020.103581

Valles, P., Fernandez-Pato, J., Morales-Hernandez, M., et al.: A 2D shallow water flow model with 1D internal boundary condition for subgrid-scale topography. Advances in Water Resources 189, 104716 (2024). https://doi.org/10.1016/j.advwatres.2024.104716

Vasileva, E.S., Aleksyuk, A.I., Belyakova, P.A., et al.: Numerical modeling of the behavior of a destructive rain flood on a mountain river.Water Resources 46(1), 45–55 (2019). https://doi.org/10.1134/S0097807819070169

Vatyukova, O., Klikunova, A., Vasilchenko, A., et al.: The problem of effective evacuation of the population from floodplains under threat of flooding: algorithmic and software support with shortage of resources. Computation 11(8), 150 (2023). https://doi.org/10.3390/computation11080150

Voevodin, V., Antonov, A., Nikitenko, D., et al.: Supercomputer Lomonosov-2: Large Scale, Deep Monitoring and Fine Analytics for the User Community. Supercomputing Frontiers and Innovations 6(2), 4–11 (2019). https://doi.org/10.14529/jsfi190201

Zhang, X., Guo, X., Weng, Y., et al.: Hybrid MPI and CUDA paralleled finite volume unstructured CFD simulations on a multi-GPU system. Future Generation Computer Systems 139, 1–16 (2023). https://doi.org/10.1016/j.future.2022.09.005

Zolfaghari, H., Becsek, B., Nestola, M.G., et al.: High-order accurate simulation of incompressible turbulent flows on many parallel GPUs of a hybrid-node supercomputer. Computer Physics Communications 244, 132–142 (2019). https://doi.org/10.1016/j.cpc.2019.06.012