The Roofline Analysis of Special Relativistic Hydrodynamics Coarray Fortran Code

Igor M. Kulikov; Igor G. Chernykh; Dmitry A. Karavaev; Vladimir G. Prigarin; Anna F. Sapetina; Ivan S. Ulyanichev; Oleg R. Zavyalov

doi:10.14529/jsfi230301

Authors

Igor M. Kulikov Institute of Computational Mathematics and Mathematical Geophysics SB RAS, Novosibirsk, Russian Federation https://orcid.org/0000-0002-1959-780X
Igor G. Chernykh Institute of Computational Mathematics and Mathematical Geophysics SB RAS, Novosibirsk, Russian Federation
Dmitry A. Karavaev Institute of Computational Mathematics and Mathematical Geophysics SB RAS, Novosibirsk, Russian Federation
Vladimir G. Prigarin Institute of Computational Mathematics and Mathematical Geophysics SB RAS, Novosibirsk, Russian Federation
Anna F. Sapetina Institute of Computational Mathematics and Mathematical Geophysics SB RAS, Novosibirsk, Russian Federation
Ivan S. Ulyanichev Institute of Computational Mathematics and Mathematical Geophysics SB RAS, Novosibirsk, Russian Federation
Oleg R. Zavyalov Institute of Computational Mathematics and Mathematical Geophysics SB RAS, Novosibirsk, Russian Federation

DOI:

https://doi.org/10.14529/jsfi230301

Keywords:

HPC analysis, Coarray Fortran, high performance computing, roofline analysis

Abstract

Our previous papers are dedicated to the development of the first code for computational astrophysics using Coarray Fortran technology. The main result of the study of the developed code is the achievement of weak scalability at the level of MPI implementations, which allows to fully concentrate on using Coarray Fortran for developing new program codes. Coarray Fortran is based on the MPI directives, and helps software developer to create simple code without Send/Receive or synchronization commands. At the same time, such scalability can be achieved due to the weak implementation of the sequential part of the program code, which is characterized by frequent cache misses, inefficient memory usage and poor overall performance. In this article, we propose a method for analyzing program code performance using the roofline analysis. We used Intel Advisor software package from Intel oneAPI toolkit. High performance and efficient work with the memory of both individual key procedures and the code as a whole are demonstrated.

References

Sotomayor, P., Romero G.: Nonthermal radiation from the central region of super-accreting active galactic nuclei. Astronomy & Astrophysics 664, A178 (2022). https://doi.org/10.1051/0004-6361/202243682

Kulikov, I.: A new code for the numerical simulation of relativistic flows on supercomputers by means of a low-dissipation scheme. Computer Physics Communications 257, 107532 (2020). https://doi.org/10.1016/j.cpc.2020.107532

Reshetova, G., Cheverda, V., Khachkova, T.: A comparison of MPI/OpenMP and Coarray Fortran for digital rock physics application. In: Parallel Computing Technologies. PaCT 2019. Lecture Notes in Computer Science, vol. 11657, pp. 232–244. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-25636-4_19

Reshetova, G., Cheverda, V., Khachkova, T.: Numerical experiments with digital twins of core samples for estimating effective elastic parameters. In: Supercomputing. RuSCDays 2019. Communications in Computer and Information Science, vol. 1129, pp. 290–301. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-36592-9_24

Reshetova, G., Cheverda, V., Koinov, V.: Comparative efficiency analysis of MPI blocking and non-blocking communications with Coarray Fortran. In: Supercomputing. RuSCDays 2021. Communications in Computer and Information Science, vol. 1510, pp. 322–336. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-92864-3_25

Kulikov, I., Chernykh, I., Karavaev, D., et al.: A new parallel code based on a simple piecewise parabolic method for numerical modeling of colliding flows in relativistic hydrodynamics. Mathematics 10(11), 1865 (2022). https://doi.org/10.3390/math10111865

Wang, Y., Li, Z.: GridFOR: a domain specific language for parallel grid-based applications. International Journal of Parallel Programming 44, 427–448 (2016). https://doi.org/10.1007/s10766-014-0348-z

Kataev, N., Kolganov, A.: The experience of using DVM and SAPFOR systems in semi automatic parallelization of an application for 3D modeling in geophysics. The Journal of Supercomputing 75, 7833–7843 (2019). https://doi.org/10.1007/s11227-018-2551-y

Shterenlikht, A., Cebamanos, L.: MPI vs Fortran coarrays beyond 100k cores: 3D cellular automata. Parallel Computing 84, 37–49 (2019). https://doi.org/10.1016/j.parco.2019.03.002

Guo, P., Wu, J.: One-sided communication in Coarray Fortran: performance tests on TH-1A. In: Algorithms and Architectures for Parallel Processing. ICA3PP 2018. Lecture Notes in Computer Science, vol. 11337, pp. 21–33. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-05063-4_3

The Roofline Model. https://en.wikipedia.org/wiki/Roofline_model (2023), accessed: 2023-10-25

Intel Advisor tutorial. https://www.intel.com/content/www/us/en/docs/advisor/tutorial-roofline/2021-1/run-a-roofline-analysis.html (2021), accessed: 2023-10-25

Intel oneAPI overview. https://www.intel.com/content/www/us/en/developer/tools/oneapi/overview.html (2023), accessed: 2023-10-25

Chernykh, I., Vorobyov, E., Elbakyan, V., Kulikov, I.: The impact of compiler level optimization on the performance of iterative Poisson solver for numerical modeling of protostellar disks. In: Supercomputing. RuSCDays 2021. Communications in Computer and Information Science, vol. 1510, pp. 415–426. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-92864-3_32