Making Large-Scale Systems Observable - Another Inescapable Step Towards Exascale

Dmitry A. Nikitenko; Sergey A. Zhumatiy; Pavel A. Shvets

doi:10.14529/jsfi160205

Authors

Dmitry A. Nikitenko Moscow State University, Moscow Research Computing Center, Moscow
Sergey A. Zhumatiy Moscow State University, Moscow Research Computing Center, Moscow
Pavel A. Shvets Moscow State University, Moscow Research Computing Center, Moscow

DOI:

https://doi.org/10.14529/jsfi160205

Abstract

The effective mastering of extremely parallel HPC system is impossible without deep understanding of all internal processes and behavior of the whole diversity of the components: computing processors and nodes, memory usage, interconnect, storage, whole software stack, cooling and so forth in detail. There are numerous visualization tools that provide information on certain components and system as a whole, but most of them have severe issues that limit appliance in real life, thus becoming inacceptable for the future system scales.

Predefined monitoring systems and data sources, lack of dynamic on-the-fly reconfiguration, inflexible visualization and screening options are among the most popular issues.The proposed approach to monitoring data processing resolves the majority of known problems, providing a scalable and flexible solution based on any available monitoring systems and other data sources. The approach implementation is successfully used in every-day practice of the largest in Russia supercomputer center of Moscow State University.

References

Zabbix - The Enterprise-class Monitoring Solution for Everyone, http://www.zabbix.com.

Nagios - The Industry Standard in IT Infrastructure Monitoring, http://www.nagios.org.

Long J.W. Lorenz: Using the Web to Make HPC Easier. 2013. 15.

OpenLorenz - Web-Based HPC Dashboard and More, https://github.com/hpc/OpenLorenz.

Showerman M. Real Time Visualization of Monitoring Data for Large Scale HPC Systems // 2015 IEEE International Conference on Cluster Computing. IEEE, 2015. Pp. 706-709.

Dmitry Nikitenko, Vladimir Voevodin, and Sergey Zhumatiy. Octoshell: Large Supercomputer Complex Administration System // Russian Supercomputing Days International Conference, Moscow, Russian Federation, 28-29 September, 2015, Proceedings. CEUR Workshop

Proceedings, 2015. Vol. 1482. pp. 69-83.

Pavel Shvets, Vladimir Voevodin, Sergey Sobolev, Vadim Voevodin, Konstantin Stefanov, Sergey Zhumatiy, Artem Daugel-Dauge, Alexander Antonov and Dmitry Nikitenko. An Approach for Ensuring Reliable Functioning of a Supercomputer Based on a Formal Model.

Parallel Processing and Applied Mathematics. 11th International Conference, PPAM 2015, Krakow, Poland, September 6-9, 2015. Revised Selected Papers, Part I (2016), vol. 9573 of LECTURE NOTES IN COMPUTER SCIENCE, Springer International Publishing, pp. 12-22.

Dmitry Nikitenko, Vladimir Voevodin, Sergey Zhumatiy, Konstantin Stefanov, Alexey Teplov, Pavel Shvets, and Vadim Voevodin. Supercomputer Application Integral Characteristics Analysis for the Whole Queued Job Collection of Large-Scale HPC Systems. Parallel Computational Technologies (PCT'2016): Proceedings of the International Scientic Conference. Chelyabinsk, Publishing of the South Ural State University, 2016. pp. 20-30.