Performance Assessment of InfiniBand HPC Cloud Instances on Intel Haswell and Intel Sandy Bridge Architectures

Jonathan Low; Jakub Chrzeszczyk; Andrew Howard; Andrzej Chrzeszczyk

doi:10.14529/jsfi150303

Authors

Jonathan Low A*STAR Computational Resource Centre, Singapore
Jakub Chrzeszczyk National Computational Infrastructure, ANU, Canberra
Andrew Howard National Computational Infrastructure, ANU, Canberra
Andrzej Chrzeszczyk Jan Kochanowski University, Kielce

DOI:

https://doi.org/10.14529/jsfi150303

Abstract

This paper aims to establish a performance baseline of a HPC installation of OpenStack. We created InfiniCloud - a distributed High Performance Cloud hosted on remote nodes of InfiniCortex. InfiniCloud compute nodes use high performance Intel (R) Haswell and Sandy Bridge CPUs, SSD storage and 64-256GB RAM. All computational resources are connected by high performance IB interconnects and are capable of trans-continental IB communication using Obsidian Longbow range extenders.

We benchmark the performance of our test-beds using micro-benchmarks for TCP bandwidth, IB bandwidth and latency, file creation performance, MPI collectives and Linpack. This paper compares different CPU generations across virtual and bare-metal environments.

The results show modest improvements in TCP and IB bandwidth and latency on Haswell; performance being largely dependent on the IB hardware. Virtual overheads were minimal and near-native performance is possible for sufficiently large messages. From the Linpack testing, users can expect more than twice the performance in their applications on Haswell-provisioned VMs. On Haswell hardware, native and virtual performance differences is still significant for MPI collective operations. Finally, our parallel filesystem testing revealed virtual performance coming close to native only for non-sync/fsync file operations.

References

Kenneth Ban, Tin Wee Tan, Jakub Chrzeszczyk, Andrew Howard and Dongyang Li. InfiniCloud: Leveraging Global InfiniCortex Fabric and OpenStack Cloud for Borderless High Performance Computing of Genomic Data and Beyond. Submitted to Supercomputing Frontiers 2015 conference proceedings, Singapore.

Jakub Chrzeszczyk, Muhammad Atif, Joseph Antony, Dongyang Li, Matthew Sanderson and Allan Williams. Perspectives on implementation of a high performance scientific cloud backed by a 56G high speed interconnect. HPC Advisory Council Event, Singapore. http://www.hpcadvisorycouncil.com/events/2014/singapore-workshop/preso/12_ANU.pdf, November 2014.

Tiago Pais Pitta de Lacerda Ruivo, Gerard Bernabeu Altayo, Gabriele Garzoglio, Steven Timm, Hyun Woo Kim, Seo-Young Noh and Ioan Raicu. Exploring infiniband hardware virtualization in opennebula towards efficient high-performance computing. In Cluster, Cloud and Grid Computing (CCGrid), 2014 14th IEEE/ACM International Symposium on, pages 943–948. IEEE, 2014. DOI: 10.1109/ccgrid.2014.90.

J Dongarra. Luszczek and A. Petitet (2001): ”The LINPACK Benchmark: Past, Present and Future”, University of Tennessee. Technical report, mimeo.

Jan Heichler. An introduction to BeeGFS. http://www.beegfs.com/docs/Introduction_to_BeeGFS_by_ThinkParQ.pdf, November 2014.

Marius Hillenbrand, Viktor Mauch, Jan Stoess, Konrad Miller and Frank Bellosa. Virtual InfiniBand clusters for HPC clouds. In Proceedings of the 2nd International Workshop on Cloud Computing Platforms, page 9. ACM, 2012. DOI: 10.1145/2168697.2168706.

Wei Huang, Jiuxing Liu, Bulent Abali and Dhabaleswar K Panda. A case for high performance computing with virtual machines. In Proceedings of the 20th annual international conference on Supercomputing, pages 125–134. ACM, 2006. DOI: 10.1145/1183401.1183421.

Jithin Jose, Mingzhe Li, Xiaoyi Lu, Krishna Chaitanya Kandalla, Mark Daniel Arnold and Dhabaleswar K Panda. SR-IOV support for virtualization on infiniband clusters: Early experience. In Cluster, Cloud and Grid Computing (CCGrid), 2013 13th IEEE/ACM International Symposium on, pages 385–392. IEEE, 2013. DOI: 10.1109/ccgrid.2013.76.

Glenn K. Lockwood. High-Performance Virtualization: SR-IOV and InfiniBand. http://glennklockwood.blogspot.sg/2013/12/high-performance-virtualization-sr-iov_14.html

Richard Moore, Luca Clementi, Dmitry Mishin, Phil Papadopoulos, Mahidhar Tatineni and Rick Wagner. Comet: Realizing High-Performance. Virtualized Clusters using SR-IOV Technology. HPC Advisory Council Event, China, http://www.hpcadvisorycouncil.com/events/2014/china-workshop/preso/3_Moore_Comet.pdf, November 2014.

Tin Wee Tan, Dominic S.H. Chien, Yuefan Deng, Seng Lim, Sing-Wu Liou, Jonathan Low, Marek Michalewicz, Gabriel Noaje, Yves Poppe and Geok Lian Tan. InfiniCortex: A path to reach Exascale concurrent supercomputing across the globe utilising trans-continental InfiniBand and Galaxy of Supercomputers. Submitted to Supercomputing Frontiers 2015 conference proceedings, Singapore.