The Parallel Performance of SLNE Atmosphere–Ocean–Sea Ice Coupled Model

Rostislav Yu. Fadeev

doi:10.14529/jsfi230305

Authors

Rostislav Yu. Fadeev Marchuk Institute of Numerical Mathematics of the Russian Academy of Sciences, Hydrometcenter of Russia, Moscow Institute of Physics and Technology, Russian Federation https://orcid.org/0000-0002-3928-9986

DOI:

https://doi.org/10.14529/jsfi230305

Keywords:

numerical weather prediction, coupled model, parallel performance, NEMO ocean model, SLAV model, OASIS3-MCT coupler

Abstract

The paper presents the first version of SLNE coupled model. SL and NE here are the first two letters from SLAV (Semi-Lagrangian, based on Absolute Vorticity equation) atmospheric model and NEMO (Nucleus for European Modelling of the Ocean) ocean model that have been coupled using OASIS3-MCT software. SLAV uses 0.9°x0.72° regular lat-lon grid with 96 vertical levels. NEMO incorporates SI3 sea ice model. Both of them use the same ORCA025 tripolar grid. Flux adjustments to correct inconsistencies at the interface between coupled atmosphere–ocean models have not been applied in SLNE. The model design and coupling particularities are described here in detail. A series of numerical experiments with SLNE model were performed to measure its parallel performance. We also investigated the scalability of SLNE model and its components in terms of simulation speed. Based on these results, an optimum configurations of SLNE were identified. It was found that the coupled model showed scaling efficiency of about 85% on 4000 computational cores of Cray XC40-LC in comparison to the SLNE configuration running on 224 cores. Simulations with lead times ranging from a few days to several years showed that there are no significant systematic errors in the coupled model.

References

Aleksandrov, V.V., Arkhipov, P.L., Parkhomenko, V.P., Stenchikov, G.L.: Globalnaia model sistemy okean-atmosfera i issledovanie ee chuvstvitelnosti k izmeneniiu kontsentratsii CO2 [A global model of the ocean-atmosphere system and the study of its sensitivity to changes in CO2 concentration]. Izvestiia AN SSSR. Fizika Atmosfery i Okeana 19(5), 451–458 (1983) (in Russian).

Beljaars, A., Dutra, E., Balsamo, G., Lemarié, F.: On the numerical stability of surface–atmosphere coupling in weather and climate models. Geoscientific Model Development 10(2), 977–989 (2017). https://doi.org/10.5194/gmd-10-977-2017

Byrne, N.J., Shepherd, T.G., Polichtchouk, I.: Subseasonal-to-seasonal predictability of the Southern Hemisphere eddy-driven jet during austral spring and early summer. J. of Geop. Res.: Atmospheres 124, 6841–6855 (2019). https://doi.org/10.1029/2018JD030173

Callendar, G.S.: The artificial production of carbon dioxide and its influence on temperature. Q.J.R. Meteorol. Soc. 64, 223–240 (1938). https://doi.org/10.1002/qj.49706427503

Chamberlin, T.C.: A group of hypotheses bearing on climatic changes. J. Geol. 5, 653–683 (1897). http://www.jstor.org/stable/30054630

Chou, M.-D., Suarez, M.J.: A solar radiation parameterization (CLIRAD-SW) for atmospheric studies. NASA Tech. Memo. 10460(15) (1999). https://ntrs.nasa.gov/api/citations/19990060930/downloads/19990060930.pdf

Croll, J.: Climate and Time in Their Geological Relations. Edinburgh: Adam and Charles Black (1885).

Davis, P., Ruth, C., Scaife, A.A., Kettleborough, J.: A large ensemble seasonal forecasting system: GloSea6. In: AGU Fall Meeting Abstracts, vol. 2020, pp. A192-05 (2020).

Doose, K.: Modelling the future: climate change research in Russia during the late Cold War and beyond, 1970s–2000. Climatic Change 171(6) (2022). https://doi.org/10.1007/s10584-022-03315-0

Doyle, J.D., Hodur, R.M., Chen, S., et al.: Tropical cyclone prediction using COAMPS-TC. Oceanography 27(3), 104–115 (2014). https://doi.org/10.5670/oceanog.2014.72

Durán, I.B., Geleyn, J-F., Váña, F.: Compact Model for the Stability Dependency of TKE Production–Destruction–Conversion Terms Valid for the Whole Range of Richardson Numbers. J. Atmos. Sci. 71, 3004–3026 (2014). https://doi.org/10.1175/JAS-D-13-0203.1

Edwards, P.N.: History of climate modeling. WIREs Clim Change 2, 128–139 (2011). https://doi.org/10.1002/wcc.95

Eyring, V., Bony, S., Meehl, G.A., et al.: Overview of the Coupled Model Intercomparison Project Phase 6 (CMIP6) experimental design and organization. Geosci. Model Dev. 9, 1937–1958 (2016). https://doi.org/10.5194/gmd-9-1937-2016

Fadeev, R.Yu., Ushakov, K.V., Tolstykh, M.A., Ibrayev, R.A.: Design and development of the SLAV-INMIO-CICE coupled model for seasonal prediction and climate research. Rus. J. of Num. An. and Math. Mod. 33(6), 333–340 (2018). https://doi.org/10.1515/rnam-2018-0028

Fadeev, R.Yu., Tolstykh, M.A., Volodin, E.M.: Climate version of the SL-AV global atmospheric model: development and preliminary results Russ. Meteorol. Hydrol 44(1), 13–22 (2019). https://doi.org/10.3103/S1068373919010023

Fadeev, R.Y., Alipova, K.A., Koshkina, A.S., et al.: Glacier parameterization in SLAV numerical weather prediction model. Rus. J. of Num. An. and Math. Mod. 37(4), 189–201 (2022). https://doi.org/10.1515/rnam-2022-0016

Fadeev, R.Yu.: Wind gustiness parameterization and long-range weather prediction. Proc. of Hydrometcentre of Russia 2(388), 35–55 (2023) (in Russian). https://doi.org/10.37162/2618-9631-2023-2-35-54

Fraedrich, K., Kirk, E., Luksch, U. Lunkeit, F.: The Portable University Model of the Atmosphere (PUMA): stormtrack dynamics and low-frequency variability. Meteorologische Zeitschrift 14, 735–745 (2005). https://doi.org/10.1127/0941-2948/2005/0074

Fultz, D.: Dynamics of Climate. New York: Pergamon Press, 71–77 (1960).

Gerard, L., Piriou, J., Brožková, R., et al.: Cloud and Precipitation Parameterization in a Meso-Gamma-Scale Operational Weather Prediction Model Mon. Wea. Rev. 137, 3960–3977 (2009). https://doi.org/10.1175/2009MWR2750.1

Global Deterministic Prediction System (GDPS): Update from version 4.0.1 to version 5.0.0. Canadian Meteorological Centre Tech. Note 59 (2015). https://collaboration.cmc.ec.gc.ca/cmc/cmoi/product_guide/docs/tech_notes/technote_gdps-500_20151215_e.pdf

Golubeva, E.N., Ivanov, U.A., Kuzin, V.I., Platov, G.A.: A numerical modeling of the world ocean circulation with allowance for the upper quasi-homogeneous layer. Oceanologia 32(3), 395–405 (1992).

Golubeva, E.N., Platov, G.A.: On improving the simulation of Atlantic water circulation in the Arctic Ocean. J. Geophys. Res. 112, C04S05 (2007). https://doi.org/10.1029/2006JC003734.

Gradov, V.S., Platov, G.A.: Overview of SCM Coupler and Its Application for Constructing Climate Models. Supercomputing Frontiers and Innovations 10(1), 58–76 (2023). https://doi.org/10.14529/jsfi230106

Guérémy, J.-F., Dubois, C., Viel, C., et al.: Documentaton of the METEO-FRANCE seasonal forecastng system 8 ECMWF Copernicus report. http://www.umr-cnrm.fr/IMG/pdf/system8-technical.pdf

Hallberg, R.: Numerical instabilities of the ice/ocean coupled system. CLIVAR Exchanges 19(69), 38–42 (2014). https://doi.org/10.1016/j.ocemod.2008.05.005

Hersbach, H., Bell, B., Berrisford, P., et. al.: The ERA5 global reanalysis. Quart. J. Roy. Meteor. Soc. 146(730), 1999–2049 (2020). https://doi.org/10.1002/qj.3803

Hirahara, S., Kubo, Y., Yoshida, T., et al.: Japan Meteorological Agency/Meteorological Research Institute Coupled Prediction System Version 3 (JMA/MRI–CPS3). Journal of the Meteorological Society of Japan. Ser. II, 101(2), 149–169 (2023). https://doi.org/10.2151/jmsj.2023-009

Hoskins, B.: The potential for skill across the range of the seamless weather-climate prediction problem: a stimulus for our science. Q.J.R. Meteorol. Soc. 139, 573–584 (2013). https://doi.org/10.1002/qj.1991

Hunke, E.C., Lipscomb, W.H., Turner, A.K., et al.: CICE: the Los Alamos Sea Ice Model documentation and software users manual, version 5.1. Technical report LA-CC-06-012. Los Alamos National Laboratory: Los Alamos, NM. 2015. http://www.ccpo.odu.edu/~klinck/Reprints/PDF/cicedoc2015.pdf

Ibrayev, R.A., Ushakov, K.V., Khabeev, R.N.: Eddy-resolving 1/10° model of the World Ocean Izvestiya. Atmosphere and Ocean Phys. 48, 37–46 (2012).

Kalmykov, V.V., Ibrayev, R.A., Kaurkin, M.N., Ushakov, K.V.: Compact modeling framework v3.0 for high-resolution global ocean–ice–atmosphere models. Geosci. Model Dev. 11, 3983–3997 (2018). https://doi.org/10.5194/gmd-11-3983-2018

Kasahara, A,Washington, W.M.: NCAR global general circulation model of the atmosphere. Mon. Weather Rev. 95, 389–402 (1967).

Larson, J., Jacob, R., Ong, E.: The Model Coupling Toolkit: A New Fortran90 Toolkit for Building Multiphysics Parallel Coupled Models. Int. J. High Perf. Comp. App. 19(3), 277–292 (2005). https://doi.org/10.1177/1094342005056115

Lemarie, F., Blayo, E., Debreu, L.: Analysis of ocean–atmosphere coupling algorithms: Consistency and stability. Procedia Computer Science 51, 2066–2075 (2015). https://doi.org/10.1016/j.procs.2015.05.473

Lin, H., Merryfield, W.J., Muncaster, R., et al.: The Canadian Seasonal to Interannual Prediction System Version 2 (CanSIPSv2).Weather and Forecasting 35(4), 1317–1343 (2020). https://doi.org/10.1175/WAF-D-19-0259.1

MacLachlan, C., Arribas, A., Peterson, K.A., et al.: Global seasonal forecast system version 5 (GloSea5): a high-resolution seasonal forecast system. Quart. J. Roy. Meteor. Soc. 141, 1072–1084 (2015). https://doi.org/10.1002/qj.2396

Madec, G., Bell, M., Blaker, A., et al.: NEMO Ocean Engine Reference Manual. Zenodo (2023). https://doi.org/10.5281/zenodo.8167700

Madec, G., Imbard, M.: A global ocean mesh to overcome the North Pole singularity. Climate Dynamics 12, 381–388 (1996). https://doi.org/10.1007/BF00211684

Magnusson, L., Bidlot, J.-R., Bonavita, M., et al.: ECMWF activities for improved hurricane forecasts. Bulletin of the American Meteorological Society 100(3), 445–458 (2019). https://doi.org/10.1175/BAMS-D-18-0044.1

Manabe. S, Bryan, K.: Climate calculations with a combined ocean-atmosphere model. J. Atmos. Sci. 26, 786–789 (1969). https://doi.org/10.1175/1520-0469(1969)026<0786:CCWACO>2.0.CO;2

Meleshko, V.P., Matyugin, V.A., Sporyshev, P.V., et al.: MGO general circulation model (version MGO-03 T63L25). Proc. of Voeikov Main Geop. Obs. 571, 5–87 (2014) (in Russian).

Mirvis, V.M., Meleshko, V.P., Lvova, T.Yu., et al.: Forecast experiments based on MGO coupled ocean-atmosphere model. Proc. of Voeikov Main Geop. Obs. 583, 129–148 (2016) (in Russian).

Mlawer, E.J., Taubman, S.J., Brown, P.D., et al.: RRTM, a validated correlated-k model for the longwave. J. Geophys. Res. 102(16), 663–682 (1997). https://doi.org/10.1029/97JD00237

Moiseev, N.N., Aleksandrov, V.V., Tarko, A.M.: Opyt sistemnogo analiza i eksperimenty s model’u [Experience with systems analysis and experimentation with models]. Chelovek i biosfera, Moscow, Nauka (1983) (in Russian).

Molod, A., Salmun, H.: A global assessment of the mosaic approach to modeling land surface heterogeneity. J. Geophys. Res. 107(D14) (2002). https://doi.org/10.1029/2001JD000588

Phillips, N.A.: The general circulation of the atmosphere: a numerical experiment. Q.J.R. Meteorol. Soc. 82, 123–164 (1956). https://doi.org/10.1002/qj.49708235202

Piacentini, A., Maisonnave, E., Jonville, G., et al.: A parallel SCRIP interpolation library for OASIS. https://oasis.cerfacs.fr/wp-content/uploads/sites/114/2021/08/GLOBC_WN_Piacentini_Parallel_SCRIP_cmgc_18_34_2018.pdf

Platov, G., Krupchatnikov, V., Martynova, Y., et al.: A new earths climate system model of intermediate complexity, PlaSim-ICMMG-1.0: description and performance. IOP Conference Series: Earth and Environmental Science 96(1), 012005 (2017). https://doi.org/10.1088/1755-1315/96/1/012005

Russell, R.J.: Climatic change through the ages. United States Department of Agriculture, ed. Climate and Man. Washington: U.S. Government Printing Office, 67–97 (1941).

Saha, S., Moorthi, S., Wu, X., et al.: The NCEP climate forecast system version 2. Journal of Climate 27, 2185–2208 (2014). https://doi.org/10.1175/JCLI-D-12-00823.1

Shashkin, V.V., Fadeev, R.Yu., Tolstykh, M.A., et al.: Simulation of stratosphere processes with the atmosphere general circulation model SLAV072L96. Rus. Meteorol. and Hydrol. 49, 5–20 (2023). https://doi.org/10.3103/S1068373923060018

Schneider, S.H., Dickinson, R.E.: Climate modeling. Rev. Geophys. Space Phys. 12(3), 447–493 (1974). https://doi.org/10.1029/RG012i003p00447

Schwartz, C., Garfinkel, C.I.: Troposphere–stratosphere coupling in subseasonal–to–seasonal models and its importance for a realistic extratropical response to the Madden-Julian oscillation. J. of Geop. Res.: Atmospheres 125, e2019JD032043 (2020). https://doi.org/10.1029/2019JD032043

SCRIP project at GitHub, https://github.com/SCRIP-Project/SCRIP

Smith, G.C., Bélanger, J., Roy, F., et al.: Impact of Coupling with an Ice–Ocean Model on Global Medium-Range NWP Forecast Skill. Monthly Weather Review 146(4), 1157–1180 (2018). https://doi.org/10.1175/MWR-D-17-0157.1

Stepanov, V.N., Resnyanskii, Y.D., Strukov, B.S., et al.: Large-scale Ocean Circulation and Sea Ice Characteristics Derived from Numerical Experiments with the NEMO Model. Russ. Meteorol. Hydrol. 44, 33–44 (2019). https://doi.org/10.3103/S1068373919010047

Strukov, B.S., Resnyanskii, Y.D. Zelenko, A.A.: Relaxation Method for Assimilation of Sea Ice Concentration Data in the NEMOLIM3 Multicategory Sea Ice Model. Russ. Meteorol. Hydrol. 45, 96–104 (2020). https://doi.org/10.3103/S1068373920020053

Tarasevich, M., Sakhno, A., Blagodatskikh, D., et al.: Scalability of the INM RAS Earth System Model. In: Voevodin, V., Sobolev, S., Yakobovskiy, M., Shagaliev, R. (eds) Supercomputing. RuSCDays 2023. Lecture Notes in Computer Science, vol. 14388, pp. 202–216. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-49432-1_16

Tarasova, T., Fomin, B.: The Use of New Parameterizations for Gaseous Absorption in the CLIRAD-SW Solar Radiation Code for Models. J. Atmos. and Oceanic Tech. 24(6), 1157–1162 (2007). https://doi.org/10.1175/JTECH2023.1

The Working Group on Numerical Experimentation (WGNE). https://wgne.net/

Thompson, B., Sanchez, C., Heng, B.C.P., et al.: Development of a MetUM (v 11.1) and NEMO (v 3.6) coupled operational forecast model for the Maritime Continent – Part 1: Evaluation of ocean forecasts. Geosci. Model Dev. 14, 1081–1100 (2021). https://doi.org/10.5194/gmd-14-1081-2021

Tolstykh, M.A., Volodin, E.M., Kostrykin, S.V., et al.: Development of the multiscale version of the SL-AV global atmosphere model. Rus. Meteorol. and Hydrol. 40, 374–382 (2015). https://doi.org/10.3103/S1068373915060035

Tolstykh, M.A., Shashkin, V.V., Fadeev, R.Yu., Goyman, G.S.: Vorticity-divergence semi-Lagrangian global atmospheric model SL-AV20: dynamical core. Geoscientific Model Development 10(5), 1961–1983 (2017). https://doi.org/10.5194/gmd-10-1961-2017

Tolstykh, M.A., Fadeev, R.Yu., Shashkin, V.V., et al.: Multiscale Global Atmosphere Model SL-AV: the Results of Medium-range Weather Forecasts. Russ. Meteorol. Hydrol. 43, 773–779 (2018). https://doi.org/10.3103/S1068373918110080

Tolstykh, M.A., Fadeev, R.Yu., Shashkin, V.V., et al.: The SLAV072L96 system for long range meteorological forecasts. Submitted to Rus. Meteorol. and Hydrol. (2023).

Tsyrulnikov, M.D., Svirenko, P.I., Gayfulin, D.R., et al.: Development of the data assimilation scheme of the hydrometcentre of Russia. Proc. of Hydrometcentre of Russia 4(374), 112–126 (2019) (in Russian).

Valcke, S.: The OASIS3 coupler: a European climate modelling community software. Geosci. Model Devel. 6, 373–388 (2013). https://doi.org/10.5194/gmd-6-373-2013

Vancoppenolle, M., Rousset, C., Blockley, E., et al.: SI3 – Sea Ice modelling Integrated Initiative – The NEMO Sea Ice Engine. Zenodo (2023). https://doi.org/10.5281/zenodo.7534900

Volodin, E.M., Lykossov, V.N.: Parametrization of Heat and Moisture Transfer in the Soil–Vegetation System for Use in Atmospheric General Circulation Models: 1. Formulation and Simulations Based on Local Observational Data. Izv., Atm. and Oceanic Phys. 34, 402–416 (1998).

Volodin, E.M., Mortikov, E.V., Kostrykin, S.V., et al.: Simulation of the present day climate with the climate model INMCM5. Clim. Dyn. 49, 3715–3734 (2017). https://doi.org/10.1007/s00382-017-3539-7

Volodin, E.M.: Possible Climate Change in Russia in the 21st Century Based on the INMCM5-0 Climate Model. Rus. Meteorol. and Hydrol. 47(5), 327–333 (2022). https://doi.org/10.3103/S1068373922050016

Vorobyeva, V., Volodin, E.: Evaluation of the INM RAS climate model skill in climate indices and stratospheric anomalies on seasonal timescale Tellus A: Dynamic Meteorology and Oceanography 73(1), 1892435. https://doi.org/10.1080/16000870.2021.1892435

Zhang, S., Xu, S., Fu, H., et. al.: Toward Earth system modeling with resolved clouds and ocean submesoscales on heterogeneous many-core HPCs. National Science Review 10(6) (2023). https://doi.org/10.1093/nsr/nwad069

WCRP Coupled Model Intercomparison Project (CMIP). https://www.wcrp-climate.org/wgcm-cmip

Wood, N., Staniforth, A., White, A., et al.: An inherently mass-conserving semi-implicit semi-Lagrangian discretization of the deep-atmosphere global non-hydrostatic equations. Q.J.R. Meteorol. Soc. 140, 1505–1520 (2014). https://doi.org/10.1002/qj.2235