Large Language Models versus Native Speakers in Emotional Assessment of Russian Words

Polina V. Iaroshenko; Natalia V. Louckachevitch

doi:10.14529/jsfi250302

Authors

Polina V. Iaroshenko Research Computing Center, Lomonosov Moscow State University, Moscow, Russian Federation https://orcid.org/0000-0002-9425-5842
Natalia V. Louckachevitch Research Computing Center, Lomonosov Moscow State University, Moscow, Russian Federation https://orcid.org/0000-0002-1883-4121

DOI:

https://doi.org/10.14529/jsfi250302

Keywords:

Large Language Models, human-likeness, emotional intelligence, Russian language

Abstract

The paper presents a comparative analysis of emotional evaluation of Russian nouns by large language models and native speakers. Based on the ENRuN (Emotional Norms for Russian Nouns) database, which contains ratings of 1,800 nouns across five basic emotions (happiness, sadness, anger, fear, and disgust), the research compares human assessments with evaluations provided by seven large language models (Llama-3-70B, Qwen 2.5-32B, YandexGPT 5 Lite, RuadaptQwen2.5-7B, RuadaptQwen2.5-32B-Pro-Beta, T-pro, T-lite). Although some models demonstrated relatively high correlation with human assessments, persistent systematic deviations were observed across all tested models. The analysis reveals significant differences in emotional perception during word evaluation: the models demonstrate a tendency to hyperbolise negative emotions and show variability in assessing positive emotions, particularly when analysing words related to sensitive topics (violence, religion, obscene vocabulary). The findings indicate that the closest alignment with human evaluations is achieved when there is a balance between the model’s size and the quality of its language adaptation.

References

Acheampong, F.A., Wenyu, C., Nunoo-Mensah, H.: Text-based emotion detection: Advances, challenges, and opportunities. Engineering Reports 2, e12189 (2020). https://doi.org/10.1002/eng2.12189

Plaza-del Arco, F.M., Cercas Curry, A.A., Cercas Curry, A., Hovy, D.: Emotion analysis in NLP: Trends, gaps and roadmap for future directions. In: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation. pp. 5696–5710 (2024)

Plaza-del Arco, F.M., Curry, A.C., Curry, A., et al.: Angry men, sad women: Large language models reflect gendered stereotypes in emotion attribution. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. vol. 1, pp. 7682–7696. Association for Computational Linguistics, Bangkok (2024). https://doi.org/10.18653/v1/2024.acl-long.415

Bostan, L.A.M., Klinger, R.: An analysis of annotated corpora for emotion classification in text. Tech. rep., Otto-Friedrich-Universität, Bamberg (2024)

Cavicchio, F.: Emotion Detection in Natural Language Processing. Springer, Cham (2025). https://doi.org/10.1007/978-3-031-72047-5

Chen, Y., Yan, S., Liu, S., et al.: EmotionQueen: A benchmark for evaluating empathy of large language models. In: Findings of the Association for Computational Linguistics: ACL 2024. pp. 2149–2176. Association for Computational Linguistics, Bangkok, Thailand (2024). https://doi.org/10.18653/v1/2024.findings-acl.128

Dalal, D., Negi, G., Picca, D.: LLMs and emotional intelligence: Evaluating emotional understanding through psychometric tools. In: Proceedings of the 33rd ACM Conference on User Modeling, Adaptation and Personalization. pp. 323–328. UMAP ’25 (2025). https://doi.org/10.1145/3699682.3728315

Huang, J., Lam, M.H., Li, E.J., et al.: Apathetic or empathetic? Evaluating LLMs’ emotional alignments with humans. Advances in Neural Information Processing Systems 37, 97053–97087 (2024)

Kazyulina, M., Babii, A., Malafeev, A.: Emotion classification in Russian: Feature engineering and analysis. In: Analysis of Images, Social Networks and Texts, AIST 2020. Lecture Notes in Computer Science, vol. 12602, pp. 135–148 (2021). https://doi.org/10.1007/978-3-030-72610-2_10

Lyashevskaya, O.N., Sharov, S.A.: Frequency Dictionary of Modern Russian Language (based on the materials of the Russian National Corpus). Azbukovnik, Moscow (2009), (in Russian)

Lyusin, D., Sysoeva, T.A.: ENRuN database: Emotional ratings of Russian nouns. Experimental Psychology 18(2), 206–219 (2025). https://doi.org/10.17759/exppsy.2025180212, (in Russian)

MacCann, C., Roberts, R.D.: New paradigms for assessing emotional intelligence: theory and data. Emotion 8(4), 540–551 (2008). https://doi.org/10.1037/a0012746

Raj, P.: A literature review on emotional intelligence of large language models (LLMs). International Journal of Advanced Research in Computer Science 15(4) (2024). https://doi.org/10.26483/ijarcs.v15i4.7111

Sabour, S., Liu, S., Zhang, Z., et al.: EmoBench: Evaluating the emotional intelligence of large language models. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. vol. 1, pp. 5986–6004. Association for Computational Linguistics, Bangkok (2024). https://doi.org/10.18653/v1/2024.acl-long.326

Salnikov, M., Korzh, D., Lazichny, I., et al.: Geopolitical biases in LLMs: what are the "good" and the "bad" countries according to contemporary language models. https://arxiv.org/abs/2506.06751 (2024)

Sysoeva, T.A., Lyusin, D.V.: Development of an extended database with emotional ratings of nouns ENRuN-2: successes, problems and prospects. In: Vladimirov, I., Korovkin, S. (eds.) Psychology of Cognition: Proceedings of the All-Russian Scientific Conference. pp. 316–320. YARSU, Yaroslavl (2024), (in Russian)

Tikhomirov, M., Chernyshov, D.: Facilitating large language model Russian adaptation with learned embedding propagation. Journal of Language and Education 10(4), 130–145 (2024). https://doi.org/10.17323/jle.2024.22224

Wang, X., Li, X., Yin, Z., et al.: Emotional intelligence of large language models. Journal of Pacific Rim Psychology 17, 18344909231213958 (2023). https://doi.org/10.1177/18344909231213