Vital Issues of Evaluating the Social Impact of Scientific Research
DOI:
https://doi.org/10.15407/sofs2024.02.052Keywords:
public impact of scientific research, research engagement, research assessment, alternative metrics, Altmetric, Altmetric Attention Score, Dimensions, SciVal, Lens, research in mass media and communicationsAbstract
The sound assessment of scientific research has gained the increased topicality in the past decade due to the challenges of open science, enhancing the research productivity, transparency and impartiality of researchers’ scores. However, the measurement of quality and social impact of research is complicated because of its specifics across research disciplines and countries. Numerous initiatives call for new approaches to the measurement of the social impact of research, to keeping the balance between quantitative and qualitative indicators, to introducing industry-normalized or alternative metrics, especially in view of the challenges of open science. The article contains a review of advanced methods for measuring the social impact of scientific research in selected countries (U.K., Spain, U.S., Canada, Australia), and alternative metrics and tools for measuring this impact ((Dimensions, The Lens, SciVal), with focus on research in mass media and communications. The authors’ research builds on general (abstraction, analysis and synthesis, study of sources, etc.) and specialized methods (content analysis, media discourse analysis, etc.), added by selected altmetric indicators. Top-10 articles with the highest Altmetric Attention Score (AAS) and the highest citation were determined by data from Dimensions. The highest AAS were revealed for articles devoted to COVID-19, giving evidence of the increased public attention to the topic of global pandemic. The topic of the Russian-Ukrainian war and respective research (publications in 2022—2023) in mass media and communications (based on data from Dimensions) was chosen to measure the public interest in alternative metrics of impact. Ten articles with the highest AAS were determined in the sample of 514 articles, with showing citation figures for them. The impact of research in mass media and communications even for a sensitive topic like a war in the 21st century in the center of Europe was found to be much weaker than the one for medical research, and it had a substantially lower citation figure. It is argued that drawbacks of the alternative metrics are rooted in nontransparent methodology and hidden estimation algorithm: the alternative metrics is an indicator of online attention to an article (not always positive) rather than the quality of respective research, and it cannot be fi t for cross-discipline comparisons of articles.
References
Yaroshenko, T.O., & Zharinova, A.H. (2023). Scientific citation: historical and theoretical landscape. Science and Science of Science, 3 (121), 41—67. https://doi.org/10.15407/sofs2023.03.041 [in Ukrainian].
Yaroshenko, T., Serbin, O., & Yaroshenko, O. (2022). Open science: the role of universities and libraries in modern changes in scientific communication. Digital platform: information technologies in the socio-cultural sphere, 5 (2), 277–292. https://doi.org/10.31866/2617-796X.5.2.2022.270132 [in Ukrainian].
Garfield, E. (1979). Citation Indexing: Its Theory and Application in Science, Technology, and Humanities. New York, NY: Wiley.
Elkana, Y. (1978). Toward a Metric of Science: The Advent of Science Indicators. New York, NY: Wiley.
Curry, S., Gadd, E. & Wilsdon, J. (2022). Harnessing the Metric Tide: indicators, infrastructures & priorities for UK responsible research assessment. Report of The Metric Tide Revisited panel, December 2022. https://doi.org/10.6084/m9.figshare.21701624
Dobrov, G. (1989). Science about science. Introduction to General Science of Science. Kyiv: Nauk. dumka [in Russian].
Malitsky, B. (2017). Who and how should assess research results of a scientist. Science and Science of Science, 3 (97), 34—53 https://doi.org/10.15407/sofs2017.03.034 [in Russian].
Malitsky, B. (2007). Applied science studies. Kyiv: Phenix [in Ukrainian].
Malitsky, B., Popovych O., Soloviov V., Yegorov, V., & Bulkin, I. (2004). The rational R&D financing: A premise for building up the knowledge-based society in Ukraine. Kyiv: Phenix [in Ukrainian].
Popovych O., & Ryzhko L. (2022). Modern challenges to the research system and ways to overcome them. Science and Science of Science, 1 (115), 88—104. https://doi.org/10.15407/sofs2022.01.088 [in Ukrainian].
Rybachuk, V. (2013). Methodological problems in the productivity assessment of research activities. Science and Science of Science, 2, 46—52 [in Ukrainian].
Yehorov, I.Yu. (Ed.) (2016). Indicators of R&D and innovation: international standards and the practice of their use in Ukraine. Kyiv: Institute of Economics and Forecasting of the National Academy of Sciences of Ukraine [in Ukrainian].
Gasanov, S., Kotlyarevsky, Y., Melnikov, O., Kniaziev, S., Shtangret, A., & Semenyuk, E. (2019). Methodological approaches to labor norming in scholarly research and development. Science and Innovation, 15 (1), 5—24. https://doi.org/10.15407/scine15.01.005
Horovyi, V.M. (2015). Criteria for the quality of scientific research in the context of ensuring national interests. Visn. Nac. Akad. Nauk Ukr., 6, 74—80 [in Ukrainian].
Pavliuk, K.V., & Kaminska, O.S. (2019). Foreign experience in assessing the quality of scientific activity. RFI Scientific Papers, 3, 25—40 [in Ukrainian]. https://doi.org/10.33763/npndfi2019.03.025
Pylypenko, H.M., & Fedorova, N.Ye. (2020). Science as a factor of socio-economic development of society. Dnipro: NTU “DP” [in Ukrainian].
Yaroshenko, T., & Yaroshenko, O. (2020). Highly cited documents of Ukrainian scientists in citation databases: correlation of bibliometric indicators. Ukrainian Journal of Library Science and Information Sciences, 5, 108—126. https://doi.org/10.31866/2616-7654.5.2020.205734 [in Ukrainian].
Reed, M. (2018). The research impact handbook. 2nd edn. Fast Track Impact, Huntly.
Reed, M., & Cairney, P. (2021). Using research to influence policy: a handbook for researchers. Leeds: University of Leeds.
Vutsova, A., Yalamov, T., & Arabadzhieva, M. (2023). In search of excellent research assessment. Baden-Baden: Nomos. https://doi.org/10.5771/9783748937203
Alstete, J.W., Beutell, N.J., & Meyer, J.P. (2018). Evaluating scholarship and research impact: history, practices, and policy development. Bingley, UK: Emerald Group Publishing Limited. https://doi.org/10.1108/9781787563872
Roemer, R.C. & Borchardt, R. (2015). Meaningful Metrics: A 21st Century Librarian’s Guide to Bibliometrics, Altmetrics, and Research Impact. Chicago: Association of College and Research Libraries, A division of the American Library Association.
Bornmann, L. (2013). What is societal impact of research and how can it be assessed? A literature survey. Journal of the American Society for Information Science and Technology, 64 (2), 217—233. https://doi.org/10.1002/asi.22803
Servaes, J. (2014). On impact factors and research assessment. Telematics and Informatics, 31 (1), 1—2. https://doi.org/10.1016/j.tele.2013.09.005
Samuel, G.N., & Derrick, G.E. (2015). Societal impact evaluation: Exploring evaluator perceptions of the characterization of impact under the REF2014. Research Evaluation, 24 (3), 229—241. https://doi.org/10.1093/reseval/rvv007
Peruginelli, J., & Pölönen, J. (2023). The legal foundation of responsible research assessment: An overview on European Union and Italy. Research Evaluation, 32 (4), 670—682. https://doi.org/10.1093/reseval/rvad035
МacFadden, B. (2019). Broader Impact of Science on Society. United Kingdom: Cambridge University Press, 2019. https://doi.org/10.1017/9781108377577
Molas-Gallart, J. (2012). Research Governance and the Role of Evaluation. A Comparative Study. American Journal of Evaluation, 33, 583—598. https://doi.org/10.1177/1098214012450938
Hammarfelt, B., & Haddow, G. (2018). Conflicting Measures and Values: How Humanities Scholars in Australia and Sweden Use and React to Bibliometric Indicators. Journal of the Association for Information Science and Technology, 69, 924—935. https://doi.org/10.1002/asi.24043
Wouters, P., Thelwall, M., Kousha, K., Waltman, L., de Rijcke, S., Rushforth, A., & et al. (2014). The Citation: From Culture to Infrastructure. B. Cronin, C.R. Sugimoto (Eds). Beyond Bibliometrics: Harnessing Multidimensional Indicators of Scholarly Impact. Cambridge, MA: MIT Press, 47—66. https://doi.org/10.7551/mitpress/9445.003.0006
Hinze, S., Butler, L., Donner, P., & McAllister, I. (2019). Different Processes, Similar Results? A Comparison of Performance Assessment in Three Countries. Springer Handbook of Science and Technology Indicators. Cham: Springer, 465—84. https://doi.org/10.1007/978-3-030-02511-3_18
Kostoff, R.N. (1997). Handbook of research impact assessment. 7th ed. Arlington: NTIS.
Pontika, N., Klebel, T., Correia, A., Metzler, H., Knoth, P., & Ross-Hellauer, T. (2022). Indicators of Research Quality, Quantity, Openness, and Responsibility in Institutional Review, Promotion, and Tenure Policies across Seven Countries. Quantitative Science Studies, 3, 888—911. https://doi.org/10.1162/qss_a_00224
Loktiev, V.M. (2021). Does basic science have a proper status in Ukraine? Visn. Nac. Akad. Nauk Ukr., 6, 52—58. https://doi.org/10.15407/visn2021.06.052 [in Ukrainian].
Samuel, G.N., & Derrick, G.E. (2015). Societal impact evaluation: Exploring evaluator perceptions of the characterization of impact under the REF2014. Research Evaluation, 24 (3), 229—241. https://doi.org/10.1093/reseval/rvv007
Jiménez-Contreras, E., de Moya Anegón, F., & Delgado-López-Cózar, E. (2003). The Evolution of Research Activity in Spain: The Impact of the National Commission for the Evaluation of Research Activity (CNEAI). Research Policy, 32, 123—142. https://doi.org/10.1016/S0048-7333(02)00008-2
Feenstra, R., & López-Cózar, E. (2023). The footprint of a metrics-based research evaluation system on Spain’s philosophical scholarship: An analysis of researchers’ perceptions. Research Evaluation, 32 (1), 32—46. https://doi.org/10.1093/reseval/rvac020
Orfila, M. (2023). Spain wants to change how it evaluates scientists — and end the “dictatorship of papers”: Officials aim to use wider range of research outputs to assess researchers at public universities. Science, 29 November. https://doi.org/10.1126/science.zehwqlq
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Science and Science of Science
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
