Quality
V.A Traag 
Description
Quality is a very complicated concept, and in the context of academic work, is very challenging to measure. To start with, it should be clarified what object is being considered, which could range from data to peer review. In most such cases, quality cannot be defined on the basis of any easily measurable data, and rather require some form of manual assessment.
For the most traditional academic output, a scholarly publication, such a manual assessment is typically provided through peer review (Bornmann 2011). Peer review is much discussed in science studies, and there are discussions about its reliability (Cole, Cole, and Simon 1981) and its biases (Lee et al. 2013), but also about its positive effects (Goodman et al. 1994) and complementaries (Goyal et al. 2024).
Quality is typically considered to be a multidimensional concept (Aksnes, Langfeldt, and Wouters 2019), composed of various other concepts. For instance, in peer review of manuscripts submitted to journals, it is common to assess the novelty and the rigour of the manuscript. Yet even if quality is considered a multidimensional concept, in practice, quality is sometimes still considered to be unidimensional. For example, in the UK REF research articles are assigned a number of stars, varying from “recognised nationally” (1 star) to “world-leading” (4 stars).
In the context of exercises such as the UK REF there have also been discussions about the possibility to use citations as a proxy for quality. Indeed, there are substantial correlations between peer review results and citations, but this depends on the level of aggregation. At the individual paper level the correlation is typically low, yet at higher levels, such as institutional, the correlations are substantially higher (Traag, Malgarini, and Sarlo 2023). Overall, as summarised in the reputable “Metrics Tide” report (Wilsdon et al. 2015, viii), “Metrics should support, not supplant, expert judgement.”, and this is particularly relevant at the individual paper level.
References
Reuse
Citation
@online{apartis2024,
author = {Apartis, S. and Catalano, G. and Consiglio, G. and Costas,
R. and Delugas, E. and Dulong de Rosnay, M. and Grypari, I. and
Karasz, I. and Klebel, Thomas and Kormann, E. and Manola, N. and
Papageorgiou, H. and Seminaroti, E. and Stavropoulos, P. and Stoy,
L. and Traag, V.A. and van Leeuwen, T. and Venturini, T. and
Vignetti, S. and Waltman, L. and Willemse, T.},
title = {Open {Science} {Impact} {Indicator} {Handbook}},
date = {2024},
url = {https://handbook.pathos-project.eu/sections/2_academic_impact/quality.html},
doi = {10.5281/zenodo.14538442},
langid = {en}
}