Open Science Indicator Handbook

This is the Open Science Indicator Handbook by PathOS. In this handbook we cover various indicators measuring various aspects around Open Science itself, their academic, societal and economic impacts, and reproducibility.

Executive summary

In the PathOS project we take a causal perspective on studying Open Science. This necessitates making a distinction between impact itself, and the effect of Open Science on impact. For instance, we could very well see an Open Source research tool being used frequently by industry. In that sense, the Open Source research tool can be said to have a type of economic impact. However, it could very well be that the research tool would have been similarly used by industry had it been released as closed software under a commercial licence. We are interested in the difference between its actual impact under the Open Science principles and its counterfactual impact under a closed principle. That is, we are interested in the causal effect of Open Science on the impact of the science.

Causal inference is not straightforward. We have included an introductory chapter on causality in science studies to explain some of the challenges around causal inference in Open Science. At the same time, this chapter also divulges the possibilities for inferring causality from observational data. Sometimes, we will learn that it will be impossible to correctly identify a certain causal effect. Although this limits our possible conclusions, we believe it is better to be clear about the impossibility of identifying a causal effect in some cases than to pretend we did identify some causal effect.

The challenge of causal inference also clarifies that we cannot provide straightforward indicators of effects of Open Science. There are many aspects of Open Science, including Open Access, Open Data and Open Code, but also elements such as Citizen Science, Open Science infrastructure, policies and training. At the same time, there are many different types of impacts in each domain of academia, society and economy. This leads to a combinatorial number of possible effects. For each such an effect it necessitates to carefully reason about its causal inference, and what other factors should be controlled for, or should not be controlled for. This is a daunting task, which goes well beyond our capacities. For that reason, the indicator handbook provides guidance on how to operationalise various indicators in order to facilitate studies on the effect of Open Science. We hope this is useful to the research community, and that we together face the challenges of causal inference in Open Science studies.

Finally, not all indicators wil be equally well-developed. Some indicators, like citation impact, are already long established and studied in scientometrics. Other indicators, such as on data usage or reproducibility are much more recent. Such indicators may be under active development, or may actually not be worked on at all yet. We still include such indicators in this handbook if we have identified this indicator to be vital for studying the causal effect of Open Science. This indicator handbook should therefore not only be seen as an inventory of what is possible today, but also what we believe is necessary tomorrow.

We hope this handbook will be a central hub to keep track of Open Science related indicators. Within the PathOS project the indicator handbook will remain in continuous development until january 2025. We hope to be able to contribute to keeping the indicator handbook up-to-date also afterwards. The handbook is open to community contributions. Together we may create a central resource that is useful to all.

Abbreviations

Abbreviation	Description	Explanation
APC	Article Processing Charge	Fee sometimes charged for publishing articles.
API	Application programming interface	Interface allowing to make use of certain service or program
DMP	Data Management Plan	Document describing how data will be managed
DOI	Digital Object Identifier	A DOI is a digital identifier of an object, any object — physical, digital, or abstract. Core element of academic infrastructure.
EOSC	European Open Science Cloud	An initiative aimed at developing an infrastructure to provide users with services promoting Open Science practices.
EU	European Union	Supranational political union consisting of member states that are located primarily in Europe
FAIR	Findability, Accessibility, Interoperability and Reusability	Principles for academic data management
NCI	Normalised Citation Impact	Citation impact that is normalised for field differences
IPR	Intellectual Property Rights	Rights to intangible creations of the human mind.
NLP	Natural Language Processing	Computer techniques and algorithm to automatically process and analyse natural language
OA	Open Access	Principles and practices around distributing research outputs, free from any barriers.
OS	Open Science	Practices and principles for making science more open in various ways
OSF	Open Science Framework	Platform to support and enable collaboration
REST API	Representational state transfer API	Type of API often used for web services
UNESCO	United Nations Educational, Scientific and Cultural Organization	Specialized agency of the United Nations (UN) aimed at promoting world peace and security through international cooperation in education, sciences, culture, communication and information.

Acknowledgements

This project has received funding from the European Union’s Horizon Europe framework programme under grant agreement No. 101058728. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Executive Agency. Neither the European Union nor the European Research Executive Agency can be held responsible for them.