Open Science Indicator Handbook
Introduction
This is the work-in-progress Open Science Indicator Handbook by PathOS. In this handbook we cover various indicators measuring various aspects around Open Science itself, their academic, societal and economic impacts, and reproducibility.
Executive summary
In the PathOS project we take a causal perspective on studying Open Science. This necessitates making a distinction between impact itself, and the effect of Open Science on impact. For instance, we could very well see an Open Source research tool being used frequently by industry. In that sense, the Open Source research tool can be said to have a type of economic impact. However, it could very well be that the research tool would have been similarly used by industry had it been released as closed software under a commercial licence. We are interested in the difference between its actual impact under the Open Science principles and its counterfactual impact under a closed principle. That is, we are interested in the causal effect of Open Science on the impact of the science.
Causal inference is not straightforward. We will include an introductory chapter to explain some of the challenges around causal inference in Open Science. At the same time, this chapter also divulges the possibilities for inferring causality from observational data. Sometimes, we will learn that it will be impossible to correctly identify a certain causal effect. Although this limits our possible conclusions, we believe it is better to be clear about the impossibility of identifying a causal effect in some cases than to pretend we did identify some causal effect.
The challenge of causal inference also clarifies that we cannot provide straightforward indicators of effects of Open Science. There are many aspects of Open Science, including Open Access, Open Data and Open Code, but also elements such as Citizen Science, Open Science infrastructure, policies and training. At the same time, there are many different types of impacts in each domain of academia, society and economy. This leads to a combinatorial number of possible effects. For each such an effect it necessitates to carefully reason about its causal inference, and what other factors should be controlled for, or should not be controlled for. This is a daunting task, which goes well beyond our capacities. For that reason, the indicator handbook provides guidance on how to operationalise various indicators in order to facilitate studies on the effect of Open Science. We hope this is useful to the research community, and that we together face the challenges of causal inference in Open Science studies.
Finally, not all indicators wil be equally well-developed. Some indicators, like citation impact, are already long established and studies in scientometrics. Other indicators, such as on data usage or reproducibility are much more recent. Such indicators may be under active development, or may actually not be worked on at all yet. We still include such indicators in this handbook if we have identified this indicator to be vital for studying the causal effect of Open Science. This indicator handbook should therefore not only be seen as an inventory of what is possible today, but also what we believe is necessary tomorrow.
We hope this handbook will be a central hub to keep track of Open Science related indicators. At the moment, the handbook contains only a first draft of Open Science indicators and Reproducibility indicators. It also contains an outline of the various impact indicators that we intend to cover. Within the PathOS project the indicator handbook will remain in continuous development until january 2025. We hope to be able to contribute to keeping the indicator handbook up-to-date also afterwards. The handbook is open to community contributions. Together we may create a central resource that is useful to all.
Abbreviations
| Abbreviation | Description | Explanation |
|---|---|---|
| APC | Article Processing Charge | Fee sometimes charged for publishing articles. |
| API | Application programming interface | Interface allowing to make use of certain service or program |
| DMP | Data Management Plan | Document describing how data will be managed |
| DOI | Digital Object Identifier | A DOI is a digital identifier of an object, any object — physical, digital, or abstract. Core element of academic infrastructure. |
| EOSC | European Open Science Cloud | An initiative aimed at developing an infrastructure to provide users with services promoting Open Science practices. |
| EU | European Union | Supranational political union consisting of member states that are located primarily in Europe |
| FAIR | Findability, Accessibility, Interoperability and Reusability | Principles for academic data management |
| NCI | Normalised Citation Impact | Citation impact that is normalised for field differences |
| IPR | Intellectual Property Rights | Rights to intangible creations of the human mind. |
| NLP | Natural Language Processing | Computer techniques and algorithm to automatically process and analyse natural language |
| OA | Open Access | Principles and practices around distributing research outputs, free from any barriers. |
| OS | Open Science | Practices and principles for making science more open in various ways |
| OSF | Open Science Framework | Platform to support and enable collaboration |
| REST API | Representational state transfer API | Type of API often used for web services |
| UNESCO | United Nations Educational, Scientific and Cultural Organization | Specialized agency of the United Nations (UN) aimed at promoting world peace and security through international cooperation in education, sciences, culture, communication and information. |