Open Science Impact Indicator Handbook

Author

Affiliation

V.A Traag

Leiden University

This is the Open Science Impact Indicator Handbook by PathOS. In this handbook we cover various indicators measuring various aspects around Open Science itself, their academic, societal and economic impacts, and reproducibility.

Executive summary

In the PathOS project we take a causal perspective on studying Open Science. This necessitates making a distinction between impact itself, and the effect of Open Science on impact. For instance, we could very well see an Open Source research tool being used frequently by industry. In that sense, the Open Source research tool can be said to have a type of economic impact. However, it could very well be that the research tool would have been similarly used by industry had it been released as closed software under a commercial licence. We are interested in the difference between its actual impact under the Open Science principles and its counterfactual impact under a closed principle. That is, we are interested in the causal effect of Open Science on the impact of the science.

Causal inference is not straightforward. We have included a separate section on causality, including an introduction to causality in science studies to explain some of the challenges around causal inference in Open Science. At the same time, this introduction also divulges the possibilities for inferring causality from observational data. Sometimes, we will learn that it will be impossible to correctly identify a certain causal effect. Although this limits our possible conclusions, we believe it is better to be clear about the impossibility of identifying a causal effect in some cases than to pretend we did identify some causal effect.

The challenge of causal inference also clarifies that we cannot provide straightforward indicators of effects of Open Science. There are many aspects of Open Science, including Open Access, Open Data and Open Code, but also elements such as Citizen Science, Open Science infrastructure, policies and training. At the same time, there are many different types of impacts in each domain of academia, society and economy. This leads to a combinatorial number of possible effects. For each such an effect it necessitates to carefully reason about its causal inference, and what other factors should be controlled for, or should not be controlled for. This is a daunting task, which goes well beyond our capacities. For that reason, the impact indicator handbook provides guidance on how to operationalise various indicators in order to facilitate studies on the effect of Open Science. In addition, we have explored three effects in greater detail, studying the effect of Open Data on citations and on Cost Savings, and issues of inferring causality in the context of societal impact. We hope this is useful to the research community, and that we together face the challenges of causal inference in Open Science studies.

Finally, not all indicators are equally well-developed. Some indicators, like citation impact, are already long established and studied in scientometrics. Other indicators, such as on data usage or the level of replication are much more recent. Such indicators may be under active development, or may actually not be worked on at all yet. We still include such indicators in this handbook if we have identified them to be vital for studying the impact of Open Science. This impact indicator handbook should therefore not only be seen as an inventory of what is possible today, but also what we believe is necessary tomorrow.

We hope this handbook will be a central hub to keep track of Open Science related indicators. We hope to be able to contribute to keeping the impact indicator handbook up-to-date as part of the PathOS project. The handbook is open to community contributions. Together we may create a central resource that is useful to all.

Acknowledgements

The PathOS project has received funding from the European Union’s Horizon Europe framework programme under grant agreement No. 101058728. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Executive Agency. Neither the European Union nor the European Research Executive Agency can be held responsible for them.