Uptake by policy makers
R. Costas 
Description
The main concept of the indicator “Uptake by Policy Makers” revolves around the extent to which policy makers engage with and incorporate scientific publications into their decision-making processes. A recently introduced metric for measuring uptake by policy makers is the number of citations to scientific publications coming from policy documents, also referred to as “policy citations” (Szomszor and Adie 2022).
Policy citations are usually extracted from platforms or tools dedicated to index policy sources, being Altmetric.com and Overton (https://www.overton.io/) the most important platforms currently available (Murat, Noyons, and Costas 2023). These platforms aggregate data from various policy sources and provide overviews on how scientific publications or other scholarly entities (e.g. journals, publishers, repositories, universities, researchers, or even scientific concepts) have been mentioned in policy documents, having the potential to capture the uptake of science by policy makers.
Similar to other altmetric indicators, like academic readership indicators or social media uptake, metrics related to policy citations are computed at the level of individual scientific publications, often identified by a DOI or other unique identifiers. Altmetric, for instance, employs a rnge of identifiers, including DOIs, ISBNs, and PubMed IDs, while Overton uses DOIs to identify mentions in policy documents to scientific publications. The counts and mentions can be extracted through the APIs and search engines implemented by these platforms, however currently none of them is entirely open available, although they provide free tools and demos that can be used for small sets of publications.
From a theoretical point of view, the count of policy citations can be leveraged similarly to citation counts in scholarly contexts. Derived indicators could include metrics like the mean policy citation score, mean normalized policy citation scores, identification of highly cited publications in policy circles, and other relevant measures that gauge the impact of research on policy-making processes, although none of them have been proposed or implemented yet in the literature.
Metrics
Policy citations
As discussed above, to the best of our knowledge there are no specific proposals of concrete metrics of policy citation indicators beyond the current counts of policy citations provided in Altmetric.com and Overton. From a theoretical point of view, similar indicators as for citations and academic readership would be possible, however given the relatively low uptake of policy citations across scientific publications (Fang et al. 2024; Bornmann, Haunschild, and Marx 2016), the most sensible metric to be consider would be the counts of policy citations, perhaps complemented with more qualitative exploration of the types of mentions in policy documents. Similarly, policy citations metrics could be complemented with the identification of mentions in policy documents to other larger scholarly entities like journals, repositories, individual researchers, universities, or scientific concepts, to name a few.
Measurement
For any given set of scientific publications, utilizing their DOIs (or other unique publication identifiers), one can employ the Altmetric or Overton APIs, or use their search tools, to retrieve the total count of policy citations to each of the scientific publications. These counts can then be aggregated as the total count of citations for the set, or even averages or other statistics.
Existing datasources
Altmetric
Altmetric.com was founded in 2011 by Euan Adie (Murat, Noyons, and Costas 2023) with the aim of collecting the influence of scientific publications across different media and social media platforms, including policy document citations (Bornmann, Haunschild, and Marx 2016; Tattersall and Carroll 2018). Altmetric curates policy sources from organizations around the world, and that are designed to influence policy or practice1. Policy sources are collected directly from organizations’ websites.
Overton
Overton was founded in 2019 also by Euan Adie. It is a novel database with the goal to increase the coverage and comprehensiveness of policy-focused data and information. Overton indexes policy documents from all over the world, and it is created by the web crawling of publicly accessible documents published by a curated list of over 30,000 organizations including governments, intergovernmental organizations, think tanks, and charities (Szomszor and Adie 2022).
Existing methodologies
Policy citations impact
Beyond the methodologies of the different data providers mentioned above (Overton and Altmetric), and other past applications related to the extractions of mentions to scientific concepts, there are no other large methodologies focused on extracting mentions to scientific publications or scholarly entities from policy sources.
Known correlates
Previous studies have reported mostly positive weak correlations between policy citations and other metrics, including scholarly citations (Fang et al. 2024; Bornmann, Haunschild, and Marx 2016; Szomszor and Adie 2022).
References
Footnotes
https://help.altmetric.com/support/solutions/articles/6000236695-policy-documents↩︎
Reuse
Citation
@online{apartis2024,
author = {Apartis, S. and Catalano, G. and Consiglio, G. and Costas,
R. and Delugas, E. and Dulong de Rosnay, M. and Grypari, I. and
Karasz, I. and Klebel, Thomas and Kormann, E. and Manola, N. and
Papageorgiou, H. and Seminaroti, E. and Stavropoulos, P. and Stoy,
L. and Traag, V.A. and van Leeuwen, T. and Venturini, T. and
Vignetti, S. and Waltman, L. and Willemse, T.},
title = {Open {Science} {Impact} {Indicator} {Handbook}},
date = {2024},
url = {https://handbook.pathos-project.eu/sections/3_societal_impact/uptake_by_policy_makers.html},
doi = {10.5281/zenodo.14538442},
langid = {en}
}