Uptake by policy makers
R. Costas 
History
| Version | Revision date | Revision | Author |
| 1.0 | 2023-02-07 | Finalised indicator template | T.P. Willemse, V.A. Traag |
Description
The main concept of the indicator “Uptake by Policy Makers” revolves around the extent to which policy makers engage with and incorporate scientific publications into their decision-making processes. A recently introduced metric for measuring uptake by policy makers is the number of citations to scientific publications coming from policy documents, also referred to as “policy citations” (Szomszor & Adie, 2022, Overton, 2023).
Policy citations are usually extracted from platforms or tools dedicated to index policy sources, being Altmetric.com and Overton (https://www.overton.io/) the most important platforms currently available (Murat et la, 2023). These platforms aggregate data from various policy sources and provide overviews o how a particular scientific publication has been used in policy documents, having the potential to capture the uptake of science by policy makers.
Similar to other altmetric indicators, like academic readership indicators or social media uptake, metrics related to policy citations are computed at the level of individual scientific publications, often identified by a DOI or other unique identifiers. Altmetric, for instance, employs a range of identifiers, including DOIs, ISBNs, and PubMed IDs, while Overton uses DOIs to identify mentions in policy documents to scientific publications. The counts and mentions can be extracted through the APIs and search engines implemented by these platforms, however currently none of them is entirely open available, although they provide free tools and demos that can be used for small sets of publications.
From a theoretical point of view, the count of policy citations can be leveraged similarly to citation counts in scholarly contexts. Derived indicators could include metrics like the mean policy citation score, mean normalized policy citation scores, identification of highly cited publications in policy circles, and other relevant measures that gauge the impact of research on policy-making processes, although none of them have been proposed or implemented yet in the literature.
Metrics
Policy citations
As discussed above, to the best of our knowledge there are no specific proposals of concrete metrics of policy citation indicators beyond the current counts of policy citations provided in Altmetric.com and Overton. From a theoretical point of view, similar indicators as for citations and academic readership would be possible, however given the relatively low uptake of policy citations across scientific publications (see Bornmann et al, 2016l Fang et al, 2020), the most sensible metric to be consider would be the counts of policy citations, perhaps complemented with more qualitative exploration of the types of mentions in policy documents.
Measurement.
For any given set of scientific publications, utilizing their DOIs (or other unique publication identifiers), one can employ the Altmetric or Overton APIs, or use their search tools, to retrieve the total count of policy citations to each of the scientific publications. These counts can then be aggregated as the total count of citations for the set, or even averages or other statistics.
Existing datasources:
Altmetric
Altmetric.com was founded in 2011 by Euan Adie (Murat et al, 2023) with the aim of collecting the influence of scientific publications across different media and social media platforms, including policy document citations (Bornmann et al, 2016; Tattersall & Carroll, 2018). Altmetric curates policy sources from organizations around the world, and that are designed to influence policy or practice (Altmetric, 2021). Policy sources are collected directly from organizations’ websites.
Overton
Overton was founded in 2019 also by Euan Adie. It is a novel database with the goal to increase the coverage and comprehensiveness of policy-focused data and information. Overton indexes policy documents from all over the world, and it is created by the web crawling of publicly accessible documents published by a curated list of over 30,000 organizations including governments, intergovernmental organizations, think tanks, and charities (Szomszor & Adie, 2022).
Existing methodologies
Policy citations impact
Beyond the methodologies of the different data providers mentioned above (Overton and Altmetric), there no other extensive methodologies that extract citations to scientific publications from policy sources.
Known correlates
Previous studies have reported mostly positive weak correlations between policy citations and other metrics, including scholarly citations (see Fang et al, 2020; Bornmann et al, 2022; Szomszor & Adie, 2022).