A new feature added to the DevPubMetric system in May 2021 that links the publication lists generated by DevPubMetric with the 2021 SDG Mapping methodology released as part of the SciVal system in April 2021.
Further information about SciVal’s SDG mapping is available at: https://data.mendeley.com/datasets/9sxdykm8s4/2 (doi: 10.17632/9sxdykm8s4.2)
The first SciVal analysis for SDG mapping was released in April 2021 using a data extract from the Scopus database in January 2021. From June 2021 it is expected that new data extracts will be available through SciVal once every two months until late 2021 when a bi-weekly data extract is expected.
The methodology currently being applied in DevPubMetric requires a set of manual steps outlined below. It is expected that this process will be automated once regular bi-weekly data extracts are available from SciVal in 2021.
Step 1. Create list of publications to be analysed
This step is carried out on a monthly basis for relevant programmes within the DevPubMetric system. Data for the Newton Fund and GCRF programmes are currently available on this website at: pvgglobal.uk/data.
The list of publications need to include either the publications digital object identifier (doi) or the electronic identifier (eid) used by the Scopus system. Where possible, it is advisable to use the eid value as some publications will not have a doi value.
Step 2. Create a Publication Set in SciVal
A list of publication identifiers (eid or doi), with one id per line is pasted into SciVal using the “Import a Publication Set” command. In most cases, SciVal will need to process this list offline before the set is available for additional analysis. An email is generated by SciVal when the new Publication Set is available.
Step 3. Extract the new SDG Mapping from Scival
The new Publication Set is opened in SciVal and an appropriate date range is selected (the DevPubMetric analysis is currently working with publications dates >= 2015.
The resulting publication subset is then viewed so that the results can be exported as a CSV file. Only two data items are exported, the publication’s eid and the field “Sustainable Development Goals (2021)”.
Step 4. Exclude publications in Scival that were added after the most recent SDG analysis.
The data export produced by SciVal (Step 3, above) is likely to include a small number of publications that have been added to the system since the date of the most recent data extract used for the SDG mapping. It is currently not possible to differentiate between publications that do not map to any SDGs, from those which are recent publications that were not included in the most recent SciVal SDG analysis.
The approach used by the DevPubMetric system is to use the original publication list generated in Step 1 as part of a query to the Scopus system to determine when the publication was first added to Scopus and hence available to SciVal for the SDG analysis. The reason for this is that SciVal utilises data derived from Scopus for analysis.
This process utilises the Scopus Application Interface (API) dev.elsevier.com/scopus.html using the “Scopus Search Option) to return a list of publications that were added to Scopus before the most recent SciVal SDG analysis.
The search term used combined a list of Scopus eid values and a function to only return publications that were added to Scopus before a set date.
The Scopus query used in this process is constructed as : EID(comma separated list of eids)AND ORIG-LOAD-DATE BEF date-in-unix-format
where: – comma separated list of eids is generated from the list of publications. – date-in-unix-format is a numeric value representing the number of seconds since 1 January 1970.
A list of publication identifiers (eids) is then generated by the data returned from the API call.
Step 5. Identify publications that have contributed to Covid-19 research areas.
The Corvid-19 pandemic has generated a very significant surge of publications which could be argued to be relevant to the SDGs or at least the theme of global challenge research.
A list of publications where generated using the query published in SciVal using the Scopus Search API (See above, Step 4).
The Scopus query used in this process is constructed as : EID(comma separated list of eids)AND (TITLE-ABS-KEY(“coronavirus disease 2019” OR covid19 OR covid OR ncov OR sars-cov-2 OR {novel coronavirus}) AND (PUBYEAR > 2018))
where: – comma separated list of eids is generated from the list of publications.
Step 6. Generate list of publications and the SDGs that they have been mapped against.
This task is processed by reading the data returned from Steps 3, 4 and 5 into a MySQL database where the resulting data tables are joined using a condition to will only return data if the eid is contained in the table generated in Step 4. A standard SQL INNER JOIN condition is used for this step. The resulting query contains two columns, eid and SDG, with one row of data for each SDG that a publication has been mapped against. If a publication has contributed to Covid-19 research the DevPubMetric system adds a row with a dedicated unique value (SDG=20) There is also a dedicated value (SDG=100) used to identify publications that did not map against a SDG and did not contribute to COVID-19 research.
The resulting data table is then available for further analysis. The DevPubMetric system currently provides a diagrammatic analysis of the contributions of both the GCRF and Newton Fund programmes to the SDGs. Additional analysis will be provided in future releases of the DevPubMetric system.
Information about current activities and projects is summarised below:
Reform of Higher Education, Research and Innovation in Kurdistan and Iraq.
Reform of Higher Education, Research and Innovation in Kurdistan and Iraq.
Supporting the Prime Minister’s Office of the Kurdistan Regional Government (KRG), building on previous activities supporting HE and research reform in Africa with the African Union and NEPAD and Rwanda See my keynote presentation on the topic at an 2018 OECD event on “Strengthening linkages between STI and ODA policies in order to address the grand challenges and promote inclusive growth”.
DevPubMetric. Capturing the Impact of INvestment in Development Research
DevPubMetric was designed and built during 2020 to collect data on the academic impacts of the UK’s Ecosystem Services for Poverty Alleviation Programme (ESPA). As ESPA’s first Director (2010-2016) I pioneered a “Big-Data” approach to build a set of progressive Key Performance Indicators for the programme which were used up to the closure of the programme in 2018. DevPubMetric was designed to update the process using publicly available sources of data and uses more sophisticated analysis benefiting for advances in data collection and the shift to support open access to data and publications.
The system is currently being applied to analyse ESPA+10 (ten years of progress) and to review outcomes from the first phases of the UK’s Newton Fund and the Global Challenges Research Fund (GCRF). Results from this work will be published on this website in the last quarter of 2020.
ESPA Results: Ten Years ON
The DevPubMetric system is being applied to update the information collected through the previous Monitoring Evaluation and Learning system used by ESPA. The initial results will be published in October 2020 to coincide with the 10th anniversary of ESPA’s first Science Conference.
Reviewing Outcomes from the Newton Fund and Global Challenges Research Fund
The DevPubMetric system is being applied to review the academic outcomes and levels of engagement with developing country researchers from the UK’s Newton Fund (2014-) and Global Challenges Research Fund (2016-). This work in ongoing and will feed into discussions about potential next phases of both programmes currently being considered by the UK Government. As part of this research the DevPubMetric system is being extended to include the ability to map academic publications against the UN’s Sustainable Development Goals, using an adapted version of the SDG search terms published by Elsevier and used as part of the Times Higher Education Supplement’s Impact Rankings.
Innovation Systems: Higher Education, Science, Technology and Innovation
Innovation Systems: Higher Education, Science, Technology and Innovation
Work is ongoing to update the provocation paper used at the GCRF-funded Narobi meeting in 2017 that then contributed to material for the keynote at papers presented the OCED event in 2018. It is intended that this work will feed into a wider discussion during 2021 including an event on Innovation Systems as part of the University of Nairobi’s Innovation Week in March 2021, 4 years after the original worksop in Nairobi.
The objective for this work is to set out a new type of relationship between higher education, governments and business to promote innovation and enterprise with a strong link to economic growth and creating jobs for the rapidly growing numbers of young people in the emerging economies in sub-Saharan Africa and South Asia.
Plastic Waste in Africa
Collaborative research with the University of Malawi and University of Nairobi on the use and fate of plastics in the environment. More Detail:
DevPubMetricprovides an objective and quantitative method to measure and compare the academic impact of development or global challenge research, whilst also estimating the levels of engagement of developing country researchers and research institutions.
In the September 2000, the United Nations (UN) adopted the Millennium Declaration and in the following year the Millennium Development Goals (MDGs). The MDG’s set out an ambitious agenda for global development and poverty reduction up to 2015 when the current set of Sustainable Development Goals (SDGs) were adopted by the UN.
The resulting focus on sustainable development and addressing the world’s shared global challenges has led to very significant growth in research that addresses these challenges.
A number of major research initiatives addressing global challenges and sustainable development have been launched by governments, philanthropic and multilateral organisations along with a number of major charities.
Most of these new programmes have shared objectives that include:
Contributing to global knowledge (publications);
Driving both academic and development impact;
Engaging with and empowering developing country researchers and research organisations.
There are well established methodologies for measuring and documenting development impact, though in many cases these can be challenged by the complex non-linear pathways that link research and impact, the time-lag to build impact and the question of how to evidence attribution.
There are many fewer examples of systematic processes to measure the academic impact of development research, and virtually none that map the global engagement and partnerships that deliver the research and its impact.
In 2013 the ESPA Directorate developed a quantitative monitoring framework for the programme, that included measures of both academic impact (numbers of publications and citations) and engagement with developing country researchers (proxy indicators based on the proportion of academic publications with a developing country first author and the proportion of with at least one developing country author. These data formed part of ESPA’s monitoring and evaluation leading to standard reporting against its Theory of Change and Logical Framework. An example of regular reporting that used these data is presented below
Publication metrics presented as part of monthly reporting produced by the ESPA Directorate.
Monthly reporting was collated by the ESPA Directorate up to the point that the programme closed in July 2018. ESPA’s final highlight document included the last set of data collated by the programme at which time the programme had captured information about 419 journal articles which had been cited 8,516 times.
The original system developed by the ESPA Directorate utilised data and systems that were available in 2013-15 with a high level of reliance on internal data and extensive manual processing that required every paper to be assessed by a team member. This meant that data collection and reporting could not continue automatically once the programme had closed. It also meant that ESPA’s approach could not be extended to be used by much larger research programmes that followed including the UK’s Newton Fund and Global Challenges Research Fund.
Defining the challenge: Capturing the opportunities of “open” big data.
During 2020, work commenced to implement a new system, DevPubMetric that could provide an update to the data previously collected by ESPA in a way that was essentially automatic and could easily be extended to other research programmes and research funders. The motivation for this work was to update the statistics from ESPA to capture and quantify the very significant academic impact of research after projects and programmes closed. It is often said that up to half of a programme’s academic impact may be invisible after funding ceases.
Whilst this work was originally intended to update data for ESPA, it was realised that the same approach could be used for other programmes, providing the opportunity to do quantitative comparative analysis between and within programmes.
In order to deliver these objectives DevPubMetric was designed to address a set of challenges an opportunities as detailed below:
Challenges and Opportunities
Responses
Automate data collection and analysis.
This has become possible through the evolution of data products and systems. • Extensive access to downloadable data through web-based APIs ( Application Programming Interface) • Enhanced data availability for UK funded research through the Gateway to Research (GTR) • Enhanced data content available from commercial bibliometric databases (including open access metrics and information on authors and their institutional affiliations.
Remove the requirement for manual processing.
The development and application of a standardised system for data processing and analysis has removed the requirement for routine manual interventions. It has not been possible to remove this requirement entirely because of some issues relating to data compatibility, for example in the names of some countries and institutions in different systems.
Where possible these issues linked to integrating data from different sources have been addressed through the application of look-up tables. These tables need to be checked and updated at intervals.
Develop new approaches to measure the levels of open access publication and developing country engagement.
The original ESPA system linked measures of developing country engagement to data stored about funded researchers which are not available in the public domain and could not be applied to other research programmes.
For this reason the new system of analysis tracks engagement through the institutional links of authors, which are data readily available from the current generation of bibliometric analysis including the Elsevier Scopus system that is used in the new analysis.
Enhance the range of publications used for analysis
The system developed for ESPA used data downloaded from the Web of Science bibliometric system delivered by Clarivate Analytics.
The updated system has moved to use data provided by Scopus provided by Elsevier. Both systems are constantly evolving and whilst they have many common functions, there are aspects where each can be considered to have an advantage.
One of the perceived strengths of Scopus is its broader coverage of current academic journals, especially for social sciences, arts and humanities. This is the main reason that the choice of data provider has changed for the current system of analysis.
Both Scopus and Web of Science have added additional information in the period since ESPA’s original system was designed. Of most relevance is the ability to identify open access publications and the provision of detailed information about the institutional affiliations of all authors of a publication
Enable objective comparison with other programmes
The design of the new data collection and analysis system utilises a set of standard data that should be readily available for most major research programmes. This required the ability to link publications to programmes using unique identifiers of either by programme name or project reference codes. These in turn can be used as search terms in biometric systems.
In addition, if the programme publishes a downloadable record of publications these can also be used to supplement analysis in case authors have not acknowledged links to funding programmes.
DevPubMetric utilises data from a variety of open access data sources and then combines it with data extracted from the commercially provided Scopus bibliometric system. The results provided here are restricted to those available from open access sources, or the synthesis of results obtained from Scopus along with information that Scopus lists in the public domain.
The system then generates lists of projects associated with key research programmes and full lists of publications that have been captured through the core search protocols. These data have been collated in the data repository for the system: https://pvgglobal.uk/devpubmetric/repository. These data extracts are updated on the first day of each month with data generated from the system after new publications and projects have been extracted.
Canadian International Development Research Centre (IDRC/CRDI): idrc.ca
One of the search criteria used in analysis is to extract abstracts from publications that acknowledge the programme. This information needs to represent a unique identifier and for this reason acronyms have been found to be unreliable
SCOPUS Search terms have been developed for each programme to guide the capture of publications from each programme to balance the capture of relevance publications against rejection of those that should not be attributed at programme level.
Some programmes, such as the Newton Fund have multiple names in difference contexts. The search process has been designed to cover all known variants of the programme name (currently restricted to English language versions.)
Publications are downloaded from Scopus using an abstract search based on unique project identifiers (where available) or by using queries of funder information using a set of uniquely defined programme names.
All projects funded by UKRI are required to report outcomes including publications through the ResearchFish reporting system. These data are then checked by UKRI systems before being replicated on their Gateway to Research System. These data are in the public domain and can be downloaded using using unique identifiers for a programme or project. One constraint with this approach is that it relies on accurate and timely reporting by projects.
In order to partially address the constraint of under-reporting, Scopus is used to capture additional publications, by searching for publications using unique programme and projects identifiers where these are available.
Scopus was used to provide the descriptive information about publications for further analysis. This included a full list of authors, their institutional affiliation and the countries that those institutions were operating from. Scopus also provided data on which publications were open access.The disadvantage of reliance on Scopus is that some lesser-know journals are not covered (including many published in developing countries) and that the system does not capture open-access publications provided through institutional repositories.
An alternative approach utilising data that are collected through the ResearchFish / Gateway to Research portals was found to have insufficient detail to meet the needs of the analysis. In some types of publication, lists of authors were incomplete and institutional detail did not have the level of information required to link back to map engagement with developing countries.
Publication metadata: Corresponding Authors and their affiliation.
The first phase of analysis using DevPubMetric used authorship as a proxy measure of the levels of engagement of developing country researchers.
It was noted that first authorship is not always the most appropriate measure of scientific leadership in publications and that protocols for deciding who acts as first author varies between disciplines.
The Scopus system stores information about the corresponding author for most publications. These data have collected from May 2021 using the Scopus “Abstract” API (Application Interface)
Publication metadata: Funding organisations and project references.
The Scopus system uses pattern matching and AI techniques to extract information extracted from publications through their acknowledgements of funding.
Data describing funders and project reference codes has been collected from May 2021 using the Scopus “Abstract” API (Application Interface). Earlier versions of the DevPubMetric system collated funding data from the Scopus “Search” API, which was found to be incomplete during the quality assurance process of the data extract.
Development status of institutional affiliations for authors.
Current data on World Development Indicators, including the income status of all nations recognised by the United Nations has been downloaded for integration into the analysis. The analysis is based on the institutional affiliation of each author as recorded by the publication. In some cases authors will list more than one institutional affiliation. In this case, the author was linked to the country with the lowest income status.(High income > Middle income > Low Income)
Geographic and SDG groups and additional population data
Integrating data: Generic approaches, challenges, complexity and solutions
DevPubMetric utilises data provided from up to four distinct data sources. These data need to be integrated before the analysis can be completed. There are three stages to this process:
Most of the data sources used in DevPubMetric have pre-existing Unique Identifiers (UIDs) for core data. These have been retained for the analysis and where necessary look-up tables have been created to link data sources. These are outlined below:
Data type
Unique Identifier(s)
Programme Names
As published by funders.
The SCOPUS search terms needed to be adjusted to maximise the probability that publications were captured whilst also minimising the likelihood of false positives.
Derived from Gateway to Research (GTR) ∙ Published UKRI project reference ∙ GTR Project UID.
From May 2021, DevPubMetric has also extracted project references via Scopus which are derived from the acknowledgement sections of most publications.
Publications
· Scopus Electronic Identifier (eid) · Gateway to Research UID · Digital Object Identifiers (DOI) included in bibliographic information · The ESPA programme produced a downloadable database of publications produced up to July 2018.
DOIs are used to create a look-up table linking systems
Gateway to Research data does not currently have comprehensive data for non-UK institutions. It is currently not possible to cross-link these data sources
Countriesand Income Status
The World Bank’s World Development Indicators include a comprehensive list of countries recognised by the United Nations. This list includes the current income-status for each territory.
The Scopus system does not publish a list of countries, but this can be extracted from data downloaded from Scopus. These data include a small number of territories not currently recognised by the United Nations or with names that do not directly map onto those used by the United Nations.
Scopus These data are stored for every publication at monthly intervals, permitting tracking of the time course of academic impact for individual publications, projects and programmes
Each of the systems publishing data used in DevPubMetric has implemented a form of internal quality assurance with details published along with the data. the development of the analysis system has identified a number of areas where existing quality assurance currently acts as a constraint.
Publication Coverage (Scopus)
The Scopus system has comprehensive coverage of academic publications, but this does not include all publications likely to be used for reporting results from development or global challenge research. Known gaps include:
Lower impact publications, including many national or regional publications originating from developing countries;
Relatively low coverage of books, book chapters and monographs;
Lower coverage of publications in languages other than English.
Poor coverage of pre-prints.
Whilst these issues are a constraint, there is currently no better alternatives that can provide the systematic analysis used in the current system. As with all commercial bibliometric systems, Scopus (and possible alternatives) are under constant development and some the current constraints may be mitigated in future releases.
The advantages of using a comprehensive commercial bibliometric system is that there are very high levels of internal quality assurance and internal consistency. One area key importance is the data data model which has four core components: Publications,
Reporting Gaps and Over-reporting (Gateway To Research & Scopus)
DevPubMetric makes full use of UKRI project reporting from the Gateway to Research System, which in turn relies on reporting by projects and researchers. A subset of publications reported through ResearchFish were tested as an additional quality assurance during the development of the current system. The following issues emerged from the QA process:
Reporting Gaps: This is a result of researchers failing to (or being unable to ) report publications linked to a project or programme. These gaps are identified when results are compared with those resulting to a direct Scopus search by programme name or project identifiers.
Over-reporting: There are also examples of potential over-reporting, which represent publications that should realistically not be considered to be attributable to the programme or project that they are linked to. This can be seen in Gateway to Research records, but also can apply to records derived from Scopus if the authors have provided the relevant programme name or project codes.
Issues relating to under and over-reporting are largely linked to behaviour of researchers and authors. Research funders provide guidance and encouragement to facilitate the adoption of good practice, but it is clear that both issues remain to some extent. One possible application of the analytical system would be to use a random subset of publications within a programme to provide an assessment of the scale of the issue
Lack of Institutional Affiliation in some Publications
The analysis and reporting in DevPubMetric uses institutional affiliations of authors to link publications with countries. There are a small number of publications where the publisher does not include this information and hence it is not collected by abstracting services and bibliometric analysis.
Analysis of the scope of this issue is ongoing. Initial results suggests that it is most prevalent in papers with large numbers of authors in specific academic disciplines.
Institutions and Countries Linked to Research (Gateway to Research)
Data extracted from Gateway to Research were examined to evaluate how these could be used to enable additional analysis of the levels of engagement with researchers and institutions in developing countries. This was not possible for two reasons:
Gateway to Research is currently not recording links between researchers and institutions listed on a grant, with the exception of the one Principal Investigator listed on the grant application. (Data are available of the current institution linked to individuals, but this is know to change with time and may not relate to a grant at any time.
Gateway to Research records details of the main institutions linked to the grant, including those outside the UK. Details recorded for non-UK institutions are inconsistent, with the majority of records not listing which country those institutions are located in. There are also instances of duplication of institutional records which means that there may not be a unique institutional identifier for non-UK institutions. This issue also applies to some non-traditional UK institutions.
Current Applications for DevPubMetric
Current applications of the DecPubMetric system are provided on the website at pvgglobal.uk/devpubmetric. More detailed analysis of results are provided on the following topis:
Application of DevPubMetric to other development research programmes, nows including FCDO/DFID, USIAD and IDRC.
SDG mapping of research publications captured by the system
Analysis of academic outputs from the Nairobi Alliance (A research alliance joining the Universities of Nairobi, Rwanda, Malawi, Witwatersrand and Leicester)
Potential future developments and applications for DevPubMetric
The data collected for DevPubMetric analysis can be extended to build additional applications. The following potential applications are being consider for the next phase of applications:
Institutional performance for development research, either at programme or full institutional level.
Gender analysis of publications.
Linking the analysis at programme or institutional level with emerging processes to link academic outputs to relevance to the Sustainable Development Goals.
Change/update log
Date
Change
Comments
10 October 2021
Added links to full publication lists extracted by DevPubMetric for the ESPA, GCRF and Newton Fund programmes
Updated data derived from the World Bank’s World Development Indicators. This update includes changes in the income status of 10 countries.
Two additional fields of data were added to country summaries (pvgglobal.uk/country-list/): ● National Gini coefficient, a measure of inequity; ● Proportion (%) of the youth population (15-24) not in employment, education or training (NEET).
8 June 2021
Published information about a wider range of development research programmes
Published data on: FCDO/DFID Research (UK) USAID (United States) IDRC (Canada).
Comparison of DevPubMetric KPIs for all programmes currently captured by the system.
18 May 2021
Identification of publications that contribute to Covid-19 research
The methodology used to generate SDG mapping was extended to identify publications relevant to Covid-19. These data, where available, are currently included in the diagrammatic presentations of SDG mapping. pvgglobal.uk/activity/sdg-mapping/
18 May 2021
SDG Analysis provided in DevPubMetric.
SDG analysis and mapping has been including by linking publication lists generated by DevPubMetric with the 2021 SDG Mapping Protocol provided in SciVal.
Publication of a full list of projects that have been captured by the DevPubMetric system for the GCRF and Newton programmes derived from the UKRI Gateway To Research system.
Data are provided as a CSV file that can be read directly by most spreadsheet and database systems
Enhanced data capture of publications derived from the UKRI Gateway To Research (GTR) database
Data capture methodology modified to ensure that all publications listed in GTR are assessed for inclusion in the DevPubMetric analysis on a monthly basis.
This change was required to ensure that data were synchronised as GTR reporting dates can be several months after publication and data capture by the Scopus system.
1 Feb 2021
Created archives of monthly extracts of recently captured or recorded awards and publications for GCRF and the Newton Fund
The ESPA Directorate, based at the University of Edinburgh provided strategic academic leadership for the programme. The Directorate also delivered a programme of monitoring evaluation and learning to measure both the academic and development impacts of the programme. One of the innovations introduced by the ESPA Directorate was to link information reported by projects and ESPA’s funders with bibliometric data to to develop a set of regular metrics to track both academic impact and the levels of engagement by developing country researchers and institutions in knowledge production and dissemination.
New Analysis: New Results
This information note presents updated analysis using a newly developed automated system DevPubMetric that derives a set of quantitative measures using publicly available data.
Highlights from this analysis include:
The academic outcomes have continued to increase after the programme closed. By October 2020, number of publications listed in the Scopus online bibliography had increased to 710 with nearly 27,000 citations. This represents a doubling of the number of publications and nearly three times the number of citations record when the programme closed in 2018.
The programme had significant success in moving towards open-access publications. In 2009 no publication was open access, but by 2017 nearly 70% of publications were open access. There has been a significant reduction since that time with 2020 data currently (October 2020) recording just over 40% of publications as full open access.
The programme promoted engagement with academics affiliated with developing country institutions. Throughout the programme up to one third of the papers had a first author from a developing country organisation and around 60% of papers had at least one author affiliated to a developing country organisation.
When all authors were examined, there was a steady increase in the proportion affiliated to low-income organisations (0% in 2019 to a maximum of 11% in 2019 and 2020).
The general trends showed that there was limited progress in enhanced empowerment or equitable contributions for developing county authors in a number of measures. Two are highlighted: – Throughout the programme over 40% of publications had now authorship from a developing country organisation. – Throughout the programmeover 60 % of authors listed on ESPA papers were affiliated to institutions located in high-income countries
This analysis will be extended in the future to compare with other similar research programmes address global development challenges. Please make suggestions for additional analysis in the comments section below.
Background
ESPA’s approach to recording academic impact was the first example of its kind for regular monitoring of a major UK research programme. The metrics highlighted in the programme’s highlights document gave an example with results from 2018 recording that the programme had produced 419 journal publications which in turn had been cited 8,516 times. These data had not been updated since the programme closed
It is widely recognised that there is a significant lag in the time to maximum academic impact in research programmes and projects. A major constraint to this, is that most monitoring of impact tends to end when projects or programmes close. This was the case for ESPA.
Since the closure of the programme in 2018 projects have continued to report academic outcomes to funders and this information is collated a a list of publications on the UK Government’s Gateway to Research portal. Data on publications also are collated through commercial abstracting services.
The growing availability of information in the public domain about research outcomes in the United Kingdom, combined with enhanced capabilities of commercially provided abstracting systems, together create the opportunity to redress this constraint.
In June 2020, a new system for bibliometric analysis of ESPA was launched with the results presented on this website (www.pvgglobal.uk/devpubmetric). The results presented below, show that since 2018 there has been continued significant growth in both the number of publications and citations of academic publications. The data also show trends in open access to publications and the levels of engagement of developing country institutions in ESPA’s research.
Number of publications (and levels of open access publications)
Figure 1: ESPA’s publications (number per year) and the proportion of these that were published in a full open access format.
The results show that the programme’s academic output peaked in its final year (2018) with a total of 100 publications. There was a reduction in the number of publications in 2019 and the 2020 data are still in progress. These data represent the total number of publications represent journal papers (and some books and book chapters) that have been captured on Elsevier’s Scopus system.
The data reporting levels of open access publication show a significant increase during the lifetime of the programme, which mirrored UK Government policy and investment by funders in supporting enhanced open access to research outcomes. ESPA achieved its highest proportion of open access publication in 2017 (68 %). The data show marked reductions in the level of open access publication in both 2019 and 2020 (since the programme closed in 2018). It is not currently clear why this has happened.
Total numbers of publications and citations
Figure 2: Cumulative total of ESPA’s publications and citations. (using papers captured by the Scopus system).
The results presented in Figure 2 highlight the extent to which academic impact (as measured by output and citations) continues to grow after the formal closure of a programme. In the two years after the programme closed, the number of recorded publications increased by over 70% (from 373 in 2018 to 714 in October 2020). Over the same period citations increased by nearly 200% (from 8516 in 2018 to 265462 in October 2020). Monthly monitoring is now being captured and updated results will be presented automatically on this page of the website.
Participation by developing country organisations and researchers
ESPA was designed from its inception to promote effective collaboration with developing country institutions and researchers. The need to engage partners from developing country institutions was captured in ESPA’s research and impact strategies. One of the indicators of this process developed by the ESPA Directorate was to record the affiliations of authors of ESPA publications.
This process has been renewed with the new approach adopted on this site which has analysed the institutional affiliations of all authors of ESPA papers that have been captured on the Elsevier Scopus system. The results of this analysis are shown below as Figure 3.
Figure 3. Developing country authorship on ESPA publications. The analysis records authors with an institutional affiliation in a developing country (low or middle income). Data marked as “any author” represent a paper where at least one author has an affiliation with an institution located in a developing country.
The results show that typically between 20 and 30% of ESPA’s publications in any year had first authors with an affiliation to an institution in a developing country (low or middle income). The broader analysis shows that in most years around 60% of all papers had some level of developing country academic input. The was measured as a paper that had at least one developing country author, and includes those papers with a developing country first author.
Whilst at first this seems a positive outcome, it does mean that around 40% of ESPA’s publications did not have evidence of academic input from developing country partners. It is also important to recognise that the measure used of “at least one developing country author” tends to mask the relative levels of academic contribution. Further research in now ongoing to explore these issues in more detail.
In order to explore this further a new indicator has been produced as part of the current analysis. This collates the affiliations of all authors on papers and then calculates the proportions of authors in each of current development income groups. The results shown as Figure 4 show that there was limited change in the overall equity of academic contribution to publications, with over 60% of authors being affiliated to institutions in high-income countries throughout the lifetime of the programme. A much more positive outcome was growth of authorship affiliated to low-income institutions growing from 0 in 2009 over ten years to 11% recorded in 2019 and 2020. A key question will be if this progress continued to be observed in the future, especially in newer and much larger programmes such as the UK’s Global Challenges Research Fund.
Figure 4. Changes in the proportions of the affiliations of all authors on ESPA publications currently recorded on Scopus.
Notes on methodology and data
The methodology used in this study differs in a number of ways from the early approach used by the ESPA Directorate and ESPA’s funders. This results in small changes in the number and types of publications recorded and a shift in process to measure engagement with developing country researchers through the country of institutional affiliations.
Data presented on this site are updated automatically on a monthly basis (day 1 of each month).
Data were captured and processed by the DevPubMetric system. The list of publications was generated from data reported by projects directly to UKRI through ResearchFish Publication data were captured from the UKRI Gateway to Research system. Open Government Licence
Additional records were generated using the Scopus database through structured searches for new documents by programme name and project reference.All records were then processed to collate full bibliographic data downloaded from Scopus.
This list of publications only contains documents that can be matched against source publications that are included in the Scopus database and the identification process requires researchers to have either reported the outcome to ResearchFish or to have included the programme name or project reference number in a publication's acknowledgement.
It is recognised that a small number of publications reported by researchers in this way and then captured by the DevPubMetric process may have limited levels of direct attribution to the programme.
This list of publications is generated through an automated process. It is possible that a few publications may be included which should not be attributed to this list and that some others may have been missed. Please send requests for corrections to admin@pvgglobal.uk
Open Government Licence