The ESPA programme was a global environmental research programme which was active from 2009 through to 2018, funded by the Government of the United Kingdom through the Department for International Development, Natural Environment Research Council and Economic and Social Research Council.
The ESPA Directorate, based at the University of Edinburgh provided strategic academic leadership for the programme. The Directorate also delivered a programme of monitoring evaluation and learning to measure both the academic and development impacts of the programme. One of the innovations introduced by the ESPA Directorate was to link information reported by projects and ESPA’s funders with bibliometric data to to develop a set of regular metrics to track both academic impact and the levels of engagement by developing country researchers and institutions in knowledge production and dissemination.
New Analysis: New Results
This information note presents updated analysis using a newly developed automated system DevPubMetric that derives a set of quantitative measures using publicly available data.
Highlights from this analysis include:
- The academic outcomes have continued to increase after the programme closed. By October 2020, number of publications listed in the Scopus online bibliography had increased to 710 with nearly 27,000 citations. This represents a doubling of the number of publications and nearly three times the number of citations record when the programme closed in 2018.
- The programme had significant success in moving towards open-access publications. In 2009 no publication was open access, but by 2017 nearly 70% of publications were open access. There has been a significant reduction since that time with 2020 data currently (October 2020) recording just over 40% of publications as full open access.
- The programme promoted engagement with academics affiliated with developing country institutions. Throughout the programme up to one third of the papers had a first author from a developing country organisation and around 60% of papers had at least one author affiliated to a developing country organisation.
- When all authors were examined, there was a steady increase in the proportion affiliated to low-income organisations (0% in 2019 to a maximum of 11% in 2019 and 2020).
- The general trends showed that there was limited progress in enhanced empowerment or equitable contributions for developing county authors in a number of measures. Two are highlighted:
– Throughout the programme over 40% of publications had now authorship from a developing country organisation.
– Throughout the programme over 60 % of authors listed on ESPA papers were affiliated to institutions located in high-income countries
This analysis will be extended in the future to compare with other similar research programmes address global development challenges. Please make suggestions for additional analysis in the comments section below.
Background
ESPA’s approach to recording academic impact was the first example of its kind for regular monitoring of a major UK research programme. The metrics highlighted in the programme’s highlights document gave an example with results from 2018 recording that the programme had produced 419 journal publications which in turn had been cited 8,516 times. These data had not been updated since the programme closed
It is widely recognised that there is a significant lag in the time to maximum academic impact in research programmes and projects. A major constraint to this, is that most monitoring of impact tends to end when projects or programmes close. This was the case for ESPA.
Since the closure of the programme in 2018 projects have continued to report academic outcomes to funders and this information is collated a a list of publications on the UK Government’s Gateway to Research portal. Data on publications also are collated through commercial abstracting services.
The growing availability of information in the public domain about research outcomes in the United Kingdom, combined with enhanced capabilities of commercially provided abstracting systems, together create the opportunity to redress this constraint.
In June 2020, a new system for bibliometric analysis of ESPA was launched with the results presented on this website (www.pvgglobal.uk/devpubmetric). The results presented below, show that since 2018 there has been continued significant growth in both the number of publications and citations of academic publications. The data also show trends in open access to publications and the levels of engagement of developing country institutions in ESPA’s research.
Number of publications (and levels of open access publications)
Figure 1: ESPA’s publications (number per year) and the proportion of these that were published in a full open access format.
The results show that the programme’s academic output peaked in its final year (2018) with a total of 100 publications. There was a reduction in the number of publications in 2019 and the 2020 data are still in progress. These data represent the total number of publications represent journal papers (and some books and book chapters) that have been captured on Elsevier’s Scopus system.
The data reporting levels of open access publication show a significant increase during the lifetime of the programme, which mirrored UK Government policy and investment by funders in supporting enhanced open access to research outcomes. ESPA achieved its highest proportion of open access publication in 2017 (68 %). The data show marked reductions in the level of open access publication in both 2019 and 2020 (since the programme closed in 2018). It is not currently clear why this has happened.
Total numbers of publications and citations
Figure 2: Cumulative total of ESPA’s publications and citations. (using papers captured by the Scopus system).
The results presented in Figure 2 highlight the extent to which academic impact (as measured by output and citations) continues to grow after the formal closure of a programme. In the two years after the programme closed, the number of recorded publications increased by over 70% (from 373 in 2018 to 714 in October 2020). Over the same period citations increased by nearly 200% (from 8516 in 2018 to 265462 in October 2020). Monthly monitoring is now being captured and updated results will be presented automatically on this page of the website.
Participation by developing country organisations and researchers
ESPA was designed from its inception to promote effective collaboration with developing country institutions and researchers. The need to engage partners from developing country institutions was captured in ESPA’s research and impact strategies. One of the indicators of this process developed by the ESPA Directorate was to record the affiliations of authors of ESPA publications.
This process has been renewed with the new approach adopted on this site which has analysed the institutional affiliations of all authors of ESPA papers that have been captured on the Elsevier Scopus system. The results of this analysis are shown below as Figure 3.
Figure 3. Developing country authorship on ESPA publications.
The analysis records authors with an institutional affiliation in a developing country (low or middle income). Data marked as “any author” represent a paper where at least one author has an affiliation with an institution located in a developing country.
The results show that typically between 20 and 30% of ESPA’s publications in any year had first authors with an affiliation to an institution in a developing country (low or middle income). The broader analysis shows that in most years around 60% of all papers had some level of developing country academic input. The was measured as a paper that had at least one developing country author, and includes those papers with a developing country first author.
Whilst at first this seems a positive outcome, it does mean that around 40% of ESPA’s publications did not have evidence of academic input from developing country partners. It is also important to recognise that the measure used of “at least one developing country author” tends to mask the relative levels of academic contribution. Further research in now ongoing to explore these issues in more detail.
In order to explore this further a new indicator has been produced as part of the current analysis. This collates the affiliations of all authors on papers and then calculates the proportions of authors in each of current development income groups. The results shown as Figure 4 show that there was limited change in the overall equity of academic contribution to publications, with over 60% of authors being affiliated to institutions in high-income countries throughout the lifetime of the programme. A much more positive outcome was growth of authorship affiliated to low-income institutions growing from 0 in 2009 over ten years to 11% recorded in 2019 and 2020. A key question will be if this progress continued to be observed in the future, especially in newer and much larger programmes such as the UK’s Global Challenges Research Fund.
Figure 4. Changes in the proportions of the affiliations of all authors on ESPA publications currently recorded on Scopus.
Notes on methodology and data
- The methodology used in this study differs in a number of ways from the early approach used by the ESPA Directorate and ESPA’s funders. This results in small changes in the number and types of publications recorded and a shift in process to measure engagement with developing country researchers through the country of institutional affiliations.
- Data presented on this site are updated automatically on a monthly basis (day 1 of each month).
Data were captured and processed by the DevPubMetric system. The list of publications was generated from data reported by projects directly to UKRI through ResearchFish
Publication data were captured from the UKRI Gateway to Research system.
Open Government Licence
Additional records were generated using the Scopus database through structured searches for new documents by programme name and project reference.All records were then processed to collate full bibliographic data downloaded from Scopus.
This list of publications only contains documents that can be matched against source publications that are included in the Scopus database and the identification process requires researchers to have either reported the outcome to ResearchFish or to have included the programme name or project reference number in a publication's acknowledgement.
It is recognised that a small number of publications reported by researchers in this way and then captured by the DevPubMetric process may have limited levels of direct attribution to the programme.
This list of publications is generated through an automated process. It is possible that a few publications may be included which should not be attributed to this list and that some others may have been missed. Please send requests for corrections to admin@pvgglobal.uk