The Youth Empowerment Initiative is in its inception stage (2021). The Initiative has emerged from the recognition that Youth (15-24 years) need to be at the heart of future economic, social and environmental sustainable development. This is most important in countries that currently have large youth cohorts on their population demographic (most parts of Asia) or have significant growth projected over the remainder of the current century (Sub-Saharan Africa).
The Initiative begins with the premise that youth in developing and emerging economies need to be empowered to act as positive agents for change in their society and economy. The Initiative is focused on providing the skills and jobs to youth (to become job-ready), as well as creating the opportunity for young people to act as innovators and entrepreneurs (driving innovation, creating new jobs and projects).
During the inception phase of the Initiative there are two main activities:
Generating evidence. Data analysis and review of relevant literature
Designing the Response. Working with stakeholders in developing and emerging economies as well as with governments, the private sector, development agencies and civil society to design an approach that will Empower Youth to deliver Sustainable Development.
Information about current activities and projects is summarised below:
Reform of Higher Education, Research and Innovation in Kurdistan and Iraq.
Reform of Higher Education, Research and Innovation in Kurdistan and Iraq.
Supporting the Prime Minister’s Office of the Kurdistan Regional Government (KRG), building on previous activities supporting HE and research reform in Africa with the African Union and NEPAD and Rwanda See my keynote presentation on the topic at an 2018 OECD event on “Strengthening linkages between STI and ODA policies in order to address the grand challenges and promote inclusive growth”.
DevPubMetric. Capturing the Impact of INvestment in Development Research
DevPubMetric was designed and built during 2020 to collect data on the academic impacts of the UK’s Ecosystem Services for Poverty Alleviation Programme (ESPA). As ESPA’s first Director (2010-2016) I pioneered a “Big-Data” approach to build a set of progressive Key Performance Indicators for the programme which were used up to the closure of the programme in 2018. DevPubMetric was designed to update the process using publicly available sources of data and uses more sophisticated analysis benefiting for advances in data collection and the shift to support open access to data and publications.
The system is currently being applied to analyse ESPA+10 (ten years of progress) and to review outcomes from the first phases of the UK’s Newton Fund and the Global Challenges Research Fund (GCRF). Results from this work will be published on this website in the last quarter of 2020.
ESPA Results: Ten Years ON
The DevPubMetric system is being applied to update the information collected through the previous Monitoring Evaluation and Learning system used by ESPA. The initial results will be published in October 2020 to coincide with the 10th anniversary of ESPA’s first Science Conference.
Reviewing Outcomes from the Newton Fund and Global Challenges Research Fund
The DevPubMetric system is being applied to review the academic outcomes and levels of engagement with developing country researchers from the UK’s Newton Fund (2014-) and Global Challenges Research Fund (2016-). This work in ongoing and will feed into discussions about potential next phases of both programmes currently being considered by the UK Government. As part of this research the DevPubMetric system is being extended to include the ability to map academic publications against the UN’s Sustainable Development Goals, using an adapted version of the SDG search terms published by Elsevier and used as part of the Times Higher Education Supplement’s Impact Rankings.
Innovation Systems: Higher Education, Science, Technology and Innovation
Innovation Systems: Higher Education, Science, Technology and Innovation
Work is ongoing to update the provocation paper used at the GCRF-funded Narobi meeting in 2017 that then contributed to material for the keynote at papers presented the OCED event in 2018. It is intended that this work will feed into a wider discussion during 2021 including an event on Innovation Systems as part of the University of Nairobi’s Innovation Week in March 2021, 4 years after the original worksop in Nairobi.
The objective for this work is to set out a new type of relationship between higher education, governments and business to promote innovation and enterprise with a strong link to economic growth and creating jobs for the rapidly growing numbers of young people in the emerging economies in sub-Saharan Africa and South Asia.
Plastic Waste in Africa
Collaborative research with the University of Malawi and University of Nairobi on the use and fate of plastics in the environment. More Detail:
DevPubMetricprovides an objective and quantitative method to measure and compare the academic impact of development or global challenge research, whilst also estimating the levels of engagement of developing country researchers and research institutions.
In the September 2000, the United Nations (UN) adopted the Millennium Declaration and in the following year the Millennium Development Goals (MDGs). The MDG’s set out an ambitious agenda for global development and poverty reduction up to 2015 when the current set of Sustainable Development Goals (SDGs) were adopted by the UN.
The resulting focus on sustainable development and addressing the world’s shared global challenges has led to very significant growth in research that addresses these challenges.
A number of major research initiatives addressing global challenges and sustainable development have been launched by governments, philanthropic and multilateral organisations along with a number of major charities.
Most of these new programmes have shared objectives that include:
Contributing to global knowledge (publications);
Driving both academic and development impact;
Engaging with and empowering developing country researchers and research organisations.
There are well established methodologies for measuring and documenting development impact, though in many cases these can be challenged by the complex non-linear pathways that link research and impact, the time-lag to build impact and the question of how to evidence attribution.
There are many fewer examples of systematic processes to measure the academic impact of development research, and virtually none that map the global engagement and partnerships that deliver the research and its impact.
In 2013 the ESPA Directorate developed a quantitative monitoring framework for the programme, that included measures of both academic impact (numbers of publications and citations) and engagement with developing country researchers (proxy indicators based on the proportion of academic publications with a developing country first author and the proportion of with at least one developing country author. These data formed part of ESPA’s monitoring and evaluation leading to standard reporting against its Theory of Change and Logical Framework. An example of regular reporting that used these data is presented below
Publication metrics presented as part of monthly reporting produced by the ESPA Directorate.
Monthly reporting was collated by the ESPA Directorate up to the point that the programme closed in July 2018. ESPA’s final highlight document included the last set of data collated by the programme at which time the programme had captured information about 419 journal articles which had been cited 8,516 times.
The original system developed by the ESPA Directorate utilised data and systems that were available in 2013-15 with a high level of reliance on internal data and extensive manual processing that required every paper to be assessed by a team member. This meant that data collection and reporting could not continue automatically once the programme had closed. It also meant that ESPA’s approach could not be extended to be used by much larger research programmes that followed including the UK’s Newton Fund and Global Challenges Research Fund.
Defining the challenge: Capturing the opportunities of “open” big data.
During 2020, work commenced to implement a new system, DevPubMetric that could provide an update to the data previously collected by ESPA in a way that was essentially automatic and could easily be extended to other research programmes and research funders. The motivation for this work was to update the statistics from ESPA to capture and quantify the very significant academic impact of research after projects and programmes closed. It is often said that up to half of a programme’s academic impact may be invisible after funding ceases.
Whilst this work was originally intended to update data for ESPA, it was realised that the same approach could be used for other programmes, providing the opportunity to do quantitative comparative analysis between and within programmes.
In order to deliver these objectives DevPubMetric was designed to address a set of challenges an opportunities as detailed below:
Challenges and Opportunities
Responses
Automate data collection and analysis.
This has become possible through the evolution of data products and systems. • Extensive access to downloadable data through web-based APIs ( Application Programming Interface) • Enhanced data availability for UK funded research through the Gateway to Research (GTR) • Enhanced data content available from commercial bibliometric databases (including open access metrics and information on authors and their institutional affiliations.
Remove the requirement for manual processing.
The development and application of a standardised system for data processing and analysis has removed the requirement for routine manual interventions. It has not been possible to remove this requirement entirely because of some issues relating to data compatibility, for example in the names of some countries and institutions in different systems.
Where possible these issues linked to integrating data from different sources have been addressed through the application of look-up tables. These tables need to be checked and updated at intervals.
Develop new approaches to measure the levels of open access publication and developing country engagement.
The original ESPA system linked measures of developing country engagement to data stored about funded researchers which are not available in the public domain and could not be applied to other research programmes.
For this reason the new system of analysis tracks engagement through the institutional links of authors, which are data readily available from the current generation of bibliometric analysis including the Elsevier Scopus system that is used in the new analysis.
Enhance the range of publications used for analysis
The system developed for ESPA used data downloaded from the Web of Science bibliometric system delivered by Clarivate Analytics.
The updated system has moved to use data provided by Scopus provided by Elsevier. Both systems are constantly evolving and whilst they have many common functions, there are aspects where each can be considered to have an advantage.
One of the perceived strengths of Scopus is its broader coverage of current academic journals, especially for social sciences, arts and humanities. This is the main reason that the choice of data provider has changed for the current system of analysis.
Both Scopus and Web of Science have added additional information in the period since ESPA’s original system was designed. Of most relevance is the ability to identify open access publications and the provision of detailed information about the institutional affiliations of all authors of a publication
Enable objective comparison with other programmes
The design of the new data collection and analysis system utilises a set of standard data that should be readily available for most major research programmes. This required the ability to link publications to programmes using unique identifiers of either by programme name or project reference codes. These in turn can be used as search terms in biometric systems.
In addition, if the programme publishes a downloadable record of publications these can also be used to supplement analysis in case authors have not acknowledged links to funding programmes.
DevPubMetric utilises data from a variety of open access data sources and then combines it with data extracted from the commercially provided Scopus bibliometric system. The results provided here are restricted to those available from open access sources, or the synthesis of results obtained from Scopus along with information that Scopus lists in the public domain.
The system then generates lists of projects associated with key research programmes and full lists of publications that have been captured through the core search protocols. These data have been collated in the data repository for the system: https://pvgglobal.uk/devpubmetric/repository. These data extracts are updated on the first day of each month with data generated from the system after new publications and projects have been extracted.
Canadian International Development Research Centre (IDRC/CRDI): idrc.ca
One of the search criteria used in analysis is to extract abstracts from publications that acknowledge the programme. This information needs to represent a unique identifier and for this reason acronyms have been found to be unreliable
SCOPUS Search terms have been developed for each programme to guide the capture of publications from each programme to balance the capture of relevance publications against rejection of those that should not be attributed at programme level.
Some programmes, such as the Newton Fund have multiple names in difference contexts. The search process has been designed to cover all known variants of the programme name (currently restricted to English language versions.)
Publications are downloaded from Scopus using an abstract search based on unique project identifiers (where available) or by using queries of funder information using a set of uniquely defined programme names.
All projects funded by UKRI are required to report outcomes including publications through the ResearchFish reporting system. These data are then checked by UKRI systems before being replicated on their Gateway to Research System. These data are in the public domain and can be downloaded using using unique identifiers for a programme or project. One constraint with this approach is that it relies on accurate and timely reporting by projects.
In order to partially address the constraint of under-reporting, Scopus is used to capture additional publications, by searching for publications using unique programme and projects identifiers where these are available.
Scopus was used to provide the descriptive information about publications for further analysis. This included a full list of authors, their institutional affiliation and the countries that those institutions were operating from. Scopus also provided data on which publications were open access.The disadvantage of reliance on Scopus is that some lesser-know journals are not covered (including many published in developing countries) and that the system does not capture open-access publications provided through institutional repositories.
An alternative approach utilising data that are collected through the ResearchFish / Gateway to Research portals was found to have insufficient detail to meet the needs of the analysis. In some types of publication, lists of authors were incomplete and institutional detail did not have the level of information required to link back to map engagement with developing countries.
Publication metadata: Corresponding Authors and their affiliation.
The first phase of analysis using DevPubMetric used authorship as a proxy measure of the levels of engagement of developing country researchers.
It was noted that first authorship is not always the most appropriate measure of scientific leadership in publications and that protocols for deciding who acts as first author varies between disciplines.
The Scopus system stores information about the corresponding author for most publications. These data have collected from May 2021 using the Scopus “Abstract” API (Application Interface)
Publication metadata: Funding organisations and project references.
The Scopus system uses pattern matching and AI techniques to extract information extracted from publications through their acknowledgements of funding.
Data describing funders and project reference codes has been collected from May 2021 using the Scopus “Abstract” API (Application Interface). Earlier versions of the DevPubMetric system collated funding data from the Scopus “Search” API, which was found to be incomplete during the quality assurance process of the data extract.
Development status of institutional affiliations for authors.
Current data on World Development Indicators, including the income status of all nations recognised by the United Nations has been downloaded for integration into the analysis. The analysis is based on the institutional affiliation of each author as recorded by the publication. In some cases authors will list more than one institutional affiliation. In this case, the author was linked to the country with the lowest income status.(High income > Middle income > Low Income)
Geographic and SDG groups and additional population data
Integrating data: Generic approaches, challenges, complexity and solutions
DevPubMetric utilises data provided from up to four distinct data sources. These data need to be integrated before the analysis can be completed. There are three stages to this process:
Most of the data sources used in DevPubMetric have pre-existing Unique Identifiers (UIDs) for core data. These have been retained for the analysis and where necessary look-up tables have been created to link data sources. These are outlined below:
Data type
Unique Identifier(s)
Programme Names
As published by funders.
The SCOPUS search terms needed to be adjusted to maximise the probability that publications were captured whilst also minimising the likelihood of false positives.
Derived from Gateway to Research (GTR) ∙ Published UKRI project reference ∙ GTR Project UID.
From May 2021, DevPubMetric has also extracted project references via Scopus which are derived from the acknowledgement sections of most publications.
Publications
· Scopus Electronic Identifier (eid) · Gateway to Research UID · Digital Object Identifiers (DOI) included in bibliographic information · The ESPA programme produced a downloadable database of publications produced up to July 2018.
DOIs are used to create a look-up table linking systems
Gateway to Research data does not currently have comprehensive data for non-UK institutions. It is currently not possible to cross-link these data sources
Countriesand Income Status
The World Bank’s World Development Indicators include a comprehensive list of countries recognised by the United Nations. This list includes the current income-status for each territory.
The Scopus system does not publish a list of countries, but this can be extracted from data downloaded from Scopus. These data include a small number of territories not currently recognised by the United Nations or with names that do not directly map onto those used by the United Nations.
Scopus These data are stored for every publication at monthly intervals, permitting tracking of the time course of academic impact for individual publications, projects and programmes
Each of the systems publishing data used in DevPubMetric has implemented a form of internal quality assurance with details published along with the data. the development of the analysis system has identified a number of areas where existing quality assurance currently acts as a constraint.
Publication Coverage (Scopus)
The Scopus system has comprehensive coverage of academic publications, but this does not include all publications likely to be used for reporting results from development or global challenge research. Known gaps include:
Lower impact publications, including many national or regional publications originating from developing countries;
Relatively low coverage of books, book chapters and monographs;
Lower coverage of publications in languages other than English.
Poor coverage of pre-prints.
Whilst these issues are a constraint, there is currently no better alternatives that can provide the systematic analysis used in the current system. As with all commercial bibliometric systems, Scopus (and possible alternatives) are under constant development and some the current constraints may be mitigated in future releases.
The advantages of using a comprehensive commercial bibliometric system is that there are very high levels of internal quality assurance and internal consistency. One area key importance is the data data model which has four core components: Publications,
Reporting Gaps and Over-reporting (Gateway To Research & Scopus)
DevPubMetric makes full use of UKRI project reporting from the Gateway to Research System, which in turn relies on reporting by projects and researchers. A subset of publications reported through ResearchFish were tested as an additional quality assurance during the development of the current system. The following issues emerged from the QA process:
Reporting Gaps: This is a result of researchers failing to (or being unable to ) report publications linked to a project or programme. These gaps are identified when results are compared with those resulting to a direct Scopus search by programme name or project identifiers.
Over-reporting: There are also examples of potential over-reporting, which represent publications that should realistically not be considered to be attributable to the programme or project that they are linked to. This can be seen in Gateway to Research records, but also can apply to records derived from Scopus if the authors have provided the relevant programme name or project codes.
Issues relating to under and over-reporting are largely linked to behaviour of researchers and authors. Research funders provide guidance and encouragement to facilitate the adoption of good practice, but it is clear that both issues remain to some extent. One possible application of the analytical system would be to use a random subset of publications within a programme to provide an assessment of the scale of the issue
Lack of Institutional Affiliation in some Publications
The analysis and reporting in DevPubMetric uses institutional affiliations of authors to link publications with countries. There are a small number of publications where the publisher does not include this information and hence it is not collected by abstracting services and bibliometric analysis.
Analysis of the scope of this issue is ongoing. Initial results suggests that it is most prevalent in papers with large numbers of authors in specific academic disciplines.
Institutions and Countries Linked to Research (Gateway to Research)
Data extracted from Gateway to Research were examined to evaluate how these could be used to enable additional analysis of the levels of engagement with researchers and institutions in developing countries. This was not possible for two reasons:
Gateway to Research is currently not recording links between researchers and institutions listed on a grant, with the exception of the one Principal Investigator listed on the grant application. (Data are available of the current institution linked to individuals, but this is know to change with time and may not relate to a grant at any time.
Gateway to Research records details of the main institutions linked to the grant, including those outside the UK. Details recorded for non-UK institutions are inconsistent, with the majority of records not listing which country those institutions are located in. There are also instances of duplication of institutional records which means that there may not be a unique institutional identifier for non-UK institutions. This issue also applies to some non-traditional UK institutions.
Current Applications for DevPubMetric
Current applications of the DecPubMetric system are provided on the website at pvgglobal.uk/devpubmetric. More detailed analysis of results are provided on the following topis:
Application of DevPubMetric to other development research programmes, nows including FCDO/DFID, USIAD and IDRC.
SDG mapping of research publications captured by the system
Analysis of academic outputs from the Nairobi Alliance (A research alliance joining the Universities of Nairobi, Rwanda, Malawi, Witwatersrand and Leicester)
Potential future developments and applications for DevPubMetric
The data collected for DevPubMetric analysis can be extended to build additional applications. The following potential applications are being consider for the next phase of applications:
Institutional performance for development research, either at programme or full institutional level.
Gender analysis of publications.
Linking the analysis at programme or institutional level with emerging processes to link academic outputs to relevance to the Sustainable Development Goals.
Change/update log
Date
Change
Comments
10 October 2021
Added links to full publication lists extracted by DevPubMetric for the ESPA, GCRF and Newton Fund programmes
Updated data derived from the World Bank’s World Development Indicators. This update includes changes in the income status of 10 countries.
Two additional fields of data were added to country summaries (pvgglobal.uk/country-list/): ● National Gini coefficient, a measure of inequity; ● Proportion (%) of the youth population (15-24) not in employment, education or training (NEET).
8 June 2021
Published information about a wider range of development research programmes
Published data on: FCDO/DFID Research (UK) USAID (United States) IDRC (Canada).
Comparison of DevPubMetric KPIs for all programmes currently captured by the system.
18 May 2021
Identification of publications that contribute to Covid-19 research
The methodology used to generate SDG mapping was extended to identify publications relevant to Covid-19. These data, where available, are currently included in the diagrammatic presentations of SDG mapping. pvgglobal.uk/activity/sdg-mapping/
18 May 2021
SDG Analysis provided in DevPubMetric.
SDG analysis and mapping has been including by linking publication lists generated by DevPubMetric with the 2021 SDG Mapping Protocol provided in SciVal.
Publication of a full list of projects that have been captured by the DevPubMetric system for the GCRF and Newton programmes derived from the UKRI Gateway To Research system.
Data are provided as a CSV file that can be read directly by most spreadsheet and database systems
Enhanced data capture of publications derived from the UKRI Gateway To Research (GTR) database
Data capture methodology modified to ensure that all publications listed in GTR are assessed for inclusion in the DevPubMetric analysis on a monthly basis.
This change was required to ensure that data were synchronised as GTR reporting dates can be several months after publication and data capture by the Scopus system.
1 Feb 2021
Created archives of monthly extracts of recently captured or recorded awards and publications for GCRF and the Newton Fund
