Health Emergency Research Preparedness: An Analysis of National Pre‑COVID Research Activity and COVID Research Output

Introduction

Research capacity is a critical element of health emergency preparedness. Following the large Ebola outbreak in West Africa in the mid‑2010s, the World Bank issued the Money and Microbes report, which lamented the “tragic delays” in medical countermeasure development owing to the lack of research capacity in the most‑affected countries and called for sustained efforts to strengthen clinical research capacity in low‑ and middle‑income countries (LMICs) [1]. The Coalition for Epidemic Preparedness Innovations (CEPI), which co‑sponsored the report, subsequently articulated a “100‑day mission” with a global goal of being able to develop “safe, effective, globally accessible vaccines” to new infectious disease threats in as little as 100 days [2].

Since then, the Global Preparedness Monitoring Board, convened by the World Health Organization (WHO) and World Bank, has developed indicators for “capacity building to develop research and development capacities at the regional and national levels” as well as indicators for national assessments for “adequate research capacity deployable at the start of a health emergency” [3]. For example, one indicator asks: “Globally, what is the state of national R&D innovation, development and access to medical countermeasures?” Components of this assessment include: “Do countries globally have adequate research capacity deployable at the start of a health emergency, including genetic sequencing capacity?” and “Do countries globally have independent national ethics review committees with rapid review procedures for research on health emergencies?” However, these have not been fully implemented or comprehensively reported. In addition, the Global Health Security Agenda established a Research & Development Task Force to create metrics to assess national health research capacity [4]. Potential metrics, which could be included in Joint External Evaluations, have been pilot tested in select countries. In 2022, WHO member states adopted resolution WHA75.8 on strengthening clinical trials, and in 2023, they convened the WHO Global Clinical Trials Forum with key stakeholders to advance that goal [5].

In the context of these nascent efforts, the COVID‑19 pandemic provided a unique opportunity to make a global assessment of national‑level research preparedness, productivity, and response. All countries were heavily affected and at least had sufficient numbers and rates of COVID‑19 cases to conduct basic, clinical, and public health research.

We assessed pre‑pandemic research output as a potential predictor of COVID‑19 research productivity. We examined: (1) the feasibility of using easily available data on national research activity as potential indicators of national research capacity, (2) the feasibility of using of easily available data on national disease‑specific research outputs as potential indicators of national pandemic research response, and (3) the extent to which national pre‑pandemic research capacity can predict national pandemic research response in comparison with other national economic and demographic characteristics.

Methods

We created a metric of each country’s national pre‑pandemic research activity with two elements. One was the percentile ranking of the average of each country’s annual number of health science publications in the Scopus citation database [6] for 2018 and 2019. Publications in Scopus are counted by the country of the authors’ affiliations, and one publication may be counted in more than one country. The second element was the percentile ranking of the average of each country’s annual number of clinical trials in the WHO International Clinical Trials Reporting Platform (ICTRP) [7] for 2018 and 2019. Clinical trials in the ICTRP are counted by the location of participant enrollment sites, and, similarly, one clinical trial may be counted in more than one country. We averaged the two percentile rankings to create an aggregate measure for pre‑COVID‑19 research activity. We limited all analyses to countries with population greater than 100,000 (N = 180) at the time of the analysis.

We created a similar metric of national COVID‑19 research output. We calculated the percentile ranking of the average of each country’s annual number of COVID‑19‑related publications in the Scopus citation database for 2020 and 2021. The search terms were created using synonyms from the COVID‑19 MeSH terms of the National Library of Medicine [8] (Appendix 1). Then we calculated the percentile ranking of the average of each country’s annual number of COVID‑19‑related clinical trials in the ICTRP for 2020 and 2021. A listing of COVID‑19‑related clinical trials was downloaded from the ICTRP website. Then we averaged the two percentile rankings to create an aggregate measure for COVID‑19 research output.

We conducted descriptive univariate analyses of these data and analyzed Pearson’s correlation coefficients to assess the relatedness of the two elements of each of the two metrics. We calculated the R‑squared coefficients of determination and Kendall’s tau rank correlation coefficients of the two metrics—pre‑COVID‑19 research activity and COVID‑19‑related research output. We also analyzed these correlation statistics of COVID‑19 research output with 2022 national population and gross domestic product (GDP) [9], Human Development Index (HDI) in 2022 [10], and national COVID‑19 cumulative case number and rate per capita through December 31, 2021 [11]. We also analyzed these data by the World Bank Income Group [5] and by WHO region [3]. GDP value was missing for two countries, which were excluded from analyses of that variable.

Lastly, we conducted a multivariable linear regression analysis to evaluate predictors of COVID‑19‑related research output during 2020–2021. The predictors examined were pre‑COVID‑19 research activity (2018–19), HDI, COVID‑19 case number, GDP, and population. We computed regression coefficients and assessed their direction and magnitude. The model’s goodness‑of‑fit was evaluated using R‑squared and adjusted R‑squared values; statistical significance was determined at a threshold of p < 0.05.

Results

In the two years before the COVID‑19 pandemic (2018–19) in the 180 countries with population greater than 100,000, the median of the average annual number of publications was 415, and the median of the average number of clinical trials was 21. These figures were higher in high‑income countries and varied by WHO region, with the highest in Europe and the lowest in Africa and the Western Pacific (Table 1). In the first two years of the COVID‑19 pandemic (2020–21), the median of the average annual number of COVID‑19‑related publications in these countries was 85 and the median of the average number of COVID‑19‑related clinical trials was 1.5 (Table 1). These figures were also higher in high‑income countries and varied by WHO region with the highest in Europe and lowest in Africa and the Western Pacific.

There was a strong positive relationship between the national numbers of pre‑COVID publications and clinical trials, with a Pearson’s correlation coefficient of 0.927 (Figure S1). These correlations were moderately to strongly positive across all World Bank income groups and strongly to very strongly positive across all WHO regions (data not shown). Similarly, there was a strong positive relationship between the national numbers of COVID‑19‑related publications and clinical trials, with a Pearson’s correlation coefficient of 0.864 (Figure S2). Also, similarly, these correlations were moderately to strongly positive across all World Bank income groups and moderately to very strongly positive across all WHO regions (data not shown).

As noted in the Methods section, for the main analyses, the raw numbers of publications and clinical trials were converted into rank percentiles and aggregated into aggregate measures, one each for pre‑COVID‑19 research activity and for COVID‑19‑related research output, for each country. Maps of these aggregate measures show the higher research activity and output generally in North America, Western Europe, and East Asia (Figures 1A and 1B). The full data set is available in a Supplemental Data file.

Figure 1A

Figure 1B

The aggregate measure of pre‑COVID‑19 research activity was very strongly positively correlated with the COVID‑19‑related research output, with an R‑squared value of 0.889 and a Kendall’s tau of 0.811 (Figure 2). The other country characteristics we explored (HDI, COVID‑19 case number, COVID‑19 case rate, GDP, and population) were not as strongly correlated with COVID‑19‑related research output (Table 2 and Supplemental Figures S3–S7).

Figure 2

In the multivariable linear regression analysis, taken together, pre‑COVID‑19 research activity (2018‑19), HDI, COVID‑19 case number, GDP, and population accounted for 90.2% of the variance in COVID‑19‑related research output (2020–21). Pre‑COVID‑19 research activity was the only factor with a substantial or statistically significant contribution to explaining variations in COVID‑19‑related research output. Other predictors contributed very little and were not statistically significant (Table 3). Paradoxically, HDI and GDP showed negative trends when accounting for other factors in the model, but these were weak or negligible and not statistically significant.

Discussion

We used easily available data on national numbers of health science publications and clinical trials as metrics of national research activity, and national numbers of COVID‑19‑related publications and clinical trials as metrics of national pandemic research response. There was a wide range of values, with higher numbers in higher income countries and certain WHO world regions. The national aggregate measure of pre‑pandemic health research activity was very strongly correlated with the national aggregate measure of pandemic research response, much more so than countries’ HDI, burden of COVID‑19 (case number or rate), GDP, or population. These findings strongly support the global efforts to strengthen research capacity as a critical element of preparedness for health emergencies. Notably, in the pandemic, scientists in diverse areas of health research pivoted to conduct critical COVID‑19 studies as part of the emergency response, suggesting that the critical, common elements of clinical research across a range of disease areas can be brought to bear in a health emergency [12].

COVID‑19‑related research outputs were correlated with economic and demographic characteristics such as HDI, GDP, and population, but these were much weaker correlations compared with pre‑pandemic research activity. Future research could examine outliers in the scatterplots as in‑depth case studies of countries that “punch above their weight” with more COVID‑19 research output than expected based on their economic and demographic characteristics.

COVID‑19‑related research outputs were not strongly correlated with either COVID‑19 case numbers (which are confounded with population size) or COVID‑19 case rates. Overall, more COVID‑19 did not necessarily drive more COVID‑19 research; nor, inversely, did more COVID‑19 research seem to correlate with a lower burden of disease.

This previous point raises an important limitation to this analysis. Many clinical trials and publications did not contribute useful evidence that was able to be linked to effective public health or clinical interventions. During the pandemic, it was noted that only about 5% of clinical trials registered in the ICTRP could have been described as randomized and adequately powered [13]. The number of publications and the number of clinical trials are at best crude indicators of research activity and research response to health emergencies. Much more detailed country‑level information is needed, such as the metrics being used by the Global Preparedness Monitoring Board [3] and the Global Health Security Agenda [4], including information on research clinics, laboratory testing, biorepositories, data management, biostatistics, research ethics review, and research regulatory capacity. This information is not readily available for most countries, but it can be inferred from the number of publications and clinical trials for global analyses such as this.

Data on these research outputs may be useful for funders of research capacity strengthening to target resources for countries with lower levels of research capacity [14]. For example, special funding could be made available for north–south or south–south collaborations to build capacity in countries in the lowest quartile or lowest half of these metrics of research activity. These indicators may also be helpful to assess global, regional, and national‑level progress in research capacity strengthening as the number of publications and clinical trials in countries with lower activity increases over time.

We found that pre‑COVID numbers of publications and clinical trials were strongly correlated, as were numbers of COVID‑19‑related publications and clinical trials. This is as expected and validates our combining each of these two elements to create aggregate indicators of pre‑COVID research activity and COVID‑19‑related research output, respectively. The raw data of the number of publications and clinical trials were highly skewed, with very high values for a small number of countries. We therefore used medians rather than means to compare groups in Table 1 and log scales to visualize the data in scatterplots in Figures S1 and S2. We then used percentile rankings in the aggregate measures and the Kendall’s tau statistic as a correlation measure that indicates the strength of the relationship between the two rankings.

The COVID‑19 pandemic provided the opportunity to conduct this analysis, with sufficient case rates and case numbers in almost every country to be able to conduct research and participate in clinical trials. However, in many disease outbreaks, there may be few or no cases in countries with highly developed research capacity, as in the West Africa Ebola outbreak, when it took several months to initiate vaccine clinical trials even after the pathogen and vaccine candidates were identified. As we are unable to predict where the next outbreak may occur, it is important to strengthen clinical research capacity in all countries to ensure national, regional, and global preparedness.

In conclusion, we found that it was a relatively straightforward, reproducible, and feasible process to examine national‑level health science publications and clinical trials, pre‑COVID‑19 and COVID‑19 related, as potential metrics of research capacity and health emergency research response. National COVID‑19‑related research response was very strongly correlated with pre‑COVID‑19 research activity, much more than with other national economic or demographic characteristics. These findings indicate that research capacity strengthening is an important element of preparedness for health emergencies. Efforts to increase capacity, especially in countries with lower levels of research activity, should continue as a global public health priority for the health security of all nations.

Competing Interests

The authors have no competing interests to declare.

Funding

This work was conducted by the Fogarty International Center with no specific funding source.

Data Accessibility

The complete dataset has been made available as a supplemental file.

Disclaimer

The findings and conclusions in this paper are those of the authors and do not necessarily represent the views of their institutions.

Additional Files

The additional files for this article can be found as follows: