Enhancing Health Policy Administration in LMICs: Dr. LJW Fellowship Program Insights (2021–2023)

Program outline

The Health Policy Administrator course within the Dr. LJW Fellowship Program targets key health policymakers and administrators working in central and regional health organizations in LMICs. This course aims to enhance the effectiveness of fellowship programs and expand future collaboration opportunities, ultimately improving residents’ health in partner countries.

The selection of training institutions for the Dr. LJW Fellowship Program is conducted through a structured, merit‑based, and competitive process designed to ensure both educational excellence and long‑term programmatic sustainability. Institutions are assessed and selected based on their demonstrated expertise and operational capacity within their respective fields of specialization, utilizing Korea’s national competitive procurement platform. Notably, KOFIH implements this process through the Korean government’s procurement system, adhering to principles of transparency, fairness, and efficiency. Contracts are awarded for a period of three to four years, providing institutional continuity while upholding high standards of accountability and performance. The detailed process for selecting host institutions and fellows is presented in Supplementary File 1 and Figure 1.

Figure 1

The course commenced in 2007, when two Uzbekistanis received initial training, and since then, 169 fellows have participated in this training program from various institutions in South Korea. In terms of follow‑up, four consecutive years (2021–2024) have been conducted at our affiliated university. The training program consisted of three modules: pre‑training, training implementation (coursework, individual training, and field visits), and post‑training management (Figure 1). The detailed instructions for the training program are presented in Supplementary Table 1.

Since 2021, the Dr. LJW Fellowship Program has trained public health professionals from LMICs through a structured and collaborative international program. Each year, 12 to 14 participants, selected from the Ministry of Health, regional or district‑level health sectors, national research institutions, and primary healthcare services, engage in a 3‑month training program in South Korea, followed by a 1‑year post‑training follow‑up. Fellows from 19 countries participated in this study. Laos had the largest number of participants (20), followed by Ghana and Tanzania (18 each).

Study design and participants

A mixed‑ methods design is adopted, focusing on quantitative and qualitative approaches. The development and assessment of this training course necessitate a modern, comprehensive evaluation approach. To enhance the reliability of training courses, it is crucial to implement a program and utilize an authentic performance evaluation model that employs an objective and specific evaluation methodology. The developmental approach is suitable for evaluating satisfaction with the course. Accordingly, the Dr. LJW Fellowship Program’s training course for health policy administrators adopts Kirkpatrick’s four‑level evaluation model, which proposes four levels of program evaluation for an educational module: reaction, learning, behavior, and results to the product phase [13–15]. Participants included 39 public health policy administrators who underwent the Dr. LJW Fellowship training course between 2021 and 2023.

Donald Kirkpatrick developed the Kirkpatrick evaluation model in 1959 to assess training programs and course performance. Kirkpatrick’s model is one of the most widely adopted and frequently referenced frameworks for training evaluation. Known for its simplicity and proven effectiveness over time, it offers a foundational approach to assessing training outcomes. While the model provides limited detail on which specific elements should be evaluated or how to align results with organizational strategy, it serves as a useful guide for the overall evaluation process [15–17]. The model is referred to as “Kirkpatrick’s levels” or the “four levels” because it comprises four evaluation levels (Figure 2).

Figure 2

1. Evaluation indicators

   This study highlighted the key indicators as a survey‑based evaluation (Supplementary Table 2‑5) within Kirkpatrick’s four levels of success that have positive implications for the learning process [18]. The four evaluation levels consist of Level 1 (reaction), Level 2 (learning), Level 3 (behavior), and Level 4 (results) to assess the short‑term outcomes of the Dr. LJW Fellowship Program. The final selected indicators comprised six items: training satisfaction, knowledge and competency improvement, competency achievement, competency improvement, work enhancement, and job adoption of learned knowledge. The four levels of the evaluation model and their corresponding indicators are listed in Table 1.

   Table 1

   The evaluation indicators of this study.

   KIRKPATRICK’S FOUR LEVELS	INDICATORS	DESCRIPTION OF INDICATOR	TIME POINT
   Reaction	Training satisfaction	Satisfaction with training content Evaluates the satisfaction of participants with the goals, content, and environment of the training, as well as their willingness to re‑participate, recommends the training, and their awareness of Korea.	At the end of training in Korea
   Learning	Knowledge and competency improvement	Improvement of knowledge and competencies Evaluates the improvement in participants’ knowledge and competencies through common education, action plan guidance, and Korean language education received during the training in Korea.	Every start/end point of each lecture
   	Competence achievement/ improvement	Achievement of learning objectives Assesses the achievement of participants in educational and research competencies (knowledge, skills, and attitudes).	2–3 months after the end of training in Korea
   Behavior	Work enhancement/job adoption of learned knowledge	Utilization and transfer of learning outcomes Assesses, 6 months after the training, whether participants’ job performance has improved due to the training and the extent to which the competencies acquired during the training have been applied in their work.	2–3 months after the end of training in Korea
   Results	Self‑reported reliability of training and professional enhancement	Linking training satisfaction to practical application Self‑reported satisfaction to assess the consistency and trustworthiness of course delivery and its relevance to professional roles, alongside the application of acquired knowledge integration.	2–3 months after the end of training in Korea
   		The impact of training results in improving the performance of training alumni and/or organizations. Improved performance productivity value‑driven practices as part of the learning outcomes.

   The number of items and subcategories is listed in Table 2. The questionnaire demonstrated high reliability across all four indicators. This table presents the internal consistency (Cronbach’s alpha) of the self‑administered questionnaire used to assess the four levels of training evaluation indicators: reaction, learning, behavior, and results.

   Table 2

   Reliability and consistency of the self‑administered questionnaire.

   INDICATORS	TOOL (LIKERT SCALE)	NO. OF ITEMS (SUBCATEGORIES)	CRONBACH’S ALPHA	RELIABILITY/APPROPRIATENESS
   Training satisfaction	5 points	7 (33)	0.771	Highly reliable
   Knowledge and competency improvement	5 points	8 (20)	0.784	Highly reliable
   Competence achievement/ improvement	10 points	2 (3)	0.857	Highly reliable
   Work enhancement/ job adoption of learned knowledge	5 points	2 (10)	0.805	Highly reliable

2. Data harmonization and scale standardization of evaluation

   Evaluation indicators were partially revised each year to reflect the specific professional backgrounds, job positions, and experience levels of the selected fellows. Following the submission of the year‑end program report, revisions to both the training curriculum and the evaluation questionnaires continuously refined and disseminated updated evaluation indicators to affiliated training institutions, based on feedback from the external panel committee and KOFIH. Consequently, indices associated with identical evaluation indicators may vary in the structure or scale of their four‑level measurements from 2021 to 2023. These Likert‑scale‑year inconsistencies pose challenges to longitudinal comparisons and program continuity. To address this, the present study standardizes all evaluation indicators by converting them to a 100‑point scale. The use of a unified 100‑point scale enhances the interpretability of the results and ensures comparability, despite minor changes in the evaluation instruments. This methodological adjustment facilitates a more reliable assessment of the trends in program performance and participant‑reported outcomes over time. This approach is consistent with survey research practices in which Likert‑type responses are frequently converted to a percentage scale to improve clarity and comparability across instruments and time points [19].

3. Qualitative evaluation

   The results component centered on fellows’ self‑reported perceptions of training reliability and professional enhancement, gathered through open‑ended questionnaires administered online two to three months after program completion (via Zoom meetings and Google surveys). Fellows were invited to articulate how the training influenced their roles and practices in their respective countries and workplaces, adhering to the Consolidated Criteria for Reporting Qualitative Research guidelines. The qualitative design facilitated fellows’ reflections on the impact of the training on their roles and workplaces. The responses were thematically analyzed by two independent researchers utilizing an inductive approach to ensure analytical rigor. These narrative responses provided contextual insights into the real‑world impact of the Dr. LJW program and served to complement the quantitative findings.

Statistical analysis

Descriptive statistics were employed to assess the profiles of participants in the Dr. LJW Fellowship Program training course for health policy administration. Means and standard deviations were calculated for all indicators. Paired t‑tests were utilized to compare knowledge and competence before and after the program. The results were derived through a meticulous review of responses collected via Google surveys and participant interviews. Verbalized interview transcripts were coded and analyzed using inductive thematic analysis. Two authors independently coded each transcript to ensure reliability and to capture meaningful user perspectives. The coding framework was refined iteratively through repeated discussions, leading to the development of a set of coherent themes. These final themes were subsequently defined and described, supported by illustrative quotes from participants to provide contextual depth and clarity. Statistical significance was established at p < .05 and was deemed significant. Data analysis was conducted using SAS 9.4.

Characteristics of participants

Thirty‑nine health policymakers participated in the Dr. LJW Fellowship Program training course for health policy administrators from 2021 to 2023; their descriptive statistics are presented in Table 3. Among the participants, 24 (61.5%) were male, and 15 (38.5%) were female. Most participants were in their 30s (59.0%), followed by those in their 40s or older (35.9%), and those in their 20s (5.1%). Their fields of work included chronic disease management (20.5%), personnel management and education (18.0%), maternal and child health (15.4%), health information systems (12.8%), health education (12.8%), disease prevention and health promotion (10.3%), and health management (10.3%). Participants’ work experience was distributed as follows: 6–10 years (38.5%), 11–15 years (28.2%), more than 15 years (20.5%), and 5 years or less (12.8%).

Level 1: Reaction

1. Training satisfaction

   The overall results of training satisfaction are presented in Figure 3. Training satisfaction serves as an indicator that evaluates not only the fellows’ satisfaction with the educational objectives, content, and environment, but also their willingness to participate in the training, recommendations to others, and their awareness of Korea. The mean overall training satisfaction score was 92.9. Satisfaction with the training objectives recorded a mean score of 91.8 (±11.9), while satisfaction with the training curriculum content had a mean score of 89.9 (±8.2). Satisfaction with the training environment was rated at 92 (±7.1). Furthermore, the safety management assistance provided by the training institutions scored 91.3 (±15.1), and the response from the safety management agency was rated at 90.9 (±11.7). The intention to re‑participate and recommend the training recorded the highest score at 97.4 (±5.9), while awareness of Korea was rated at 96.9 (±7.3).

   

   Figure 3

Level 2: Learning

• 1. Knowledge and competency improvement

The enhancement in knowledge scores across various subjects, evaluated through a comparative analysis of the pre‑ and post‑training scores, is illustrated in Figure 4. Overall, the average score rose from 2.7 prior to the training to 4.4 following the training, indicating a 61.0% improvement. When examining specific subjects, the field of epidemiology demonstrated an increase from 2.7 before the training to 4.1 after the training, resulting in a 48.8% improvement. The health statistics field recorded a noteworthy enhancement, with scores escalating from 2.5 before the training to 4.4 after the training, marking a 77.0% increase. The healthcare systems field exhibited an improvement from 2.9 to 4.5, reflecting a 56.3% increase, while the healthcare resources field showed a 60.8% improvement, with scores rising from 2.6 to 4.2.

Figure 4

In the health policy field, scores increased from 2.7 before the training to 4.3 after the training, resulting in a 60.5% improvement, whereas the health planning field experienced an increase from 2.9 to 4.5, reflecting a 55.2% improvement. The health financial management field also demonstrated progress, with scores ascending from 2.9 before the training to 4.4 after the training, indicating a 51.7% improvement. Finally, the Korean language education field exhibited the highest level of enhancement, with scores rising from 1.7 before the training to 4.3 after the training, resulting in a 150.0% increase.

• 2. Competence achievement/improvement

  Details of the levels of competence achievement during the training period in Korea are presented in Figure 5, demonstrating that the level of competence achievement, as reported by course advisors, had a mean score of 91.0 (±6.2). Conversely, the level of competence achievement, as self‑reported by fellows, was slightly higher, with a mean score of 93.9 (±6.7). The average level of competence achievement across these two assessments was 92.5 (±4.4). Furthermore, the overall competence achievement percentage was calculated to be 26.9% (±18.3).

  

  Figure 5

  The highest competence achievement was observed in the health prevention and promotion departments, which recorded a score of 96.2. This was followed by the maternal and child health department, which scored 94.4, and the manpower management and education department, which scored 93.0. The administration of chronic disease and healthcare information systems departments demonstrated similar levels of competence achievement, scoring 92.9 and 92.7, respectively. The healthcare management and healthcare education departments exhibited relatively lower competence achievement scores, at 89.4 and 87.8, respectively (Figure 5).

Level 3: Behavior

1. Work enhancement

   The results of work enhancement following the return to work are presented in Figure 6, which displays assessments from the fellows themselves, their supervisors, and coworkers. Specifically, the level of work enhancement, as self‑reported by fellows, had a mean score of 90.4 (±9.9). The level of work enhancement, as reported by supervisors, was slightly higher, with a mean score of 92.0 (±9.7). The level of work enhancement assessed by coworkers was 92.2 (±8.8). The average level of work enhancement across all three assessments was calculated to be 91.8 (±7.4).

   

   Figure 6

2. Job adoption of learned knowledge

   The results of the evaluation of how well the fellows employed the knowledge gained during the training period in their work, as assessed by their supervisors and coworkers, are shown in Figure 6. Specifically, the level of job adoption of learned knowledge, as assessed by supervisors, had a mean score of 70.9 (±7.0). Meanwhile, the level of job adoption of learned knowledge, as assessed by coworkers, was slightly higher, with a mean score of 72.1 (±6.4).

Level 4: Results

Level 4 (results) reflects the tangible impact of the training on performance outcomes. Participants reported that the training program significantly enhanced their work productivity and professional competencies. Conversely, fellows were able to deliver outcomes aligned with organizational goals and meet the performance standards and accreditation requirements set by the government. The training contributed to improvements in technical and conceptual skills, effective use of information technology, and adherence to standardized accreditation procedures. The responses of the fellows are transcribed and described in Table 4.