Lessons Learnt Delivering a Novel Infectious Diseases National Training Programme to Timor-Leste’s Primary Care Workforce

Curriculum development

The Advancing Surveillance and Training to Enhance Recognition of Infectious Diseases (ASTEROID) project was led and implemented by Maluk Timor, a non‑governmental organisation (NGO) with a focus on the capacity‑building of Timorese healthcare staff. It was designed to be delivered through multiple teaching methods, including didactic lectures, multimedia presentations and an integrated mobile application. The strategic plan underpinning the project is shown in Figure 1. Prior to the development of the curriculum, key areas of need were identified through discussion with stakeholders with experience working in the Timorese healthcare system. These stakeholders formed a technical working group (TWG), with representation from directors of the Instituto Nacional de Saúde Publica (INSP; the national institute for health), staff specialist physicians from Hospital Nacional Guido Valadares (HNGV; the national hospital), the Maluk Timor project leads and representatives from the World Health Organization (WHO), Menzies School of Health Research and St John of God Health Care (an Australian healthcare provider working in Timor‑Leste).

Figure 1

A 17‑module curriculum was proposed by a specialist infectious diseases physician with experience in clinical practice and teaching in Timor‑Leste (assisted by a paediatrician with similar local experience for the paediatric modules). Modules were developed using content from existing locally available guidelines, e.g., National Standard Treatment Guidelines [18], with supplementation from relevant international guidance, e.g., available WHO guidelines, and knowledge regarding local practice. Guidance on disease notification was taken from the Timor‑Leste Integrated Disease Surveillance and Response guidelines, which had not yet been widely disseminated.

The TWG agreed upon this core curriculum and had regular meetings to discuss the content of each individual module as it was developed. This was particularly helpful in resolving queries around content, for example, where there were disparities between guideline recommendations and ‘on the ground’ practice, e.g., lack of availability of certain medications or investigations. The feedback provided by the TWG was collated and forwarded on to Maluk Timor staff for further action before the curriculum content was finalised. This core content was then used as the source material for the 17 modules and mobile application content. All resources were initially developed in English and were translated into Tetun‑Dili (Timor‑Leste’s most common language) with the help of an independent professional medical translator.

After the modules received approval from INSP, local Timorese doctors were recruited as facilitators to deliver the content to Timorese healthcare workers. They were provided with formal training to prepare them for their role. This training was conducted by Maluk Timor and involved the facilitators trialling several modules in community health centres (CHCs) over several weeks. At the end of each session, formal feedback was provided by both training participants and Maluk Timor staff and fed back to the facilitator to improve the quality of their presentation skills.

For each CHC training week, monitoring and evaluation (M&E) would begin before the training week with a rapid facility assessment (RFA) to establish the physical and personnel resources of each CHC. This would enable tailoring of content delivery and identify facility deficits that would hinder immediate implementation of training practices, e.g., the availability of hand‑washing facilities.

Description of training

Each module consisted of a 1‑hour lecture incorporating video content, a microlearning module on the mobile application, small group case studies and multiple choice question (MCQ) exams. The 17 modules were delivered over a week, and were as follows:

1. Introduction to infectious diseases

2. Infection prevention & control

3. Disease surveillance & outbreak recognition

4. Rheumatic fever & rheumatic heart disease

5. Dengue & other important causes of fever

6. Measles, rubella and other fevers with rash

7. Tetanus

8. Envenoming, stings & bites

9. Neurological infections

10. Acute respiratory infections

11. Tuberculosis

12. Sexually transmitted infections

13. HIV

14. Viral hepatitis

15. Tropical dermatology

16. Helminth infection

17. Diarrhoeal illness

Each lecture was delivered in Tetun‑Dili by the local facilitators. All lectures followed a similar format – with objectives outlined at the start, followed in order by discussion of disease presentation, assessment, management, prevention and notification. Short video segments (2–5 minutes), presented by Timorese specialist clinicians, were integrated throughout the lecture, covering topics felt to be especially important [e.g. diagnosis of tuberculosis (TB)], or topics that benefited especially from visual demonstration (e.g. first aid for snakebite).

Also integrated throughout the lecture were case presentations, usually ‘typical’ presentations of a case. These provided a chance for audience participation and would be revisited throughout the lecture. Other audience participation sections included an audience response system, which enabled anonymous answering by the group of multiple‑choice questions, as well as direct questioning of the group on topics such as the availability of local diagnostics.

Other sessions throughout the week included a daily refresher section starting each day, recapping the previous day’s content, and small group sessions working through a series of short‑answer questions. The short‑answer questions encouraged use of national guidelines, with accessible sections provided as printouts (see example guideline in Figure 2).

Figure 2

A final learning supplement was the dedicated mobile application ‘Haroman’ (developed by Catalpa International), shown in Figure 2. This was designed as an educational supplement, and not to include any patient data or be used directly in healthcare provision beyond as a reference and guideline repository. The app had short versions of all 17 modules uploaded, including MCQs and accessible national guidelines, as shown in Figure 3. It was designed to require minimal data use for download and is available in English and Tetun‑Dili versions. It was introduced to participants during their training week, where they were encouraged to use its guideline library to help answer questions in small group tasks. At the time of its development, no other medical education mobile application specific to Timor‑Leste was in existence.

Figure 3

Examinations were based on MCQs, with a short ‘pre‑test’ every morning to assess the group’s existing knowledge. There were then three ‘post‑tests’ throughout the week, which formed the basis of a candidate ‘passing’ the week, and our quantitative evaluation of the programme.

Context and recipients of training

The ASTEROID programme was targeted at non‑specialist healthcare workers in Timor‑Leste. This included nurses, midwives and doctors working at CHCs. The wide range of audience backgrounds was reflected in the collected demographic data of training participants (Table 1). The audience’s demographics were considered in the preparation of the modules, with complex concepts explained using simple and familiar terminology wherever possible. Unavoidably, this one‑size‑fits‑all approach led to some of the more complicated source material being omitted from the formal teaching programme; however, all participants were encouraged to ask the facilitator questions and were directed to the Haroman mobile application for further in‑depth information.

The training has been conducted at many different CHCs across all of Timor‑Leste’s municipalities. It was envisioned that travelling to individual CHCs to deliver the training would enhance attendance and increase audience participation when compared with delivering the training at a single centralised location [19].

Monitoring and evaluation (M&E)

During the training week there was continuous modular assessment, as described above, and anonymous written feedback was gathered daily. Each week finished with a small group verbal feedback session.

Follow‑up after training (FUAT) was carried out at 3 and 12 months. In these visits, data were collected including repeat MCQ examination, self‑reflection on the impact of the training and qualitative feedback via participant interviews.

Test results

Table 2 presents results from the MCQ‑based assessments, showing aggregated results by profession. The average improvement in test score was 19.4%, from 44.5% to 63.6%. This was statistically significant (p < 0.0001, using paired t‑tests). Improvement was observed for all professions, ranging from 16% (nurse assistant) to 20.1% (pharmacist). The improvement was statistically significant (using paired t‑tests) for all professions with n > 5. Variation in baseline knowledge between professions was reflected in the results, with doctors scoring highest pre‑ and post‑test (54.2% and 74.8%). This improvement continued throughout FUAT, increasing by 7.6% from post‑test to the FUAT at 3 months (71.3%), and by a further 8% for the FUAT at 6 months (79.3%).

Interview and questionnaire findings

To date, in‑depth qualitative interviews have been carried out reviewing the training week with 17 healthcare professionals (including health facility managers as well as participants).

Of the participants interviewed, 100% reported that they felt that the training was useful for their work (including the choice of modules and the Haroman application), and 100% reported improvements in infection prevention and control (IPC) practices in their health centre after training. In all, 83% felt that it would increase their confidence in diagnosing and treating patients with infectious conditions, whilst 67% reported changes in the identification and management of infectious diseases (most particularly for sexually transmitted infections). In addition, 50% reported improvements in disease notification practices in their CHC, and 50% also reported sharing their learnings from training with other colleagues.

Feedback was also gathered from anonymous questionnaires throughout the week, and small group discussion at the end of the week.

Common positive comments involved the overall quality of the training – with the videos delivered by local specialists being especially praised, and the quality of the teaching delivery also being frequently highlighted. The variety of educational methods, such as the use of the audience response system and case study discussion, was also appreciated. Appreciation was also expressed for the dedicated training delivered in the Tetun‑Dili language, tailored to the Timorese context. Some rural CHCs reported having had no previous professional training in over 5 years.

Common negative comments involved the quantity of content being delivered during a single week making retention difficult, especially for non‑clinical staff members. It was also reported to be difficult for staff members to be released for a whole week off of clinical duties. Another common comment was on the language used in written material – which had frequently been translated from original source material in English and did not use Tetun‑Dili as colloquially used by the participants.

Successes

Combining local emphasis with national coverage, embedded governmental co‑operation and an innovative multimedia approach, the ASTEROID programme is delivering well‑received and timely training to the Timorese primary care workforce.

The ASTEROID programme showed an immediate 19% increase in test scores, which was statistically significant and represents a 43% increase in pre‑existing knowledge. This was not just sustained but improved upon in follow‑up at 3 and 12 months, returning 27% and 35% increases on baseline knowledge, respectively. The immediate increase is in‑line with published literature on LMIC health worker training programmes, with a systematic review by Abdel‑All et al. finding increases ranging from 4% to 40% [20]. In follow‑up after training, Abdel‑All et al. found on average no improvement, with three out of five studies included showing a decline. ASTEROID’s programme’s FUAT has bucked this trend in showing statistically significant improvement from even post‑test results. Positive explanations include the project’s ongoing learning resources such as the Haroman app, which showed good uptake, and an increased focus on infectious diseases stimulated by Asteroid. Possible confounding factors explaining the FUAT increase beyond a true knowledge increase include three similar tests in 12 months increasing ability at ASTEROID MCQs, and FUAT selecting for candidates more likely to perform (roughly 50% of candidates took FUAT tests).

The increase in training score was roughly consistent across different professions, with all with n > 5 showing statistically significant improvement. This is despite different baselines of knowledge as shown by average pre‑test score and suggests that the training was accessible to all participants.

Further successes and failures can be analysed using the framework provided by Hill et al. in their systematic review of LMIC healthcare professional development programmes [21]. This review identified that many training programmes showed Western bias, having been developed in higher‑income countries (HICs), thus being most relevant to the pathology and available treatments in HICs. The ASTEROID programme was developed specifically for the Timorese context, with a focus on diagnostics and treatments available in Timor‑Leste, as well as approaches such as showing common Timorese skin tones in dermatology videos. This was frequently commented on in qualitative feedback and was well‑received.

Another problem identified by Hill’s review was that delivery methods and learning opportunities were not always best suited to the LMIC learners. The ASTEROID programme ensured the best possible engagement by delivering the training in the place of work, including reaching isolated rural CHCs, and covering every municipality. Delivery methods were tailored to the Timorese context easily by experienced local facilitators.

The programme was also able to introduce multimedia learning methods, such as the multimedia use of video and the Haroman mobile application. This, as the first mobile app developed specifically for use in Timorese healthcare, has been popular and should facilitate continuing professional development by offering up‑to‑date learning post‑training. The app’s embedded sections of clinical guidelines provide easy access to national guidelines, which has been popular, given the high prevalence of smartphones set against a relative lack of paper guideline provision. Reviewing the app against Wallis et al.’s discussion of mobile health integration in LMICs [22], Haroman has not suffered from the technological pitfall of a poor‑quality mobile signal, as the app can be fully downloaded during the training week itself whilst higher‑quality internet is provided.

Further positive changes seen during the ASTEROID programme have included the enthusiastic reception of WhatsApp groups, an increasingly recognised healthcare tool [23], set up during training for notifying infectious diseases. These have been heavily used as a new peer network for second opinions on diagnoses and treatment.

Failures and learning points

A fundamental limitation in this evaluation of the ASTEROID programme is the lack of data collection on its impact on healthcare practice. It is well established that in‑service training programmes, even those demonstrating significant knowledge improvement, do not necessarily lead to improvements in quality of care [21, 24, 25]. In the Timorese context, Hou et al. found the increasing knowledge of infection control methods did not necessarily lead to improved practices such as use of hand‑washing [16]. Similarly, Wallis et al. found that use of mobile apps in LMICs often rapidly dropped off after in‑service training without dedicated support [22], which may be an issue for Haroman (user statistics are being monitored throughout the programme).

In future, a programme identifying specific, measurable, achievable, relevant and time‑bound (SMART) [26] goals for CHCs which can be measured pre‑ and post‑training will provide more evidence of benefit. An embedded research programme for local researchers has been developed alongside the ASTEROID teaching programme, looking at questions such as utilisation of GeneXpert for tuberculosis diagnosis, and hepatitis B prevalence, but for simplicity this has not been designed to comment directly on the effect of ASTEROID.

A further difficulty for the ASTEROID programme has been an unexpectedly low baseline of clinical knowledge, meaning many participants had lacked the scaffolding to absorb new clinical concepts. This does tally with previous literature and Timor‑Leste’s difficult recent history, which provides a backdrop to participants’ professional development [4, 5, 14, 27]. The gratitude expressed in some rural centres upon receiving their first training programme in many years also highlights predictable future challenges, given that decay in knowledge over time from one‑off training programmes is inevitable [28]. More input than a single training week is likely required to lead to robust, sustained change. With further resources, additional input such as annual training courses or in‑work supervision on key topics could enable these participants to truly absorb and make use of the information this training has provided.

Participants struggled with certain aspects of the training week, such as the volume of content being delivered, and their lack of familiarity with using clinical guidelines or MCQ examination. Running more pilot sessions with future participants during content development might have enabled content set at a more appropriate level. Although some degree of discomfort may be inevitable when driving change in healthcare practices, it could be that a week‑long training programme is not the appropriate avenue for introducing these changes [21].