Barriers to Applying Knowledge Gained Through an Implementation Research Massive Open Online Course: An Explanatory Qualitative Study

Abstract

This study explores the barriers encountered by participants in applying knowledge gained from a massive open online course (MOOC) on implementation research (IR), particularly among learners in low- and middle-income countries (LMICs). Despite the rising demand for quality IR training, access remains limited in LMICs. The IR MOOC, created by the Special Program for Research and Training in Tropical Diseases, seeks to bridge this gap by providing free online IR training. Twenty-three interviews were conducted with participants from the French, Spanish, and English language sessions of the MOOC, and were analyzed using a general inductive approach. Barriers were identified at individual, organizational, community, and health authority levels. Individually, learners requested opportunities for deeper training and expert guidance. Organizational barriers included a lack of understanding and resistance to adopting IR methodologies. Community barriers involved limited roles and opportunities in IR and a lack of networking. Health authorities exhibited a lack of awareness and funding to support IR. Challenges in completing the course included language barriers, strict deadlines, limited internet connectivity, and a lack of localized case examples. This study highlights the importance of pairing online training with practical opportunities to apply newly learned skills. Our findings emphasize the need for expert guidance, improved language accessibility, and diverse case studies to support the development of a robust cadre of IR practitioners. Networking opportunities are crucial for connecting learners with IR practitioners and applying the knowledge acquired. Efforts to address these barriers could improve the effectiveness of IR training programs in LMICs.

Keywords

online training implementation research infectious diseases of poverty evaluation

What do we already know about this topic?

Access to implementation research training remains limited in low- and middle-income settings, and learners often face barriers in applying the knowledge gained from such courses.

How does your research contribute to the field?

This study identifies key barriers faced by learners in applying their implementation research knowledge after participating in a massive open online course on implementation research. It highlights the need for enhanced mentorship, localized case studies, and improved access to online learning materials.

What are your research’s implications toward theory, practice, or policy?

The findings could inform future training programs and policy decisions, emphasizing the need for localized examples and opportunities for enhanced engagement with online learning content and with opportunities to apply the knowledge gained.

Background

Implementation research (IR) has emerged as a critical field for improving the uptake and scaling of evidence-based health interventions.^1,2 High quality IR identifies implementation bottlenecks for efficacious health interventions and applies this information to develop tailored strategies to overcome the identified challenges. The potential for IR to benefit health systems may be greatest within low- and middle-income countries (LMICs) where governments often lack the resources to thoroughly study implementation weaknesses and to develop and apply localized implementation solutions.^3,4

With the continued maturation of IR comes a growing demand for high quality training programs.⁵ The number and types of IR training programs has expanded nearly exponentially over the past 2 decades, with short-courses, boot camps, multi-week workshops, and university degree programs all being offered routinely.⁶ Yet, these courses generally revolve around North American and European audiences, making them out of reach both practically and financially for learners in LMICs who seek to add IR to their research toolkits. There are some notable exceptions,^7,8 yet the demand for places within these IR training programs among researchers, program implementers, and policymakers living in LMICs far outstrips the number of opportunities available.

IR is best conducted by those with intimate knowledge of the context where implementation challenges are being encountered.⁹ Due to the context-specific nature of each implementation bottleneck, IR teams who are a part of and possess in-depth knowledge of the health system and the communities accessing care and treatment are inherently best positioned to identify and develop strategies to overcome these challenges.¹⁰ Individuals with these lived experiences, be they researchers, health service providers, national disease program implementers, or community members themselves, should be at the forefront of demand-driven and responsive IR projects. Therefore, teams of aspiring IR practitioners from LMICs must be the core unit of local IR, rather than foreign researchers who possess limited in-country expertise and potentially conflicting motivations.¹¹

The application of online technology has expanded the accessibility of many forms of training globally. Through online platforms, learners globally can take part in a wide range of virtual courses, many of which are free of charge. For IR specifically, there are a number of core resources available online including open access toolkits and self-paced courses.¹² The Special Program for Research and Training in Tropical Diseases (TDR), co-funded by United Nations International Children’s Emergency Fund (UNICEF), the United Nations Development Programme (UNDP), the World Bank, and the World Health Organization (WHO), developed a massive open online course (MOOC) on IR specifically for learners in LMICs to addresses the relative absence of IR training opportunities for this audience. The content was created by recognized IR experts through the coordination of TDR. The course focuses on applying IR methodologies to improve the effectiveness of interventions for infectious diseases. Through the six-week course, learners gain a basic understanding of IR and build foundational IR skills through 5 modules comprising video lectures and case examples, supplemental readings, quizzes, and interactive online forums. The IR MOOC was piloted in 2017 and opened to learners worldwide in 2018 free of charge. Originally developed in English, the IR MOOC is now routinely offered with French, Spanish, Russian, and Arabic subtitles and facilitated by native speakers of each language. As of 2024, over 20 000 individuals have registered for the IR MOOC. Additional online content has been developed to supplement the original course, including the creation of modules on gender and intersectionality in IR, content specifically for health program implementers interested in working within IR teams, and a library of case examples illustrating the utility of IR for a number of infectious diseases across geographies.

TDR’s IR MOOC is perhaps the best studied IR training program for learners in LMICs. A series of survey-based evaluations of the English language version of the MOOC demonstrated its success in reaching its intended audience, increasing IR knowledge, and directly influencing learners’ professional behavior.^13
-17 A recent follow-up evaluation examining experiences of learners participating in the French, Spanish, and English language offerings of the IR MOOC illustrated that similar results are achieved across language groups and that this online training tool has greatly expanded the pool of researchers with basic IR capabilities.¹⁸ Yet, this most recent study also showed that only a minority of learners were able to meaningfully apply their IR knowledge and skills within their professional organizations. Even though learners reported high degrees of motivation to use their newly acquired IR knowledge, very few individuals reported being able to undertake professional IR activities within their organizations.

The current explanatory qualitative study has 2 aims: (1) to identify the barriers encountered by the IR MOOC participants in applying their knowledge professionally and (2) to examine the barriers met by participants preventing them from fully engaging with the IR MOOC content. We include here learners who participated in the French, Spanish, and English language offerings of the IR MOOC. Through better understanding the challenges learners encounter in gaining and applying IR knowledge, we hope to refine our methods of instruction and develop strategies to further support learners to more readily apply IR methodologies within their profession and/or respective health systems.

Methods

Participants of 6 sessions of the French, Spanish, and English language IR MOOC (2 sessions of each language) were surveyed anonymously in August 2022. The survey questions were developed based on the Kirkpatrick Model¹⁹ and have been piloted and used in previous examinations of IR MOOC participant experiences.^13,15,16 The interview questions were based on those used in a previous study¹⁵ and were contextualized to examine the barriers participants encounter in using the knowledge and skills gained through the course (Supplemental File 1).

Survey respondents were asked if they were willing to be contacted for a follow-up interview regarding their experiences during and after participating in the IR MOOC. From this pool of 350 individuals willing to be interviewed (70 Spanish; 57 French; 223 English), 168 (39 Spanish; 57 French; 72 English) were contacted via email between January and August 2023. The WHO’s Research Ethics Review Committee reviewed and approved this study (protocol ID: ERC.0003785).

A convenience sample was recruited beginning with an introductory recruitment email sent to each potential participant in the language corresponding to the language of the MOOC session they participated in. The email provided a brief overview of the evaluation and asked if the individual was willing to participate in an online interview. If the participant agreed to be interviewed, a meeting time and date was sent via email. Inclusion criteria were that potential participants registered for at least one of the IR MOOC sessions under evaluation, was at least 18 years of age, and able to meet for an online interview. All participants provided written informed consent prior to engaging in the interview.

Interview Methods

Interviewers were researchers with a background in health research with a mix of PhD, MD, and MS credentials and underwent training on qualitative methods and semi-structured interviewing prior to recruiting participants (Authors 1 and 2 in English, 3 and 5 in Spanish, and 6 and 7 in French). Four of the researchers identified as women and 2 identified as men; all were employed by universities or other research and training organizations. Authors 1, 2, 6, and 7 conducting the interviews had prior involvement in either developing, running, or evaluating the IR MOOC. Interview guides were developed with questions and potential probing prompts focusing on participants’ experiences in the IR MOOC itself, perceived barriers to engagement with the course, challenges in applying their IR skills within their organizations, and recommendations for improving both the delivery of the course and methods for improving post-course application of IR knowledge. Interviews were conducted via online telecommunication software such as Zoom and Google Meet. Researchers introduced themselves, explained the purpose and nature of the study, and asked background questions of the participants prior to beginning the interview to develop rapport. The semi-structured nature of the interviews allowed for participants to openly discuss the challenges that they met during and after the course, as well as any meaningful outcomes from the course. Topics outside of those on the interview guides that emerged through the discussion were able to be openly elaborated upon. Participants were offered a chance to change or amend any of the answers they provided prior to the end of the interview. No other individuals were present during the interview sessions and no repeat interviews were conducted. Interviews were audio recorded using the telecommunications software and audio files were transcribed in their original language for analysis and further informed by written notes taken by the interviewer during the conversation. Where needed, key quotes were translated into English by bilingual researchers for reporting purposes.

Data Analysis

Analysis followed a general inductive approach—an analytical technique that allows for rapid analysis of qualitative data when a priori objectives are outlined and aligned with data collection instruments.²⁰ The general inductive approach produces robust findings when analysts apply the technique rigorously. This methodology was chosen as most appropriate due to our goals of examining pervasive challenges encountered by participants in applying their knowledge after taking the course and identifying barriers to engagement with the IR MOOC. We did not aim to validate a theoretical model nor test any specific hypotheses through this evaluation.

For each language, 2 researchers began the analysis by reading the library of transcripts. The researchers first reviewed the raw data to identify core meanings within the transcripts in relation to the primary evaluation objectives. A coding framework was constructed with higher-level categories defined by the goals of the evaluation: identifying challenges in utilizing IR knowledge after the course and identifying substantial barriers to engaging in the course. Lower-level codes were more specific and created from emerging statements within the text. Transcripts were reviewed on a rolling basis as interviews were conducted and were analyzed in line with the developed coding framework. Two researchers for each language regularly discussed the overarching themes that emerged from the interviews and refined the code book as needed. Discrepancies were resolved through conversation between researchers and with direct reference to the original data. Coding was conducted using Microsoft Excel and Microsoft Word. Participants were not asked to provide feedback on the findings.

Results

A total of 23 individuals participated in the interview: 4 from the French language cohorts, 8 from the English language cohorts, and 11 from the Spanish language cohorts. The interviews lasted between 17 to 60 min. Table 1 presents the overall profile of the participants.

Table 1.

Interview Participants’ Profiles.

Spanish N = 11	Gender identity	Woman	6 (54.5%)
		Man	5 (45.5%)
	Country	Argentina	1 (9.1%)
		Brazil	1 (9.1%)
		Colombia	4 (36.3%)
		Dominican Republic	1 (9.1%)
		Ecuador	1 (9.1%)
		Haiti	1 (9.1%)
		Honduras	1 (9.1%)
		Mexico	1 (9.1%)
	Profession	Health care provider	1 (9.1%)
		Public health worker	3 (27.3%)
		Professor	3 (27.3%)
		Researcher	3 (27.3%)
		Student	1 (9.1%)
	Age	26-35	2 (18.1%)
		36-45	3 (27.3%)
		46-55	3 (27.3%)
		55+	3 (27.3%)
French N = 4	Gender identity	Woman
		Man	4 (100%)
	Country	Burkina Faso	2 (50%)
		Central African Republic	1 (25%)
		South Africa	1 (25%)
	Profession	Public health clinician	1 (25%)
		Student	2 (50%)
		Researcher	1 (25%)
	Age	26-35	3 (75%)
		36-45	1 (25%)
English (N = 8)	Gender identity	Woman	3 (37.5%)
		Man	5 (62.5%)
	Country	Afghanistan	1 (12.5%)
		Cote d’Ivoire	1 (12.5%)
		Guatemala	1 (12.5%)
		Indonesia	1 (12.5%)
		Kenya	2 (25%)
		Nigeria	1 (12.5%)
		Sudan	1 (12.5%)
	Profession	Health care provider	4 (50%)
		Public health NGO founder	1 (12.5%)
		Researcher	1 (12.5%)
		Student	2 (25%)
	Age	26-35	2 (25%)
		36-45	4 (50%)
		46-55	1 (12.5%)
		55+	1 (12.5%)

Given the wide range of possible participant contexts, the authors acknowledge that data saturation was not fully achieved with the interview pool. However, recurring themes were found, suggesting that the core findings reported in this paper reflect the broader population’s experiences with the application of IR training. A priori research questions focused on previously identified barriers from the survey, and while it was not possible to capture every individual reason behind these barriers, saturation was achieved concerning the intention to explore these barriers from the perspective of IR MOOC learners.

The following results were derived from the data and first present the barriers encountered by participants in applying the knowledge gained through the MOOC to their professional work—organized from the individual, organization, community, and authority levels. Then, we present the challenges participants faced in engaging with the MOOC content, be they related to the technological aspects of online learning or the presentation of the course materials themselves.

Individual Level Barriers: Need for Expert Guidance and Additional IR Training

A significant individual level challenge described by participants was the lack of expert support and guidance in completing their capstone project, a short research proposal. Participants’ proposals were reviewed by peers also in the course, but the inability to work with a person knowledgeable in IR was a persistent barrier to following through with the IR project:

I think that the main issue would be how to make the participants see the weaknesses in their initiatives, in their proposals [the capstone project for the course], and be able to continue working [on the proposals]. These aspects would lead to more institutional insertion, in order to have research continuity. (Man, Ecuador, Professor, 55+,).

The course was about implementation research, but proposal development also needs other skills, so maybe in the course something about proposal development could be incorporated, some lessons about that. (Woman, Indonesia, Master’s student, 26-35)

Similarly, participants encountered points of confusion within the material, such as how IR outcomes can be differentiated from each other, and would have benefited from more expert guidance on certain topics. The IR MOOC is only a starting point for acquiring knowledge on this methodology, and the need for additional in-depth training was a frequently cited issue in carrying out an IR project after the MOOC had finished. Some participants were able to access individual guidance on these points through their own professional networks, but many could not.

Yeah, that [expert guidance] might have helped. I actually have a supervisor who is knowledgeable about this. So, he helped me clear some of the confusion, because as I was not able to differentiate between those outcomes, he helped me in saying that, “yeah, you don’t have to differentiate it. All those implementation outcomes can be mingled with each other.” So, this is just part of what it is. . .Maybe because I got to know about it from him. But maybe in the course there could be some lessons that it’s not always separate. It’s connected. (Woman, Indonesia, Master’s student, 26-35)

Organizational Level Barriers: Resistance to Change and Lack of Understanding of IR

After completing the course, participants encountered pervasive barriers at the organizational level. The resistance to adopt IR, an often new form of research, by authorities was commonly noted and posed a substantial barrier to applying the approaches presented in the course. Participants frequently encountered skepticism or outright opposition to introducing IR concepts learned from the MOOC. This resistance was often rooted in a preference for traditional research practices and a reluctance to adopt new approaches:

My research is the first research relating to implementation research in that field. So, I think it shouldn’t be like this. . . I shouldn’t be the first one, because the program [I work on] is going on for a long time. So, it should have started a long time ago. So, there’s definitely a need. But in terms of demand from the implementers, I did not see that much of demand. Even they were trying to demotivate me to do implementation research. They are just trying to tell me to just stick to the normal, what other people do. So, I don’t think there’s demand from them. But there’s definitely a need. (Woman, Indonesia, Master’s student, 26-35)

Community Level Barriers: Scarcity of Applied Research Opportunities

Participants noted the shortage of employment opportunities related to IR as well as a scarcity of local, ongoing IR projects that they could participate in. Some expressed frustration at the lack of opportunities to use the knowledge they had gained via the MOOC:

It’s like giving people the theoretical framework without the field experience. . . So, along these lines, these people, NGOs, collaborators of the program, can make opportunities available for people to volunteer [to work on IR] . . . It would be a very good way to practicalize that which has been learned during the training and create a longer lasting impression of the course and its practicality. (Man, Nigeria, PhD student, 36-45)

It would be really interesting after the MOOC to provide participants with internships in certain structures, if they wish. (Man, Burkina Faso, Clinician, 26-35)

Those interviewed regularly expressed their interest in IR and desire to use the knowledge they acquired through the MOOC, but also conveyed their dismay with the lack of support received from their employers to do so:

I haven’t been able to use most of it, only because I am limited by what I am doing [at work]. I haven’t used most of what I learned. I haven’t been able to use it because I am not getting any support. So, I’m limited to what I’m doing. I’m just: “do this, do this, do this,” then you go home. The resources and the encouragement is not there. (Man, Kenya, Clinical/Healthcare provider, 26-35)

Community Level Barriers: Need for Networking and In-person Learning

Participants pointed to the importance of creating networks of IR professionals and that the lack of personal communication that comes with online learning can dilute their learning experience:

I’m not aware. If [an IR network] does exist, I’m not part of it. But we could start with the students. . . I would love to be a part of that community. (Man, Nigeria, PhD student, 36-45)

What was surely missing is how to close the gap among each other, the students of the course, to share close experiences and create continuity groups. (Man, Mexico, Public health worker, 55+)

I think that in the next MOOCs to come, we need to develop this community participation because it is really essential. If we do not take the population into account, the different interventions will not work. (Man, Burkina Faso, Clinician, 26-35

Furthermore, the interviewees also drew attention to the perceived weak ties among research centers, universities, and health institutions. Participants discussed the need to promote the IR MOOC on a country and regional level:

Through the country offices [the IR MOOC] should be promoted. . .basically I think that this would achieve a lot, that many people would find out and learn about it. (Man, Dominican Republic, Public Health Worker, 36-45)

After completing the MOOC, I think that it is important to support the participants; we can first group them by country, and also give them means and train them even more, so that they can really intervene. Supervising them so that they can spread the knowledge they received during the MOOC training will also be very important. (Man, Burkina Faso, Clinician, 26-35)

Many of the interviewees highlighted that virtual education is necessary and effective, however, it has limitations for students to meaningfully interact with colleagues in different countries and may be a second option compared to in-person courses. They suggested that the course include face-to-face meetings (national or regional) that favor knowledge exchange, build networks, and produce collective work.

I insist on face-to-face meetings and discussions, above all, this is a topic that needs to be looked at from different points of view, not only of the teachers, but of the participants. (Woman, Colombia, Professor 46-55)

With all this coldness that these forms of communication have, imagine when it’s one person’s turn to listen to the voice of a stranger, because for me that increases the distance, decreases the strength of the connection, especially when we work with such a delicate thing as life. (Woman, Argentina, Researcher 36-45)

I think [the MOOC] is useful, but it’s always better to have face-to-face trainings, except for those who really cannot have the free time to be present somewhere to have the lessons. (Man, Burkina Faso, Master’s student, 26-35)

Health Authority Level Barriers: Lack of Understanding of IR

Compounding with the resistance to adopting IR as a research methodology among leaders, an overarching lack of awareness of IR at the health level authority was a routinely cited barrier to applying knowledge from the IR MOOC. Participants stated that the lack of IR uptake was based on low levels of interest or understanding of IR among health authorities:

I have tried to explain it to them to see if they are also motivated and to see how we can use it in our actions, and they ask me a lot of questions, because discussing it raises many questions, but when I try to explain it on my own, they just don’t understand it. (Man, Dominican Republic, Public health worker, 36-45)

Several interviewees underlined major barriers related to the lack of knowledge of IR among government institutions that oversee research and training programs:

In the institution where I work, colleagues do not have knowledge of IR. It was my university professor who talked to me a lot about it, and I found it to be really important. It was the opportunity I had been waiting for. So I took the opportunity. (Man, Burkina Faso, Clinician, 26-35)

There is poor qualification of staff [on IR]. (Man, Burkina Faso, Master’s student, 26-35)

One participant stated that the high-level health authorities in their country lacked knowledge of IR. Another participant mentioned that despite the authorities recognizing IR as a science that works, this interest does not translate into action, and it is difficult to convince their co-workers about the value of IR.

I think that initially, the biggest challenge is how to empower or convince people [of the value of IR]. That will take time, to raise awareness, meet with people, so that they have an understanding of the subject, but later, it can really flow. (Man, Dominican Republic, Public health worker, 36-45)

I think that nowadays, in the context of my country, only researchers are interested in implementation research. So, the problem I talked about earlier is at the organization or the people who actually have the power to make a change in the field - they are not interested in it. (Woman, Indonesia, Master’s student, 26-35)

First of all, in low-income countries, we really need to train staff, because those who decide most often don’t really have this skill. (Man, Burkina Faso, Clinician, 26-35)

One participant suggested that training in IR be introduced to governmental actors to improve their understanding and hence reduce resistance to conducting IR:

I know that we are all going to face resistance when applying work of this type, right? Probably, for this reason, I think that [IR training] should be done with new generations to train future governmental public health actors. (Man, Mexico, Public health worker, 55+)

Health Authority Level Barriers: Lack of Funding

Participants discussed openly challenges related to funders and financing opportunities. In addition to the general lack of funding for health research, participants stated that a lack of awareness about IR among funders makes receiving grants for IR projects very difficult:

If someone were to present, for example, to [high-level health authorities], a research project using this methodology, are there enough evaluators there that are really going to fund these kinds of projects? We also often have difficulties with the evaluators. (Woman, Colombia, Professor, 55+)

One participant explained how a lack of understanding of the local context by funders and the role it plays in IR is not always understood by partner organizations abroad:

We’ve seen so many things being kind of pushed down our throats to say it that way. When we see a design of something, that we are there, in the real-world, and maybe it’s thought out in an airconditioned office somewhere in US by somebody [to be carried out in] Africa. And then I think “Oh, gosh, what are they thinking?” . . . they really want questions, answers to every question. And today, I had to tell them, “I’m sorry we can’t go to the field because there were demonstrations.” And then they ask, “Okay, and what about tomorrow?” And then I said, “well, let’s let tomorrow come first,” you know. They want so much to be sure about everything. They get concerned if there’s a day that maybe the attendance was not so good, and then we have to start explaining. Yes, because there was a funeral in the home that the person was facilitating, so we could not continue. You know, we have all these practical things, which it’s too far away from their lives. (Woman, Kenya, Public health NGO founder, 55+)

Challenges Accessing and Completing the MOOC

The lack of time to participate in the course due to obligations in other professional and personal roles, such as teaching and family caregiving, was a consistent barrier to completing the IR MOOC. Balancing the demands of the MOOC with other responsibilities also led to a rushed learning experience, hindering comprehension and retention of the information presented in the course.

I did it [the IR MOOC] two times. The first time I didn’t finish, because that was the time that COVID came. . . So I kind of dropped out from the first one, but then I saw it coming up again, and I joined again. But I didn’t have time to read all those articles. I literally quickly browsed through and learned some knowledge. I just wanted to get the skills. (Woman, Kenya, Public health NGO founder, 55+)

One participant felt that there was not enough time to complete the course and assignments, especially with their daily workload.

Unfortunately, despite my best efforts, I wasn’t able to complete the course. It takes a lot of time to do the required assignments. . . it’s not easy with everyday work. (Man, Central African Republic, Researcher, 26-45)

Language Barriers

The interviewees pointed out the language of instruction as a barrier to fully engaging with the information within the MOOC as well as the need to include examples from geographical and cultural contexts that they are more familiar with. The French and Spanish versions of the course videos were originally recorded in English and then subtitled in the second language, but there is a clear preference for additional context- and language-specific content.

The need for materials in Spanish, include regional experiences. Having the course translated directly by a Spanish-speaking person would allow more people to access it. (Woman, Colombia, Researcher, 26-35)

Experiences and content in Spanish would be more appropriate to better understand. If, for example, someone is doing it in Spanish, who finds content entirely in Spanish or French then it would be easier for them to assimilate [the information]. (Man, Haiti, Master’s Student, 26-35)

Additionally, supporting texts for the French and Spanish courses were in English, making them difficult to read for many learners.

I continued to investigate a little [on the MOOC platform] and almost all the texts were only in English. Sometimes it is difficult to learn a new science in a language that is not your native one. (Man, Dominican Republic, Public health worker, 36-45)

Strict Course Schedule and Deadlines

The MOOC’s structure, designed to be carried out over several weeks, was not only content-rich but also featured facilitated discussion forums and quizzes to enhance the learning experience. Each week, new sections were released, meaning learners needed to be steadily engaged in order to stay up to date with the content. However, for some participants, this schedule proved challenging:

I think the experience was good, but it is systematized by a schedule which prevents many from continuing. Right now, as many of these topics can be self-studied, by each person, this may happen at different times and perhaps at different speeds. We don’t have to close each topic on certain days and at certain times, this would have made everything easier, right? (Man, Mexico, ublic health worker, 55+)

Poor Internet Connectivity

Additional key challenges encountered by participants centered around poor internet connectivity. This technological barrier not only hindered consistent access to the video-based lectures, but also impacted the ability of participants to submit their quizzes and assignments on time:

Yes, the main problem in Afghanistan is weak internet connectivity. (Man, Afghanistan, Public health worker, 46-55)

The following quote comes from a participant who experienced some issues with connectivity, but was able to complete the MOOC due to it being asynchronous:

I used my laptop to access the course. I had a Wi-Fi connection at home, but sometimes it disconnected. I had to reconnect again, so that was the challenge. (Woman, Indonesia, Master’s student, 26-35)

Another participant stated that, although they were able to work around the connectivity challenges, it was not certain that all participants would be able to so:

However, the challenges that abound is switching between your digital devices and a laptop, because I had to download some of the videos using my laptop. Remember, I’m a media person, so I can be media savvy, but other people may not. (Man, Nigeria, PhD student, 36-45)

Furthermore, the costs of internet data needed to participant in the IR MOOC can also be a financial burden:

And these [course content] are on the internet. Like we say jokingly here, that the internet is a new scam. The price you pay for data is a new scam. It’s so expensive, and a lot of people do not have access to data. (Man, Nigeria, PhD student, 36-45)

Lack of Diverse Range of Case Studies

One participant highlighted the relatively limited range of examples and case studies used in the MOOC, and that the ones presented were not always relevant to their context in East Africa. They suggested using additional examples taken from contexts globally to appeal to a wider audience:

What I found a little bit irritating in the course was, there was a lot of examples about ivermectin, and that it’s not so common in this area. And that was like, of course, there we go again with another ivermectin story. So, a bit more diverse and looking at what is in East Africa. What is in Asia. What is in western Africa. To make it appeal to everyone in the world. (Woman, Kenya, Public health NGO founder, 55+)

Participants in the Spanish language version of the IR MOOC would have liked to have seen more examples from Latin America. This would have made the MOOC more engaging and may have encouraged more local networking:

Usually when there are courses a little bit European centric, then they tend to forget a little of bit about examples from Latin America. Then they focus on Africa, Asia, and that is what happened to me. . . There were some. But maybe improve a little bit more. Try to make a little bit more examples of the region. (Man, Guatemala, Researcher, 36-45)

With the new experiences that are being developed on other continents, it could also be very useful to incorporate examples of research carried out in our context, Latin America, for example. I think that this can enrich and make the context of application of the examples and proposals made in the course better. (Man, Ecuador, Professor, 55+)

Add a little experience from the regions. For example, if the program is going to be offered in Colombia, include an experience with leishmaniasis. Include a researcher in the region, to feel closer. (Woman, Colombia, Student/Professor, 26-35)

Lastly, participants would have liked more varied examples from within the health system:

There must be, so to speak, more than an example of some problem in some community, problems of the health systems of a country. For example, the issue of public hospital administration. (Man, Honduras, Researcher, 36-45)

Discussion

This study investigated the complex web of challenges encountered by learners after participating in the IR MOOC in French, Spanish, and English. We examined the issues learners faced while attempting to engage with the course and their desire for additional and more in-depth IR training opportunities. Many of the challenges in applying their IR knowledge professionally, such as the MOOC serving an introductory role in IR training, time constraints while taking the course, and language barriers, mirror the obstacles common to online learning.²¹ MOOCs are by definition easily accessible, and generally aim to provide a baseline learning opportunity that can help participants decide if further commitment in the area would be useful to build a thorough understanding of a complex topic.

At the organizational level, resistance to change and lack of understanding of IR’s value by supervisors were identified as persistent challenges. This resistance is typically compounded by hierarchical institutional structures where many MOOC learners find themselves working or studying.²² These environments are less conducive to “bottom-up” innovation, and make adopting and integrating new methodologies difficult.

Community-level barriers included a lack of IR-related job opportunities and limited avenues to apply the skills and knowledge acquired through the MOOC. These issues are reinforced by the resistance to change and lack of understanding of IR previously mentioned. To many, IR comes with the reputation as a still-burgeoning field. Together, this results in a gap between the supply of those interested in pursuing IR as a career and the demand for IR professionals. The IR MOOC remains an introductory course and is not designed to be the stopping point in an individual’s IR learning journey. Rather, the course should be one at the start of a learner’s engagement with IR training content, yet, approval and encouragement from supervisors would be meaningful as learners contemplate taking further steps in becoming more highly trained in IR methodologies.

The barriers at the health authority level, including limited availability of funding for IR projects, reflect the broader issues impacting health research within many LMICs.²³ These challenges highlight the critical role of policy and advocacy in shaping the future of IR. Developing cross-institutional collaborations, where researchers and health practitioners can collaborate on projects and learn from each other are limited yet in demand. Concentrated efforts, for example, social media campaigns, to raise the awareness of the value of IR among individuals within leadership positions will help elevate the profile of IR internationally and can spur demand for IR skill-building initiatives at governmental and academic research institutions.

The need for mentorship and expert guidance consistently emerged as a crucial element missing from learners’ experiences. Participants expressed a clear desire for more direct interaction with IR professionals as well as with other learners, highlighting how gaps can remain between theoretical knowledge and practical application, particularly in online learning environments. In a recent series of trainings focused on IR for non-communicable diseases, engagement with the faculty and other learners was a highly rated component of the course, with approximately 84% of these participants indicating that they intended to leverage the connections they made after the course was finished.²⁴ These findings align with educational theories emphasizing the need for guided and interactive components to enhance educational experiences.²⁵ A more pointed effort to include mentorship within the IR MOOC could increase personalized feedback and facilitate deeper understanding through helping learners navigate complex IR concepts.²⁶ The integration of such support mechanisms has been limited in the context of MOOCs but are particularly vital where access to IR expertise may be limited.

Participants reported several pervasive challenges while taking the MOOC itself. Language barriers were persistent, as the course videos were developed originally in English and later subtitled in other languages. This led learners in the French and Spanish versions of the course wanting for more content originally produced in the language of delivery, including the supplemental texts and presentation of the slides within the videos themselves—the text on these slides remained in English within the subtitled versions of the course. Poor internet connectivity was also a hinderance to remaining engaged in the course. Learners in LMICs must often manage with lower quality connections and may experience greater relative cost for mobile internet data than learners in high-income contexts, which can make the IR MOOC financially inaccessible, even if the course itself is free of charge. Electrical brownouts also interfere with online learning as they typically cause internet interruptions and are commonly encountered in some resource limited settings.²⁷

Of special note is learners’ desire for additional, contextualized case examples that are both geographically diverse and describe research on a broad range of diseases. This finding has also been described previously,¹⁴ and, in response to this, TDR is now in the process of releasing strategic case examples from Ecuador, Ethiopia, and Nepal to accompany future sessions of the IR MOOC. Through adding in-depth focus on critical diseases and contexts, learners may be stimulated to engage in further education or research linked to these.

Our findings underscore the need for strategic improvements in the design and delivery of future online training initiatives, particularly for learners in LMICs, to achieve health impacts. The results align with those from a recently conducted evaluation of a water, sanitation, and hygiene MOOC targeting learners in LMICs.²⁸ Here, the authors conclude that improvements are needed around making the course accessible in the face of expensive yet unreliable internet connections, providing the materials in additional languages, and improving contextualization through meaningful partnerships with collaborators in LMICs. The next generation of IR training for learners living and working in LMICs should similarly prioritize creating more interactive, engaging, and contextually relevant learning experiences. TDR’s newly created case examples and teaching videos featuring native French and Spanish speakers cater specially to the global audience who access the IR MOOC. Such strategic adaptations are essential in maintaining engagement among participants and in ensuring learners are empowered to bridge the gap between the theoretical knowledge gained through the course and practical application of this knowledge to real-world health settings to work toward the ultimate goal of health impact.

Addressing low awareness of IR and resistance to adopt IR methodologies within organizations is critical for the broader utilization of IR and to generate demand for professionals with these research skills. Comprehensive awareness-raising campaigns aimed at leadership and staff within governmental and university research organizations are a first step in educating decision-makers about the value and potential impacts of IR. Such sensitization work is now underway at TDR, utilizing multimedia channels to emphasize how IR methodologies can directly impact an organization’s goals, and showcasing successful examples of how IR has improved the health of communities. This approach aims to increase awareness of and demand for IR skills as well as influence the shift of an organization’s culture toward a more receptive view on IR.

Conclusion

In conclusion, our study interrogated the multiple barriers faced by learners in integrating the IR knowledge from the MOOC into their professional work. The findings demonstrate the need for greater engagement with IR communities, localized content, and opportunities to apply IR skills in practice. Addressing access issues, such as those related to internet connectivity and language barriers, are needed to ensure learners remain engaged in and motivated to complete the course. Additionally, raising the profile of IR with research organizations and overcoming resistance to adopting IR methodologies are critical in broadening the adoption of this valuable and potentially lifesaving approach. By creating solutions to these barriers now, as TDR has begun doing, future IR training programs can be better aligned with the needs of learners in disease-endemic settings. It would be possible to apply IR frameworks to modifying future iterations of the IR MOOC, such as the RE-AIM (reach, effectiveness, adoption, implementation, maintenance) framework or the Consolidated Framework for Implementation Research (CFIR).²⁹ These well-established IR frameworks could help to align adaptations of the content and delivery of the course to the organizational and contextual factors of learners in LMICs. Regardless of the exact approach, future revisions of the IR MOOC will focus on further strengthening of health research capacity, positively impacting community health outcomes.

Study Limitations

Among limitations worth noting, this study relies on self-reported data from a self-selected subset of the MOOC participants and may not convey the full range of experiences and challenges encountered by learners. While we have included learners who engaged with the French, Spanish, and English language versions of the IR MOOC, these findings may not be generalizable to participants of all sessions of the online course. Additionally, our rich qualitative data does not expressly quantify the degree of impact of each of the barriers investigated. Lastly, we acknowledge that gaining the perspectives of employers and organizations who utilize IR methodologies would have strengthened the study.

Supplemental Material

sj-docx-1-inq-10.1177_00469580241284916 – Supplemental material for Barriers to Applying Knowledge Gained Through an Implementation Research Massive Open Online Course: An Explanatory Qualitative Study

Supplemental material, sj-docx-1-inq-10.1177_00469580241284916 for Barriers to Applying Knowledge Gained Through an Implementation Research Massive Open Online Course: An Explanatory Qualitative Study by Michael J. Penkunas, Bella Ross, Charlotte Pana Scott, Anna Thorson, Luis Fernando Baron, Wafa Kammoun Rebai, Hind Bouguerra and Pascal Launois in INQUIRY: The Journal of Health Care Organization, Provision, and Financing

Footnotes

Acknowledgements

The authors would like to thank the participants for their time and candor.

Author Contributions

PL, MJP, and BR conceived of the study. MJP drafted the initial form with assistance from BR. MJP, BR, CPS, LFB, WKR, and HB contributed to the data analysis and interpretation. AT contributed to the interpretation of the data and modified drafts.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This evaluation was internally funded by the Special Programme for Research and Training in Tropical Diseases (TDR).

Ethical Statement

This study was reviewed and approved by the World Health Organization’s Research Ethics Review committee (protocol ID: ERC.0003785). All participants provided written informed consent.

ORCID iDs

Michael J. Penkunas

Luis Fernando Baron

Supplemental Material

Supplemental material for this article is available online.

References

Peters

Adam

Alonge

Agyepong

Tran

. Implementation research: what it is and how to do it. BMJ. 2013;347:f6753. doi:10.1136/bmj.f6753

Bauer

Kirchner

. Implementation science: what is it and why should I care? Psychiatry Res. 2020;283:112376. doi:10.1016/j.psychres.2019.04.025

Ridde

. Need for more and better implementation science in global health. BMJ Glob Health. 2016;1(2):e000115. doi:10.1136/bmjgh-2016-000115

Sheikh

Agyepong

Jhalani

, et al. Learning health systems: an empowering agenda for low-income and middle-income countries. Lancet. 2020;395(10223):476-477. doi:10.1016/S0140-6736(19)33134-4

Turner

Bogdewic

Agha

, et al. Learning needs assessment for multi-stakeholder implementation science training in LMIC settings: findings and recommendations. Implement Sci Commun. 2021;2(1):134. doi:10.1186/s43058-021-00238-2

Davis

D’Lima

. Building capacity in dissemination and implementation science: a systematic review of the academic literature on teaching and training initiatives. Implement Sci. 2020;15(1):97-26. doi:10.1186/s13012-020-01051-6

Galaviz

Narayan

KMV

Manders

, et al. The public health leadership and implementation academy for noncommunicable diseases. Prev Chronic Dis. 2019;16:E49. doi:10.5888/pcd16.180517

Osanjo

Oyugi

Kibwage

, et al. Building capacity in implementation science research training at the University of Nairobi. Implement Sci. 2016;11(1):30. doi:10.1186/s13012-016-0395-5

Theobald

Brandes

Gyapong

, et al. Implementation research: new imperatives and opportunities in global health. Lancet. 2018;392(10160):2214-2228. doi:10.1016/S0140-6736(18)32205-0

10.

Balane

Palafox

Palileo-Villanueva

McKee

Balabanova

. Enhancing the use of stakeholder analysis for policy implementation research: towards a novel framing and operationalised measures. BMJ Glob Health. 2020;5(11). doi:10.1136/bmjgh-2020-002661

11.

Doherty

Lewin

Kinney

, et al. Addressing the tensions and complexities involved in commissioning and undertaking implementation research in low- and middle-income countries. BMJ Glob Health. 2018;3(4):e000741. doi:10.1136/bmjgh-2018-000741

12.

World Health Organization, Ogundahunsi

Kamau

. Implementation Research Toolkit. 2nd ed. World Health Organization; 2018. Accessed January 26, 2024. https://iris.who.int/handle/10665/363749

13.

Launois

Maher

Certain

Ross

Penkunas

. Implementation research training for learners in low- and middle-income countries: evaluating behaviour change after participating in a massive open online course. Health Res Policy Syst. 2021;19(1):59. doi:10.1186/s12961-021-00703-3

14.

Launois

Allotey

Reidpath

Maher

Certain

Ross

. Lessons Learnt from a Professional Development MOOC: Engaging Culturally and Linguistically Diverse Learners from Low- and Middle-Income Countries. Eur J Open Distance E-Learn. 2019;22 (2): 1-16. Accessed October 25, 2022. https://www.academia.edu/40470523/Lessons_learnt_from_a_professional_development_MOOC_Engaging_culturally_and_linguistically_divers_learners_from_low_and_middle_income_countries

15.

Ross

Penkunas

Maher

Certain

Launois

. Evaluating results of the implementation research MOOC using Kirkpatrick’s four-level model: a cross-sectional mixed-methods study. BMJ Open. 2022;12(5):e054719. doi:10.1136/bmjopen-2021-054719

16.

Nwameme

Dako-Gyeke

Asampong

, et al. Improving understanding of disease control implementation research through a mooc with participants from low- and middle-income countries: evaluating participant reactions and learning. PLoS Negl Trop Dis. 2023;17(3):e0011139. doi:10.1371/journal.pntd.0011139

17.

Allotey

Reidpath

Certain

, et al. Lessons learned developing a massive open online course in implementation research in infectious diseases of poverty in low- and middle-income countries. Open Prax. 2021;13(1):127. doi:10.5944/openpraxis.13.1.1172

18.

Penkunas

Ross

Echavarria

, et al. Professional outcomes and barriers experienced by participants of an implementation research massive online open course: descriptive results from French, Spanish and English speaking learners in low- and middle-income settings. Under Review.

19.

Kirkpatrick

. Evaluating Training Programs: The Four Levels. Berrett-Koehler Publishers; 2006.

20.

Thomas

. A general inductive approach for analyzing qualitative evaluation data. Am J Eval. 2006;27(2):237-246. doi:10.1177/1098214005283748

21.

Gillett-Swan

. The challenges of online learning: supporting and engaging the isolated learner. J Learn Des. 2017;10(1):20-30. Accessed January 29, 2024. https://eprints.qut.edu.au/102750/

22.

Chandler

. Reasons and forms of organizational resistance to change in the higher education sector. Pract Theory Syst Educ. 2010;5(1):87-104.

23.

Beran

Byass

Gbakima

, et al. Research capacity building-obligations for global health partners. Lancet Glob Health. 2017;5(6):e567-e568. doi:10.1016/S2214-109X(17)30180-8

24.

Aziz

Haregu

Kyobutungi

, et al. Strengthening capacity for implementation research amid COVID-19 pandemic: learnings from the global alliance for chronic diseases implementation science school. Int J Public Health. 2022;67: 1604944. doi:10.3389/ijph.2022.1604944

25.

Cavinato

Hunter

Ott

Robinson

. Promoting student interaction, engagement, and success in an online environment. Anal Bioanal Chem. 2021;413(6):1513-1520. doi:10.1007/s00216-021-03178-x

26.

Penkunas

Chong

Rhule

ELM

Berdou

Allotey

. Designing the next generation of implementation research training for learners in low- and middle-income countries. Global Health. 2021;17(1):63. doi:10.1186/s12992-021-00714-3

27.

Pasha

Abidi

Ali

. Challenges of offering a MOOC from an LMIC. Int Rev Res Open Distance Learn. 2016;17(6):221-228. doi:10.19173/irrodl.v17i6.2696

28.

Suter

Lüthi

. Delivering WASH education at scale: evidence from a global MOOC series. Environ Urban. 2021;33(1):99-116. doi:10.1177/0956247820987759

29.

King

Shoup

Raebel

, et al. Planning for implementation success using RE-AIM and CFIR frameworks: a qualitative study. Front Public Health. 2020;8:59. doi:10.3389/fpubh.2020.00059

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.01 MB