Long-Term Effectiveness of an E-Learning Program in Improving Health Care Professionals' Attitudes and Practices on Breastfeeding: A 1-Year Follow-Up Study

Abstract

Introduction:

In-service continuing education offers a unique opportunity to improve knowledge, skills, attitudes, and practices regarding breastfeeding. It has been shown that an online approach to in-service education is effective at improving practices and attitudes toward breastfeeding among health care professionals (HCPs) in the short term.

Aim:

To evaluate the long-term effectiveness of an online national program on infant nutrition for HCPs.

Materials and Methods:

We carried out a follow-up study using data from three time points: T0 (pretraining), T1 (immediately post-training), and T2 (1 year after training). The differences between T0, T1, and T2 were tested using repeated-measures ANOVA. Statistical analysis was performed using SPSS version 22.0.

Results:

The final sample was comprised of 4,582 participants, mainly women (87.4%). At T2, we observed a worsening of attitudes and practices (APs) as compared with T1, though those APs almost never reached the low levels observed at T0. The greatest changes over time concerned the use of drugs during breastfeeding (T0: 3.00 ± 1.33 versus T1: 1.74 ± 1.03 versus T2: 2.64 ± 1.35) and dietary restriction (T0: 2.77 ± 1.35 versus T1: 1.76 ± 1.12 versus T2: 2.57 ± 1.35). The differences between the means of APs at T0, T1, and T2 were significant (p < 0.01).

Conclusion:

This e-learning program was effective in improving APs regarding the protection, promotion, and support of breastfeeding. The improvement, higher immediately after training, decreased over time. E-learning project managers should propose strategies to facilitate the retention of knowledge related to the main training objectives.

Introduction

Although a strong knowledge of breastfeeding by health care professionals (HCPs) is pivotal for the success of breastfeeding, a widespread lack of training on this topic is reported in the literature.^1,2 HCPs' preservice education on breastfeeding is often limited to a few hours of curricular content, which is clearly insufficient to ensure the effective support of mothers.³ Therefore, in-service continuing education (CE) is in most cases a unique opportunity to improve knowledge, skills, attitudes, and practices regarding breastfeeding. Since 1991, the World Health Organization (WHO) and the United Nations Children's Fund (UNICEF) have promoted courses on breastfeeding within the framework of Baby-Friendly Initiatives (BFIs) tailored to “informed,” “involved,” and “dedicated” health personnel.⁴

Innovative teaching strategies such as e-learning⁵ can successfully support BFIs, and prior studies have shown that online methods are likely to be just as effective as alternative methods for training HCPs.^6–8 In light of this, the Italian National Institute of Health, UNICEF, and the Local Health Authority of Milan have instituted a national educational program on infant nutrition based upon e-learning. While this online approach has been effective at improving short-term practices and attitudes toward breastfeeding among HCPs who have completed the course,⁹ there is as of yet little evidence that e-learning is effective at producing long-term changes to practices and attitudes.^7,10

To evaluate the long-term effectiveness of this online national program on infant nutrition, a follow-up study was conducted that measures the extent to which changes in attitudes and self-reported practices are maintained one year after training.

Materials and Methods

Design

This follow-up study measured attitudes and self-reported practices at three time points: T0 (pretraining), T1 (immediately after training), and T2 (1 year after training) (Fig. 1).

FIG. 1.

Flowchart of sample selection.

Participants

In June 2015, HCPs included in the T1 sample (N = 15,004) were invited by e-mail to participate in a follow-up study. In cases of acceptance, participants could access an online questionnaire.

Tools

Data collection at T2 was performed using the attitudes and practices (APs) questionnaire administered at T0 and T1,⁹ a satisfaction questionnaire, and a questionnaire for the training needs assessment on advanced skills for breastfeeding support.

The AP questionnaire items focused on the duration and exclusivity of breastfeeding, the use of medications, maternal nutrition (so-called food taboos), health risks derived from nonbreastfeeding and physiology and normal management of lactation. The attitudes section of the AP questionnaire consists of 10 items measured with a 5-point Likert scale, ranging from “1—totally agree” to “5—completely disagree” (Cronbach's alpha = 0.76). Lower scores indicated more positive attitudes toward breastfeeding. The practices section consists of 10 questions measured with a 4-point Likert scale ranging from “1—often” to “4—never/I have no chance” (Cronbach's alpha = 0.94). Lower scores indicated more frequent professional practices.

The satisfaction questionnaire was composed of 3 items measured with a 5-point Likert scale and focused on relevance (from “1—nonrelevant” to “5—very relevant”), quality (from “1—poor” to “5—excellent”), and effectiveness (from “1—not effective” to “5—very effective”). Satisfaction was also indirectly explored by asking users if they would suggest the same online course to their colleagues and if they would apply for an advanced breastfeeding training course based upon the same methodology. When participants answered in the affirmative, a training needs assessment was carried out by asking those participants to indicate their preferences within a list of advanced skills.

Ethical consideration

In accordance with Italy's institutional regulations regarding this type of observational survey, ethics committee approval was not requested. In accordance with Italian legislation on data protection, participants were informed about and accepted the use of anonymous data.

Data analysis

The demographic data and variables related to training needs assessment were described using frequency and descriptive statistics. The total average scores of APs and the average scores for each item were calculated for time points T0, T1, and T2. Repeated-measures ANOVA was used to test the differences between T0, T1, and T2 mean scores, and a conservative correction factor was applied to take into account the lack of independence of observations. Statistical analysis was performed using SPSS software (version 22, SPSS, Inc., Chicago, IL). By convention, p-values <0.05 were considered significant.

Results

Demographic characteristics

At T0, the AP questionnaire was completed by 26,009 HCPs, while at T1 and T2 it was completed by 15,004 and 4,963 HCPs, respectively (T2 response rate: 33.1%). Considering the HCPs who responded comprehensively to all AP items, the final sample at T2 comprised of 4,582 users (Fig. 1). Of these, 4,456 completed the questionnaire in full, including satisfaction items and the training needs assessment section.

The first five most represented HCPs are nurses (N = 1,820), midwives (N = 1,067), physicians (N = 632), pharmacists (N = 195), and pediatric nurses (N = 189). Respondents were mostly women (87.4%), and their mean age was 39 years (Table 1).

Table 1.

Characteristics of the Cohort (N = 4582)

Variable	Mean ± SD	n (%)
Age	37.0 ± 9.9
Gender
Female		4004 (87.4)
Male		578 (12.6)
Health care professionals
Nurses		1820 (39.7)
Midwives		1067 (23.3)
Physicians		632 (13.8)
Pharmacists		195 (4.3)
Pediatric nurses		189 (4.1)
Health care assistants		152 (3.3)
Radiology technicians		86 (1.9)
Medical laboratory technicians		86 (1.9)
Dietitians		82 (1.8)
Physiotherapists		78 (1.7)
Other health care professionals^a		195 (4.2)

Biologists, psychologists, speech therapists, neuropsychomotor developmental therapists, professional educators, dentists, technicians in prevention techniques for the environment and the workplace, psychiatric rehabilitation therapists, orthoptist/ophthalmic assistants, technicians in neurophysiopathology, occupational therapists, audiology technicians, physicists.

Attitudes and practices

We found a general increase of the averages at T2 compared with the values obtained at T1, corresponding to less positive attitudes and less frequent practices. However, at T2 we almost never obtained values higher than those obtained at T0 (Table 2).

Table 2.

Attitudes and Practices (N = 4582)

	Pretest, mean score ± SD	Post-test, mean score ± SD	Follow-up, mean score ± SD
Attitudes^*
In general, I think that:
1. Nonbreastfed infants have greater health risks (otitis, diabetes, asthma, cardiovascular diseases) compared with breastfed infants^a	1.98 ± 1.04	1.58 ± 0.93	1.78 ± 1.02
2. In the first six months, babies should take only human milk^a	1.77 ± 1.08	1.26 ± 0.63	1.63 ± 1.02
3. Breastfeeding can continue beyond one year of life^a	1.77 ± 0.94	1.46 ± 0.80	1.70 ± 0.93
4. Breastfeeding can continue beyond baby's second year of life^a	2.69 ± 1.29	2.12 ± 1.23	2.62 ± 1.31
5. Most medications are compatible with breastfeeding^a	3.00 ± 1.33	1.74 ± 1.03	2.64 ± 1.35
6. If a breastfeeding mother has a pain, it is better to endure it and to avoid taking medications^a,b	2.39^* ± 1.06	2.18^* ± 1.15	2.18^* ± 1.05
7. Manual expression of human milk is an effective method to maintain its production^a	2.41 ± 1.10	1.94 ± 1.10	2.26 ± 1.11
8. Many foods, that is, garlic, onion, cabbage, broccoli, must be avoided during breastfeeding^a,b	2.77^* ± 1.35	1.76^* ± 1.12	2.57^* ± 1.35
9. Baby-led breastfeeding is the normal way to manage the feeds^a	1.75 ± 0.98	1.44 ± 0.79	1.64 ± 0.93
10. If after being breastfed the baby does not seem satisfied, it is advisable to offer a complement of human milk substitute^a,b	2.43^* ± 1.20	2.01^* ± 1.21	2.30^* ± 1.20
Practices^**
In my professional practice:
1. I actively support breastfeeding^a	1.88 ± 1.10	1.78 ± 1.07	1.64 ± 1.09
2. I inform women about the risks associated with the use of human milk substitutes and not breastfeeding^a	2.03 ± 1.09	1.87 ± 1.07	1.95 ± 1.13
3. I inform women about the risks related to tobacco use and to the exposure to secondhand smoke during pregnancy and lactation^a	1.66 ± 0.97	1.68 ± 0.97	1.58 ± 0.98
4. I provide information on legislation for working mothers and on policies that support breastfeeding in public places^a	2.31 ± 1.11	2.15 ± 1.12	2.18 ± 1.15
5. I observe and assess the breastfeeds^a	2.31 ± 1.24	2.23 ± 1.26	2.30 ± 1.30
6. I demonstrate to women how to express human milk manually^a	2.53 ± 1.18	2.40 ± 1.22	2.48 ± 1.24
7. I offer information on medication use during lactation using reliable sources when necessary^a	2.26 ± 1.08	2.12 ± 1.09	2.18 ± 1.13
8. I use counseling skills in supporting pregnant women and mothers^a	1.94 ± 1.12	1.87 ± 1.12	2.04 ± 1.21
9. I adhere to the International Code of Marketing of Breast milk Substitutes^a	2.19 ± 1.26	1.87 ± 1.19	1.96 ± 1.23
10. I offer information on the community care network (family pediatrician, community mother and child centers, peer support, etc.)^a	1.99 ± 1.12	1.88 ± 1.11	1.92 ± 1.14

p < 0.01 significant differences between T0, T1, and T2, F-test of ANOVA for repeated measures.

Inverted score.

Likert scale: “1, completely agree”; “2, agree”; “3, neither agree nor disagree”; “4, disagree”; “5, completely disagree.”

Likert scale: “1, often”; “2, sometimes”; “3, never”; “4, never/I don't have the opportunity.”

With regard to particular items on the attitudes, the greatest differences between averages were found with regard to item number five, “Most drugs are compatible with breastfeeding” (T0: 3.00 ± 1.33 versus T1: 1.74 ± 1.03 versus T2: 2.64 ± 1.35; p < 0.05), and item number eight, “Many foods, such as garlic, onions, cabbage, broccoli, should be avoided during lactation” (T0: 2.77 ± 1.35 versus T1: 1.76 ± 1.12 versus T2: 2.57 ± 1.35; p < 0.05). Almost stable values at T1 and T2 emerged for attitude item number six, “If a nursing mother has a pain it is better to bear rather than take drugs” (T0: 2.39 ± 1.06 versus T1: 2.18 ± 1.15 versus T2: 2.18 ± 1.05; p < 0.05).

On the contrary, in the evaluation of the practices there are two cases in which the improvement has been growing at T2: practice number one, “I feel actively involved in breastfeeding support” (T0: 1.88 ± 1.10 versus T1: 1.78 ± 1.07 versus T2: 1.64 ± 1.09; p < 0.05); practice number three, “I inform women about the risks associated with smoking and exposure to passive smoking during pregnancy and lactation” (T0: 1.66 ± 0.97 versus T1: 1.68 ± 0.97 versus T2: 1.58 ± 0.98; p < 0.05).

Practice item number eight, “I use active listening skills to support pregnant women and mothers” (T0: 1.94 ± 1.12 versus T1: 1.87 ± 1.12 versus T2: 2.04 ± 1.21; p < 0.05), was the only one for which the average score increased when compared with both T1 and T0.

Level of satisfaction

The satisfaction levels declared at T2 were as follows: the topic was rated as relevant or very relevant or by 82.8% of the participants, 91.8% rated the quality of the course as good or excellent, and 80.7% rated the training as effective or very effective. About 79.2% of respondents indicated that they did not take part in any breastfeeding refresher courses, while 20.2% indicated that they did.

Furthermore, 98.7% of professionals would suggest to their colleagues that they take part in the same breastfeeding course they had, and 92% would participate in another course on advanced breastfeeding skills.

Training needs regarding breastfeeding

Regarding the effectiveness of the training methods used for the course on advanced skills on breastfeeding, it was found that e-learning was considered effective by 87.3% of professionals, while a mixed course (e-learning and training in attendance) and a residential course were considered by 81.8% and 79.5%, respectively.

The topics HCPs would like to treat in a course on advanced breastfeeding skills mainly concern the use of drugs (97.3%) and breastfeeding in premature infants (89.5%). Breastfeeding and work and child diseases were chosen by 85.5% and 85.2% of participants, respectively. Good interest has also been shown in positioning, latch-on and effective suckling (83.8%), maternal diseases (80.6%), and use of natural products (74.9%), while code, ethics, and conflict of interest were chosen by less than half of the sample. Figure 2 shows that most participants would prefer for the advanced course to be conducted through e-learning rather than through a traditional residential course. Only in one case (positioning, latch-on and effective suckling), HCPs would prefer a traditional residential course rather than e-learning (Fig. 2).

FIG. 2.

Training needs assessment (N = 4518).

Discussion

Our study, in line with the available literature, shows a general decrease in APs developed over time, but these APs remain higher than they were before training. Furthermore, this trend applies differently to several topics and is mainly coherent with the training needs assessment.

The course has increased the involvement of participants in breastfeeding support, confirmed by a positive trend over time, improved upon at T1, and further improved upon at T2. Regarding the main findings on attitudes, the best results, already appreciable before the course, concern the exclusivity of breastfeeding for the first six months, on demand and continued breastfeeding for over a year. These results further improved immediately after training and then decreased after one year, while still remaining higher than at T0. In general, we can note an overall worsening of the level of APs at T2 as compared with T1, although (with some exceptions) without ever reaching the scores obtained before training. This finding is in line with the literature on learning retention, which shows that knowledge learned during a course tends to be lost over a short period of time.^11,12 Moreover, because practical applications are important for learning, the lack of practice opportunities causes a decrease of knowledge and skills.¹³

Regarding the use of medications during breastfeeding, the idea that lactating mothers should “endure the pain” rather than taking a drug persists after 1 year. While HCPs did declare at the end of the course that they more often offered information from reliable sources concerning drugs and their compatibility with breastfeeding, this practice was only partially retained at T2, though it did remain higher than at T0, a finding also in line with the literature on long-term retention. The training needs assessment confirms that the use of drugs during breastfeeding arouses great interest among HCPs. In fact, the literature shows that HCPs need to be supported by evidence-based information on lactation and maternal drugs, probably due to the lack of specific preservice education on this topic.^14–16

Moreover, in our training needs assessment, interest has been expressed regarding the use of natural products during breastfeeding, suggesting that HCPs know too little about this topic.

Also of interest is compliance with the International Code on the Marketing of Breast Milk Substitutes (the Code).¹⁷ The significant improvement found at T1 was partially retained at T2, while maintaining a higher level than at T0. However, the code, ethics, and conflict of interest seem not to be perceived as a pivotal training need by participants, as this item obtained a low score among areas for future training. To strengthen enforcement of the code, it is thus recommended that HCPs be further trained regarding lactation and breastfeeding protection.^18,19

Following the same trend as the previous topics are professional attitudes regarding both dietary restrictions during breastfeeding and the practices of offering information about the community care network (family pediatricians, community health centers, mother-to-mother support groups) and about legislation for working mothers and policies that support breastfeeding in public places. A specific training need has been expressed by several HCPs concerning breastfeeding management for working mothers.

Regarding the practice of informing women about the risks related to tobacco consumption and to exposure to passive smoking during pregnancy and lactation, while no improvement occurred at T1, an improvement was found at T2. This could be the result of crosscutting factors, such as the recent community policies to combat smoking,²⁰ which were the subject of discussion in Italy throughout 2015 and were finally implemented in 2016. That relevant legislation includes setting additional antismoking limits. In particular, smoking is prohibited in hospitals, research centers, and cars when minors or pregnant women are present.

There was only a clear decrease when it came to the use of active listening skills to support pregnant women and mothers. As expected, a slight improvement was reported at T1, followed at T2 by a reduction beyond even the levels found at T0. “Active listening” is a characteristic of counseling, defined by WHO in 1989 as “a process that, through dialogue and interaction, helps people to solve and manage problems and make decisions.”²¹ Counseling is an activity that tends to guide, support, and develop the potential of the person, promoting active, proactive attitudes and stimulating the ability to choose. As such, it can only be a “relational” process, the learning of which must necessarily pass through an interaction. The reason for this negative finding could be sought in the specific training method used (low-interaction e-learning, absence of individual or group practice). In fact, while the course offered a theoretical explanation of the basic counseling techniques, HCPs had no opportunity to experience them. One of the limits of e-learning is the lack of interaction, so the training objectives and the related topics should be set taking into account the methodology that will be adopted. The same assumption is applicable also for the following two practical skills: observing and assessing a breastfeed (including laid-back positioning) and demonstrating how to express human milk manually. This result suggests that passive viewing of video and multimedia materials is not sufficient to improve practical and communication skills. In fact, interactive practice exercises, repetition, and feedback play pivotal roles in e-learning and seem to be associated with improved learning outcomes.²²

The training needs expressed by participants seem to support this hypothesis. In fact, concerning the advanced course, an e-learning methodology is preferred in all cases except for positioning, latch-on and effective suckling, for which HCPs are more likely to choose a traditional residential course. In line with the two previous cases, this is a specific practice that requires technical skills.

It is interesting to point out that one of the lowest scores before training concerns the attitude toward the continuation of breastfeeding for 2 years or for as long as the mother and child desire is. Although it improves after the course, this attitude, like some others, tends to return to its pretraining level.

According to the literature, one-third of knowledge learned for the first time but never used, revisited, or repeated will be lost within 1 year, and half or more will be forgotten within 2 years. After 2 years, the process of knowledge loss continues, but in a more gradual way.¹¹ If, on the contrary, knowledge is used after learning, our retention of it increases while the speed with which it is lost slows down. Furthermore, in terms of quality, relearning information is more effective than acquiring information for the first time. We can thus achieve better long-term retention of knowledge by using active recall.

E-learning project managers should perhaps include strategies aimed at strengthening the retention of knowledge related to the main training objectives. Using reminders at specific intervals could be an effective strategy not only during the course but also post-training. Offering participants the opportunity to receive key messages in the months following the end of the course should also be on the e-learning project managers' agenda. Post-training reminders based on alternative methods (quiz challenges or games like crosswords) could be sent through email to participants who have chosen this option during their initial registration. In the absence of sound evidence,²³ one suggestion for future e-learning programs would be to build and test post-training reminders to maintain a high level of knowledge of key concepts.

Limitations of the study

This study is subject to certain limitations, including the lack of a control group and a selection bias due to the voluntary participation of the HCPs.

According to the CE standard procedures, knowledge scores were measured only post-test, and for this reason we could not assess the degree of improvement between the period before training and after. Health care performance and health outcomes were not measured due to the geographical (national) distribution of participants.

Although the sample at T2 is just a third the size of the sample at T1, the response rate is in line with that of other web-based surveys.^23,24 Further studies are needed to evaluate the effectiveness of long-term e-learning to find useful strategies for maintaining the improvement in knowledge, skills, and practices that is found in the short term, immediately after training.

Conclusion

The known delivery advantages of an e-learning program include lower costs, widespread distribution, increased accessibility to information, frequent content updates, and personalized instruction in terms of content and pace of learning.²² This e-learning program for HCPs seems effective in improving APs regarding the protection, promotion, and support of breastfeeding. According to the literature, this kind of improvement, which is greater immediately after online training, tends to decrease over time.²² For this reason, learning retention should be emphasized by any educational program, in person or online, and every effort should be made to link educational interventions with the past experiences and specific needs of adult learners.²² Among other strategies, training managers could limit the physiological loss of knowledge by using simple reminder methods at regular intervals.

Footnotes

Disclosure Statement

No competing financial interests exist.

Funding Information

No funding was received for this article.

References

Pound

, Williams

, Grenon

, et al. Breastfeeding knowledge, confidence, beliefs, and attitudes of Canadian physicians. J Hum Lact, 2014; 30:298–309.

Spear

. Baccalaureate nursing students' breastfeeding knowledge: A descriptive survey. Nurse Educ Today, 2006; 26:332–337.

Webber

, Serowoky

. Breastfeeding curricular content of family nurse practitioner programs. J Pediatr Health Care, 2017; 31:189–195.

Bettinelli

, Chapin

, Cattaneo

. Establishing the Baby-Friendly Community Initiative in Italy: Development, strategy, and implementation. J Hum Lact, 2012; 28:297–303.

Semenic

, Childerhose

, Lauzière

, et al. Barriers, facilitators, and recommendations related to implementing the Baby-Friendly Initiative (BFI): An integrative review. J Hum Lact, 2012; 28:317–334.

Lathi

, Hätönen

, Välimäki

. Impact of e-learning on nurses' and student nurses knowledge, skills, and satisfaction: A systematic review and meta-analysis. Int J Nurs Stud, 2014; 51:136–149.

Richmond

, Copsey

, Hall

, et al. A systematic review and meta-analysis of online versus alternative methods for training licensed health care professionals to deliver clinical interventions. BMC Med Educ, 2017; 17:1–14.

McCutcheon

, Lohan

, Traynor

, et al. A systematic review evaluating the impact of online or blended learning vs. face-to-face learning of clinical skills in undergraduate nurse education. J Adv Nurs, 2015; 71:255–270.

Colaceci

, Giusti

, Chapin

, et al. E-learning to improve healthcare professionals' attitudes and practices on breastfeeding. Breastfeed Med, 2017; 12:629–636.

10.

Salter

, Karia

, Sanfilippo

, et al. Effectiveness of E-learning in pharmacy education. Am J Pharm Educ, 2014; 78:16–22.

11.

Custers

, Ten Cate

. Very long-term retention of basic science knowledge in doctors after graduation. Med Educ, 2011; 45:422–430.

12.

Custers

. Long-term retention of basic science knowledge: A review study. Adv Health Sci Educ Theory Pract, 2010; 15:109–128.

13.

Ameh

, White

, Dickinson

, et al. Retention of knowledge and skills after Emergency Obstetric Care training: A multi-country longitudinal study. PLoS One, 2018; 13:4–15.

14.

Colaceci

, Giusti

, De Angelis

, et al. Medications, “Natural” products, and pharmacovigilance during breastfeeding: A mixed-methods study on women's opinions. J Hum Lact, 2016; 32:324–332.

15.

Ramos

, Sebastian

, Sebesta

, et al. Missed opportunities in the outpatient pediatric setting to support breastfeeding: Results from a mixed-methods study. J Pediatr Health Care, 2019; 33:64–71.

16.

Colaceci

, Giusti

, Chapin

, et al. The difficulties in antihypertensive drug prescription during lactation: Is the information consistent?. Breastfeed Med, 2015; 10:468–473.

17.

World Health Assembly (WHA). Implementation and monitoring. International Code of Marketing of Breast-milk Substitutes 2012. Available at: www.who.int (accessed June 29, 2019).

18.

Barennes

, Slesak

, Goyet

, et al. Enforcing the international code of marketing of breast-milk substitutes for better promotion of exclusive breastfeeding: Can lessons be learned?. J Hum Lact, 2016; 32:20–27.

19.

Soldavini

, Taillie

. Recommendations for adopting the international code of marketing of breast-milk substitutes into U.S. policy. J Hum Lact, 2017; 33:582–587.

20.

European Parliament. Directive 2014/40/EU of the European Parliament and of the Council of 3 April 2014 on the approximation of the laws, regulations and administrative provisions of the Member States concerning the manufacture, presentation and sale of tobacco and related products and repealing Directive 2001/37/EC. Available at https://eur-lex.europa.eu (accessed June 29, 2019).

21.

World Health Organization, United Nations Children's Fund & International Baby Food Action Network. Protecting, promoting and supporting breast-feeding: The special role of maternity services. 1989. Available at https://apps.who.int/iris/bitstream/handle/10665/39679/9241561300.pdf?sequence=1&isAllowed=y (accessed April 10, 2019).

22.

Vaona

, Banzi

, Kwag

, et al. E-learning for health professionals. Cochrane Library, 2018; 1:1–32.

23.

Keusch

. How to increase response rates in list-based web survey samples. Soc Sci Comput Rev, 2011; 30:380–388.

24.

Cunningham

, Quan

, Hemmelgarn

, et al. Exploring physician specialist response rates to web-based surveys. BMC Med Res Methodol, 2015; 15:4–11.