Student Evaluation of Distance Learning for Health Care Professionals

Introduction

Infectious diseases remain a public health concern in Brazil. ¹ Although some researchers ² have identified a decrease from 50% to 5% between 1930 and 2010 in the total number of deaths from such diseases, this reduction did not occur in a balanced way, because there was a decrease in deaths related to some diseases but not all. A large proportion of deaths and hospitalizations registered in Brazil are the result of chronic infectious diseases such as tuberculosis and acquired immune deficiency syndrome (AIDS). There are many reasons for the occurrence of chronic infectious diseases; however, several studies have reported the occurrence of an epidemiological transition in Brazil, which is characterized by a reduction in the number of infectious diseases and an increase in chronic noncommunicable diseases. This has led to reduced public policy actions focused on infectious diseases, which could potentially result in their emergence and/or re-emergence. ³

Other reasons for the emergence of chronic infectious diseases include uncontrolled urban growth and socioeconomic conditions, both of which are determinants for infectious diseases related to poverty—such as AIDS and tuberculosis—which have higher mortality rates. Furthermore, in low- and middle-income neighborhoods, there are cases of neglected tropical diseases. From the 17 diseases characterized as neglected by the World Health Organization (WHO) in 2010, at least 12 occur endemically in Brazil, among which are malaria, Chagas disease, leishmaniasis, dengue, and leprosy, and parasitic diseases such as schistosomiasis. ^{4 –6}

Currently, Brazil is better equipped to detect and intervene in situations of epidemics as well as in cases of diseases with epidemic potential. ³ However, to achieve success in this process and to implement surveillance and public policy action, it is necessary to have health professionals who are highly trained and updated with the latest skills so that they can detect and treat these types of diseases in a timely manner, so as to circumvent morbidity and mortality. For this, it is necessary to use participatory and dialogic teaching and learning methodologies capable of training professionals who prioritize health promotion and disease prevention. ⁷

Considering the professional qualifications of public health system workers, the Brazilian government, through the Open University of Brazilian National Health System (UNA-SUS), in partnership with public universities in several states, offers specialization courses, further training and extension courses with free access; these courses aim to serve strategic areas of the healthcare service. This government initiative also facilitates learning through the use of a variety of mechanisms and educational tools.

To raise the quality of their distance learning (DL) courses, the UNA-SUS pedagogical team of the Federal University of Maranhão (UFMA) sought to establish measures and implement plans that could mitigate the problems inherent to the DL modality, such as dropout. One of the measures adopted was the creation of the Monitoring System (Monsys), patented by UNA-SUS/UFMA at the National Institute of Industrial Property (NIIP), with registration process number BR 51 2014001542 4. Monsys is a tool that enables data mining on the Moodle virtual learning environment (VLE); this platform is used by UNA-SUS/UFMA to offer DL courses. Monsys is used to collect data on student and teacher activities on the Moodle platform; the data are then organized on an interface that is easily viewable by the pedagogical team. The team uses this information to detect recurrent problems with the DL modality, such as grade repetition and dropout, and implement speedy solutions.

Therefore, to assess the DL courses offered by the institution, UNA-SUS/UFMA created the Integrated Management System (SIGU) questionnaire, an auxiliary subsystem inserted in the UNA-SUS SIGU. The SIGU questionnaire facilitates the processing and interpretation of data that are collected when students assess modules. This study presents the results elicited from the SIGU questionnaire regarding students' evaluation of the communicable diseases module, offered by two postgraduate DL courses at the institution.

This study aims to analyze students' perceptions of the didactic and pedagogical aspects related to the content and activities, educational resources, and tutoring in the communicable disease module offered in two DL postgraduate courses at UNA-SUS/UFMA. Students evaluated the course content, available resources, and interaction in the VLE and in the modules of the course, with the intention of identifying possible problems in the two DL postgraduate courses.

Materials and Methods

This quantitative study was analytical and observational in nature. Students evaluated the didactic and pedagogical aspects of the course content, online course resources, and tutoring provided. The project was approved (approval No. 668.838) by the Research Ethics Committee of University Hospital, affiliated with UFMA.

Our analysis was based on secondary data collected directly from the SIGU, the UNA-SUS Registration System, and from the Moodle VLE, through the SIGU questionnaire, which is an auxiliary subsystem designed to process and interpret data pertaining to students' evaluation of DL courses offered by UNA-SUS/UFMA. The SIGU questionnaire facilitates the processing and interpretation of data collected when students evaluate modules.

We analyzed data on the didactic and pedagogical aspects of tutoring, educational resources, and course content, and activity, among other aspects, for the courses available on the Moodle VLE. The questionnaire completed by the students was prepared by the educational sector and adapted to the VLE by the instructional design team at UNA-SUS/UFMA. The final questionnaire consisted of objective questions, divided into five categories pertaining to the main aspects of the courses: (1) Overall Evaluation of the Module, (2) Didactic and Pedagogical, (3) Tutoring, (4) Content and Activities, and (5) Educational Resources.

A Likert scale was used to measure attitudes, comprising a series of statements related to the object searched. The scale used consisted of five points, which were “great,” “good,” “bad,” “insufficient,” and “not applicable” or “I have not done it.” However, for this analysis, the options of “not applicable” and “I have not done it” have been omitted. Analyses were performed according to module type and by complete course. By way of a cross-analysis, the evaluation variables were dichotomized into positive outcome, which included in the categories of “great” and “good,” and negative outcome, which included the “bad” and “insufficient” categories.

The sample consisted of 319 students from a total of 1,552 individuals enrolled in the following classes: two Postgraduate in Family Health (classes 3 and 4) classes, and two Postgraduate Classes in Primary Care (groups 3 and 4).

We analyzed the data using Version 18 of the Statistical Package for the Social Sciences (SPSS^®; SPSS Inc. Released 2009. PASW Statistics for Windows, Version 18.0. Chicago; SPSS Inc.). Variables were summarized by frequency measurements and medium and standard deviations. We used the chi-square test or Fisher exact test to compare categorical variables among groups. Numerical variables were initially analyzed using the Shapiro–Wilk test, and groups were compared using either the t test or Mann–Whitney test, depending on the normal distribution evaluation. The Spearman correlation coefficient was calculated to estimate the relationship between the different categories of the questionnaire. We adopted a 5% significance level (p < 0.05).

Results

Figure 1 is a pie chart of the students' overall evaluation of the module. Sixty-five percent of the students rated the module as “great,” 34% rated it as “good,” and only 1% rated it as “bad.” Then, we compared the overall assessment percentages with those for specific aspects of the module.

OPEN IN VIEWER

Fig. 1.

Overall evaluation of the module by students.

Table 1 lists the percentages associated with students' evaluations of the didactic and pedagogical aspects in the communicable diseases module. There were no statistically significant differences (p > 0.050) between the evaluations of these items and the students' overall evaluations. More than 50% of students rated all items as “great,” particularly in the cases of tutor participation in the forum and posting, for which the accompanying percentages were 71.1% (214) and 69.5% (221), respectively. From among the items rated as “great,” the one with the lowest rating was posting, with 59.2% (187). Table 1 also clarifies that in terms of the book options (online versus PDF), the PDF option was rated as better by 98.7% (310) of the students.

Table 2 contains the students' assessments of the tutoring and the tutor for the communicable diseases module. None of the items were rated as “insufficient,” and in fact, >60% of students rated all items as “great.” Tutor's performance was rated as “great” and “good” by 71.8% (229) and 27.3% (87) of the sample, respectively. There were no differences in students' perception of mentoring quality to the module's overall rating.

Table 3 lists the students' evaluations regarding the module's content and activities. The item workload of the course educational units was evaluated as “great” by 53.0% (169), “good” by 43.6% (139), and “bad” by 3.4% (11) of students, but no statistically significant difference appeared in the overall evaluation (p = 0.011). The highest rated item (by 71.2% of students) was “relevance of the module to your professional practice.”

Students' evaluations of the educational resources provided are presented in Table 4. The item of “DL platform: access and functionality” was rated as “great” by 57.4% (183), “good” by 38.9% (124), “bad” by 3.4% (11), and “insufficient” by 0.3% (1) of the sample (p = 0.076). With regard to whether the educational resources provided were adequate for the activities performed, 63% (201), 35.7% (114), and 0.9% (3) of students rated this issue as “great,” “good,” and “bad,” respectively. There was no statistically significant difference in the module's overall evaluation (p = 0.707).

Discussion

In this study, we aimed to analyze students' perceptions regarding didactic and pedagogical aspects related to the content and activities, educational resources, and tutoring in the communicable disease module in two DL postgraduate courses at UNA-SUS/UFMA. The students' perceptions were elicited through the SIGU questionnaire in the Moodle VLE.

Completion of the SIGU questionnaire was voluntary, which resulted in some challenges such as the nonparticipation of all students. Therefore, the UNA-SUS pedagogical team had to take certain actions to encourage students to participate, such as keeping the questionnaire open even after the module was completed, giving students more time to fill them.

The students who composed the sample were also participants in two Federal Government programs: Primary Care Professionals Valorization Program (PROVAB) and the More Doctors Program. Both focus on supporting health professionals working in deprived areas and areas with inadequate medical care. UNA-SUS/UFMA, in partnership with the ministry of health, offers DL courses for the ongoing education of professionals participating in these programs. The development and preparation of courses offered to participants are a complex task due to the heterogeneity of the target audience. PROVAB, for example, has national coverage, and it includes professionals from various Brazilian states. The More Doctors Program has an even larger public profile: in addition to having participants from different Brazilian states, it also has foreign participants, especially from Cuba.

Some researchers ⁸ emphasize that the target audience's heterogeneity should be an important consideration for pedagogical teams developing DL courses, since the courses should account for students' varying learning styles, promoting accessibility and inclusion. After analyzing the results of this study, we observed that even with such a heterogeneous student body, the students' evaluations were mostly positive, and their overall evaluation of the module, shown in Figure 1, was almost unanimously positive.

Based on the results in Table 1, we can see that in terms of didactic and pedagogical aspects, students preferred the PDF book to the online book, a result that is probably related to study access in the offline mode, after completing the download. The use of offline materials must have increased the accessibility and learning efficiency of SUS professionals in the DL course, mainly because of the large number of these students living and/or working in remote areas with limited internet access. ⁹ Another contributing factor could be that most of the students came from a generation that primarily used printed material; therefore, they were not as familiar with digital learning resources.

As also seen from Table 1, the activity of posting received the lowest “great” rating. This could be related to the fact that “forum” is an interactive tool that depends on participation among students and teachers, as well as internet access. In their research on interactive concepts in VLE, some authors ¹⁰ affirmed that students considered the “forum” activity to be a tool that allowed for greater interactivity because it elicited answers from the teacher, promoted the exchange of ideas with other students, and facilitated debate on the subject proposed by the tutor, thus allowing for feedback and the exchange of experiences between tutor and student.

The results presented in Table 2 reveal students' perceptions of the UNA-SUS' mentoring team. All the items received a “great” evaluation from >60% of the sample. In fact, a tutor's performance and support are essential in DL and can directly influence students' performance, motivation, and drive to complete a course, thereby ensuring learning acquisition. ¹¹ Furthermore, tutors have a multifunctional role in DL learning because not only do they require scientific knowledge but they must also master technological, didactic, pedagogical, and up-to-date knowledge. ¹²

From Table 3, we see that the item with the highest “great” rating (from 71.2% of students) was the module's relevance to students' professional practice. This could be related to the professional needs faced by SUS professionals on a daily basis, because communicable diseases still persist as a public health problem in Brazil. ¹³

The students' interest in the module's theme (communicable diseases) is also related to the characteristics of their workplaces, which in most cases consist of municipalities located in the northeast of Brazil, areas with a low human development index and with poverty or extreme poverty. According to a report published by the ministry of health, poor living conditions are responsible for numerous cases of communicable diseases, such as infectious diseases associated with widespread poverty in marginalized populations (AIDS and tuberculosis), and neglected diseases that occur mainly in low- to middle-income areas of countries (malaria, Chagas disease, leprosy, trachoma, and parasitic diseases). ³

Based on the students' assessments of the educational resources provided (Table 4), we observe that item “DL platform: access and functionality” obtained very positive feedback from 96.3% (307) of students. UNA-SUS/UFMA offers distance courses through the Moodle VLE platform, which is used by educational institutions in >200 countries. ¹⁴ This platform allows educators to create a learning environment that can combine sound and visual media; this facilitates learning through content presentations that are more dynamic and organized. The use of technological innovations such as VLEs allows students to have greater interaction with their teaching staff, thereby decreasing the distance barriers that are inherent to DL. The exchange of information and dialogue among participants in the learning process—students, teachers, and tutors—carried out in VLE, directly influences success in the teaching–learning process. ¹⁵

However, despite its benefits, VLEs can only be accessed over the internet, which can pose a problem for students living in areas with poor internet services. Other issues with VLEs relate to access and functionality problems arising from the technical limitations of the platform.

The DL modality also allows for a number of different learning resources to be made available to students through VLE. ¹⁶ In fact, 98.7% (315) of the students mentioned that they had adequate educational resources to perform the activities required from the course. The organization and adequate use of these resources, as well as their versatility, were directly associated with user satisfaction and learning outcome. Therefore, it is essential for educators to develop creative and dialogical course material that can establish effective interaction with and among students ⁸ in order for students/professionals to be motivated to face the daily challenges of public health in Brazil, such as communicable chronic diseases.

However, it is worth noting that the full spectrum of health education extends beyond clinical and curative treatments. Teaching and learning methodology must be used in a participative and dialogical way so that health professionals can be trained to prioritize health promotion and disease prevention. ³ Health professionals must also use educational strategies that take into consideration their daily routines, thereby enabling their professional development and encouraging their participation in strategies for continuing medical education. ¹⁷ This fact justified the realization of this study involving DL, since it is an effective strategy for the continuous training of SUS professionals, since this method of teaching allows flexibility in the moment and location.

But despite its advantages, DL faces, among other problems, the challenge of keeping students engaged in e-learning platforms, circumventing the so-called dropout phenomenon, which is defined as student withdrawal from a course or educational program without successfully completing it, ¹⁸ that is, without achieving the approval certification of completion. In fact, dropout in DL has, in general, been the major concern for educators. ^{19 –22}

Apart from Monsys, the pedagogical team also realized the need to provide assessment tools that students could use to reflect upon completed modules; these assessment tools would help the team gain knowledge on students' views about the methodology and resources used in courses. Researchers ²³ have highlighted that assessment procedures are essential if pedagogical teams are to improve teaching–learning processes, because the collected data can be transformed into grants for teaching and administrative staff. The evaluation of DL courses is also necessary because it can help educators detect major difficulties encountered by DL students. ²⁴

Conclusions

Despite the plethora of studies indicating the effectiveness of DL, it is still necessary to collect data and develop new evaluative tools such as the SIGU questionnaire and Monsys. Data obtained through these tools are essential for pedagogical teams to improve existing methods and understand and solve recurrent problems associated with DL, such as dropout.

This study enabled us to observe how evaluation is an essential step in the teaching–learning process, especially in the case of DL, where classes comprise an extremely diverse audience. It is only through feedback and evaluation that educators will hear the opinions of the participants and determine actions to enhance the educational experience for everyone involved, regardless of their specialties. In fact, the established logic is that institutions should adapt to students' needs, not vice versa. Furthermore, assessment tools such as the SIGU questionnaire become even more necessary in a DL modality where there is no personal contact between students and the teaching staff.

The results obtained from the SIGU questionnaire provided valuable information for the UNA-SUS/UFMA teaching staff, allowing them to systematically and objectively reflect on this very multifaceted teaching process. The study also highlights the correlation between student satisfaction and reduced dropout rates, an aspect that needs to be further explored in future studies.