Abstract
The aim of this study was to assess the preliminary effectiveness and acceptability of an inter-professional e-consultation on vaccines. We used a quasi-experimental design to introduce an application in electronic health records that allowed primary health providers to launch e-consults to the hospital vaccines unit. A total of 135 e-consults were received during 10 months. E-consults were more frequently about subjects with chronic diseases (82.2%). Most of the e-consults were global (60.7%), that is, to revise a patient’s complete vaccination schedule, whereas 39.3% were specific, that is, to request information regarding a concrete vaccine or serology, with hepatitis vaccines leading the ranking (42.9%). The e-consultation avoided hospital referral in 85.4% of the global e-consults and in 100% of the specific e-consults. Indicators of acceptability were all above nine points (10 points indicated the maximum). The best-rated aspect was the level of recommendation (9.7 ± 0.68 points). In summary, linking primary health-care providers with specialists in vaccines through an e-consultation tool is effective and well-accepted by users.
Keywords
Introduction
Telemedicine is a broad term that encompasses different applications of telecommunication technologies for health-care purposes, including the diagnosis, treatment and prevention of health problems, as well as the evaluation and continued education of health-care workers. 1 The use of telemedicine has rapidly grown in recent years, thanks to its efficiency. Since early 2020, the strict control on the movement of people forced by the severe acute respiratory syndrome coronavirus 2 pandemic has triggered health institutions to convert part of their face-to-face activity rapidly to telemedicine. 2 , 3 Health-care systems that were not using telemedicine have been forced to introduce it, and those that were already using it satisfactorily have reaped the benefits of its many advantages sooner. 4
Given the growing body of evidence on the effectiveness of telemedicine and the possible new waves of the coronavirus disease 2019 pandemic – or of other pandemics caused by other communicable diseases – undoubtedly, telemedicine is here to stay. Currently, telemedicine is represented in most –although not all – health science disciplines, 5 albeit with different levels of development and under different formats. The e-consultation service is one of the most widely used formats within telemedicine. It is classically defined as an asynchronous consultative communication between health professionals via an electronic health registry or equivalent web platform. 6
E-consultations between professionals of primary and specialised health care have been described as one of the most beneficial applications of telemedicine. Thus, they can reduce waiting times, face-to-face appointments and health costs. Furthermore, patients access the health-care system through their primary health care (PHC) provider – physician or nurse – with whom they are more familiar because clinical relations are more continuous and take place in their social environment. Finally, when a regulated e-consultation is used, all clinical information is recorded in the electronic health record (EHR), providing greater security for patients and professionals, especially when compared to informal consultations between professionals, such as curb-side consultations. 7 , 8
The increase in chronic pathologies – with and without immunodeficiency 8 – and the constant development of new vaccines 9 have increased the need for specialised medical assessments on vaccinology. However, to our knowledge, no use of e-consultation on vaccines (e-CV) has been documented in the scientific literature. Therefore, the aims of this study were to describe the applicability of an e-CV in a rural area of northern Spain, to explain the characteristics of the consultations and to assess the effectiveness and acceptability of this tool in a real setting.
Methods
Study design and setting
We performed a non-controlled quasi-experimental study to estimate the effectiveness and the acceptability of PHC providers with an e-CV developed by vaccinology experts from a hospital vaccines unit. Experts were preventive medicine doctors with postgraduate training and expertise in infectious diseases and vaccinology. The study was carried out between May 2019 and February 2020 in a rural health district of northern Spain. In this health district, health care is covered by a secondary referral hospital, including the vaccines unit, and 25 PHC centres, with 132 PHC providers (67 general practitioners and 65 nurse practitioners). In Spain, health-care expenditure is financed by general state taxes. Therefore, access to primary or specialised health care is universal and free. The whole population has access to both specialised health care and PHC. The resident population (N = 61,267) is characterised by being older (>26% of people are >65 years), having multi-morbidity and receiving poly-medication.
The primary aim of the e-CV was to connect the hospital’s vaccines unit with PHC providers, physicians and nurses, in order to resolve global or specific questions about vaccines which may emerge during the clinical management of patients. Global e-consults involved requests for the complete assessment of a patient's vaccination schedule, whereas specific e-consults involved queries on a concrete vaccine, procedure or serology, for instance regarding a potential adverse event following immunisation, proper administration of a vaccine or the adequate interpretation of a serological test. An update of the software used in PHC introduced a new functionality in the EHR. This application allowed PHC providers to launch a consult to the experts of the vaccines unit regarding the clinical course of patients. Consults were recorded in the EHR, and a response was expected within 72 hours. Therefore, the effectiveness of the e-CV was measured in terms of decreasing patients’ referrals from PHC to specialised care. We also estimated the potential altered costs due to the decreased need for referrals using the officially established public fees for the provision of health services (specialised health-care consultation = €116 and PHC consultation = €59 – a difference of €57 according to the setting of the consultation). Moreover, the acceptability of PHC providers was measured using an anonymous online questionnaire delivered in March 2020. The questionnaire included questions on usability, handling, difficulty and overall satisfaction. A numerical scale ranging from 1 to 10 was applied, with 10 representing the most positive value. The survey was elaborated by a multidisciplinary group composed of seven nurses, one paediatrician, one preventive medicine physician and one psychologist. We also reviewed patients’ EHRs to collect certain clinical information and the vaccines that led to the e-consult.
Informed consent was given by all study participants. The study was approved by the Clinical Research Ethics Committee of Asturias (Spain; ref. 2020.153).
Data analysis
First, a descriptive statistical analysis of each variable was performed (univariate analysis), calculating absolute and relative (%) frequencies for qualitative variables and means and standard deviations (SD) for quantitative variables. Subsequently, a bivariate analysis was performed to assess the association between the study variables. For dichotomous qualitative variables, the chi-square test was used. For the quantitative variables studied, the unpaired t-test was used. Lastly, we estimated the monthly trend of the consults by means of a linear regression. The data analysis was performed using IBM SPSS Statistics for Windows v23.0 (IBM Corp., Armonk, NY). Results were considered statistically significant with p < 0.05.
Results
Characteristics of the virtual consultations on vaccines
A total of 137 e-consults on vaccines were received during the study period. Of these, two were excluded because they corresponded to request errors by PHC providers. Most e-consults were global (60.7%), meaning that PHC providers most often requested the assessment of a complete vaccination schedule rather than specific queries on a concrete vaccine or serology (39.3%). Among the specific e-consults, the vaccines that accumulated the highest number of questions were those related to hepatitis A and B (42.9%), followed by pneumococcus (14.3%) and the live-attenuated vaccines for measles, mumps and rubella and varicella (10.7%). In 58.3% of the specific e-consults, the specialists from the hospital’s vaccines unit deemed it necessary to amplify the indications for vaccination. Thus, in these cases, a complete assessment of the vaccination schedule was carried out in addition to the response for the specific query (Figure 1).

Flow diagram of e-consults registered by the electronic consultation on vaccines (e-CV) during the study period.
Characteristics of patients
Table 1 shows the main characteristic of patients involved in the e-CV. Although the target population of the e-CV was people with some kind of disease, especially clinically complex patients, it was observed that 17.8% of the questions involved healthy subjects. Within the group of e-consults regarding patients, 25.1% corresponded to consults on patients with autoimmune diseases, mainly inflammatory bowel diseases, followed by patients with cancer and those with chronic diseases without immunodeficiency.
Characteristics of participants (N = 135).
Monthly trend of virtual consults
The monthly evolution of e-consults experimented a linear decreasing trend (Y = 15.7–0.41X; R2 = 0.03) throughout the first 10 months of e-CV implementation (Figure 2), although this was not statistically significant (p = 0.448).

Monthly trend of the number of e-consults registered by the e-CV during the study period.
Effectiveness and acceptability
All of the e-consults (100%) were responded to within the first 72 hours. The e-CV avoided referral to the hospital in 85.4% of the global e-consults and in 100% of the specific e-consults (Figure 1). Given that the e-CV avoided 122 referrals to the hospital’s vaccines unit, we estimate an overall cost reduction of €6954 (122 × €57). The patients who were most referred to hospital were paediatric patients (p < 0.001), those with immunodeficiency (p = 0.014) or those who needed a comprehensive review of their vaccine schedule (p = 0.049). No differences were found according to sex (p = 0.509).
The acceptability survey was conducted on 100 of the 132 PHC providers (response rate: 75.8%). Among the respondents, only 34% had used the e-CV. Table 2 shows the mean scores and score ranges. All responses were above nine points. The best-rated aspect corresponded with the level of recommendation (9.7 ± 0.68 points).
Acceptability scores for the e-CV.
e-CV: electronic consultation on vaccines; EHR: electronic health record.
Discussion
According to the results of our study, a new computer functionality hosted in the EHR for PHC to launch e-consults to the hospital’s vaccines unit was effective and accepted by those who used it. To the best of our knowledge, this study constitutes one of the first approaches of telemedicine in the field of vaccinology. This is especially relevant, as telemedicine is occupying an increasingly important position in the current development of health sciences, 10 and all specialties should take advantage of its potential. Some promising uses of e-health to improve performance of vaccination programmes have been documented previously. 11 , 12 However, these were all people-centred tools to reach optimal coverage targets (e.g. email reminders, personal EHR, etc.). Our approach was different. Our e-CV was centred on PHC providers who had doubts on vaccines for people with a diagnosed disease, not for healthy people in the context of a vaccination programme.
The development of new vaccines and adapted guidelines for patients in special clinical situations has increased the doubts of PHC providers and patients. 13 In other medical specialties, e-consults do not often require a complete review of the recommendations, perhaps because there were fewer doubts or they are more specific than in vaccinology. For example, Phadke et al. 14 found that only 13.4% of e-consults on allergy/immunology required a global patient assessment, whereas in our e-CV, the corresponding figure was 60%. This is consistent with the type of patient included in our study. The profile of the patient who required more virtual consultations is a middle-aged person with primary or secondary immunodeficiency. Autoimmune diseases are usually diagnosed in middle-aged adults, and the clinical management of these cases is often complicated in primary-care services, both regarding treatment and prevention measures. 15
The ability to resolve doubts via the e-consult service is of particular interest to both patients and health institutions. In our study, only 1 out of 10 patients required a face-to-face appointment at the hospital. In comparison with other studies, with a referral percentage between 34% and 59.8%, 16 , 17 our results suggest that e-CV can be highly efficient. In addition, the virtual consultation on vaccines answered all queries within the first 72 hours, which is similar to other studies. 17 Thus, PHC providers were given a response within a reasonable time and were able to advise their patients to execute the clinical decision recommended by the vaccines unit. Considering that the waiting time in PHC is significantly shorter than in specialised care, it is highly likely that faster and more satisfactory solutions were achieved for all parties concerned. Moreover, PHC providers are often physically closer to patients than specialised health-care providers are. Therefore, the replacement of a face-to-face appointment at the hospital with a PHC consult could also lead to an appreciable cost reduction for patients in terms of time expenditure and transportation costs. Finally, although it is difficult to evaluate, receiving a solution from the PHC provider could be satisfying for patients because the personal relationship with the PHC provider is often closer than with specialised care providers, as well as for the health-care professionals because their competence in the management of vaccines could be enhanced, together with their reputation.
Regarding acceptability, although the e-CV achieved very high scores, the percentage of e-consult users was lower than in other studies (25.8% vs. 32.5%), 17 suggesting that the dissemination of the e-CV is essential. Particularly noteworthy was the high consideration of e-CV usefulness for daily clinical activity. In this regard, some studies recognise that e-consults represented a change in the comprehensive approach to clinical cases. For example, in a retrospective review of 3872 e-consults using a platform covering 34 US states, 75% of PHC providers admitted an improvement in their clinical strategy. 18 In Massachusetts (USA), a six-month pilot programme of an e-consult on cardiology was considered helpful by 100% of physicians. 19 In another pilot project developed in Ontario (Canada), with more than 300 e-consults launched by PHC providers referred to 24 specialties, 92% of users rated the e-consults at four or five points (indicating that this service was perceived as valuable and very valuable), and only 2% of responses to doubts were deemed not useful. 20 Lastly, the trend towards less use of e-CV during the study period, although not statistically significant, suggested a possible loss of interest by PHC providers. Indeed, we would have expected a gradual increase in the number of e-consults as the tool became better known. Typically, the use of new tools depends largely on the effort to promote their use until they are completely embedded in routine clinical activity. Thus, a practical recommendation for developers is to make efforts towards regularly promoting e-consultations and the training of potential users during the first months of implementation. We must be aware that telehealth is a disruptive process which requires effective change-management strategies for health-care providers. 3
This study had some limitations. First, the number of e-consults analysed and the number of PHC providers using the tool was limited. However, it was a preliminary assessment and was sufficient to meet the objectives of this study. Second, to have a control group would have been optimal. However, in evaluation studies conducted in real settings, a control group is sometimes not considered appropriate. Finally, a more complete evaluation of effectiveness, including a comprehensive cost analysis, would have enriched our results.
Conclusion
For the first time in the scientific literature, this preliminary study tested an e-CV in the real context of application. According to our findings, linking PHC with the vaccines unit of specialised care through an e-CV embedded in the EHR is effective and accepted by both physicians and nurses. However, this e-CV should be interpreted as an additional tool, which in no case should completely replace the practice of preventive medicine in face-to-face consultations.
Footnotes
Acknowledgements
The authors thank Luis M. Martínez-Escotet and the staff of the Systems Service of Directorate-General for Planning (Department of Health of Asturias) for their support during the design and implementation of the pilot project.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.
Funding
This research received no financial support for the research, authorship and/or publication of this article.
