A Mixed Methods Evaluation Finds Virtual Training Among Health Professions Education Trainees and Supervisors as Effective as In-Person Training

Abstract

Objectives

The COVID-19 pandemic led to a rapid adoption of telehealth technologies, particularly in terms of treatment delivery and trainee education. Prior work has found that virtual training (in which supervisors and trainees are not co-located and interact via virtual communication platforms) is well-received and supports effective supervisor/trainee working relationships. However, there is limited assessment of the effectiveness of virtual training in supporting clinical skill development or working relationships. Given the growing adoption of telehealth in health professional education, an evaluation was conducted in collaboration with the Clinical Resource Hubs (CRH), a national telehealth program within the Veterans Health Administration (VHA) offering clinical training opportunities within a virtual work environment.

Methods

Survey instruments were developed and administered to CRH trainees and supervisors. Information was collected related to trainee and supervisor demographics, CRH training experiences, perceived quality of training and supervision, supervisor/trainee relationships, and perspectives on virtual training.

Results

Trainees and supervisors perceived virtual supervisor/trainee interactions to be equivalent or superior to in-person interactions in supporting the development of clinical competencies and experienced effective supervisory working relationships in a virtual environment. Trainees and supervisors differed in their preferences for virtual versus in-person training and views on the ideal combination of these modalities to support skill development.

Conclusion

Results suggest that virtual supervisor/trainee interactions can effectively support trainee clinical skill development and effective supervisor/trainee relationships. For many aspects of training, virtual training for health profession trainees within large healthcare settings can provide an effective experience that is an acceptable and viable alternative to in-person training.

Keywords

telesupervision virtual training trainees veterans telehealth

Introduction

The COVID-19 pandemic led to a rapid adoption of telehealth technologies within health care systems, including training activities involving Health Professions Education (HPE), which are designed to support the development of the next generation of clinicians (ie, physicians, registered nurses, physician assistants, pharmacists, and psychologists).^1–6 The incorporation of telehealth into HPE has taken the form of virtual training (virtual interactions between supervisors and trainees, wherein supervisors and trainees are not co-located and interact via virtual communication platforms) and virtual care delivery by trainees to patients. This transition has led to a growing interest in assessing virtual training's equivalence to more traditional, in-person training in supporting HPE within large health care environments and, in particular, the effectiveness of virtual interactions between trainees and supervisors in supporting skill development and the formation of effective working relationships.^7,8

Prior work has found that virtual training is well-received by trainees,⁹ can support the development of effective supervisor/trainee working relationships,^10–12 and is technologically feasible for a wide range of clinical trainee specialties, levels, and settings.⁸ Virtual training also provides flexibility, allowing trainees to draw on the professional expertise of clinicians who may work in different locations,^6,12 and helps increase trainees’ exposure to patients in rural or underserved areas while eliminating the burden of travel.^12,13 However, there has been limited assessment of the effectiveness of virtual supervisor/trainee interactions in supporting clinical skill development or the creation of specific aspects of supervisory training relationships, such as shared clinical decision-making and trust building.

Given the growing interest in virtual training in HPE, an evaluation was conducted in collaboration with the Veterans Health Administration (VHA) Clinical Resource Hubs (CRH), a national program designed to provide telehealth to Veterans, primarily those residing in rural or underserved areas. Each Veteran Integrated Service Network (VISN) (ie, regional area of the VHA) has a CRH that helps fill gaps in healthcare by connecting Veterans with VHA clinicians remotely, ensuring they receive consistent, high-quality, and evidence-based care regardless of location. Each CRH is staffed with a diverse group of clinicians, including physicians, advanced practice providers (eg, nurse practitioners, physician assistants, and clinical pharmacist practitioners), psychologists, registered nurses, social workers, and others. The regional CRHs provide contingency staffing and longitudinal support primarily through telehealth modalities to complement services at local Veterans Affairs Medical Centers (VAMCs) and outpatient clinics to maintain Veteran access to primary, mental health, and specialty care.^14–16 The CRH also offer clinical training opportunities for health profession trainees (HPTs) across multiple disciplines within a virtual work environment. Key training opportunities offered by CRH include clinical pharmacy, psychology, internal medicine (eg, physicians), and primary care (eg, advance practice nurses).

This evaluation explores the effectiveness of virtual supervisor/trainee interactions, relative to traditional in-person interactions, in terms of clinical skill development for CRH trainees. This evaluation also seeks to understand how virtual interactions influence the establishment and quality of working relationships between supervisors and trainees, with the goal of identifying potential benefits, challenges, and best practices.

Methods

Evaluation Design

Surveys were created and collected using Qualtrics, a web-based survey platform approved for VHA use.¹⁷ VHA REDCap was used to distribute the Qualtrics surveys to ensure delivery of surveys to VHA emails. Surveys were created to capture CRH trainee and supervisor perspectives related to the outcomes of interest. This evaluation was guided by an adapted version of the CDC's Framework for Program Evaluation in Public Health and the Office of Management and Budget's evaluation standards.¹⁸

Virtual Training Environment

All VHA HPT Programs are hosted and managed by VAMCs. These trainee programs include affiliate-sponsored and VHA-sponsored trainee programs commensurate with the professional training accreditation standards. Trainees’ target degrees include PharmD, MD, NP, PsyD, and PhD. In addition, the CRH hosts postgraduate residency experiences for pharmacists and physicians. Pharmacy trainees rotate daily for 4 to 6 weeks, physician trainees rotate half day per week for 3 weeks, nurse practitioner trainees rotate one day per week for 3 months, and psychology trainees rotate for 8 months to 1 year. Supervisors include clinical pharmacist practitioners with over 3 years of supervisory experience, board-certified internal medicine physicians, licensed clinical psychologists with at least one year of supervisory experience, or advanced practice nurse practitioners with over 3 years of supervisory experience. For all VHA trainees, the VHA training program director determines if the trainees are appropriate for virtual training and if adding a CRH learning experience to their training options is beneficial to the trainee and training program. For example, a trainee could request to gain competency in delivering evidence-based treatment protocols via telehealth modalities. To meet this training goal, the trainee could elect to complete a major or minor rotation in a CRH training program to gain these skills.

The virtual training experience varies by profession and training levels. Pharmacy CRH trainees report to a private office at their local VAMC, where oftentimes their supervisors are not located in the building, but the trainees have access to pharmacists nearby for questions and emergencies. All patient care appointments are conducted through virtual modalities, including telephone and video. Pharmacy supervisors are always present with the trainee during the virtual visit (eg, 3-way call). Other required aspects of the rotation are completed virtually through Microsoft Teams meetings, such as topic discussions, patient case presentations, journal clubs, and formative and summative feedback. Otherwise, brief video chats are conducted multiple times throughout each clinic day to guide the course of the day. Psychology CRH trainees report to a private office at their local VAMCs, where they are always co-located with at least one supervisory staff member. The full supervisory team is available ad hoc and as needed via telephone or Microsoft Teams, and supervisors always have the capability to join a virtual telehealth session in real time. All clinical services are delivered via clinical video technology (utilizing telephone as a backup in the event of a technological failure that renders video unavailable). A clinical supervisor is always on-site and on-call, and primary supervisors are accessible as needed and ad hoc during all training days. Supervision consists of a minimum of 1-hour a week of individual supervision and 1-hour a week of group supervision.

Participant Selection

A total of 115 CRH trainees and 77 CRH supervisors were identified and recruited for survey participation. Only trainees who completed a CRH rotation in academic year (AY) 2023 to 2024 and supervisors who worked within CRH training environments during AY 2023 to 2024 were included in this evaluation.

Ethical Considerations

This study was designated as nonresearch quality improvement per official VHA guidance and was exempt from IRB oversight.¹⁹ However, steps were taken to ensure the protection of survey respondents. Trainees and supervisors were informed their participation was voluntary, that their responses were confidential, and that survey results would be reported in aggregate. Additionally, survey content and methodology were reviewed and approved by the VHA quality improvement oversight groups prior to administration. The reporting of this study conforms to the SQUIRE 2.0 (Standards for Quality Improvement Reporting Excellence) guideline (Supplemental Appendix A).²⁰

Survey Instrument

Parallel survey instruments were developed for CRH trainees (Supplemental Appendix B) and supervisors (Supplemental Appendix C). Survey questions were designed to collect information on demographics, CRH training experiences, the perceived quality of training and supervision experiences, supervisor/trainee relationships, and preferences for virtual versus in-person interactions. Questions assessing clinical competency were adapted from guidance related to expected milestones for skill development within the Common Program Requirements as proposed by the Accreditation Council for Graduate Medical Education (ACGME) to fit a broader trainee population. Modest revisions of ACGME competency domain definitions were provided to survey respondents to ensure consistent interpretation and understanding.²¹ Survey instruments were reviewed by CRH leadership and piloted with trainees and supervisors in February 2024 and reviewed by CRH leadership. Piloting the survey instruments allowed for the refinement of the language and definitions used throughout the instruments, and updates to the survey distribution plan to include additional reminders via Microsoft Teams messages.

Procedure

Outreach emails containing information about the study and a link to the survey were sent to CRH supervisors and trainees targeted for inclusion in this study. CRH leadership was enlisted to raise awareness of the project via outreach emails to CRH supervisors and trainees and to support survey completion by encouraging CRH supervisors to provide time for trainees to complete the survey during regularly scheduled meetings. Incentives for survey completion were not provided to participants. Survey data was collected from March 2024 through April 2024 using Qualtrics. Survey completion rates were tracked, and follow-up reminders were sent via email and VHA-based clinical messaging programs (Microsoft Teams) to those who had not completed the survey. Two email reminders were sent, as well as 2 waves of Microsoft Teams-based completion prompts, before the survey was closed after 4 weeks of fielding.

Analysis

Analyses were conducted by staff at the Center for Evaluation and Implementation Resources (CEIR) based in Ann Arbor, Michigan, USA. Descriptive analyses were conducted to describe the population and assess trainee and supervisor perspectives related to outcomes of interest. Nonparametric chi-square tests were used to assess group equivalence, with decisions about group significance based on a P < .01 alpha level. Cases with missing data were excluded via listwise deletion. All statistical analysis was conducted in Stata 18 MP.²² Open-ended survey responses were analyzed using MAXQDA qualitative analysis software using inductive qualitative content analysis.²³ First, one coder (KB) reviewed all open-ended responses to identify and develop codes from the data. Next, one additional coder (GY) reviewed the coding scheme to identify discrepancies and modify or expand the codebook. Following initial coding, coded responses were further refined and sorted into meaningful domains. Discrepancies were resolved by consensus between the 2 coders.

Results

Population Characteristics

A total of 115 trainees and 77 supervisors were invited for survey participation. A total of 43 trainees (37.5%) and 57 supervisors (74.0%) completed the survey (Table 1). Among trainees, most reported being white (61.9%), non-Hispanic (87.5%), women (76.2%), and between the ages of 25 and 34 (83.3%). In terms of clinical discipline, the largest proportion was psychology trainees (60.5%), followed by clinical pharmacists (23.3%). The largest proportion of trainees (48.8%) had been with VHA for 6 to 12 months. Most trainees (74.4%) had experienced only virtual interactions with their supervisors in their CRH training, while 25.6% had both in-person and virtual interactions.

Table 1.

Demographic Characteristics of Respondents.

CHARACTERISTIC	TRAINEE (N = 43)	SUPERVISOR (N = 57)
CHARACTERISTIC	N (%)	N (%)
Race
Asian	8 (19.1)	9 (15.8)
Black or African American	4 (9.5)	1 (1.8)
White	26 (61.9)	38 (66.7)
Mixed Race	1 (2.4)	4 (7.0)
Other	3 (7.1)	5 (8.8)
Ethnicity
Hispanic	5 (12.5)	4 (7.0)
Not Hispanic	35 (87.5)	53 (93)
Gender
Woman	32 (76.2)	41 (71.9)
Man	7 (16.7)	14 (24.6)
Nonbinary	1 (2.4)	2 (3.5)
Other/prefer to self-describe	1 (2.4)	NA
Prefer not to say	1 (2.4)	NA
Age
25-34	35 (83.3)	NA
35-44	4 (9.5)	NA
45 and up	3 (7.1)	NA
26-29	NA	1 (1.8)
30-39	NA	35 (61.4)
40-49	NA	16 (28.1)
50-59	NA	3 (5.3)
60 years or older	NA	1 (1.8)
Discipline
Clinical pharmacist	10 (23.3)	20 (35.1)
Psychology	26 (60.5)	23 (40.4)
Other	7 (16.3)	14 (24.6)
Months as a VA health professions trainee
< 6 months	3 (7.0)	NA
6-12 months	21 (48.8)	NA
13-18 months	1 (2.3)	NA
> 18 months	18 (41.9)	NA
Participated in a VA training or educational program before becoming a VA employee
Yes	NA	55 (96.5)
No	NA	2 (3.5)
Proportion of VA time to train trainees
< 10%	NA	25 (43.9)
11%-25%	NA	23 (40.4)
26%-50%	NA	8 (14.0)
51%-75%	NA	1 (1.8)
Time served as a supervising practitioner of CRH trainees in VA
< 6 months	NA	4 (7.0)
6-12 months	NA	4 (7.0)
1-2 years	NA	23 (40.4)
3-5 years	NA	18 (31.6)
6-10 years	NA	8 (14.0)
Supervision modality experience
Some in-person training	11 (25.6)	21 (36.8)
Only virtual training	32 (74.4)	36 (63.2)

Among supervisors, most reported being white (66.7%), non-Hispanic (93.0%), women (71.9%), between the ages of 30 and 39 (61.4%), and being former VHA trainees (96.5%). Psychology supervisors were the largest group of supervisor respondents (40.4%), and the largest proportion of supervisors reported training VHA trainees for 1 to 2 years (40.4%). The majority of supervisors (63.2%) had interacted with CRH trainees using only virtual interactions, while 36.8% had experience interacting with trainees both in-person and virtually.

Quantitative Results

Trainee Skill Development

A large proportion of trainees (over 93%) and supervisors (over 90%) endorsed the perspective that virtual interactions offered equivalent or greater contributions to skill development than in-person interactions for each of the ACGME clinical skill areas. Trainees and supervisors were equivalent in their perceptions of the value of virtual interactions in supporting ACGME skill development across competency areas (Table 2).

Table 2.

Proportion of Respondents Who Perceive Virtual Training to be Equivalent or Better Than In-Person Training Related to Skill Development on ACGME Competency Areas, Stratified by Role (Trainees [N = 42] Versus Supervisors (N = 57)).

ACGME COMPETENCY AREAS	TRAINEES	SUPERVISORS	CHI-SQUARE	SIGNIFICANCE^a
ACGME COMPETENCY AREAS	N (%)	N (%)	CHI-SQUARE	SIGNIFICANCE^a
Patient care	42 (100)	56 (98.2)	0.74	NS
Knowledge for practice	42 (100)	57 (100)	NA	NS
Interpersonal and communication skills	41 (97.6)	51 (89.5)	2.44	NS
Professionalism	41 (97.6)	54 (94.7)	0.52	NS
Practice-based learning and improvement	40 (95.2)	53 (93.0)	0.22	NS
Systems-based practice	39 (92.9)	51 (89.5)	0.34	NS

Note: ^a Group comparisons did not meet statistical significance based on P < .01. Degrees of freedom for all measures = 1.

Supervisor/Trainee Working Relationships

Across all supervisory relationship aspects, both trainees and supervisors endorsed strong agreement (over 91%) with the view that virtual supervisory interactions were collaborative, supportive, comfortable, and effective in supporting learning. Trainee and supervisor perspectives related to these measures were not found to differ from one another (Table 3).

Table 3.

Proportion of Respondents Who Rated Aspects of Virtual Supervisory Relationship as Positive (Trainees [N = 42] Versus Supervisors [N = 57]).

ASPECTS OF SUPERVISORY RELATIONSHIP	TRAINEES	SUPERVISORS	CHI-SQUARE	SIGNIFICANCE^a
ASPECTS OF SUPERVISORY RELATIONSHIP	N (%)	N (%)	CHI-SQUARE	SIGNIFICANCE^a
Effective collaboration on patient care decisions	41 (97.6)	55 (96.5)	0.10	NS
Supervisor has trainee best interest in mind	41 (97.7)	56 (98.3)	0.10	NS
Trainees feel respected and heard	39 (92.9)	55 (96.5)	0.67	NS
Comfort within the supervisory relationship	40 (95.2)	54 (94.7)	0.01	NS
Supervisor trust in trainee decisions	40 (95.2)	55 (96.5)	0.10	NS
Overall satisfaction with virtual training activities	41 (97.6)	53 (93.0)	1.08	NS
Supervisor has accurate view of trainee performance	41 (97.6)	52 (91.2)	1.73	NS
Supervisor provides useful feedback about performance	42 (100.0)	54 (94.7)	2.28	NS
Supervisor models professional behaviors in virtual settings	41 (97.6)	54 (94.7)	0.52	NS
Supervision includes professional development	42 (100.0)	54 (94.7)	2.28	NS
Satisfaction with frequency of virtual meetings	41 (97.6)	52 (91.2)	1.73	NS

Note: ^a Group comparisons did not meet statistical significance based on P < .01. Degrees of freedom for all measures = 1. Agreement combines “Somewhat Agree” and “Completely Agree” response options.

Perspectives on Virtual Training: Modality Preferences

Supervisors were significantly more likely to endorse a preference for fully virtual supervisor/trainee interactions (49.1%), while this preference was endorsed by only a minority of trainees (9.3%). Conversely, trainees predominantly preferred equal amounts of virtual and in-person interactions (46.5%) while this was less preferred by supervisors, 12.3%; χ(3) = 22.99, P < .01. There was a similar difference in opinion related to perceptions of the ideal mix of in-person and virtual interactions to support clinical training, with supervisors more likely to endorse fully virtual interactions (46.4%) relative to trainees (11.9%), and trainees more likely to endorse equivalent amounts of virtual and in-person interactions (54.8%) than supervisors, 16.1%; χ(3) = 20.36, P < .01 (Table 4).

Table 4.

Virtual Training Modality Preferences, Stratified by Role (Trainees vs Supervisors).

	TRAINEES	SUPERVISORS	CHI-SQUARE^a
Amount of preferred virtual training
	N (%)	N (%)
Prefer fully virtual	4 (9.3)	28 (49.1)	22.99*
Prefer mostly virtual	15 (34.9)	17 (29.8)
Prefer equal virtual/in-person	20 (46.5)	7 (12.3)
Prefer mostly in-person	4 (9.3)	5 (8.8)
Amount of virtual training thought to be most effective
	N (%)	N (%)
Fully virtual	5 (11.9)	26 (46.4)	20.36*
Mostly virtual	9 (21.4)	15 (26.8)
Equal virtual/in-person	23 (54.8)	9 (16.1)
Mostly in-person	5 (11.9)	6 (10.7)

Note: ^a Statistical significance determination based on P < .01. Degrees of freedom for all measures = 3.

Qualitative Results

Of the respondents who completed the survey, 17 (39.5%) trainees and 29 (50.8%) supervisors provided feedback to the open-ended question,

What feedback do you have for the Clinical Resource Hubs office about your experience with virtual training at CRH sites in VA?

Both trainees and supervisors reported that the virtual environment was a beneficial training modality, with several respondents noting that the virtual environment provided greater opportunities for skill and professional development relative to a traditional in-person training setting. Similarly, both groups reported that the virtual training environment allowed for expanded opportunities in available training experiences by providing greater variation in training scenarios, practice areas, resources, and interactions with different patient populations, with one trainee stating:

It has been very beneficial to learn about different sites and resources, and expand the practice areas to gain experience in.

Similarly, supervisors reported that the virtual training environment provided trainees with expanded and more immediate access to technology and resources, as well as supervisor support during patient crises, compared to in-person settings:

One distinct advantage of virtual training, over in-person training, is the immediate accessibility to information/resources that is available to the trainee as a result of being on their own laptop/desktop. In in-person training, a resource might be mentioned and then emailed to the trainee, or the trainee awkwardly looks on the supervisor's screen. In both cases, the information must be recalled or is often forgotten to be sent.

Interestingly, supervisors also reflected on the efficacy of virtual supervisor/trainee interactions. Overall, supervisors reported that virtual interactions allowed for greater training opportunities than in-person interactions and that the real-time practice associated with a virtual environment was more effective for “training, education, and supervision.”

Precepting in the virtual setting has allowed for educational enrichment that precepting in-person in clinic has not been able to provide.

While several supervisors reported virtual supervisor/trainee interactions to be equally if not more effective than in-person interactions, one participant noted that “virtual is ok for some aspects of training” but stated “in-person is usually better all around.”

Despite reporting many benefits, some respondents noted significant limitations associated with virtual supervisor/trainee interactions; however, trainees and supervisors differed in the types of limitations discussed. Specifically, trainees noted greater interruptions and communication challenges within the virtual training environment. Additionally, some trainees noted limited time for professional development and mentorship in the virtual space. One trainee stated,

The virtual space feels like it has a different etiquette, and so while there's plenty of time to cover clinical topics, it sometimes feels like there's less space for mentorship and professional development related discussions.

Supervisors reported that while the virtual environment provided greater opportunities for real-time practice and assessment, this modality introduced concerns regarding trainees’ development of physical assessment skills related to important clinical factors such as patients’ smell, height, and so on. Similarly, supervisors noted that virtual training is not well-suited for all trainees, particularly those training in specialty care, which requires more hands-on training and skill development. For example, one supervisor noted that “much of sleep care that [trainees] should develop competencies in is hands-on with durable medical equipment,” therefore indicating that some specialty care training should include in-person interactions.

Lastly, supervisors spoke to the time, coordination, and the relationship development aspect of interactions within a virtual environment. Supervisors reported that virtual interactions required careful planning when coordinating and planning sessions with trainees and that virtual interactions often included clinical activities, which were challenging to supervise, shadow, or demonstrate in a virtual environment:

Supervision and shadowing in the virtual world takes additional time to coordinate given we are not physically in the same location.

Additional information on the benefits and limitations of virtual training reported by supervisors and trainees can be found in Table 5.

Table 5.

Benefits and Limitations of Virtual Training Stratified by Role.

DOMAIN	KEY CHARACTERISTICS STRATIFIED BY ROLE		EXEMPLAR QUOTES
DOMAIN	SUPERVISORS	TRAINEES	EXEMPLAR QUOTES
Benefits of virtual training	Provides greater professional and skill development opportunities Increases trainees’ access to technology, supervisor support, and other resources Promotes more options in training opportunities (eg, greater variety in VA sites, resources, and patient populations, expands practice areas, etc) Reduces the need for trainees to travel for training opportunities	Provides greater professional and skill development opportunities Promotes greater number of options in training opportunities (eg, greater variety in VA sites, resources, and patient populations, expands practice areas, etc)	“Given the increase in virtual care across so many different practice areas, these CRH training experiences were really beneficial and much needed for my professional development.” (Trainee)“I am particularly grateful for how my supervisor struck an impactful balance between supporting me in achieving my developmental goals and expanding my clinical autonomy.” (Trainee)“I love that the residents can see more global views of our health system, can see differences in practice across sites, and think of their projects from a more VISN-level view to potentially create change at various sites.” (Supervisor)“Feel it has greatly expanded access to high quality precepting and learning experience opportunities for sites that wouldn't otherwise have access to those training opportunities” (Supervisor)
Limitations of virtual training	Takes more time and coordination 100% virtual training is not suited for all specialties May impact trainees’ development of physical assessment skills Inhibits relationship development between supervisors and trainees and within trainee cohorts	Limits time for professional development or mentorship Presents greater connection issues, interruptions, and challenges in communication	“The limitations and barriers presented by virtual training such as connection issues, interruptions, and collaborating about complex question and issues over teams are greater than those I have experienced in person.” (Trainee)“I do wish there was a bit more time allotted for supervisions with virtual supervisors, since often the thing that suffers is time for professional development.” (Trainee)“The only major disadvantage I see is losing some patient data with virtual clinical work (eg, smells, height, etc).” (Supervisor)“Less cohort cohesion with less in person collaboration together.” (Supervisor)

Discussion

This evaluation explored the experiences of multidisciplinary trainees and supervisors participating in HPE within learning settings that included high levels of virtual interactions. Information related to this topic is essential, given the increasing use of virtual training in HPE and the need for refinement of virtual training approaches. Trainees and supervisors perceived virtual supervisor/trainee interactions to be equivalent or superior to in-person interactions in supporting the development of clinical competency across multiple clinical areas, a finding which has not been well-explored in prior work on this topic. Both trainees and supervisors also described virtual interactions as equivalent or superior to in-person interactions in supporting the development of effective supervisor/trainee working relationships. These findings are consistent with prior studies that have highlighted positive experiences during virtual supervision related to communication and rapport building, trainee autonomy, and expanded educational opportunities.^{8–13,23–26}

Trainees and supervisors noted several benefits of a virtual training environment, including opportunities for expanded training opportunities by providing greater variation in training scenarios, practice areas, resources, and interactions with different patient populations. However, there were also some points of disagreement between the groups. Supervisors and trainees had significantly different perspectives regarding their preferences for virtual versus in-person interactions, as well as their views of the ideal combination of virtual and in-person interactions in supporting effective skill development. Notably, a larger proportion of supervisors preferred having fully virtual interactions, whereas a larger proportion of trainees preferred equal amounts of virtual and in-person interactions. Similarly, a larger proportion of supervisors perceived fully virtual interactions to be most effective at supporting clinical skill development, whereas a larger proportion of trainees perceived equal amounts of virtual and in-person interactions to be most effective. The potential implications of this difference in perspective are unclear, although both groups rated virtual interactions as overall effective at supporting skill development and the creation of effective supervisor/trainee relationships. Future studies should further investigate reasons for these differences in perspectives and any potential implications of these differences on HPE training effectiveness in environments which may include both in-person and virtual training activities.

This study utilized novel enhanced methods to support high levels of survey completion by supervisors and trainees, including the use of front-end advertising via organizational leadership channels and the use of embedded clinical communication tools (Microsoft Teams) to directly prompt survey completion. These methods may have contributed to much higher survey completion rates than those found in similar evaluations (38% of trainees; 74% of supervisors, compared to 12.7% of trainees responding to a national survey of VHA trainees).²⁷ These results point to the potential value of similar enhanced front-end awareness-raising and back-end tracking and prompting via embedded communication channels for other evaluation projects utilizing survey methods in large organizations to support elevated survey completion rates.

This evaluation has limitations. Despite modest response rates, the number of trainees and supervisors were both relatively small (54 trainees and 57 supervisors), and all participating in training within a unique setting (the VHA CRH). Because of this, caution is warranted in generalizing the results beyond this sample. The modest response rate also raises the potential for nonresponse bias, which may limit the representativeness of our sample, especially if those with strong positive or negative experiences were more likely to respond. As such, results should be interpreted with caution, recognizing the possibility that certain perspectives may be underrepresented. Future studies should consider ways to increase response rates, such as the use of incentives or additional outreach messaging. Additionally, when assessing outcomes related to skill development, trainees and supervisors were asked to consider their VHA training experiences outside of CRH as the basis of their responses, which was used as a proxy for in-person training experiences. Trainee and supervisor data were collected independently and were not analyzed based on supervisory dyads, limiting the ability to draw conclusions of supervisor/trainee relationships within the context of direct interactions with one another. Without linked supervisor/trainee responses, it was not possible to assess the extent to which trainee perceptions align with those of their supervisors or evaluate how specific supervisory approaches may influence trainee experiences. The assessment of clinical competency areas was based on ACGME global clinical areas, which may have led to some other important clinical skills being omitted from this evaluation. Similarly, given the broad focus on ACGME clinical areas across multiple clinical disciplines, there may be discipline-specific clinical competencies which were not assessed but might not be well-suited for training in a virtual environment. Lastly, the open-ended responses are limited, with only 39.5% of surveyed trainees and 50.8% of surveyed supervisors providing feedback.

There has been an increasing movement toward the integration of virtual training into HPE, with initial assessments of this integration finding that virtual supervisor/trainee interactions can effectively support trainee clinical skill development and effective supervisor/trainee supervisory relationships in a manner similar to traditional in-person interactions. Our findings suggest that, for many aspects of training, a virtual environment can provide a positive experience that is acceptable and a viable alternative to in-person HPE training. Additionally, the use of virtual modalities within HPE may expand learning opportunities for trainees interested in HPE within large healthcare settings like VHA who might otherwise be unable to participate in training due to logistical barriers. Further, HPE training within a virtual environment may help to prepare these trainees to eventually serve as healthcare clinicians within a mixed in-person and virtual care environment, as well as allow them to address broader institutional concerns such as clinician shortages through their ability to provide coverage via the use of virtual care delivery. It is important to recognize that the implications of a virtual training environment may vary considerably across HPE disciplines. Studies have found variations in participation rates for telehealth and virtual supervision among different health professions and specialties.^28–31 In one VHA study, associated health professions such as psychology, social work, and pharmacy had the highest participation rates, followed by physicians and nurses, indicating that certain disciplines may be more inclined to engage in virtual supervision and more amenable to virtual supervision than others.²⁸

As telehealth becomes an increasingly prominent mode of delivering care, HPTs should be offered exposure to diverse telehealth models when relevant to their training goals. Expanding HPT experience offerings within an interfacility hub and spoke model, such as the CRH, can mitigate training gaps and disruptions when facility-based trainee programs experience supervisor gaps due to extended leave and attrition. Delivering telehealth services in an evidence-based and effective way is an important core competency for HPTs, given the increasing prevalence of this modality and the infusion of virtual care services throughout VHA nationally.

There are many aspects of virtual HPE training that merit additional attention. Future evaluations into this topic would benefit from the use of qualitative methods to investigate factors that influence the effectiveness and quality of virtual training and assessment of patient outcomes or satisfaction, and explore the potential implications of supervisor/trainee differences in perspective related to the use of virtual supervisor/trainee interactions in HPE. Similarly, additional information related to best practices can help to standardize practices and further pave the way for additional adoption and refinement of virtual training as a standard aspect of HPE. Further, we did not collect information on the modality of supervisor/trainee interactions (group vs individual), which may have been differentially distributed across in-person and virtual training activities. If the modality of trainee/supervisor interactions is strongly related to trainee competency development or perceived quality of supervisory working relationships, it could represent an unmeasured third variable factor which may have led to misrepresentative results in our evaluation. Because of this, future evaluations on this topic would benefit from assessing the impact of trainee/supervisor interaction modality.

Conclusion

Given the growing use of telehealth and the changing landscape of health care delivery, VHA provides a unique opportunity to evaluate virtual training across multiple professions in a national clinical education setting. The VHA CRH supports health professions training by offering hands-on experience with telehealth technologies and practices under experienced VHA clinicians, exposing trainees to a wide range of patient populations and conditions. Findings from this study suggest that virtual training is seen as comparable, if not superior, to in-person methods in fostering clinical competencies and effective supervisor/trainee relationships within HPE. The CRH's virtual training model and telehealth platform can be replicated by other large health systems to reduce the need for physical presence and facilitate training across dispersed locations, ensuring high-quality education regardless of location.

Supplemental Material

sj-docx-1-mde-10.1177_23821205251380043 - Supplemental material for A Mixed Methods Evaluation Finds Virtual Training Among Health Professions Education Trainees and Supervisors as Effective as In-Person Training

Supplemental material, sj-docx-1-mde-10.1177_23821205251380043 for A Mixed Methods Evaluation Finds Virtual Training Among Health Professions Education Trainees and Supervisors as Effective as In-Person Training by Gezan M. Yahya, Kaylee W. Burgan, Linda M. Kawentel, Addison Ragan, Jennifer M. Doran and Nicholas W. Bowersox in Journal of Medical Education and Curricular Development

Footnotes

Acknowledgments

The evaluation team wishes to thank the VHA Clinical Resource Hub (CRH) Health Professional Trainee Workgroup members for providing feedback on the survey instruments and helping to promote the surveys to the field.

ORCID iDs

Gezan M. Yahya

Kaylee W. Burgan

Linda M. Kawentel

Addison Ragan

Nicholas W. Bowersox

Ethical Considerations

This study was designated as a nonresearch quality improvement study per official VHA guidance 1200.21 and was therefore exempt from IRB oversight. Trainees and supervisors were informed their participation was voluntary, that their responses were confidential, and that survey results would be reported in aggregate. Additionally, survey content and methodology were reviewed and approved by the VHA Organizational Assessment Committee (OAC) and VHA Office of Labor Management Relations (LMR).

Consent to Participate

Participation in the survey was completely voluntary.

Consent for Publication

I certify that all authors have seen and approved the manuscript, that each author has contributed significantly to it, and that this manuscript has not been published and is not under review elsewhere.

Author's Contributions

NB (lead), GY (supporting), KB (supporting), and LK (supporting): conceptualization; GY (lead), KB (equal), and LK (supporting): data curation; GY (lead) and KB (equal): formal analysis; LK (lead) and NB (equal): funding acquisition; LK (lead), GY (equal), and KB (equal): investigation; LK (lead), GY (equal), KB (equal), AR (supporting), JD (supporting), and NB (supporting): methodology; LK (lead), GY (supporting), KB (supporting), and NB (supporting): project administration; LK (lead), GY (supporting), and KB (supporting): resources; GY (lead) and KB (equal): software; LK (lead) and NB (supporting): supervision; NB (lead), LK (equal), GY (supporting), and KB (supporting): validation; GY (lead) and KB (equal): visualization; GY (lead), KB (supporting), LK (supporting), AR (supporting), JD (supporting), and NB (supporting): writing–original draft; GY (lead), KB (supporting), LK (supporting), AR (supporting), JD (supporting), and NB (supporting): writing–review and editing.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The author(s) received financial support for the program evaluation from the VHA Office of Connected Care.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data Availability Statement

The data that support the findings of this evaluation are available from the VHA QUERI Center for Evaluation and Implementation Resources (CEIR) and are available from the authors upon reasonable request with permission of CEIR and the Clinical Resource Hub (CRH).

Supplemental Material

Supplemental material for this paper is available online.

References

Shaver

. The state of telehealth before and after the COVID-19 pandemic. Prim Care. 2022;49(4):517-530.

Connolly

Stolzmann

Heyworth

, et al. Rapid increase in telemental health within the Department of Veterans Affairs during the COVID-19 pandemic. Telemed J E Health. 2021;27(4):454-458.

Phillips

Logan

Mather

. COVID-19 and beyond: telesupervision training within the supervision competency. Train Educ Prof Psychol. 2021;15(4):284-289.

Watters

Northey

. Online telesupervision: competence forged in a pandemic. J Fam Psychother. 2020;31(3-4):157-177.

Soheilian

O’Shaughnessy

Lehmann

, et al. Examining the impact of COVID-19 on supervisees’ experiences of clinical supervision. Train Educ Prof Psychol. 2023;17(2):167-175.

Perle

Zheng

. A primer for understanding and utilizing telesupervision with healthcare trainees. J Technol Behav Sci. 2023;9(1):1-7.

Vogt

Welch

Bunnell

, et al. Quantifying the impact of COVID-19 on telemedicine utilization: retrospective observational study. Interact J Med Res. 2022;11(1):e29880.

Kang

Shin

Han

, et al. Virtual supervision in graduate medical education: a systematic review. J Grad Med Educ. 2024;16(4):415-426.

Bernhard

Camins

. Supervision from afar: trainees’ perspectives on telesupervision. Couns Psychol Q. 2021;34(3-4):377-386.

10.

Tarlow

McCord

Nelon

, et al. Comparing in-person supervision and telesupervision: a multiple baseline single-case study. J Psychother Integr. 2020;30(2):383-393.

11.

Inman

Soheilian

Luu

. Telesupervision: building bridges in a digital era. J Clin Psychol. 2019;75(2):292-301.

12.

Jordan

Shearer

. An exploration of supervision delivered via clinical video telehealth (CVT). Train Educ Prof Psychol. 2019;13(4):323-330.

13.

Shore

Thurman

Fujinami

, et al. A resident, rural telepsychiatry service: training and improving care for rural populations. Acad Psychiatry. 2011;35(4):252-255.

14.

Perdew

Erickson

Litke

. Innovative models for providing clinical pharmacy services to remote locations using clinical video telehealth. Am J Health Syst Pharm. 2017;74(14):1093-1098.

15.

Litke

Spoutz

Ahlstrom

, et al. Impact of the clinical pharmacy specialist in telehealth primary care. Am J Health Syst Pharm. 2018;75(13):982-986.

16.

Burnett

Stockdale

Yoon

, et al. The clinical resource hub initiative: first-year implementation of the Veterans Health Administration regional telehealth contingency staffing program. J Ambul Care Manage. 2023;46(3):228-239.

17.

Qualtrics . Qualtrics [Version 2024]. Qualtrics; 2024. Retrieved from https://www.qualtrics.com.

18.

Bowersox

Jedele

Kawentel

. Quality Enhancement Research Initiative (QUERI) Evaluation Guide: Developing and Implementing a High-Quality Evaluation. https://www.queri.research.va.gov/ceir/ (2021).

19.

VHA Operations Activities That May Constitute Research. Program Guide 1200.21, Department of Veterans Affairs, Office of Research and Development.

20.

Ogrinc

Davies

Goodman

, et al. SQUIRE 2.0 (standards for quality improvement reporting excellence): revised publication guidelines from a detailed consensus process. BMJ Qual Saf. 2016;25(12):986-992.

21.

Kavic

. Competency and the six core competencies. JSLS. 2002;6(2):95-97.

22.

StataCorp . Stata statistical software: Release 18. StataCorp LLC; 2023.

23.

Elo

Kyngäs

. The qualitative content analysis process. J Adv Nurs. 2008;62(1):107-115.

24.

Thompson

Keenan-Miller

Dunn

, et al. Preferences for and acceptability of telesupervision among health service psychology trainees. Train Educ Prof Psychol. 2023;17(3):221-230.

25.

Augusterfer

O’Neal

Martin

, et al. The role of telemental health, tele-consultation, and tele-supervision in post-disaster and low-resource settings. Curr Psychiatry Rep. 2020;22(12):85.

26.

Chipchase

Hill

Dunwoodie

, et al. Evaluating telesupervision as a support for clinical learning: an action research project. Int J Pract Based Learn in Health Soc Care. 2014;2(2):40-53.

27.

Northcraft

Bai

Griffin

, et al. Association of the COVID-19 pandemic on VA resident and fellow training satisfaction and future VA employment: a mixed methods study. J Grad Med Educ. 2022;14(5):593-598.

28.

Harada

Falco

Bowman

, et al. Telehealth and virtual supervision practices for health professions education in the Department of Veterans Affairs. BMC Med Educ. 2025;25(1):314.

29.

Bauer

White

. Telehealth nursing: the impact of technology on healthcare delivery. J Telemed Telecare. 2020;26(8):452-461.

30.

Bashshur

Shannon

Krupinski

. The role of telemedicine in the future of health care delivery. Telemed e-Health. 2013;19(1):1-10.

31.

Kane

Gillis

. The use of telemedicine by physicians: still the exception rather than the rule. Health Aff (Millwood). 2018;37(12):1923-1930.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.05 MB