Abstract
As technology for remote learning advances, it is critical to understand how public health internship preceptors or faculty can provide engaging virtual experiential learning experiences for pre-professionals. We aimed to examine whether a virtual internship offered through a breastfeeding education company engaged learners to develop public health skills resulting in products beneficial for the internship site and learner. We provided a menu of tools to consider when developing virtual experiential opportunities. Master of Public Health students seeking dual-degrees in dietetics, nursing, and social work participated. Value of the interprofessional team, engagement, knowledge attainment, and translational skills were assessed through diverse modalities including surveys, oral communication, and products relevant to the internship site and interns’ academic program. Interns indicated that they valued the internship team and felt the team valued each intern’s opinion. Interns developed products useful to the internship site, suggesting active engagement in the virtual experience. Interns’ breastfeeding knowledge developed as the internship progressed demonstrated through oral communication as the content conveyed and discussed by interns advanced in cognitive level. The most frequently practiced translational skills reported by interns were research and communication. Virtual experiential learning can be hands-on resulting in professional skill development. This work aides in the understanding of how to feasibly implement an engaging remote internship.
To produce competent public health professionals, programs and schools accredited by the Council on Education for Public Health (CEPH) require Master of Public Health (MPH) students to develop competencies through an applied experience with an internship site (CEPH, 2016; Lockyer et al., 2017). Internship sites engage students to apply knowledge and skills to real-world projects while working in an interprofessional team (Hernandez et al., 2014; Wykoff et al., 2015) that address public health issues, and thus, are beneficial to the internship site’s mission, and also benefit the student in developing competence in the field (Comeau et al., 2019).
A significant competency that public health trainees strive for is the development of team behavior skills to function in an effective interprofessional team (CEPH, 2016). The Institute of Medicine and World Health Organization encourage inclusion of interprofessional education (IPE) within health professional curricula (UNM Health Sciences Center, n.d.), so health professionals learn from and with each other for the best population health outcomes.
For many, ideal internships may be out of reach due to personal or logistical reasons (e.g., cost, housing, family; Goodman, 2015). Virtual internships may provide students experiences that are equitable, flexible, and accessible (Banerjee & Firtell, 2017; Shapiro et al., 2017); yet many virtual experiences lack engagement and interaction required for sufficient attainment of skills to advance them as a professional in their field of work (Dumford & Miller, 2018; Gray & DiLoreto, 2016).
As remote educational experiences increase in prevalence, it is critical to understand how internship preceptors or faculty can engage students to develop relevant public health skills in a virtual experiential learning setting. We aimed to develop an engaging, hands-on virtual experiential opportunity offered by a breastfeeding education company and to determine if the experience engaged learners to develop public health skills that resulted in the creation of products mutually beneficial for the internship site and learner. We offered a menu of tools for public health preceptors or faculty to adapt in order to design and implement remote yet effective experiential learning opportunities.
Virtual Internship Framework
Internship Team
The facilitators included the preceptor and a staff member of the breastfeeding education company, one faculty member in public health, and a PhD candidate in public health. The facilitators had backgrounds in maternal–child health, social entrepreneurship, mammary gland biology, clinical lactation, and IPE. Interns included three MPH students seeking dual-degrees in nursing, social work, or dietetics.
Content Delivery
Internship communication across facilitators and interns was conducted through Slack and Google Hangouts. The facilitators applied backwards design to outline the interns’ work schedule in a “learner-centered” syllabus by aligning intended outcomes (skills and products) to activities (Nilson, 2009; Wiggins et al., 2005). Weekly content related to supportive environments for breastfeeding, collectively referred to as “public health perspectives of breastfeeding” (Rollins et al., 2016). Tasks/assignments in the first 8 weeks were designed to build interns’ knowledge and translational skills prior to product development weeks 9 through 16. Interns completed tasks and surveys designed to evaluate engagement, knowledge, and satisfaction (Table 1).
Summary Tools Used to Measure Intern Outcomes and Progress.
Note. CEPH = Council on Education for Public Health.
Logistics and Satisfaction
Interns reported the number of hours spent on activities each week. In a mid-, and postinternship survey, interns rated their agreement (6-point Likert-type: 1 = strongly disagree, 2 = disagree, 3 = somewhat disagree, . . ., 6 = strongly agree) with statements on overall team communication and clarity of learning objectives. In weekly surveys distributed on Wednesdays and Fridays, interns rated their agreement with statements about whether materials and activities were helpful for attaining learning objectives, workload, satisfaction, and value of the interprofessional team. Interns responded to open-ended questions about what worked well and what remained fuzzy. The facilitators reviewed surveys each week to identify areas needing improvement and adjusted content delivery accordingly.
Measures of Active Engagement
The facilitator–intern team met each Wednesday for 1 hour. Approximately 30 minutes was spent on self-reflection or presentations and 30 minutes on discussion with guided questions. Each meeting was audio–video recorded and then transcribed (Kaltura 2006, New York, NY). Two investigators rated the guiding questions by adapting a 3-point knowledge-level scale based on Bloom’s taxonomy cognitive domains (1 = low knowledge-level [recall]; 2 = medium [application]; 3 = high [synthesis]; Gonzalez-Cabezas et al., 2015).
In Friday surveys, interns rated their agreement (Likert-type scale, 1–6) in practicing translational skills. The number of skills practiced were aligned to CEPH competencies that fall under the three domains of: (1) analytical and assessment skills, (2) communication, and (3) interprofessional practice, leadership, and program planning. The aligned skills were quantified.
Attainment of Desired Translational Skills
In the mid-, and postinternship surveys, interns ranked five translations skills from the list of CEPH competencies that they felt most confident in and five they still hoped to gain. Facilitators tailored the tasks/assignments related to final work products to interns’ goals in order for them to obtain the indicated skills.
Development of Final Work Products
The preceptor developed projects (Supplemental File 1) with a deliverable (i.e., product) to benefit the internship site’s mission and tailored to interns’ self-identified goals. Utility of products for the internship site was measured through grant funding, publications, and social media engagement. Interns submitted products to their academic program to fulfill the applied experience requirement. Products were aligned to the CEPH competency domains and quantified.
Knowledge Attainment of Public Health Perspectives of Breastfeeding
A concept inventory is an educational assessment tool first implemented in higher education in the field of physics in 1985 to measure (1) student placement (prior to instruction), (2) instructional effectiveness, and (c) students’ misconceptions (Halloun & Hestenes, 1985; Hestenes et al., 1992). The usefulness of an inventory as an assessment tool in physics led educators to develop inventories suitable within their discipline, especially in the health and natural sciences (McCabe & Anderson, 2017). A 14-item multiple-choice concept inventory related to public health perspectives of breastfeeding was administered at the start and end of the internship to measure knowledge gains within and across the interns. In weekly surveys, interns rated their agreement (Likert-type scale, 1–6) in attaining learning objectives.
Interns completed 2-minute oral assessments by responding to an open-ended question on the week’s respective topic to test their highest level of comprehension (Girija et al., 2018; Hammonds & Schwarze, 2019). The oral assessment was completed Mondays before starting assignments and Fridays after completing assignments. Interns’ responses were recorded and transcribed (Kaltura 2006, New York NY). Two investigators rated each question and interns’ responses according to the 3-point knowledge-level scale.
Data Analysis
Descriptive statistics were used to calculate averages for agreement scores within survey instruments, hours spent each week on internship activities, and time spent discussing guiding questions on Wednesdays. The Mann–Whitney rank sum test was used to determine the average change between the pre–post oral assessment score each week and pre- and postconcept inventory. All interns consented to participation and audio–video recording. This study was approved as self-regulated through the University of Michigan Institutional Review Board (HUM00181201).
Virtual Engagement Translates to Public Health Skills and Knowledge
Interns Were Satisfied With the Virtual Framework
Interns spent an average of 24.3 hours per week on activities. Interns agreed that they valued their peers and facilitators and felt their opinion was valued in return (6.0/6.0 for both). Interns were satisfied (5.2/6.0) with the internship experience and communication between the team (5.6/6.0). Interns reported Wednesday and Friday discussions (5.9/6.0) as helpful for achieving learning objectives. Periodically, interns wrote that there was too much work or they needed clearer expectations.
Interns Were Actively Engaged
Of 22 guiding questions used in Wednesday meetings, most were at the applied or synthesis knowledge-level (n = 7 synthesis; n = 10 application; n = 5 recall). The majority of time (average 16.8/30 minutes) was spent discussing questions at the synthesis and application knowledge-level. As the internship progressed, time spent on application and synthesis knowledge-level questions increased (Figure 1).

Time spent discussing questions according to knowledge-level during the Wednesday discussion time.
Interns reported practicing skills within the three CEPH competency domains a total of 70 times. Interns agreed (average 5.3/6.0) that they practiced translational skills that aligned to weekly assignments (Supplemental File 2).
Interns Attained Translational Skills They Desired to Gain Competence in
By the midpoint, interns felt most confident in oral communication, specifically giving presentations in a virtual setting; working in teams and independently; and giving and receiving constructive feedback. Interns wanted to further develop networking and data collection and analysis. At the end of the internship, interns felt most confident in oral and written communication and qualitative and quantitative data collection and analysis.
Translational Skills Were Represented by Work Products
The internship site used interns’ sustainable, field-based products to apply for more than $10,000 in funding as of May 2020, present three abstracts, and write three research manuscripts relevant to the breastfeeding community, and engage their target audience through marketing and social media campaigns (8,700+ video views on YouTube in 5 months).
Interns submitted a total of eight products they developed during the internship to their academic program to fulfill their applied experience. The products aligned to the following CEPH competency domains: analytical and assessment skills (n = 3/8), communication (n = 3/8), and community dimensions of practice, interprofessional practice, leadership, or program planning (n = 8/8).
Interns Attained Knowledge Pertaining to Public Health Perspectives of Breastfeeding
Interns’ average score on the concept inventory increased by 3.3 points (p < .001) from 7.0/14 to 10.3/14. Interns were confident (5.1/6.0) that they met the weekly learning objectives.
Interns answered the oral assessment prompt at a higher knowledge level (+0.4/3.0, p = .007) after completing each week’s assignments. Mondays’ responses did not correlate (r2 = 0.04) while Fridays’ were highly correlated (r2 = 0.8) to the question’s knowledge level. For example, one week’s prompt asked interns to “describe how the WHO/UNICEF Baby-Friendly course prepares hospital staff to promote and support breastfeeding” (question rating = 3, synthesis). One intern initially responded, “the course is well-planned, utilized by expert consultants, and aimed at nurses” (rating = 1, recall). On Friday, the intern said, the course is aimed at nurses but could better prepare nurses for non-judgmental conversations with mothers who do not want to breastfeed, to understand reasons for reluctance . . . and to understand how empathetic conversations may [or may not] change a mother’s mind. (rating = 3).
Tools to Create an Effective, Virtual Experiential Learning Opportunity
We recognize that the internship offering reported within this article was intensive with regard to evaluation and assessment. By going through this intensive process, we were able to develop a menu of options for preceptors or faculty to engage public health trainees virtually in experiential learning. Faculty developing experiential learning opportunities within the context of the remote classroom can also adapt these elements to support student outcomes and to communicate intended goals with community partners. In addition to the tools and activities presented in Table 1, we provide Figure 2 as a “Sample Syllabus and Weekly Organizer” to assist preceptors or faculty in designing a framework for an experience that includes tools for evaluation and assessment, opportunities to define learning objectives, tips to engage team members in productive meetings, and ideas for remaining flexible.

Sample syllabus and weekly organizer.
Key Elements of the Virtual Internship Resulting in Competence of Our Future Public Health Workforce
The flow of self-directed learning to applying newly obtained skills to create work products resulted in a discourse across interns that evolved from the lowest knowledge level (recall) to the highest (synthesis). The oral assessments have learner-centered benefits in which the interns can engage and refine translational skills. Learners can self-assess presentation skills by listening to their recordings and in the process reflect on their progress and improve delivery of their message (Gong et al., 2019; Sinclair, 2016). The 2-minute oral assessment did not significantly add to the workload for the intern nor burdensome to evaluate for the preceptor.
It is common for interns to report feeling apathetic, disinterested or as if their presence was meaningless, or even worse, burdensome to the internship site (Crowell, 2018; Johansson et al., 2014). Consistent interactions throughout the internship supported a sense of community (vanOostveen et al., 2018). Like in residential learning environments where midpoint evaluations can counter elements of a course to improve student learning outcomes (Diamond, 2004; Overall & Marsh, 1979; Payette & Brown, 2018), consistent check-ins can be applied to virtual learning to create meaningful experiences for the learner and preceptor.
Given the interns’ pursuit of dual degrees or previous clinical experience, this experience resulted in interprofessional collaboration. IPE is challenging to integrate into discipline-specific curriculum due to scheduling conflicts (Anderson et al., 2019; Sunguya et al., 2014). Interns in this work immersed in IPE because of the flexible, virtual environment and mix of asynchronous and synchronous activities provided by the internship site. Given that interns reported they valued the opinions of facilitators and fellow interns and their opinion was valued in return, it is not surprising that interprofessional practice was the CEPH competency domain interns aligned to their submitted products most frequently.
Limitations
This was pilot work to test and evaluate the innovative pedagogies employed for an engaging virtual internship; thus, the sample size of three interns is small. The sample size makes it difficult to understand the generalizability of the results, yet the diverse plethora of activities and modes of evaluation allowed us to pilot and evaluate these innovative modes of learning in a safe environment with unique learners.
Conclusion
Immersive virtual learning experiences should be considered by internship preceptors or faculty as a customary option to support experiential learning, especially in a time when remote learning opportunities are a necessity. The tools we present create critical opportunities for sites to reach additional learners. Leveraging virtual technologies that support experiential learning is an innovative means to provide equitable learning opportunities for preprofessionals required to complete an internship as part of a program requirement. The immersive learning experience serves as a prime example of a public health internship site offering a virtual internship that models an engaging, virtual environment for public health students and internship sites.
Supplemental Material
sj-docx-1-php-10.1177_2373379920977537 – Supplemental material for Virtual Experiential Learning: A Description of an Internship Framework That Engages Students to Build Public Health Competencies
Supplemental material, sj-docx-1-php-10.1177_2373379920977537 for Virtual Experiential Learning: A Description of an Internship Framework That Engages Students to Build Public Health Competencies by Olivia S. Anderson, Carolyn F. McCabe, Samantha A. Chuisano, Emily Wicoff, Aria Grabowski and Anna Sadovnikova in Pedagogy in Health Promotion
Supplemental Material
sj-docx-2-php-10.1177_2373379920977537 – Supplemental material for Virtual Experiential Learning: A Description of an Internship Framework That Engages Students to Build Public Health Competencies
Supplemental material, sj-docx-2-php-10.1177_2373379920977537 for Virtual Experiential Learning: A Description of an Internship Framework That Engages Students to Build Public Health Competencies by Olivia S. Anderson, Carolyn F. McCabe, Samantha A. Chuisano, Emily Wicoff, Aria Grabowski and Anna Sadovnikova in Pedagogy in Health Promotion
Footnotes
Declaration of Conflicting Interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: AS is a shareholder, President, and Chief Executive Officer of LiquidGoldConcept, Inc. (LGC), the breastfeeding company sponsoring the internship described in this work. SC is a shareholder, Chief Operating Officer, and employee of LGC.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work and stipends for the interns were funded by The University of Michigan Center for Academic Innovation.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
