A Randomized Study Using Telepresence Robots for Behavioral Health in Interprofessional Practice and Education

Abstract

Background:

The events of the coronavirus disease 2019 (COVID-19) pandemic forced the world to adopt telemedicine frameworks to comply with isolation and stay-at-home regulations. Telemedicine, in various forms, has been used by patients and medical professionals for quite some time, especially telepsychiatry. To examine the efficacy and role of telesimulation as a method to educate health sciences students via telepresence robots. The study recruited students from the above health science disciplines. All participants were trained to administer a contextual interview to a standardized patient (SP) for mental health concerns.

Methods:

The completion of the contextual interview observation form adult (CIOF-A), National Aeronautics and Space Administration Task Load Index, self-efficacy in patient centeredness questionnaire (SEPCQ), and communication skills attitude scale with or without a telepresence robot. All participants completed baseline metrics and were trained to conduct a contextual interview to an SP. Researchers block-randomized the participants to either the telepresence robot group (TP) or in-person (IP) group.

Results:

The study recruited n = 43 participants to the IP group (n = 21) or TP group (n = 22). Mean participant demographics of age were 25.3 (±1.9) years in the IP group and 24.3 (±2.1) years for the TP group. Mean and standard deviation scores with effect sizes in CIOF-A scores IP: 0.05 (±1.91) and TP: −0.45 (±1.71), Cohen's d = 0.28; SEPCQ—Patient Domain scores IP: 0.42 (±4.69) and TP: 0.50 (±7.18), Cohen's d = 0.01; change in SEPCQ—Sharing Domain scores IP: 0.53 (±5.10) and TP: 0.91 (±9.98), Cohen's d = 0.05. These effect sizes will inform future studies and appropriate sample sizes.

Conclusion:

These data indicate that health sciences students utilizing a telepresence robot in an SP scenario to perform a behavioral health screening felt as comfortable and competent as those health sciences students performing the same behavioral health screening in person. ClinicalTrials.gov Identifier: NCT03661372.

Introduction

The recent events of a global pandemic forced the world to adopt telemedicine frameworks to comply with isolation and stay-at-home regulations.¹ Historically, improving access to health care via telemedicine leads to improved patient outcomes, increased compliance, and reduced health care costs.² While many perceived barriers are documented to offering telemedicine, including provider availability, skill set, willingness, liability insurance, reliable communication connections, and/or insurance reimbursement, rural populations have benefited from telemedicine.³ Even though telemedicine is found to be a safe, reliable, and an effective means of health care delivery for rural patients, there is still a dearth of services and practitioners offering telemedicine.⁴ Likewise, there is a dearth of health science institutions implementing telemedicine, telehealth, and/or telesimulation as part of a standardized curriculum.^5,6

To further highlight the necessity of telemedicine application in education, there continues to be further challenges in providing behavioral health via telemedicine.⁷ The decreased availability of mental health providers, combined with an increased need for behavioral health services, continues to negatively impact rural communities, which have a higher suicide rates compared with urban communities.⁸ Moreover, patients with mental illnesses in rural communities are less likely to access health care services, especially due to stigmas associated with mental health and more expenses to reach these services.⁹ As a result, more resources, providers, training, and access appear to be the recipe to tackle complex issues—but how?

More and more, the literature demonstrates the benefits of interprofessional care, practice, and education to understand the needs of a patient along the continuum of care.¹⁰ Interprofessional collaboration is associated with decreased morbidity and mortality in patients and cited to be a key strategy to promote access to care.¹¹ The importance of interprofessional collaboration using telemedicine for means of improving a patient's mental health could have a significant impact on all health outcomes.¹² Thus, health science students could benefit from early exposure to a standardized telemedicine curriculum with an interprofessional approach.

The current study aimed to assess attitudes, perceived proficiency, and confidence, and related telesimulation outcomes from a cohort of interprofessional health care students engaged in a standardized patient (SP) behavioral health screening. A secondary aim was to determine if implementing telesimulation to practice basic behavioral health interviewing, counseling, and agenda-setting skills enhances student's attitudes, proficiency, and confidence regarding telemedicine compared with in-person (IP) patient simulation. In this investigation, we seek to obtain more information on how health science students respond to behavioral health screening training and subsequently applying this knowledge to an SP via a robot.

Methods

Subjects

The Pacific Northwest University of Health Sciences (PNWU) Review Board was responsible for approving the study protocol. The study was designed as a prospective, block randomized-controlled, survey study. Participants were recruited from a variety of health care disciplines (n = 43) and academic institutions, including physician assistant, osteopathic medicine, nursing, nutrition, paramedicine, and pharmacy students. The study team recruited potential subjects by posting the study flyer on the Yakima Valley Interprofessional Education Consortium (YVIPEC) Facebook and Twitter accounts as well as e-mailing students using an approved YVIPEC e-mail distribution list from participating institutions. Participants were initially screened to ensure the following: (1) active enrollment in a YVIPEC health science program, (2) available for one 120-min study visit, and (3) aged 18–33 years (keeping the sample within generational definitions most exposed to technology).

The study staff used Research Electronic Data Capture software (REDCap) (Vanderbilt University, Nashville, TN) for tracking all screening criteria, participant information, and study outcomes. The research team provided each participant an electronic device for collection of the consent form, baseline demographics, and survey outcomes. Then, participants were block randomized (Fig. 1) to either the IP group: implementing a behavioral health assessment via face-to-face (n = 21) (health sciences student face-to-face with an SP), or the telepresence robot group (TP): implementing a behavioral health assessment via a robot (telepresence health sciences student with an SP) (n = 22).

Fig. 1.

Study design. NASA-TLX, National Aeronautics and Space Administration Task Load Index.

Interprofessional Training

The PsyD trained the participants for ∼45 min using a didactic lecture and a video instruction demonstrating different contextual interviews, comparing more effective interviews with less effective ones.^13,14 The demonstration allowed for the participants to use the contextual interview observation form adult (CIOF-A) to critique whether important elements of the contextual interview were present or not. At the conclusion of the contextual interview training, the students randomized to the TP were provided 5 min of additional instruction on how to use the robot via the web browser (Google Chrome, Mountain View, CA), a laptop camera, a laptop microphone, and a laptop's arrow buttons for direction control.

Medical Simulation

Simulation Center staff trained SPs using a traditional training process to prepare them for a mental health case. Furthermore, additional training on the CIOF-A process and in behavioral characterization appropriate for a mental health encounter was provided. The SPs were blinded to whether the encounter would be performed via robot or IP, until the actual session occurred when the robot or participant walked through the door. The SPs performed the behavioral health scenario in a simulation clinic examination room with each participant. All simulations were audio and video recorded in their entirety.

Telesimulation with SPs was performed utilizing the Ohmni Developers Kit robot (OhmniLabs, Santa Clara, CA) (Fig. 2). The student participant operated the robot in a different building on the same campus as the simulation clinic examination rooms. Technical training and assistance were provided to the robot telesimulation participants only during the 5 min before the encounter started. Once the telesimulation session engaged the SP, no assistance was provided. In addition, study and simulation staff supported the robot participants by knocking on the SP clinic room door, opening the door for the robot, and closing the door once the robot was safely in the clinic room (Fig. 3A–C).

Fig. 2.

Ohmni Developers Kit Robot (OhmniLabs, Santa Clara, CA).

Fig. 3.

Telepresence robot methods. (A) For the robot group, the study participant interfaces using the laptop webcam and browser to conduct the assessment. (B) Standardized simulation patient room and placement of the robot while the participant conducted the telemedicine session. (C) The robot participants (exactly like the in-person group) started the telesimulation scenario in the hallway before entering the standardized patient room.

In both groups, study staff stopped the simulation in one of the following conditions, whichever came first: (1) the resolution of the conversation from the health care student; (2) the resolution of the conversation from the SP, or (3) after 10 min had passed conducting the simulated scenario. Toward the end of the study visit and after the simulation, each participant was offered a debriefing session to ensure that the safety and comfort of the SP encounter and simulation experience were within normal/baseline limits.

Outcomes

Participants completed pre- and poststudy visit surveys of attitudes, perceived proficiency, complicated task assessment, and knowledge related to telemedicine using the self-efficacy in patient centeredness questionnaire (SEPCQ) and the communication skills attitude scale (CSAS).^15,16 During the SP simulation, the participant conducted the contextual interview and the SP independently completed the CIOF-A form. The SP completed a CIOF-A form after the session to aid in the assessment of the participant's performance (patient perspective). In addition, two medical professionals (PsyD and physician, DO) graded the participant using the CIOF-A form via either the synchronous two-way simulation monitoring system or the asynchronous watching of the recorded video of the simulation. In postsimulation only, the National Aeronautics and Space Administration Task Load Index (NASA-TLX) score was completed to evaluate the task load of performing the CIOF-A.

Statistical Analysis

REDCap data were imported into SPSSv25.0 software (IBM Corp.) and summarized by study group. Demographic and baseline study data were summarized by study group using appropriate descriptive statistics, including frequencies and percentages for categorical data and means and standard deviations (SDs) for numeric data. Initially, t tests were proposed; however, the pilot data revealed that a larger sample size was needed to accommodate the normality assumption. The changes in outcomes were determined by postintervention value minus preintervention value for CIOF-A summed responses and the individual domains for SEPCQ and subscales for CSAS. NASA-TLX was determined by a postintervention score using the weighted scoring method for each component and an overall total was determined for each participant.

The mean (SD) difference for each of the outcomes was displayed by the study group and Cohen's d effect sizes were determined and reported. Following the American Statistical Association's guidance on reporting p-values, only the effect sizes were displayed and interpreted for the outcomes rather than reporting nonparametric test results.¹⁷ Effect sizes were interpreted using Cohen's guide as follows: 0.20 is a small effect size, 0.50 is a medium effect size, and 0.80 is considered a large effect size.

Results

Demographically, both study groups are similar due to randomization where the groups were predominately Caucasian students who were unmarried in their mid- to late twenties in age on average (Table 1). Over 90% were full-time students with more than 75% in each group studying to become osteopathic physicians. Almost all participants had at least an associate or bachelor's degree with the modal income reported as <$40,000. The TP group experienced a small difference in the mean change in the CIOF-A summed response when compared with the IP study group (Cohen's d = 0.28, Table 2). More substantive differences were determined by mean changes in the CSAS positive subscale. The mean (SD) increase in the positive subscale was 2.50 (3.40) for the telesimulation group compared with 0.11 (3.27) for the IP group (d = 0.72). Similarly, the CSAS negative subscale showed a mean (SD) decrease for the telesimulation group, while the IP group increased on average [−0.65 (3.73) vs. 1.67 (5.34), d = 0.51]. Differences in the changes in the SEPCQ domains were considered small as each exhibited a Cohen's d effect size difference of <0.2. There were small to moderate differences in the task load as the telesimulation group had mean (SD) values less than the IP study group [52.8 (11.20) vs. 56.7 (10.67), d = 0.36].

Table 1.

Demographics/Baseline Characteristics

VARIABLE/STATISTIC	TELEPRESENCE ROBOT (n = 22)	IN-PERSON (N = 21)
Age (years)
Mean (SD)	24.9 (2.65)	27.5 (4.62)
Race, n (%)
Asian	6 (27.3)	7 (33.3)
African American	0	2 (9.5)
Caucasian	16 (72.7)	12 (57.1)
Education, n (%)
HS diploma	0	1 (4.8)
Some college	1 (4.5)	0
Associate or bachelor's degree	17 (77.3)	10 (47.6)
Master's degree	3 (13.6)	9 (42.9)
Doctoral degree	1 (4.5)	1 (4.8)
Marital status, n (%)
Married	7 (31.8)	4 (19.0)
Single	15 (68.2)	17 (81.0)
Income, n (%)
<$40K	16 (72.7)	20 (95.2)
$40K–$60K	2 (9.1)	1 (4.8)
>$60K	3 (13.6)	0
Not specified	1 (4.5)	0
Employment, n (%)
Employed part-time	2 (9.1)	2 (9.5)
Student	20 (90.9)	19 (90.5)
Student status, n (%)
Nursing	2 (9.1)	1 (4.8)
Osteopathic medicine	17 (77.3)	17 (81.0)
Pharmacy	3 (13.6)	3 (14.3)

HS, high school; SD, standard deviation.

Table 2.

Outcomes

OUTCOME/STATISTIC	TELEPRESENCE ROBOT (n = 22)	IN-PERSON (N = 21)	COHEN'S D EFFECT SIZE
Change in CIOF-A			0.28
Mean (SD)	−0.45 (1.71)	0.05 (1.91)
Change in SEPCQ—Patient Domain			0.01
Mean (SD)	0.50 (7.18)	0.42 (4.69)
Change in SEPCQ—Sharing Domain			0.05
Mean (SD)	0.91 (9.98)	0.53 (5.10)
Change in SEPCQ—Communication Domain			0.17
Mean (SD)	1.73 (4.79)	1.05 (2.78)
Change in CSAS—Positive Attitude Scale			0.72
Mean (SD)	2.50 (3.40)	0.11 (3.27)
Change in CSAS—Negative Attitude Scale			0.51
Mean (SD)	−0.65 (3.73)	1.67 (5.34)
NASA Task Load Weighted Index			0.36
Mean (SD)	52.8 (11.20)	56.7 (10.67)

CIOF-A, contextual interview observation form adult; CSAS, communication skills attitude scale; NASA, National Aeronautics and Space Administration; SEPCQ, self-efficacy in patient centeredness questionnaire.

Anecdotally, most health sciences student participants felt that conducting a behavioral health interview was challenging and that it is one of the most uncomfortable interviews they encounter. In addition, participants stated that performing the interview using a robot provided an “electronic emotional barrier” and as such, they felt more comfortable performing the task. However, the students admitted that their comfort may have come at the expense of increased difficulty establishing rapport with the patients. Finally, students who performed the telepresence robot interview felt that the conclusion of the interview was hindered by needing assistance with the door to exit the room.

Discussion

The results indicated no significant differences between the IP and TPs in NASA-TLX or CSAS scores. The NASA-TLX data suggest that the task load of using the robot for telesimulation purposed and delivering a behavioral health assessment was not significant compared with face-to-face implementation. The CSAS positive attitudes results suggest that performers using telepresence robot interviewing had more positive changes in their attitudes toward telemedicine interviewing. The differences may be associated with an increased familiarity with telesimulation procedures and a better understanding of what an interview can look like using the robotic technology. The CSAS negative attitudes data show a drop in negative attitudes toward telemedicine interviewing for the TP, which may also be due to increased familiarity.

Interestingly, live participants had a general increase in negative attitudes toward telemedicine interviewing, which may be due to establishing rapport and feeling that in person allowed better establishing of rapport, which is important in mental health. However, again both these changes were not statistically significant. The results from the SEPCQ increased for the telesimulation study group, which may relate to the amount of screen time and technology exposure current generations have before the study, or the increased comfort in conducting difficulty interviews with an “electronic emotional barrier,” however, this change was not significant.

These data indicate that participants utilizing telesimulation to perform the contextual interview felt as comfortable performing the interview in person. This provides promising data to further investigate teaching telemedicine through telesimulation in professional health care programs. The demographic data are consistent with groups that widely use social media with 88% of persons 18–29 years old utilizing some form of social media platforms such as Facebook.¹⁸ This may contribute to not only increased familiarity with communicating with other persons, including strangers online, but also increased prevalence of electronic use in day-to-day life increasing comfort in communicating electronically.

The need for health care providers to possess basic interviewing, counseling, and agenda-setting skills to ascertain patients' pertinent contextual (e.g., familial, social, cultural, spiritual) information and health behavior and risk factors is vital to identify behavioral health and chronic diseases as well as individualize treatment.^19,20 Technological advancements in psychiatry, including utilization of two-way video conferencing, have helped address the growing issue of access to mental health services.²¹

While telemedicine services in psychiatry have increased significantly, limited quantitative data exist that examine the training and implementation of the service.²² It is important that the ability of the practitioner, internal and external barriers to use, and patient satisfaction are explored when considering widespread implementation. Christensen et al. demonstrated that patient satisfaction with videoconferencing is independent of age but increases with the number of visits conducted via a video interview.²³ Another research by Taylor et al. found that external clinical facilitation was important in the initial implementation for postpartum telemental health services and also requires ongoing facilitation for the service to remain useful.²⁴ Furthermore, among psychiatric subspecialties such as childhood and adolescent psychiatry, there remain even less data about implementation of telepsychiatry by subspecialists.²⁵

This study presents several limitations, including a single-site and single-study visit, a small cohort of students from different educational backgrounds, and a robot and software system not currently being used in daily telemedicine practices. The intent of the study is pilot in nature and was intended to demonstrate the viability of using telesimulation with a robot in an SP clinic atmosphere. Furthermore, there is currently limited evidence that exists to suggest that outcomes in simulated scenarios correlate to improved understanding in health science curriculum, especially in curriculums that lack a standardized approach for telemedicine. During the time of this study, none of the participating health science programs had a standardized telemedicine curriculum. Ease of use for operating the robot may also be a limiting factor for applicability since it did not require an encrypted and secure login complying with Health Insurance Portability and Accountability Act firewalls.

Future studies should account for larger sample sizes, curriculum development, and patient perceptions with the use of telepresence robots to perform health screenings and telepresence physiological assessment. As approaches utilizing telemedicine technology increase within medicine, it is inevitable and necessary to ensure that future generations of physicians are trained on how to best utilize this approach to optimize the care provided to their patients.^26,27 Introductory curricula for medical students are developing but are still in their infancy.^28,29 Further study into the cost analysis, strengths, weakness, and pitfalls of this approach needs further investigation before widespread adoption occurs. Lastly, more studies investigating telemedicine delivery need to address and assess whether the tenets of osteopathic medicine are maintained, which may require the deployment of various countermeasures not observed in face-to-face care models.

Footnotes

Acknowledgments

The authors thank Eric McCoy, MD, MPH, Keith Monosky, PhD, Melissa Holm, David Bauman, PysD, Patrick Vigil, MD, PhD, Thomas Greenwood, DO, Derrick Owsley, Marlene Barsten, Chelan Shepherd, and Dustin Shepherd for helping with the study.

Disclosure Statement

The authors (J.R., Z.H., J.K., and L.S.) are affiliated to PNWU. The funding for the study provided supplies and stipends for participants.

Funding Information

Financial support for this study was provided by the Yakima Valley Interprofessional Practice and Education Collaborative (YVIPEC) and a grant from the American Association of Colleges of Osteopathic Medicine (AACOM) and Pacific Northwest University of Health Sciences Interprofessional Education and Practice Center.

References

Ohannessian

, Duong

, Odone

. Global telemedicine implementation and integration within health systems to fight the COVID-19 pandemic: A call to action. JMIR Public Health Surveill, 2020; 6:e18810.

World Health Organization. Telemedicine: Opportunities and developments in member states: Report on the second global survey on eHealth, Vol. 2. Geneva, Switzerland: WHO Press, 2009.

Berkman

, Sheridan

, Donahue

, Halpdem

, Crotty

. Low health literacy and health outcomes: An updated systematic review. Ann Intern Med, 2011; 155:97–107.

, Pujara

, Sutton

, Rhee

. Telemedicine in the management of type 1 diabetes. Prev Chronic Dis, 2018; 15:E13.

Chike-Harris

, Durham

, Logan

, Smith

, DuBose-Morris

. Integration of telehealth education into the health care provider curriculum: A review. Telemed J E Health, 2021; 27:137–149.

McCoy

, Sayegh

, Alrabah

, Yarris

. Telesimulation: An innovative tool for health professions education. AEM Educ Train, 2017; 1:132–136.

About Rural Health Care—NRHA. Ruralhealthweb.org. Available at https://www.ruralhealthweb.org/about-nrha/about-rural-health-care (last accessed November 3, 2019 ).

CDC Press Releases. 2016. Available at https://www.cdc.gov/media/releases/2017/p1005-rural-suicide-rates.html (last accessed November 3, 2019 ).

National Alliance on Mental Illness. Mental Health by the Numbers. Available at https://www.nami.org/Learn-More/Mental-Health-By-the-Numbers (last accessed September 18, 2017 ).

10.

Wheelan

, Burchill

, Tilin

. The link between teamwork and patients' outcomes in intensive care units. Am J Crit Care, 2003; 12:527–534.

11.

World Health Organization. Framework for action on interprofessional education and collaborative practice. Geneva, Switzerland: World Health Organization, Department of Human Resources for Health, 2010. Available at www.who.int/hrh/resources/framework_action/en (last accessed August 16, 2014 ).

12.

Nadeau

, Jaimes

, Rousseau

, Papazian-Zohrabian

, Germain

, Broadhurst

, Battaglini

, Measham

. Partnership at the forefront of change: Documenting the transformation of child and youth mental health services in Quebec. J Can Acad Child Adolesc Psychiatry, 2012; 21:91–97.

13.

Beachy

Developing an evaluation form for initial interviews: The Contextual interview observation form—adult (CIOF-A). Poster presented at the 50th Society of Teachers of Family Medicine annual conference. San Diego, CA, 2017.

14.

Bauman

, Beachy

, Ogbeide

. Stepped care and behavioral approaches for diabetes management in integrated primary care. In: O'Donahue

, Maragakis

, eds. Principle-based stepped care and brief psychotherapy for integrated care settings. New York, NY: Springer Science, Business Media, LLC, 2018.

15.

Zachariae

, O'Connor

, Lassesen

, Olesen

, Kjær

, Thygesen

, Mørcke

. The self-efficacy in patient-centeredness questionnaire—a new measure of medical student and physician confidence in exhibiting patient-centered behaviors. BMC Med Educ, 2015; 15:150.

16.

Yakhforoshha

, Shirazi

, Yousefzadeh

, Ghanbarnejad

, Cheraghi

, Mojtahedzadeh

, Mahmoodi-Bakhtiari

, Emami

SAH

. Psychometric properties of the communication skills attitude scale (CSAS) measure in a sample of Iranian medical students. J Adv Med Educ Prof, 2018; 6:14–21.

17.

Wasserstein

, Lazar

. The ASA statement on p-values: Context, process, and purpose. Am Stat, 2016; 70:129–133.

18.

Share of

U.S

. adults using social media, including

Facebook

, is mostly unchanged since 2018. Pew Research Center, 2019 . Available at https://www.pewresearch.org/fact-tank/2019/04/10/share-of-u-s-adults-using-social-media-including-facebook-is-mostly-unchanged-since-2018 (last accessed November 3, 2019 ).

19.

Rogers

CR.

The necessary and sufficient conditions of the therapeutic personality change. J Consult Psychol, 1959; 21:95–103.

20.

Egan

The skilled helper: A problem management and opportunity development approach to helping, 8th ed. Independence, KY: Thomson, 2006.

21.

Hasselberg

MJ.

The digital revolution in behavioral health. J Am Psychiatr Nurses Assoc, 2020; 26:102–111.

22.

Spivak

, Spivak

, Cullen

, Meuchel

, Johnston

, Chernow

, Green

, Mojtabai

. Telepsychiatry use in

U.S

. mental health facilities, 2010–2017. Psychiatr Serv, 2020; 71:121–127.

23.

Christensen

, Gildberg

, Sibbersen

, Skjoeth

, Nielsen

, Hansen

. Videoconferences and treatment of depression: Satisfaction score correlated with number of sessions attended but not with age. Telemed J E Health, 2020; 26:898–904.

24.

Taylor

, Kikkawa

, Hoehn

, Haydon

, Neuhaus

, Smith

, Caffery

. The importance of external clinical facilitation for a perinatal and infant telemental health service. J Telemed Telecare, 2019; 25:566–571.

25.

Khan

, Ramtekkar

. Child and adolescent telepsychiatry education and training. Psychiatr Clin North Am, 2019; 42:555–562.

26.

Kaonga

, Morgan

. Common themes and emerging trends for the use of technology to support mental health and psychosocial well-being in limited resource settings: A review of the literature: Common themes and emerging trends for eMental health in limited resource settings. Psychiatry Res, 2019; 281:112594.

27.

Wozney

, McGrath

, Gehring

, Bennett

, Huguet

, Hartling

, Dyson

, Soleimani

, Newton

. eMental healthcare technologies for anxiety and depression in childhood and adolescence: Systematic review of studies reporting implementation outcomes. JMIR Ment Health, 2018; 5:e48.

28.

Walker

, Echternacht

, Brophy

. Model for medical student introductory telemedicine education. JMIR Med Educ, 2019; 5:e12515.

29.

Waseh

, Dicker

. Telemedicine training in undergraduate medical education: Mixed-methods review. Telemed J E Health, 2019; 25:717–723.