Abstract
Background:
It is estimated that 16 to 25% of patients in hospital have diabetes and 1 in 25 inpatients with Type 1 Diabetes develop diabetic ketoacidosis (DKA). It is vital that non-specialist doctors recognize and appropriately manage diabetes emergencies. Simulation training is increasingly being used in healthcare and virtual reality (VR) based educational resources is transforming medical education. This study aimed to evaluate the use of virtual reality to help non-specialist clinicians manage clinical scenarios related to diabetes.
Methods:
This pilot project, titled ‘DEVICE’ (Diabetes Emergencies: Virtual Interactive Clinical Education) was developed in collaboration with Oxford Medical Simulation. Fully interactive immersive VR scenarios were created to stimulate real life diabetes emergencies. Users then received personalized feedback and performance metrics. Feedback surveys were provided before and after the participation in the VR scenario. Kirkpatrick’s training evaluation model was used.
Results:
Thirty-nine participants from 2 hospitals in UK provided feedback up to 3 months after attending the VR education sessions. Overall feedback was extremely positive, and participants found this immersive teaching experience very helpful. After use of virtual reality scenarios, the mean trainee confidence in managing DKA (on an 8-point Likert scale) increased from 3.92 (3.38-4.47) 95% CI to 5.41 (4.79-6.03) 95% CI (statistically significant). The VR study demonstrates Kirkpatrick level 3 in the follow up survey.
Conclusion:
VR based training scenarios in this pilot project increased confidence in managing diabetes emergencies and demonstrated positive changes in their behavior. VR education is a safe, useful and a well-liked training tool for diabetes emergencies.
Keywords
Introduction
There is an exponential increase in the number of people living with diabetes mellitus (DM), with the prevalence in adults projected to rise from 463 million (2019) to 700 million in 2045. 1 In the United Kingdom (UK), 4.7 million people live with diabetes, with a new diagnosis made every 2 minutes. 2
The National Diabetes Inpatient Audit (NaDIA) is an annual snapshot audit of Diabetes inpatient care in England and Wales. It is widely acknowledged that considerable improvements are needed in medication management, to help prevent insulin errors and Diabetic KetoAcidosis (DKA) in hospital. 3 As per the NaDIA 2017 findings, 1 in 4 to 6 hospital adult inpatients had diabetes. 4 Approximately 1 in 25 of all adult inpatients (4.3%) with type 1 diabetes mellitus (T1D) developed Diabetic Ketoacidosis (DKA) in hospital. It was estimated that 1 in 80 inpatients with diabetes (1.3%) needed injectable hypoglycemic rescue treatment, for a dangerously low blood glucose level during their stay in hospital.
The clinical care of people with diabetes (PWD) of any type is becoming increasingly complex as PWD are living longer with multiple comorbidities and polypharmacy. Although there has been a national improvement in inpatient diabetes team staffing, one-fifth of hospitals that participated in NaDIA had no diabetes specialist inpatient nurses to support patient care and other clinical teams (DISN). 5 Even when present, they are not always available out of hours to support diabetes care and hence non-specialist staff (general ward nurses and junior doctors) are the ones who address concerns regarding acute inpatient diabetes care needs, such as action on deranged blood glucose levels and diabetes emergency situations. The Trainees Own Perception of Delivery of Care (TOPDOC) Diabetes study 6 reported low confidence of trainee doctors in managing all aspects of diabetes care, especially around emergencies in diabetes.
Simulation based training is being increasingly used to health care professionals and has proven to improve preparedness of junior doctors for real world clinical situations. 7 Simulation Based Medical Education (SBME) supports the development of competencies in technical skills as well as human factors (or non-technical skills). Virtual reality (VR) is rapidly developing as a part of SBME.
Virtual reality (VR) is defined as the computer-generated simulation of a three-dimensional image or environment that can be interacted with in a seemingly real or physical way by a person using special electronic equipment, such as a Virtual Reality headset and hand controls. Virtual reality is increasingly being used in healthcare professional education.8,9 A systematic review of 31 studies concluded that VR improved post-intervention knowledge and skills outcomes of health professionals when compared with traditional learning or other types of digital education. 10 By encouraging the use of this practice in a safe learning environment, VR based training could play an important role in reducing the risk of harm to PWD.
Our aim in this pilot study was to explore the feasibility of using VR as a means of delivering effective training in diabetes emergencies for clinicians in training. This was assessed by measuring confidence in their ability to manage relevant clinical scenarios.
Methods
Setting
The DEVICE (Diabetes Emergencies: Virtual Interactive Clinical Education) study was developed to evaluate the role of VR based clinical scenarios as a diabetes training resource. 11 It was piloted at 2 neighboring hospital sites: University Hospital Southampton (UHS) and Portsmouth Hospital University Trust (PHT), England. Both sites enrolled junior doctors currently working in Emergency Medicine or Acute Medicine. Novo Nordisk, a pharmaceutical company, provided an educational grant to enable the pilot project to be developed.
The evaluation for the use of VR as a training tool was enabled through working with Oxford Medical Simulation (OMS), a professional VR simulation company. The clinical project team comprised of UK National Health Services (NHS) clinicians (2 Consultant Diabetologists, 1 trainee Consultant in Diabetes, 1 Emergency Medicine Consultant) and a person with type 1 diabetes, who collaborated with OMS to develop the diabetes emergency training scenarios. Four clinical diabetes scenarios were developed in order to provide an immersive and realistic experience. 2 were based on hyperglycaemia and 2 on hypoglycaemia. Each scenario was designed to take 15 minutes to complete by the learner. Immersion in this VR based environment allowed the user to work through the case, with the ability to interact with the “virtual” patient, “virtual nurse” nurse and to use relevant medical equipment and resources as they made their clinical assessment and formulated their clinical management plan (Figure 1).
Examining a virtual patient.
Participants and VR Training
In Southampton, doctors in training were invited to participate by email. Those that accepted the invite understood that they were committing to 2 training sessions. Session 1 provided time for a project pre-brief, time for the user to familiarize and orientate themselves in the VR environment using a VR headset with a gaming laptop and for the Diabetologist to outline what was to be expected. A safe learning environment was established, ensuring they were aware their individual performance would be anonymous, and the session was not intended to scrutinize or test their own clinical abilities, it was to look at VR as a tool for learning. Once confident to proceed, each trainee then completed one diabetes scenario to familiarize themselves with the setup. Session 2 on a separate occasion provided an opportunity for the learner to complete a different diabetes scenario (the DKA test scenario).
In Portsmouth, the trainees working in the Emergency Department (ED) were introduced to the concept by email and were informed that they would have the opportunity to participate in the study over the coming months. The educator set up the VR environment in a quiet area, close to the ED but away from real patients and approached trainees currently working in the ED to find a convenient time within their clinical shift that they could participate in the study. At a convenient time in the shift the trainee was taken to the VR environment and pre briefed like the trainees were in Southampton, establishing the safe learning environment (see above). They completed the familiarization scenario and then the test scenario back to back. They then returned to the ED to finish their shift.
The participants were able to manage the scenario in a digital environment using an Oculus Rift VR headsets. After putting on the head mounted display, the user is placed inside an experience allowing them to move around 360 degrees and can interact with the environment and virtual characters in a way that feels real. Patients can look more unwell or better making the experience even more real. Practice scenarios included 2 related to hypoglycaemia and 1 related to management of hyperglycaemia.
Each VR test case scenario started with the user/trainee being immersed with a first person view in the clinical diabetes emergency situation of DKA with a virtual patient with or without a virtual relative to help gather information. The participant could then interact using the VR technology to take a history, perform a physical examination (Figure 1), and get help from the virtual nurse with clinical management of the patient. For example, the nurse could start supplementary oxygen, fluids or take bloods only if directed by the trainee. Each immersive scenario would last a maximum of 15 minutes, and the trainee could exit the scenario earlier if needed. The patient in the scenario was unwell with DKA and would require the participant to perform a focused and thorough assessment with appropriate investigations and management for which the patient would have to correctly diagnose DKA. At the end of the case, 3 to 4 multiple choice questions. This was followed by an immediate post scenario debrief with the trainer, as well as a review of the decisions during the scenario. Each trainee attended twice and hence experienced 2 different clinical scenarios.
Data Collection
At the end of each scenario, the user was able to complete some associated relevant multiple-choice questions, before having a chance to debrief on their experience with the trainer and review their clinical performance based on feedback from the VR based resource itself. The trainees were also sent a curriculum competency mapped certificate for their ePortfolio (personal clinical training record).
The trainees were all sent an electronic survey by email to complete. This established their place of employment, grade and speciality and whether they had experience of VR in the past. They were asked (using 8-point Likert scales) about the suitability of the scenario to their level, if they felt it would improve their day to day practice and if it achieved the objectives.
Measures
The survey was collected by sending all particpants a link to a Google form to complete following participation.
Device study—initial feedback form (post VR sessions)
Demographics Training Grade Trust of employment Have you had experience of VR (gaming or education) (Lots of experience/Some experience/No experience)
The Scenario (1 to 8 Strongly disagree to Strongly agree) This scenario is suitable for your level This was useful to improving your day to day practice The scenario achieved the objectives
Confidence in managing the case discussed today BEFORE the session (1 being not confident at all and 8 being very confident)
Confidence in managing the case discussed today AFTER the session (1 being not confident at all and 8 being very confident
Are there other diabetes themed modules that you would like to try in the future (please provide brief explanations)?
How much of what you have learnt do you expect to implement at work. Please choose the statement that is closest to your expectation a. Little or nothing b. Less than half c. Half or more of what I learnt d. All or nearly all
How soon do you think you will start implementing your learning? a. More than 3 months b. Within 3 months c. Within the next month d. Within the next week
Impact of the session a. Interesting b. Irrelevant c. Inspiring d. Confusing e. Thought provoking f. Valuable g. Challenging h. Practical i. Complicated j. Boring k. Unhelpful l. Comprehensive m. None of the above
Would you recommend this teaching method to others a. Yes b. No
Your preference to this teaching method over the following (1 significantly preferring *method below* teaching and 8 significantly preferring VR a. PowerPoint b. Bedside Teaching c. Standard simulation teaching
Please list 3 things that you have learned from the session that you will apply to your clinical practice
Please state one thing that you would change in the session to improve it
Any other comments 3 month follow up survey
Training grade
Trust of employment
Speciality
Have you used the learning gained from the VR scenarios in your daily practice since you participated in the study?
How many times have you used the learning from the VR scenarios since you participated? a. None b. Once c. 2 to 4 d. 5 to 9 e. 10+ f. Other
What parts of the learning did you find most useful?
Have you shared the learning with others?
From your experience, did you retain more information from the VR scenarios than you would have from an eLearning module or lecture containing the same information?
Why do you think that is?
Comments
Data Analysis
Kirkpatrick’s model for evaluating Training Programs was also used.12,13 This evaluation tool has been used for over 60 years to look at training. The framework comprises of 4 levels - reaction, learning, behavior and results, as outlined below in Figure 2. It offers a means to grade the impact training has on an individual and wider group. It has been used in a meta-analysis 14 of team training in healthcare.
Kirkpatrick’s model for evaluating Training Programs.
For this study we used the following Kirkpatrick levels:
In any simulation based medical education, we want to improve patient care (level 4 which is a system wide change which could be measured with long term metrics like datix numbers, morbidity/mortality figures/inpatient DKA figures, etc.)
Ethics Approval and Consent Process for the Study
As per the NHS Health Research Authority decision tool, this study did not warrant NHS Research Ethics Committee Review.
Consent was taken from respondents on the form and were asked to check a box if they agreed to participate in the DEVICE study. Participants were given written material explaining the process which included VR session and pre and post questionnaires.
Statistical Analysis
Data was analyzed using Microsoft Excel. Percentages were used for assessment of training grade and for certain parts of the questionnaire. A paired t-test was used to generate the 95% confidence intervals. The level of significance is expressed in confidence intervals.
Results
Participants were from University Hospital Southampton (UHS) and Portsmouth Hospital University Trust (PHT), and 39 trainees completed the test scenario, all of whom had little or no previous experience with VR education or gaming.
The junior doctors included foundation doctors (FY1, FY2, FY3), senior house officers (SHO) in the General Practitioner Vocational Training Scheme (GPVTS) core medical trainees (CMTs), doctors in Internal medicine training (IMT), Acute Care Common Stem Specialty Trainee (ACCS ST), and trainee Advance Clinical Practitioners (ACPs). Table 1 summarizes the trainee demographics.
Demographics.
Hundred percent of participants deemed that the scenarios were suitable for their level and will improve their daily practice. 56% of participants felt that that they would use all of what they learned in daily practice. 42% of the trainees felt they would use more than half of what they learned in their daily practice.
In terms of the knowledge acquired from the VR based experience, 72% expected that they would use the knowledge within a week, 18% expected they would use the knowledge within the next month, 10% expected that they would use the knowledge within the next 3 months
Before VR the mean trainee confidence in managing DKA (on an 8-point Likert scale) was 3.92 (3.38-4.47) 95% CI.
After VR, the mean trainee confidence in managing DKA increased to 5.41 (4.79-6.03) 95% CI (statistically significant), see Figure 3.
Stacked bar chart showing change in trainee confidence after VR scenario on 8-point Likert scale.
The Likert scale scoring for whether the scenario was suitable for the level, usefulness in improving day to day practice, and achievement of scenario objectives is shown in Figure 4.
Stacked bar chart showing assessment of suitability, achievement of objectives and usefulness.
28% increase in confidence in ability to manage the diabetes emergencies after exposure to the VR scenarios. Learners went to Kirkpatrick stage 2 of 4
In terms of qualitative data, 94.87% found it interesting, 69.2% felt it was valuable, 79.5% felt it was practical, 2.5% found it confusing, 56.4% found it challenging, 43.6% found it thought provoking, 33.33% felt it was comprehensive, 38.5% found it inspiring and 7.7% found it complicated.
When the trainees were contacted for a review 3 months after their VR training session for follow up, 89% of trainees reported using the knowledge learned from their VR training in their daily practice. 75% had shared their diabetes learnings with others. 96% found that the VR scenarios were better for knowledge retention than “standard” eLearning modules. 96% would use the VR format again for clinical learning
Some of the comments made at 3 months have been included: “Really enjoyed using the VR would happily take part in a similar study again.” “Would love more VR cases to practice with” “Love the training, more sessions should be available.’
From this study we established that participants enjoyed the VR survey as per the survey (Kirkpatrick Level 1–Reaction). They learnt from it and the felt they would take it back to their workplace (Level 2—Learning) and the 3 month questionnaire demonstrated that they used the information/learning achieved in the study back to their practice (Level 3—Behavior). Level 4 is beyond the scope of this study.
Discussion
The DEVICE study not only had a positive impact on learning, but also had a positive impact in behavior in managing diabetes emergencies through VR based training scenarios. Diabetic KetoAcidosis (DKA) and Hyperosmolar Hyperglycaemic State (HHS) are potentially life-threatening emergencies which should not develop in hospital. Severe hypoglycemia apart from causing distress is also associated with increased morbidity and mortality, hence the need for proactive prevention and optimal management. It is of vital importance that the clinical workforce, though not necessarily diabetes specialists themselves, are proficient in being able to deliver basic diabetes management and respond appropriately to suspected and confirmed diabetes emergency states. This is at a time when low confidence in managing diabetes in hospital has been previously reported 6 and specialist diabetes expertise is not always readily available. Simulation is becoming a cornerstone in medical education and VR in particular has been adopted across medical fields. 15 However, there is no study to evaluate the use of VR training in diabetes emergencies. This pilot project sought to assess the value of the VR platform as a medium that could support clinical diabetes training.
A Pubmed search did not yield any prior studies on virtual reality and diabetes emergencies and our study appears to be the first such study. A pilot study to evaluate the role of VR in diabetes outpatient services in which the participant can immerse themselves as a patient with diabetes or the caregiver, concluded that diabetes immersive simulation had the potential and constitutes a specific contribution to the management of diabetes. 16 This experience was generally well received by patients and their loved ones. Another study to address the growing diabetes care demands, using VR cinema demonstrated improved cultural self-efficacy and diabetes attitudes among health care providers and administrators. 17
The feedback received following participation in the DEVICE project was extremely positive. The evaluation from participants suggested that VR is a useful, well liked educational tool to increase confidence in managing diabetes emergencies. The participants strongly felt that the knowledge gained would be applied in daily practice. Many positive comments were shared by the trainees expressing their preference to this method of training over other lecture/screen presentation based (eg, PowerPoint) and eLearning module based learning. They all recommended the training to others. One trainee said “. . ..honestly, this was the single most useful learning experience I’ve had so far in my medical training.” They felt it was immersive and helped them to retain the information better.
Delivering simulation using this virtual platform—either using headsets or on computer screens—could both serve as an educational adjunct for the practitioner at any point in their training. By using realistic simulation scenarios that can be repeated after receiving feedback, learner competence in managing such scenarios can be improved.
Implementing this as an educational approach across the entire UK and upskilling the entire junior trainee population may have a system wide effect in the long term (Kirkpatrick level 4)
Limitations
We do acknowledge several limitations to this study. Firstly, this is a small study and although the results are promising, larger studies are required and there is a plan is to roll this out nationwide. It was possible that low trainee uptake in Southampton could have been due to the fact that the sessions had to be delivered outside of working hours (due to lead diabetes clinician daytime commitments), which could have deterred willingness to participate
Secondly this VR experience does not evaluate teamwork A future approach could for example involve having multiple participants in the same scenario and we do now have the multiplayer version of the platform.
Lastly, the VR experience is relatively time consuming when trying to support a number of learners in one setting, especially if there is a trainer required to debrief each participant and it cannot be conducted in places without the equipment. Also the local guidelines may vary in terms of management and may need to be taken into account.
The next steps include evaluating larger groups and comparing a VR based approach to standard simulation training or other teaching methods. It would be useful to see if this knowledge does translate to better outcomes. Implementing this across the UK and upskilling the junior trainee population may have a significant system wide positive effect in the long term. However, if the VR kit is located in a shared learning area (eg, a library or VR training suite), once trained on how to use it, trainees could then use the platform as they would with eLearning modules by logging in to their account and selecting a module for learning. They can then repeat the module to “beat their score” and improve their skills. The alternative ‘distance learning’ option is to launch the scenarios from the trainee’s own computer without the VR headsets to use in their own time. These are both ways of encouraging deliberate practice and enhancing competence.
Conclusion
In conclusion VR is a useful and well -liked educational tool for trainee clinical staff which increases their confidence in managing Diabetes emergencies taking them to Kirkpatrick level 3. Training doctors using virtual reality is another example of modernizing the NHS, seeking to improve training, to improve patient care. “DEVICE” could be one of the key means to help educate non-specialist trainees in diabetes so they feel more confident and empowered. Future work should be targeted so that VR based training can be widely employed in improving diabetes care.
Footnotes
Acknowledgements
Oxford Medical Simulation and Novo Nordisk
Abbreviations
ACCS ST, Acute Care Common Stem Specialty Trainee; ACPs, Advance Clinical Practitioners; CI, Confidence Interval; CMTs, core medical trainees; DKA, diabetic ketoacidosis; DEVICE, Diabetes Emergencies: Virtual Interactive Clinical Education; DISN, diabetes specialist inpatient nurses; ED, Emergency Department; FY1, FY2, FY3, foundation doctors; GPVTS, General Practitioner Vocational Training Scheme; HHS, Hyperosmolar Hyperglycaemic State; IMT, Internal medicine training; NHS, National Health Service; NaDIA, National Diabetes Inpatient Audit; OMS, Oxford Medical Simulation; PWD, patients with diabetes; SBME, Simulation Based Medical Education; SHO, senior house officers; VR, virtual reality.
Declaration of Conflicting Interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: RM - none
MP - received speaker fees from AstraZeneca, Boehringer Ingelheim and Eli Lilly
BA - none
PK - none
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Novo Nordisk provided funding to support the piloting of the virtual reality scenarios but had no input into the project execution or its outcomes.
