Findings Favor Haptics Feedback in Virtual Simulation Surgical Education: An Updated Systematic and Scoping Review

Abstract

Background

Virtual simulations (VSs) enhance clinical competencies and skills. However, a previous systematic review of 9 RCT studies highlighted a paucity of literature on the effects of haptic feedback in surgical VSs. An updated systematic and scoping review was conducted to encompass more studies and a broader range of study methodologies.

Methods

A systematic literature search was conducted on July 31, 2023, in MEDLINE, Embase, and Cochrane. English language studies comparing haptic vs non-haptic conditions and using VSs were included. Studies were evaluated and reported using PRISMA-ScR guidelines.

Results

Out of 2782 initial studies, 51 were included in the review. Most studies used RCT (21) or crossover (23) methodologies with medical residents, students, and attending physicians. Most used post-intervention metrics, while some used pre- and post-intervention metrics. Overall, 34 performance results from studies favored haptics, 3 favored non-haptics, and the rest showed mixed or equal results.

Conclusion

This updated review highlights the diverse application of haptic technology in surgical VSs. Haptics generally enhances performance, complements traditional teaching methods, and offers personalized learning with adequate simulator validation. However, a sparsity of orienting to the simulator, pre-/post-study designs, and small sample sizes poses concerns with the validity of the results. We underscore the urgent need for standardized protocols, large-scale studies, and nuanced understanding of haptic feedback integration. We also accentuate the significance of simulator validation, personalized learning potential, and the need for researcher, educator, and manufacturer collaboration. This review is a guidepost for navigating the complexities and advancements in haptic-enhanced surgical VSs.

Keywords

virtual simulation haptic force feedback simulation surgery training

Introduction

Healthcare education has transitioned from conventional apprenticeship-based to a more simulation-focused model of teaching.^1–7 This more recent model allows healthcare learners to freely learn from mistakes and apply theoretical knowledge before treating patients in real-life.^1–7 Simulations can occur either in-person or virtually.^1,8 In-person simulations involve role-playing individuals (eg, standardized patient actors) and manikins.^1,8 In-person simulations thus require human resources and equipment which can be very expensive.^1,8 An alternative to in-person simulations are virtual simulations (VSs), which use computer hardware and software to achieve the same goal.^8,9 VSs provide a game-like environment where students can complete educational scenarios involving virtual patients or organs, similar in content to in-person simulations.¹⁰

Virtual simulations are effective in teaching students and can improve clinical competencies,¹¹ declarative knowledge,¹² procedural skills,^13,14 and clinical decision-making.¹⁵ However, VS technology is still emerging with new features being added to replicate realism.⁹ A large limitation in many VSs is the lack of force feedback (eg, haptics).^1,9 To guide our research, we define haptics as force feedback or any feedback that is delivered in vibrational form.¹⁶ The perception of haptics includes both tactile and somatosensory perception via the skin as well as kinesthetic proprioceptive feedback in other body parts (eg, muscles and ligaments).¹⁷ Haptics are thought of as being crucial in VSs, especially those relating to surgery as safe tissue handling and motor skill acquisition rely on haptic cues.^16,18–21

With the safer preference of laparoscopic, robotics, and minimally invasive surgical procedures over traditional surgical intervention,²² there is a clear need for surgical training that replicates realism during learning to ensure transferability of ex vivo skills to real-life situations.²³ As such, research on haptics is seen as essential, yet rarely assessed/incorporated in VSs.²³ This is evident as a prominent systematic review of haptics in VSs found only nine total studies to include in their review.²³ To the author’s knowledge, this review represents the sole review-based investigation comparing haptics vs non-haptic conditions in VSs. While the review by Rangarajan and colleagues²³ was a crucial and informative review, its search is limited up to 2019, which at the time was recent; however, many advances have been made regarding haptic integrations within VSs since. The rapid evolution of haptic technology necessitates the need to conduct an updated review.²⁴ Furthermore, the previous review focused primarily on randomized controlled trials (RCTs), and thus not capturing a wide array of studies within surgical education with other methodologies.²³ Small sample sizes due to specialized populations (eg, medical residents and physicians) are common in this field, and accordingly, many studies are unable to pursue an RCT methodology. As such, we argue that focusing on RCTs excludes many relevant studies and does not give a comprehensive picture of the research landscape. Consequently, an updated review is needed to include findings from all study types and include the plethora of studies that have proliferated since, including but not limited to RCTs.

Therefore, we conducted an updated scoping review using a systematic methodology to assess the evidence in the literature regarding the use of haptics within educational VSs. Towards this end, we were interested in the methodologies employed by studies, their use of simulators, and evidence for the use of haptics in VSs.

Material and Methods

Search Strategy

A scoping review was conducted in accordance with Arksey and O’Malley’s framework²⁵ and is reported using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews (PRISMA-ScR) guidelines.²⁶ The literature search was conducted by a health sciences librarian (AQ) to identify all relevant literature examining the use of haptics in virtual surgical simulations. The search strategy for the review was informed by the previous review as the previous review’s search was not reproducible.²³ The databases searched include Ovid MEDLINE, Ovid Embase, and Cochrane Central Register of Controlled Trials. The search period covered the databases’ inception to 31 July 2023,²³ (see supplemental material for complete search strategies).

Data Extraction and Selection Criteria

The results were compiled from the databases and duplicates were removed in EndNote 20 (Build 16860)²⁷ and Microsoft Excel.²⁸

After obtaining a list of studies from the search strategy, six reviewers were involved in blinded title/abstract and full-text screening processes. In both stages, pilot screenings were performed to mitigate any conflicts. Final screenings at each stage began once the pilot stage had achieved 80% agreement. Screening was conducted using Microsoft Excel.²⁸ After full-text screening, a data extraction chart was developed in Microsoft Excel and extraction included five reviewers (one senior and four junior reviewers). Junior reviewers formed 2 pairs. Independent calibration was also conducted prior to data extraction, and discrepancies were resolved for all reviewers by the senior reviewer. Once reaching sufficient agreement (at least 80%),²⁹ an additional pilot extraction was performed between the senior reviewer and both pairs of junior reviewers. The agreement was once again sufficient (above 80%),²⁹ and remaining data extraction was conducted by the two pairs of junior reviewers. Similar to the previous review,²³ a meta-analysis methodology was not undertaken due to the extensive heterogeneity of the data and methodologies. While studies could be grouped by experimental design, there existed considerable variability in terms of haptic devices, software choice, and metrics.

We included studies that: (1) were in English, (2) involved a VS, and (3) compared a haptic condition to a non-haptic condition. We excluded studies that (1) were in another language than English, (2) did not involve a VS, or (3) did not have a haptic-enabled and haptic-disabled condition/comparison. Gray literature (eg, conference papers) were also included if that had sufficient data and elaboration of results.

Data Synthesis and Study Assessment

Data were synthesized in a 3-step process: qualitative analysis, reporting of findings, and discussion of relevant results/conclusions.^30,31 The findings from each study were evaluated by a senior reviewer with research experience and a junior reviewer with medical expertise (undergraduate medical student). The purpose of this evaluation is to provide a broad summative overview for the readers. Study quality was also assessed systematically for all included studies. The medical education research study quality instrument (MERSQI) was utilized to assess all included studies due to its broad applicability to various study designs.³² The Critical Appraisal Skills Programme (CASP) RCT Standard Checklist³³ and The Joanna Briggs Institute (JBI) Critical Appraisal Checklist for Quasi-Experimental Studies³⁴ were also utilized to assess RCTs and experimental study designs, respectively.

Results

A total of 2782 potentially relevant studies were yielded in the initial search. After deduplication, 1707 studies remained. Following abstract and full-text screening, 51 were included for data extraction and synthesis (Figure 1).^20,35–84

Search Results and Study Characteristics

Of the 51 studies, our search captured all 9 studies in the previous review.^{20,35,39,43–46,48,64} Of the remaining 42 studies, 30 were from 2019 or earlier and 12 were new and relevant studies (Table 1). Crossover methodologies were employed most frequently (23 studies, 45.10%) followed by RCTs (21 studies, 41.18%). The remaining studies utilized quasi-experimental methodologies (7 studies, 13.73%). Notably, our review found 17 RCTs^{38,41,49,50,54,55,58,65,67–69,77,78,81–84} that were not included in the previous review, 11 of which were published before 2019^{38,41,49,50,54,55,58,65,67–69} and the rest being novel studies.^{77,78,81–84}

The most common population sampled were medical residents (14 studies), followed by novices (9 studies), physicians (8 studies), and medical students (7 studies). A minority of studies collected data from dentistry students (3 studies), and patients (2 studies). The remaining 8 studies sampled across a mosaic of populations (eg, medical students + physicians + biomedical scientists). Notably, studies describing their sample as “novices” did not provide any further information regarding participant classification (eg, medical resident, fellow or medical students). The mean sample size across studies was 24 (1 gray literature study did not explicitly report its sample size),⁶⁷ though sample sizes ranged from 1 to 79.

Most studies drew comparisons between haptics and non-haptic conditions by utilizing only post-intervention (29 studies, 56.86%) metrics, while 18 studies (35.29%) used both pre- and post-intervention metrics. A minority of studies (4 studies, 7.84%) conducted assessment of metrics over multiple days. Relatedly, prior to beginning data collection, 22 studies (43.14%) studies gave students oral instructions and allocated time for practice on the respective simulation device (eg, technological orientation). 10 studies (19.61%) gave only oral instructions while 7 studies (13.73%) only provided hands-on practice. 12 studies (23.53%) did not provide any technological orientation to participants.

Relevant Results from Included Studies

A variety of simulators were used across studies with the most used being the LapSim platform (6 studies). Other studies either built their own simulators with haptic feedback using Phantom devices (eg, a device with a handle attached to a mobile arm that participants can manipulate and feel haptic feedback)⁵² and/or utilized other VS platforms such as Lap Mentor and ProMIS. The assessed haptic and non-haptic tasks varied by studies but broadly fell into the following categories: surgical tasks (n = 101), dentistry-specific tasks (n = 31), and non-surgical tasks (n = 12). Surgical tasks included basics tasks such as tool insertion attempts, movement distance, camera angle manipulation, as well as complex tasks including economy of hand movement, and suturing. Dentistry-specific tasks also had basic and complex tasks such as mean drilling depth and mean buccolingual angle deviation, respectively. Lastly, non-surgical tasks consisted predominantly of survey-based metrics regarding the perceptions of the haptic-compatible simulator (eg, preference for haptic vs non-haptic environment). An in-depth presentation of all assessed tasks is presented in the supplemental material while a general summary is provided in Table 2.

The characteristics of haptics (eg, the sense that devices or simulators provided) were not elaborated on in twenty-four studies (n = 24). Among the studies that described these details, sense of collision or touch (n = 12),^{39,40,51,52,60–62,70,71,75,77,85} drilling (n = 7),^{49,58,59,64,78,82,83} and force changes due to interactions with different tissue types (eg, skin, adipose tissue, vessels, n = 4)^50,68,69,74 were the replicated haptic interactions. In two studies (n = 2),^47,65 haptics were used to provide guidance to participants to operate the haptic robot within a predefined path. Among studies that specified mechanical aspects of the device and delivery of haptics, the majority utilized haptic arms (n = 23) or laparoscopic simulators (eg, LapSim, Lap Mentor II; n = 10). In studies utilizing haptic arms, the most common device was the Phantom Omni arm which consisted of a mobile arm with six degrees of freedom that the participants could manipulate and feel haptic feedback generated via a control box.⁵² In laparoscopic simulator, instrument handles⁴¹ or trocars⁸⁶ were utilized to exert the haptic force. To replicate the multisensory nature of surgical practice,⁶¹ in addition to haptic, reinforcement of visual information (eg, shadows of instruments, tissue deformation) was added to enhance realism (n = 19).^{35,36,40,41,47,51,56,59–61,64,65,67,70,71,73,77,78,86} Sound effects during collisions and drilling were also reported to enhance realism (n = 8).^{49,60,71,75–77,82,83}

Studies Favoring Haptics (n = 34)

34 studies had results that, overall, favored the haptic condition over non-haptic while the summative results of only 3 studies^59,66,86 favored the non-haptic condition. The remaining studies either showed the two conditions to be equal/similar (n = 11)^{39,42,44,45,50,54,67,74,77,80,83} or had mixed results (n = 3)^52,68,81 such that some assessed variables favored the haptic condition while others favored non-haptics. The important results from included studies are summarized in Table 1 while a more nuanced picture of each task and the condition it favored is listed in Table 3.

Studies showcasing favorable outcomes for the haptic condition tended to display a variety of benefits in different aspects. For one, time to complete tasks and basic instrument handling tasks were often seen as being better in the haptic condition. Furthermore, the general trend across studies was that haptics helped students reach proficiency faster relative to non-haptics. Other general trends often yielded positive subjective ratings of haptic feedback and optimistic outlook regarding its integration into training, especially for fundamental tasks. These results are not unexpected as the previous review highlights many of the same, albeit with a much smaller number of studies (9 vs 51).

Studies With Mixed Results (n = 3)

Three studies demonstrated mixed results which favored the haptic condition only under certain aspects. Vamadevan and colleagues tasked residents to reach proficiency in either a haptic or non-haptic simulator (intervention) and then reach proficiency in the non-haptic simulator (3-6 week follow-up).⁸¹ The haptic group required less instructor assistance and reached proficiency faster during the intervention, while the non-haptic group reached proficiency faster during the follow-up test.⁸¹ Across both phases, the haptic group required significantly more instructor feedback during the follow-up than intervention while the non-haptic group required significantly less.⁸¹ The authors concluded that acquired skills were not transferable to a conventional non-haptic setting.⁸¹ Mixed results in terms of errors were demonstrated by Yovanoff and colleagues who assigned medical residents to either use a manikin (non-haptic) or a haptic robotic simulator to learn internal jugular central venous catheterization training followed by feedback from either an upper-level resident (manikin) or the simulator.⁶⁸ Results from pre- and post-tests showed that both groups improved their self-efficacy scores and the non-haptic group consistently had higher self-rating of performance.⁶⁸ However, the simulator group had significantly more feedback instances and their self-ratings of performance significantly predicted the simulator’s given score.⁶⁸ Interestingly, the haptic group had a significantly greater number of errors in general procedural steps while the non-haptic group had significantly more errors relating to pressure or torque applied.⁶⁸ Lastly, mixed results between objective and subjective outcomes were shown by LeBlanc and colleagues who tasked orthopedic residents with performing a surgical fixation of the ulna on either a virtual haptic simulator or sawbones (non-haptic) followed by the same procedure on the other simulator.⁵² Their performance was evaluated by examiners via a task-specific checklist, global rating scale, and time-to-completion.⁵² Residents in the haptic group performed significantly better in both the checklist score and global rating scale score.⁵² However, the haptic group took significantly longer (by approximately a minute) to finish the procedure.⁵² Residents found that the virtual simulator needed further improvement and that they would prefer the sawbones (non-haptic) simulator if given the choice.⁵²

Studies Favoring Non-haptics (n = 3)

Three studies identified negative results of haptic feedback in VSs. Hochman and colleagues compared printed temporal bone model (non-haptic) to a virtual haptic graphic surgical simulation of a similar model (haptic condition).⁵⁹ Participants found several mechanical properties of the physical bone model to be more comparable to a cadaveric bone than the virtual model.⁵⁹ There were no differences between models in anatomic features and both models were found to be productive resources for acquiring surgical skills.⁵⁹ While both conditions were rated similarly in terms of ease of use and a visual learning tool, the physical non-haptic model was ranked more effective for learning and considered the superior tool in other domains (eg, visual realism, bonelike properties, etc.).⁵⁹ Furthermore, a survey study conducted by Vapenstad and colleagues (2013) compared the opinions of surgeons with different levels of laparoscopic experience regarding two VS instrument ports: one with haptics, and another without.⁸⁶ While most surgeons said that haptic feedback was important, most thought that the non-haptic instrument port felt more realistic.⁸⁶ The haptic port was seen as having too high friction and most surgeons performed best without it.⁸⁶ Notably, the simulator was deemed to be the issue in the study.⁸⁶ A subsequent study by Vapenstad and colleagues (2017) trained participants either in a simulator group (with haptic feedback) or control (no training, non-haptics) prior to performing a cholecystectomy in a box trainer.⁶⁶ Video assessment of participants’ performance showed better performance of the non-haptic group in perception, bimanual dexterity, and efficiency.⁶⁶ Tissue handling was not different between conditions.⁶⁶ However, the authors noted, once again, that this negative training effect may be attributed to the poor mechanical performance of the simulator utilized.⁶⁶

Assessment of Methodological Quality

The mean MERSQI score across all papers was 12.82 (1.59). This score represents a moderate rating across studies.³² The lowest scored categories were related to sample size and validity of the evaluation instruments utilized. Scores for all categories are shown in Table 4. Results from the CASP RCT standard checklist and JBI critical appraisal checklist for quasi-experimental studies are shown in Tables 5 and 6, respectively.

Discussion

This new and more comprehensive review was an imperative update to the previous literature review that only drew on 9 studies.²³ Our search yielded 30 novel studies that were excluded from the previous review yet are relevant in providing insight into the use of haptics in vs These studies bolster this review and provide valuable insight to the field, especially as the previous review highlights a paucity of studies.²³ While the previous review was beneficial at the time, this rapidly evolving field of research necessitates a more thorough summary of the literature, especially one that better conforms to expectations and standards for a review (eg, higher number of included studies). Towards this end, since 2019, 12 additional relevant studies have been published, over double the quantity of studies included in the previous review, further demonstrating the need for an update.

Our review introduces novel investigations that separates it from the previous study.²³ While both reviews delve into study characteristics, populations, and tasks, our work stands out by encompassing a broader array of study designs, including crossover studies and quasi-experimental approaches, in addition to RCTs, providing a more comprehensive review of research methodologies. Moreover, our review delves into the characteristics of haptic feedback, examining aspects like collision and touch, drilling, and force changes during interactions with different tissue types. This detailed exploration enriches the understanding of haptic technology’s impact on learning and skill acquisition. Additionally, our review employs a more extensive range of methodological quality assessment tools, such as the MERSQI, CASP RCT standard checklist, and the JBI critical appraisal checklist for quasi-experimental studies. This rigorous evaluation enhances the reliability of our findings and emphasizes the strengths of our review in advancing the field.

Most studies utilized crossover and RCT methodologies to compare haptics and non-haptics. While RCTs are the gold standard for evaluation interventions, most studies had small samples sizes (as low as n = 8).⁸⁷ This casts doubt on whether sufficient power for analyses existed and whether statistical assumptions were met.⁸⁸ Indeed, working with highly specialized populations (eg, surgical residents, physicians) poses a barrier to conducting rigorous research with ample sample sizes; however, such a limitation ultimately impacts the validity of findings.⁸⁹ Future work needs to strategically select research designs to bolster samples size whenever possible. For example, crossover methodologies can be utilized so that each participant provides two data points rather than one.⁹⁰ This aids in achieving adequate power to detect statistical significance, providing higher quality results.⁹⁰

Results also showed that most studies used oral instructions and hands-on practice to prepare students before the intervention, but nearly 25% didn’t provide any orientation. This lack of orientation can potentially skew results because unfamiliar participants might initially perform poorly but improve with practice, leading to potential misinterpretation of findings.⁹¹ This is especially important in our sample of studies as the most common research design was crossovers. Additionally, most studies relied on post-intervention assessments for comparisons, while some used pre/post designs, and only handful used longitudinal methods. Pre/post designs provide a more nuanced understanding of learning as they can factor in variance between groups and within participants.⁹² Future research comparing haptic and non-haptic approaches should combine verbal instructions and hands-on practice before assessment, and move beyond traditional post-intervention assessments. A recommendation to follow longitudinal designs may be unreasonable given the specialized study cohorts within surgical education. As such, future VS work should at the very least utilize pre/post assessment styles whenever possible.

Our review found that haptic technology’s application in surgical VS education reveals notable heterogeneity, influenced by the specific surgical domains it serves. In areas like dentistry and orthopedic surgery, the prevalent choice is to simulate drilling sensations, although the empirical support for their realism and effectiveness is limited. Furthermore, only four studies aimed to evaluate haptic feedback delivered through complex textures of various tissues, highlighting a gap within the literature. Interestingly, within laparoscopic simulations, the primary emphasis has centered on honing fundamental technical skills, such as instrument manipulation, grasping, and precise dissection in virtual environments. However, there exists an evident need for further advancements in replicating and delivering the nuanced tactile properties of soft tissues and solid organs, suggesting a promising avenue for continued research and development in this intersection of technology and surgical training for both educators and VS manufacturers.

Furthermore, our review showcases that haptic feedback has a generally positive impact on assessed tasks. Remarkably there were only three studies that demonstrated definitive drawbacks of haptic integration—though in two, the authors inferred this to be due to their device’s poor mechanics.^66,86 The third study compared a printed temporal bone model with a virtual haptic simulator of a similar model.⁵⁹ A printed model of bone is inherently more comparable to cadaveric bone, and hence better captures realism. However, the virtual haptic model was seen as having no differences in anatomic features, ease of use, ease of visual learning, and acquiring surgical skills relative to the printed model.⁵⁹ This highlights the benefits of haptic integration within VSs as an adjunct to traditional teaching methodologies. It also demonstrates that when there is limited exposure to animal models and cadavers (eg, for novices and medical students), VSs with haptic feedback may be an invaluable alternative.⁹³ Notably, we also found that many studies with non-significant results had means that often favored the haptic condition.

Given the limited sample sizes of included studies, perhaps many were unable to attain enough statistical power to detect significance.⁹⁴ It is therefore plausible that any levels of medical learners could benefit from haptic feedback for their technical skill acquisition and maintenance. For residents, haptic feedback can be a tool to aid the early stages of skill acquisition while experienced physicians may use it to refine and enhance pre-existing skills, especially those in surgical specialties. Those in specialties where haptic cues are less critical may not benefit from haptic feedback to the same degree. This is because haptic feedback seems to benefit most in procedural training than training for other types of skills (eg, problem-solving, communication skills). Thus, surgical VSs should consider integrating haptic feedback whenever possible.

Next, few studies also showed mixed results where the haptic condition was favored in select assessed tasks but not others. Vamadevan and colleagues showcased that haptics led to accelerated learning but not the transferability of those skills to other non-haptic environments.⁸¹ Transferability of technical skill acquired during simulations into clinical practice is crucial.⁹⁵ However, since the simulation (eg, in a haptic VS simulator) and real-life practice both serve as haptic conditions, these results are not necessarily against haptic-based VS training. Further work needs to be conducted to assess transferability of skills from virtual reality simulation to another real-life environment. Next, a study by Yovanoff and group demonstrated how haptic feedback can be a great tool for learning pressure or torque cues during training.⁶⁸ However, they show that haptic feedback is not a one-size-fits-all solution for other aspects of surgical training such as procedural knowledge (eg, locating the needle on ultrasound).⁶⁸ This provides support for haptic integration as a valuable educational tool for certain aspects of surgical training but not as a replacement for traditional training. Thus, it’s crucial to choose the appropriate training method considering the specific task being trained, as haptic feedback may not be universally effective for all task types. Interestingly, Yovanoff and colleagues also reported the frequency of feedback to be significantly higher using a simulator rather than a human instructor.⁶⁸ This adds to the potential benefit of utilizing VS simulators as they can provide feedback more frequently, and in doing so, potentially alleviate human resources for more personalized teaching and learning. Importantly, the utilization of VSs for feedback should only be considered if the simulator has been adequately validated as frequent feedback does not necessarily mean good quality or accurate feedback.⁹⁶ Lastly, LeBlanc and colleagues show how VS with haptic feedback offers significantly more objective results in standardized evaluations but is undesirable to residents due to its novel nature, partly explaining the longer task completion time.⁵² As such, we suggest progressive integration of haptic simulators starting with basic skill (eg, knot tying) at early stages of curricular training followed by progression to complex surgical tasks (eg, tumor resection in a haptic-enabled VS) at later stages to allow students to gain familiarity over time. Curricular integration should always be complemented with hands-on practice with the simulators prior to training sessions. By employing these guidelines, we can best tailor to the specific needs of learners and the procedures that they are involved in while optimizing the benefits of haptic technology.

Limitation of Studies

In conducting this review, we rigorously assessed included studies. Many studies included in this review lost points in the MERSQI assessment due to a combination of small sample sizes and a lack of reporting on the validity evidence of their evaluation instruments.³² Most studies also lost points as they were single-centre studies. These limitations underscore the challenges inherent in conducting surgical education research, particularly when working with highly specialized populations. Additionally, the CASP RCT Standard Checklist and the JBI Critical Appraisal Checklist for Quasi-Experimental Studies observed that many studies fell short in achieving blinding of participants and researchers, a practical challenge often encountered in the context of medical training.^33,34 The limitations in study quality, such as small sample sizes and challenges with blinding, should be considered when interpreting the results of this review. These limitations highlight the need for future research to address methodological shortcomings to enhance the robustness of findings in this field.

Next, the studies in this review are also subject to several inherent limitations. Firstly, the use of simulators across studies was extremely heterogeneous with varying companies, software versions, and levels of validation. This impacts interpretation of results across studies. For example, discrepancies in outcomes between studies employing the same simulator may arise from the use of different software versions. Failure to disclose such details can potentially lead to erroneous conclusions regarding the efficacy of a simulator (eg, deemed falsely ineffective due to outdated software). As such, future studies should explicitly note the version of the software they used. Furthermore, there is a need for standardization in the field to ensure interoperability and consistency in validation of VS hardware, requiring collaborations between researchers and VS manufacturers. Prioritizing partnerships with academic institutions and researchers will facilitate proper integration of haptic-enabled VSs, helping to ensure their alignment with proven educational outcomes. Continuous collaboration is crucial for tailoring simulator development to evolving training needs and ensuring a more effective and user-centric virtual surgical education experience. Moreover, these inconsistencies prevail in the types of methodologies adopted for training sessions and performance assessments. While the general study design may remain the same, nuances in how data is collected, and varying use of specific instruments makes direct comparisons across studies challenging. These also present challenges when attempting to draw generalizations. As such, conducting a meta-analysis of these studies was not feasible. The wide-ranging differences across these parameters preclude meaningful statistical synthesis and underscore the importance of a qualitative assessment approach in this comprehensive review.

Conclusion

In conclusion, this updated review underscores the need for a fresh perspective on the application of haptic technology in surgical VSs. We highlight the rapid advancements made since the previous review and identify many studies not encompassed by the previous publication. Drawing on a significantly expanded pool of studies, we observed a notable heterogeneity in haptic utilization across surgical domains. Dentistry and orthopedic surgery have leaned toward simulating drilling sensations, although empirical support remains limited. Conversely, laparoscopic simulations primarily target fundamental technical skills but lack the replication of nuanced tactile properties in soft tissues and solid organs. Our findings generally highlight the positive impact of haptic feedback on assessed tasks, with only a few studies showing mixed and negative results. Importantly, these results do not undermine the value of haptic-based virtual surgical training but underscore its role as an adjunct to traditional teaching methodologies. Frequent haptic feedback in virtual simulators is advantageous for efficient and personalized learning, although careful consideration of simulator validation is essential. In summary, haptic integration in surgical education holds promise for enhancing training, emphasizing the need for continued research and development in this field.

Supplemental Material

Supplemental Material - Findings Favor Haptics Feedback in Virtual Simulation Surgical Education: An Updated Systematic and Scoping Review

Supplemental Material for Findings Favor Haptics Feedback in Virtual Simulation Surgical Education: An Updated Systematic and Scoping Review by Sayed Azher, Aralia Mills, Jinzhi He, Taliah Hyjazie, Junko Tokuno, Andrea Quaiattini, and Jason M. Harley in Surgical Innovation

Supplemental Material

Supplemental Material - Findings Favor Haptics Feedback in Virtual Simulation Surgical Education: An Updated Systematic and Scoping Review

Footnotes

Acknowledgments

The authors would like to acknowledge undergraduate volunteers TK and LJ for their help during the screening and data extraction phases.

Author Contributions

Sayed Azher: Writing of main manuscript; performing title and abstract screening; performing full-text screening; analysis and synthesis of result; working on tables & figures; contributing to final review and editing of manuscript. Aralia Mills: Performing title and abstract screening; performing full-text screening; working on tables & figures. Jinzhi He: Performing title and abstract screening; performing full-text screening; working on tables & figures. Taliah Hyjazie: Performing full-text screening; evaluation of study results; working on tables & figures. Junko Tokuno: Contributing to discussion section of manuscript relating to the use of different types of haptic feedback in included studies. Andrea Quaiattini: Providing expert guidance on the planning, formatting, and execution of the overall manuscript; conducting search for the manuscript; contributing to final review of manuscript. Jason M. Harley: Providing expert guidance on the planning, formatting, and execution of the overall manuscript; providing a supervisory role to the primary author; contributing to final review of manuscript.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by start-up funding from the Research Institute of the McGill University Health Centre (RI-MUHC) awarded to the senior author, J.M. Harley. This work was also supported in-part by funding from the Fiera Capital Awards for Diversity, Equity and Inclusion in Health Care Research from the RI-MUHC, and in-part by a Social Sciences and Humanities Research Council of Canada (SSHRC) doctoral fellowship awarded to the first author, S. Azher [Award number 767-2023-2212, 2023].

ORCID iDs

Sayed Azher

Jason M. Harley

Supplemental Material

Supplemental material for this article is available online.

References

Al-Elq

. Simulation-based medical teaching and learning. J Fam Community Med. 2010;17(1):35-40. doi:10.4103/1319-1683.68787

Mueller

Cyr

Bank

, et al. The steinberg centre for simulation and interactive learning at McGill University. J Surg Educ. 2017;74(6):1135-1141. doi:10.1016/j.jsurg.2017.05.022

Cook

Hatala

Brydges

, et al. Technology-enhanced simulation for health professions education: a systematic review and meta-analysis. JAMA. 2011;306(9):978-988. doi:10.1001/jama.2011.1234

Pickering

Swinnerton

. Exploring the dimensions of medical student engagement with technology-enhanced learning resources and assessing the impact on assessment outcomes. Anat Sci Educ. 2019;12(2):117-128. doi:10.1002/ase.1810

Banfield

Wilkerson

. Increasing student intrinsic motivation and self-efficacy through gamification pedagogy. Contemp Issues Educ Res. 2014;7(4):291-298.

Motola

Devine

Chung

Sullivan

Issenberg

. Simulation in healthcare education: a best evidence practical guide. AMEE Guide No. 82. Med Teach. 2013;35(10):e1511-e1530. doi:10.3109/0142159X.2013.818632

Waurick

Weber

Bröking

Van Aken

. The European working time directive: effect on education and clinical care. Curr Opin Anesthesiol. 2007;20(6):576-579. doi:10.1097/ACO.0b013e3282f0ef61

Azher

Cervantes

Marchionni

Grewal

Marchand

Harley

. Virtual simulation in nursing education: headset virtual reality and screen-based virtual simulation offer a comparable experience. Clin Simul Nurs. 2023;79:61-74. doi:10.1016/j.ecns.2023.02.009

Kononowicz

Woodham

Edelbring

, et al. Virtual patient simulations in health professions education: systematic review and meta-analysis by the digital health education collaboration. J Med Internet Res. 2019;21(7):e14676. doi:10.2196/14676

10.

Gorbanev

Agudelo-Londoño

González

, et al. A systematic review of serious games in medical education: quality of evidence and pedagogical strategy. Med Educ Online. 2018;23(1):1438718. doi:10.1080/10872981.2018.1438718

11.

Cook

McAloon

O’Neill

Beggs

. Impact of a web based interactive simulation game (PULSE) on nursing students’ experience and performance in life support training--a pilot study. Nurse Educ Today. 2012;32(6):714-720. doi:10.1016/j.nedt.2011.09.013

12.

Kleinert

Heiermann

Wahba

Chang

Hölscher

Stippel

. Design, realization, and first validation of an immersive web-based virtual patient simulator for training clinical decisions in surgery. J Surg Educ. 2015;72(6):1131-1138. doi:10.1016/j.jsurg.2015.05.009

13.

Aebersold

Voepel-Lewis

Cherara

, et al. Interactive anatomy-augmented virtual simulation training. Clin Simul Nurs. 2018;15:34-41. doi:10.1016/j.ecns.2017.09.008

14.

Fernández Alemán

Carrillo de Gea

Rodríguez Mondéjar

. Effects of competitive computer-assisted learning versus conventional teaching methods on the acquisition and retention of knowledge in medical surgical nursing students. Nurse Educ Today. 2011;31(8):866-871. doi:10.1016/j.nedt.2010.12.026

15.

Green

Wyllie

Jackson

. Virtual worlds: a new frontier for nurse education? Collegian. 2014;21(2):135-141. doi:10.1016/j.colegn.2013.11.004

16.

Culbertson

Schorr

Okamura

. Haptics: the present and future of artificial touch sensation. Annu Rev Control Robot Auton Syst. 2018;1(1):385-409. doi:10.1146/annurev-control-060117-105043

17.

Flesher

Collinger

Foldes

, et al. Intracortical microstimulation of human somatosensory cortex. Sci Transl Med. 2016;8(361):361ra141. doi:10.1126/scitranslmed.aaf8083

18.

Minogue

Jones

. Haptics in education: exploring an untapped sensory modality. Rev Educ Res. 2006;76(3):317-348. doi:10.3102/00346543076003317

19.

Van Bruwaene

De Win

Miserez

. How much do we need experts during laparoscopic suturing training? Surg Endosc. 2009;23(12):2755-2761. doi:10.1007/s00464-009-0498-z

20.

Cao

CGL

Zhou

Jones

Schwaitzberg

. Can surgeons think and operate with haptics at the same time? J Gastrointest Surg. 2007;11(11):1564-1569. doi:10.1007/s11605-007-0279-8

21.

Rodrigues

Horeman

Sam

Dankelman

van den Dobbelsteen

Jansen

. Influence of visual force feedback on tissue handling in minimally invasive surgery. Br J Surg. 2014;101(13):1766-1773. doi:10.1002/bjs.9669

22.

Cui

Liu

Tan

. Comparison of laparoscopic surgery versus traditional laparotomy for the treatment of emergency patients. J Int Med Res. 2019;48(3):0300060519889191. doi:10.1177/0300060519889191

23.

Rangarajan

Davis

Pucher

. Systematic review of virtual haptics in surgical simulation: a valid educational tool? J Surg Educ. 2020;77(2):337-347. doi:10.1016/j.jsurg.2019.09.006

24.

Shazhaev

Mihaylov

Shafeeg

. A review of haptic technology applications in healthcare. Open J Appl Sci. 2023;13(2):163-174. doi:10.4236/ojapps.2023.132013

25.

Arksey

O’Malley

. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19-32. doi:10.1080/1364557032000119616

26.

Tricco

Lillie

Zarin

, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467-473. doi:10.7326/M18-0850

27.

EndNote | The best reference management tool. EndNote. Accessed July 22, 2023.https://endnote.com/

28.

Microsoft Excel Spreadsheet Software | Microsoft 365. Accessed July 22, 2023.https://www.microsoft.com/en-ca/microsoft-365/excel

29.

Belur

Tompson

Thornton

Simon

. Interrater reliability in systematic review methodology: exploring variation in coder decision-making. Sociol Methods Res. 2021;50(2):837-865. doi:10.1177/0049124118799372

30.

Bussières

Al Zoubi

Stuber

, et al. Evidence-based practice, research utilization, and knowledge translation in chiropractic: a scoping review. BMC Complement Altern Med. 2016;16:216. doi:10.1186/s12906-016-1175-0

31.

Thomas

Law

. Research utilization and evidence-based practice in occupational therapy: a scoping study. Am J Occup Ther. 2013;67(4):e55-e65. doi:10.5014/ajot.2013.006395

32.

Reed

Cook

Beckman

Levine

Kern

Wright

. Association between funding and quality of published medical education research. JAMA. 2007;298(9):1002-1009. doi:10.1001/jama.298.9.1002

33.

Critical Appraisal Skills Programme . CASP Randomised Controlled Trial Checklist. Critical Appraisal Checklists. 2022. https://casp-uk.net/casp-tools-checklists/. Accessed October 6, 2023.

34.

The Joanna Briggs Institute (JBI) . Critical Appraisal Checklist for Quasi-Experimental Studies. Critical Appraisal Tools. https://jbi.global/critical-appraisal-tools. Accessed October 6, 2023.

35.

Kim

Rattner

Srinivasan

. Virtual-reality-based laparoscopic surgical training: the role of simulation fidelity in haptic feedback. Comput Aided Surg. 2004;9(5):227-234. doi:10.3109/10929080500066997

36.

Gerovich

Marayong

Okamura

. The effect of visual and haptic feedback on computer-assisted needle insertion. Comput Aided Surg. 2004;9(6):243-249. doi:10.3109/10929080500190441

37.

Gosling

Westphal

Hufner

, et al. Robot-assisted fracture reduction: a preliminary study in the femur shaft. Med Biol Eng Comput. 2005;43(1):115-120. doi:10.1007/BF02345131.[Internal Pubmed Exact] [CS: 100].

38.

Cohen

Vora

, et al. Multicenter, randomized, controlled trial of virtual-reality simulator training in acquisition of competency in colonoscopy. Gastrointest Endosc. 2006;64(3):361-368. doi:10.1016/j.gie.2005.11.062

39.

Ström

Hedman

Särnå

Kjellin

Wredmark

Felländer-Tsai

. Early exposure to haptic feedback enhances performance in surgical simulator training: a prospective randomized crossover study in surgical residents. Surg Endosc. 2006;20(9):1383-1388. doi:10.1007/s00464-005-0545-3

40.

Hedman

Ström

Andersson

Kjellin

Wredmark

Felländer-Tsai

. High-level visual-spatial ability for novices correlates with performance in a visual-spatial complex surgical simulator task. Surg Endosc. 2006;20(8):1275-1280. doi:10.1007/s00464-005-0036-6

41.

Hogle

Widmann

Ude

Hardy

Fowler

. Does training novices to criteria and does rapid acquisition of skills on laparoscopic simulators have predictive validity or are we just playing video games? J Surg Educ. 2008;65(6):431-435. doi:10.1016/j.jsurg.2008.05.008

42.

Chmarra

Dankelman

van den Dobbelsteen

Jansen

. Force feedback and basic laparoscopic skills. Surg Endosc. 2008;22(10):2140-2148. doi:10.1007/s00464-008-9937-5

43.

Panait

Akkary

Bell

Roberts

Dudrick

Duffy

. The role of haptic feedback in laparoscopic simulation training. J Surg Res. 2009;156(2):312-316. doi:10.1016/j.jss.2009.04.018

44.

Salkini

Doarn

Kiehl

Broderick

Donovan

Gaitonde

. The role of haptic feedback in laparoscopic training using the LapMentor II. J Endourol. 2010;24(1):99-102. doi:10.1089/end.2009.0307

45.

Thompson

Leonard

Doarn

Roesch

Broderick

. Limited value of haptics in virtual reality laparoscopic cholecystectomy training. Surg Endosc. 2011;25(4):1107-1114. doi:10.1007/s00464-010-1325-2

46.

Zhou

Tse

Derevianko

Jones

Schwaitzberg

Cao

CGL

. Effect of haptic feedback in laparoscopic surgery skill acquisition. Surg Endosc. 2012;26(4):1128-1134. doi:10.1007/s00464-011-2011-8

47.

Lyu

Lin

Huang

Yau

. Experience-based virtual training system for knee arthroscopic inspection. Biomed Eng Online. 2013;12:63. doi:10.1186/1475-925X-12-63

48.

Våpenstad

Hofstad

Langø

Mårvik

Chmarra

. Perceiving haptic feedback in virtual reality simulators. Surg Endosc. 2013;27(7):2391-2397. doi:10.1007/s00464-012-2745-y

49.

Liu

Fang

Wang

Yeh

. Three-dimensional, virtual technologies and haptic simulation temporal bone surgical training. Otolaryngol Neck Surg. 2013;149(2_suppl):P159-P160. doi:10.1177/0194599813496044a55

50.

Kulcsár

O’Mahony

Lövquist

, et al. Preliminary evaluation of a virtual reality-based simulator for learning spinal anesthesia. J Clin Anesth. 2013;25(2):98-105. doi:10.1016/j.jclinane.2012.06.015

51.

Sengül

Rognini

van Elk

Aspell

Bleuler

Blanke

. Force feedback facilitates multisensory integration during robotic tool use. Exp Brain Res. 2013;227(4):497-507. doi:10.1007/s00221-013-3526-0

52.

LeBlanc

Hutchison

Donnon

. A comparison of orthopaedic resident performance on surgical fixation of an ulnar fracture using virtual reality and synthetic models. J Bone Joint Surg Am. 2013;95(9):e60, S1-S5. doi:10.2106/JBJS.K.01284

53.

Bouhelal

Badiani

Alaraimi

Patel

. Laparoscopic training in virtual reality (VR): haptic vs. none haptic. Int J Surg Lond Engl. 2013;11:704. doi:10.1016/j.ijsu.2013.06.628

54.

Balci

Tas

Hazar

, et al. Applicability and effectiveness of virtual reality simulator training in urologic surgery: a double-blind randomised study. Nobel Med. 2014;10:66-71.

55.

Joseph

Jehl

Maureira

, et al. Relative contribution of haptic technology to assessment and training in implantology. BioMed Res Int. 2014;2014:413951. doi:10.1155/2014/413951

56.

Park

Nawabi

Christopher

Conditt

Ranawat

. Robotic-assisted femoral osteochondroplasty is more precise than a freehand technique in a Sawbone model. J Hip Preserv Surg. 2015;2(2):136-144. doi:10.1093/jhps/hnv007

57.

Chao

Chalouhi

Bouhanna

Ville

Dommergues

. Randomized clinical trial of virtual reality simulation training for transvaginal gynecologic ultrasound skills. J Ultrasound Med. 2015;34(9):1663-1667. doi:10.7863/ultra.15.14.09063

58.

Koo

Kim

Donoff

Karimbux

. An initial assessment of haptics in preclinical operative dentistry training. J Investig Clin Dent. 2015;6(1):69-76. doi:10.1111/jicd.12065

59.

Hochman

Rhodes

Kraut

Pisa

Unger

. End user comparison of anatomically matched 3-dimensional printed and virtual haptic temporal bone simulation: a pilot study. Otolaryngol Head Neck Surg. 2015;153(2):263-268. doi:10.1177/0194599815586756

60.

Sano

Wake

Ichinose

, et al. Tactile feedback for relief of deafferentation pain using virtual reality system: a pilot study. J Neuroeng Rehabil. 2016;13(1):61. doi:10.1186/s12984-016-0161-6

61.

Erolin

Lamb

Soames

Wilkinson

. Does virtual haptic dissection improve student learning? a multi-year comparative study. Stud Health Technol Inform. 2016;220:110-117.

62.

Girod

Schvartzman

Gaudilliere

Salisbury

Silva

. Haptic feedback improves surgeons’ user experience and fracture reduction in facial trauma simulation. J Rehabil Res Dev. 2016;53(5):561-570. doi:10.1682/JRRD.2015.03.0043.

63.

Juo

Abiri

Tao

, et al. Artificial palpation with da Vinci surgical robot using a novel haptic feedback system. Sages. 2019. Published July 19 https://www.sages.org/meetings/annual-meeting/abstracts-archive/artificial-palpation-with-da-vinci-surgical-robot-using-a-novel-haptic-feedback-system/. Accessed October 6, 2023.

64.

Hagelsteen

Langegård

Lantz

Ekelund

Anderberg

Bergenfelz

. Faster acquisition of laparoscopic skills in virtual reality with haptic feedback and 3D vision. Minim Invasive Ther Allied Technol. 2017;26(5):269-277. doi:10.1080/13645706.2017.1305970

65.

Xiong

Chng

Chui

. Shared control of a medical robot with haptic guidance. Int J Comput Assist Radiol Surg. 2017;12(1):137-147. doi:10.1007/s11548-016-1425-0

66.

Våpenstad

Hofstad

Bø

, et al. Lack of transfer of skills after virtual reality simulator training with haptic feedback. Minim Invasive Ther Allied Technol. 2017;26(6):346-354. doi:10.1080/13645706.2017.1319866

67.

Kolarski

Yao

Chen

Monteiro

Lee

Vescan

. Learning curve of a virtual reality simulator (Neurotouch) for endoscopic sinus surgery. Journal of Neurological Surgery Part B: Skull Base. 2018;79:A192. doi:10.1055/s-0038-1633607

68.

Yovanoff

Chen

Pepley

, et al. Investigating the effect of simulator functional fidelity and personalized feedback on central venous catheterization training. J Surg Educ. 2018;75(5):1410-1421. doi:10.1016/j.jsurg.2018.02.018

69.

Dwisaptarini

Suebnukarn

Rhienmora

Haddawy

Koontongkaew

. Effectiveness of the multilayered caries model and Visuo-tactile virtual reality simulator for minimally invasive caries removal: a randomized controlled trial. Oper Dent. 2018;43(3):E110-E118. doi:10.2341/17-083-C

70.

Francone

Huang

Tsao

Rosen

Hubschman

. The effect of haptic feedback on efficiency and safety during preretinal membrane peeling simulation. Transl Vis Sci Technol. 2019;8(4):2. doi:10.1167/tvst.8.4.2

71.

Camara

Mayer

Darzi

Pratt

. Intraoperative ultrasound for improved 3D tumour reconstruction in robot-assisted surgery: an evaluation of feedback modalities. Int J Med Robot. 2019;15(2):e1973. doi:10.1002/rcs.1973

72.

Hagelsteen

Johansson

Ekelund

Bergenfelz

Anderberg

. Performance and perception of haptic feedback in a laparoscopic 3D virtual reality simulator. Minim Invasive Ther Allied Technol. 2019;28(5):309-316. doi:10.1080/13645706.2018.1539012

73.

Deng

Singh

Wheeler

, et al. P1566 Evaluation of haptic feedback for interaction with volumetric image data in virtual reality. Eur Heart J - Cardiovasc Imaging. 2020;21(Supplement_1):jez319.986. doi:10.1093/ehjci/jez319.986

74.

Chen

Sonntag

Mirkin

, et al. From the simulation center to the bedside: validating the efficacy of a dynamic haptic robotic trainer in internal jugular central venous catheter placement. Am J Surg. 2020;219(2):379-384. doi:10.1016/j.amjsurg.2019.10.026

75.

Choi

Schmutz

. Usability evaluation of 3D user interface for virtual planning of bone fixation plate placement. Inform Med Unlocked. 2020;19:100348. doi:10.1016/j.imu.2020.100348

76.

Farag

Hashem

. Impact of the haptic virtual reality simulator on dental students’ psychomotor skills in preclinical operative dentistry. Clin Pract. 2021;12(1):17-26. doi:10.3390/clinpract12010003

77.

Benjamin

Sabri

. Using haptic feedback in a virtual reality bone drilling simulation to reduce plunge distance. Cureus. 2021;13(9):e18315. doi:10.7759/cureus.18315

78.

Huber

Katzky

Müller

, et al. Evaluation of a new virtual reality concept teaching k-wire drilling with force feedback simulated haptic in orthopedic skills training. J Hand Surg. 2022;47(12):1225.e1-1225.e7. doi:10.1016/j.jhsa.2021.09.008

79.

Stott

Driscoll

. Face and content validity of analog surgical instruments on a novel physics-driven minimally invasive spinal fusion surgical simulator. Med Biol Eng Comput. 2022;60(10):2771-2778. doi:10.1007/s11517-022-02635-8

80.

Awad

Häfner

Reinert

Kluba

. Plaster casts vs. intraoral scans: do different methods of determining the final occlusion affect the simulated outcome in orthognathic surgery? J Pers Med. 2022;12(8):1288. doi:10.3390/jpm12081288

81.

Vamadevan

Konge

Stadeager

Bjerrum

. Haptic simulators accelerate laparoscopic simulator training, but skills are not transferable to a non-haptic simulator: a randomized trial. Surg Endosc. 2023;37(1):200-208. doi:10.1007/s00464-022-09422-4

82.

Gani

Pickering

Ellis

Sabri

Pucher

. Impact of haptic feedback on surgical training outcomes: a Randomised Controlled Trial of haptic versus non-haptic immersive virtual reality training. Ann Med Sur. 2022;83:104734. doi:10.1016/j.amsu.2022.104734

83.

Philip

Ali

Duggal

Daas

Nazzal

. Effectiveness and student perceptions of haptic virtual reality simulation training as an instructional tool in pre-clinical paediatric dentistry: a pilot pedagogical study. Int J Environ Res Public Health. 2023;20(5):4226. doi:10.3390/ijerph20054226

84.

Domes

Coale

Weber

, et al. Can a computer-based force feedback hip fracture skills simulator improve clinical task performance? A cadaveric validation study. J Am Acad Orthop Surg Glob Res Rev. 2023;7(5):e22.00056. doi:10.5435/JAAOSGlobal-D-22-00056

85.

Bornhoft

Strabala

Wortman

Lehman

Oleynikov

Farritor

. Stereoscopic visualization and haptic technology used to create a virtual environment for remote surgery - biomed 2011. Biomed Sci Instrum. 2011;47:76-81.

86.

Våpenstad

Hofstad

Bø

, et al. Limitations of haptic feedback devices on construct validity of the LapSim® virtual reality simulator. Surg Endosc. 2013;27(4):1386-1396. doi:10.1007/s00464-012-2621-9

87.

Hariton

Locascio

. Randomised controlled trials - the gold standard for effectiveness research: Study design: randomised controlled trials. BJOG Int J Obstet Gynaecol. 2018;125(13):1716. doi:10.1111/1471-0528.15199

88.

Zhong

How to calculate sample size in randomized controlled trial?

J Thorac Dis. 2009;1(1):51-54.

89.

Sedgwick

. Randomised controlled trials: the importance of sample size. BMJ. 2015;350:h1586. doi:10.1136/bmj.h1586

90.

Lim

. Considerations for crossover design in clinical study. Korean J Anesthesiol. 2021;74(4):293-299. doi:10.4097/kja.21165

91.

McCabe

Langer

Borod

Bender

. Practice effects. In: Kreutzer

DeLuca

Caplan

, eds. Encyclopedia of Clinical Neuropsychology. Springer; 2011:1988-1989. doi:10.1007/978-0-387-79948-3_1139

92.

Harris

McGregor

Perencevich

, et al. The use and interpretation of quasi-experimental studies in medical informatics. J Am Med Inform Assoc. 2006;13(1):16-23. doi:10.1197/jamia.M1749

93.

Favier

Gallet

Subsol

Captier

. Understanding the biomechanical properties of skull base tissues is essential for the future of virtual reality endoscopic sinus and skull base surgery simulators. Clin Exp Otorhinolaryngol. 2019;12(2):231-232. doi:10.21053/ceo.2018.01627

94.

Chen

Liu

. Content analysis of statistical power in educational technology research: sample size matters. Int J Technol Teach Learn. 2019;15(1):49-75.

95.

Sturm

Windsor

Cosman

Cregan

Hewett

Maddern

. A systematic review of skills transfer after surgical simulation training. Ann Surg. 2008;248(2):166-179. doi:10.1097/SLA.0b013e318176bf24

96.

Stunt

Wulms

Kerkhoffs

Dankelman

van Dijk

Tuijthof

. How valid are commercially available medical simulators? Adv Med Educ Pract. 2014;5:385-395. doi:10.2147/AMEP.S63435

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.36 MB

0.70 MB