Initial Validation of a Virtual-Reality Learning Environment for Prostate Biopsies: Realism Matters!

Introduction

Current teaching methods for prostate biopsies, based on an apprenticeship without feedback on biopsy distribution or overall performance, have shown their limitations. Previous publications reported discrepancies between the assumed location of cancer foci based on biopsies and their actual location based on final pathology reports. ¹ Adding a visual feedback proved to improve the biopsy distribution within the gland, even for an experienced operator. ²

The emergence of targeted biopsies and focal therapy created a need for a better sampling of the prostate to avoid leaving cancer foci untreated, and new training needs to target predefined areas of the prostate, using mental reconstruction or magnetic resonance imaging (MRI)-transrectal ultrasonography (TRUS) fusion systems. ³ Simulation-based training could therefore find a place in the initial training, but also in the performance assessment of these new techniques.

Various simulators already exist in the field of urology, although most are dedicated to laparoscopy and, as stated by a recent report by the French Health Authorities, are also largely underused. ^4,5 A simulator for prostate biopsies was already described, focusing more on the gesture itself than the entire biopsy procedure, and with a limited teaching environment. ⁶

We designed a simulator dedicated to prostate biopsies, allowing visual feedback and performance assessment, enhanced by a complete learning environment and connected to a clinical case database to cover the main situations encountered in clinical practice. ^7,8 The aim of this study was to validate the reliability (reproducibility and precision), face (the simulator represents what it is supposed to represent), content (the simulator teaches what it is supposed to teach), and construct (the simulator is able to distinguish between experienced and inexperienced users) of the simulator.

Materials and Methods

Simulator design

Simulator architecture

The simulator is composed of a laptop computer connected to a haptic device (Phantom Omni, Sensable) with a stylus held as the ultrasound (US) probe (Fig. 1). Manipulation of the stylus allows the user to explore the US volume and display the corresponding two-dimensional (2D) US image. Three-dimensional (3D) TRUS volumes with corresponding prostate meshes (prostate segmentation) acquired with an end-fire 3D endorectal probe (SonoAce X8, Samsung-Medison) during actual biopsy procedures with the Urostation® (Koelis, La Tronche, France) were collected and entered after anonymization into a clinical case database. Pertinent clinical information (age, prostate volume, digital rectal examination, prostatic-specific antigen [PSA] level) and prostate MRI were also collected for each case, when available.

OPEN IN VIEWER

FIG. 1.

Simulator design.

Learning environment

The various exercises and functionalities of the simulator were developed based on a didactical study performed at the beginning of the project. The expected learning outcomes included technical skills (ability to obtain an optimal coverage of the gland by distributing randomized biopsies and ability to target suspicious areas), theoretical and decision-making knowledge (ability to estimate the risk of cancer based on clinical data and PSA level). The exercises therefore include the practice of every aspect of the biopsy procedure: TRUS image reading, prostate volume measurement, prostate cancer risk estimation, sector or area targeting, and MRI-TRUS mental fusion. Each exercise can be performed with or without assistance. Both visual feedback (3D visualization and assistance) and numerical feedback (biopsy score) are provided. Assistance consists in displaying a 3D representation of the prostate surface in which the 2D US plane can be visualized in real time. In addition, the user can choose to display the needle trajectory and the already performed biopsies (Fig. 2). All these exercises aim at helping trainees to build a 3D mental representation of the prostate based on 2D images and to improve their hand-eye coordination. Each exercise can be performed independently or can be combined into a complete learning pathway. Learning pathways are adjusted to each user's specific training needs depending on their initial evaluations.

OPEN IN VIEWER

FIG. 2.

Biopsy assistance: Two-dimensional ultrasound plane (a), needle trajectory (b), and previous biopsies (c).

Biopsy procedure

Virtual biopsies can be performed and located within the prostate volume. A biopsy procedure simulation exercise replicates the pertinent conditions of a real 12-core biopsy procedure. Users have to perform a preoperative checklist, 12 virtual biopsies using the left hand (or right hand for left handed users) for biopsy firing without immediate feedback, and have to specify the location of each core as if they were preparing it for the pathologist. The target of each of the 12 cores is one of the 12 sectors as represented by the colored boxes in Figure 3. At the end of the procedure, both a visual and a numeric feedback are provided to the user. The visual feedback consists in the representation of each biopsy core within the prostate volume. The numeric feedback consists in a percentage score (see Scoring system), as shown in Figure 3.

OPEN IN VIEWER

FIG. 3.

Visual feedback (biopsy mapping) and numerical feedback (score) (a). Division of the prostate volume into 12 sectors of equal volume (b).

Scoring system

To provide the user with a numerical feedback on his/her performance, a simple scoring system has been developed. The prostate volume is divided into 12 parts (sectors) of equal volume, as shown in Figure 3. Each biopsy is awarded four points when the targeted sector is actually reached, and only one point when it is inside the prostate but outside the targeted sector. The scores of the 12 biopsies are added and the result converted to a percentage score. This score is displayed to the user along with the visual feedback (Fig. 3a, bottom left).

Results

Face and content validity

The overall realism of the biopsy procedure was rated at a median of 9 for an interquartile range of 7.1–9.7, the quality of the graphical user interface at 9.1 (7.8–10), the realism of the US image at 9.1 (7.1–10), the realism of the range of motion of the US probe (stylus) at 7.5 (4.9–9.1), and the realism of the force feedback at 5.6 (4.9–8.8). The detailed results of the ratings attributed by novices and experts are shown in Figure 4. Experts reported the range of motion and force feedback as significantly less realistic than novices (P=0.01 and P=0.03, respectively).

OPEN IN VIEWER

FIG. 4.

Face and content validity questionnaire: Evaluation of the graphical user interface (a), the realism of the ultrasound image (b), range of motion (c), force feedback (d), biopsy procedure (e), and the usefulness of the simulator for initial training (f ); *statistically significant.

The seven experts rated the usefulness in the initial training of urology residents at 8.2 (7.9–9.6), as represented in Figure 4.

Construct validity

The median score for the 21 users was 60% (43%–68%). The 7 experts had a median score of 64% (59%–73%), and the 14 novices a median score of 52% (43%–67%), the difference not proving to be statistically significant (P=0.19). Results are displayed in Figure 5.

OPEN IN VIEWER

FIG. 5.

Comparison of the scores obtained by experts and novices.

Discussion

The results of this initial evaluation allowed us to validate the face and content of our simulator, with a median rating of 9/10 by nonexperts for the realism of the biopsy procedure, and a median rating of 8.2/10 by experts for the educational interest of the simulator. The split-halves technique showed a correlation coefficient between the right and left sides of the prostate for each user of 0.79. This correlation coefficient represents the proportion of variability in scores attributable to true differences between users. This result is very close to 0.8, the value considered as satisfactory to avoid misclassification and to validate the intrinsic reliability of the simulator. ⁹

We failed to prove the construct validity despite a 12% difference in the median scores of novices and experts. This can be partly explained by the small size of the panel and the important variability of the scores in each subgroup. Taking into account this variability, we calculated a posteriori that 44 users would have been required in each group to statistically prove a 12% difference with 80% power.

Another explanation can be found in the very simple scoring system that was used, which was probably insufficient to properly discriminate between the two groups. Defining the criteria of a “good” procedure is central to the design of a simulator and in itself a very interesting contribution to medical practice assessment. The main advantage of the simple score we used is that it is easily understandable by the users. We are currently working on the definition of a more complex score that might be not only based on the reached sectors, but also on other factors. These may include the resulting length of the needle inside each sector, the distance from the targeted sector when it was not reached, and the total duration of the procedure. This new score should allow a better discrimination between novices and experts.

We chose 50 biopsies as the cut point that defines one as being an expert in prostate biopsies, based on two articles published on the subject, ^2,10 but a modification of this cut point would not have affected our results, because the experts in our panel were senior urologists who had performed largely more than 50 procedures. Three residents were included but had no previous experience in US imaging or prostate biopsies and were therefore considered as novices.

This evaluation also showed that the simulator could improve its realism. In its current version, the user is required to manipulate a small, round, and symmetrical stylus in place of a US probe. Although a recent review comparing low- and high-fidelity simulators showed only a small learning improvement when realism is increased, the interaction with the stylus was particularly confusing for the experts. ¹¹ They had trouble maintaining a steady axial plane during their virtual biopsy procedure. This was not the case for the novices, who were not used to using a real US probe. Also, the 2D images reconstructed from the exploration of the 3D US volumes were not always strictly similar to the 2D images obtained with a classical 2D US probe. This can probably explain the lower ratings of the experts compared with the novices regarding certain aspects of the realism (probe range of motion, force feedback). This can also partly explain the smaller than expected difference of the scores between experts and novices. In the planned new version of the simulator, a mockup of a real US probe, obtained using a 3D printer, will replace the stylus.

This is, to our knowledge, the second simulator for prostate biopsies described in the literature, but the first one with such an educational environment and performance evaluation. ⁶ This initial study aimed at validating the simulator only in its ability to simulate a 12-core randomized biopsy procedure. It does, however, offer other functionalities, notably a learning pathway enabled to propose exercises that meet the user-specific training needs. This learning pathway, as well as the ability of the trainee to reproduce the same performance on an actual patient (predictive validity), will also have to be further validated.

Conclusion

Validating a simulator is a complex task that should take into account not only the intrinsic properties of the simulator, but also its realism, the scoring system used, and the conditions of its validation. The development of such a simulator necessitates the characterization of the quality of a gesture and the description of the required skills and simulated procedures. We designed a learning environment for prostate biopsies, which proved its reliability, and we validated its face and content. Improvements to the realism and scoring system used are necessary to validate the construct. Further studies will also be needed to evaluate the learning pathway and predictive validity of this simulator.