The ScanTrainer obstetrics and gynaecology ultrasound virtual reality training simulator: A cost model to determine the cost viability of replacing clinical training with simulation training

Abstract

The aim of this study was to produce an economic cost model comparing the use of the Medaphor ScanTrainer virtual reality training simulator for obstetrics and gynaecology ultrasound to achieve basic competence, with the traditional training method. A literature search and survey of expert opinion were used to identify resources used in training. An executable model was produced in Excel. The model showed a cost saving for a clinic using the ScanTrainer of £7114 per annum. The uncertainties of the model were explored and it was found to be robust. Threshold values for the key drivers of the model were identified. Using the ScanTrainer is cost saving for clinics with at least two trainees per year to train, if it would take at least six lists to train them using the traditional training method and if a traditional training list has at least two fewer patients than a standard list.

Keywords

Ultrasound training virtual reality cost model

Introduction

Virtual reality simulation is increasingly used for training in healthcare and offers a number of potential advantages over traditional training methods.¹ In the field of obstetrics and gynaecology ultrasound, the Medaphor ScanTrainer (MedaPhor Ltd., Cardiff Medicentre) combines ‘real-feel’ virtual reality simulation with recordings of real patient scans. Real-time expert guidance and feedback and curriculum-based interactive learning and assessment are incorporated into the system. A recent study¹ showed the ScanTrainer can be used to assess gynaecological ultrasound competence in a valid and reliable way. A new clinical assessment framework was recently proposed for diagnostic medical ultrasound students¹ including a simulator for teaching and formatively assessing obstetrics, gynaecology and general medical ultrasound students. The article highlighted the financial pressures on higher education institutions and the particular high cost of running ultrasound education programmes for small numbers of students.

The aim of this study was to estimate the health service costs of training in basic obstetrics and gynaecology ultrasound skills using a virtual reality training simulator (Medaphor ScanTrainer), compared with the traditional approach to training during a live scanning list. We considered training to a basic level of competence which will allow trainees to scan with confidence (initially under ‘watchful eye’ supervision).

Methods

Literature review

A rapid literature search² was undertaken to determine whether any previous economic evaluation of ultrasound training simulators had been published and find data on standard UK ultrasound practice to inform the development of an economic model. The databases searched were Medline via Ovid and NHS EED. The search strategy is shown in Table 1. The papers identified in the search were screened by title and abstract. Papers with potential relevance were obtained, and assessed in terms of their application to our project.

Table 1

Search terms used in Medline

Search term	No of papers identified
1. Exp ultrasonography/	145,404
2. simulat*.mp	235,976
3. education/	8,127
4. training.mp OR Inservice training	152,677
5. 3 OR 4	159,058
6. 1 AND 2 AND 5	147

Electronic survey

An anonymised electronic survey (Bristol online surveys) of 34 UK consultants in obstetrics and gynaecology was designed in conjunction with Medaphor to elicit expert opinion on specific questions regarding the expected change in practice that would result from introducing training using the ScanTrainer simulator. The consultants were contacts identified by Medaphor Ltd. and they were told that the purpose of the survey was to identify resources used in training in their practice. The initial questionnaire was tested by four experts and the wording of one question was modified for clarification based upon their comments. The final version of the survey was issued to the full cohort by emailing a hyperlink to an electronic survey (Bristol Online Survey). Medaphor Ltd. supplied the contact list for the electronic survey and emailed the survey hyperlink to all contacts. Medaphor Ltd. had no further input to the study. Responses were received anonymously by Cedar, a consortium of Cardiff and Vale University Health Board and Cardiff University; Medaphor had no direct access to the individual responses. The survey responses were scored and analysed using IBM SPSS. The survey questions are shown in Table 2.

Table 2

Questions in the survey

Q1. What is your estimate of the average number of scans per typical ultrasound scan list, with no trainees present?

Q2. If an inexperienced trainee is being trained in basic obstetric/gynaecology scanning skills during a ‘live’ scanning list, how many scans will typically be carried out during that list?

Q3. In a typical year how many trainees do you train within your Department in basic obstetric/gynaecology ultrasound scanning skills?

Q4. On average, how many scan lists will each trainee need to attend before they have reached a basic level of competence which will allow them to scan with confidence (initially under ‘watchful eye’ supervision)?

Q5. How many staff per year in your Department require competency assessment, continuing professional development or ongoing specialist training in obstetric/gynaecology ultrasound via ‘hands-on’ sessions?

Cost model

The economic model was developed from the perspective of the NHS or healthcare service provider. The setting was secondary care and the patient population considered was women undergoing transvaginal ultrasound for obstetric or gynaecological purposes. The staff group potentially using the ScanTrainer was obstetricians and gynaecologists undergoing training. The time horizon considered was the period of initial ‘hands-on’ training up to the point where the trainee is considered to reach a level of competency which will allow them to scan with confidence (initially under ‘watchful eye’ supervision). Discounting to present value is applied in health economics to adjust for costs occurring at different times in the future. Because of the short time period considered, discounting was not applied. The economic approach is a simple cost model method, which means that the costs of the proposed arrangement using the ScanTrainer are compared with the costs of the traditional approach to training. Health or other benefits that might arise from the change are identified where possible, to be considered separately to the costs by decision makers.

The model structure is a decision tree with one branch representing the traditional approach to training and the other branch for the ScanTrainer. The model calculates the cost per year of each training route from the resources and costs identified. For the standard training route, the costs of training result from fewer patients scanned in each list multiplied by the cost per patient of an ultrasound scan,³ the number of lists to achieve competence and the number of trainees requiring training. For the virtual reality training route, the cost of training is the capital equipment cost divided by the lifespan of the equipment.

A number of sources were used to (1) identify resources, (2) quantify the resources used and (3) estimate the costs, including data from the e-survey. The base case model, developed in Excel, incorporated the best estimate for each input in the model. Uncertainties in the inputs were explored using sensitivity analysis, in which the inputs took values that gave the plausible worst case and best case scenarios for the ScanTrainer.

Results

Literature review

No economic studies of ultrasound simulation training were found during the literature search. The literature search did not reveal any studies that could be used to inform resource use.

Electronic survey

Survey responses were received from 19 (56%) of those contacted. For question 1 (Figure 1), a respondent who checked the box ‘other’ added that in their clinic they do not have ultrasound lists, but incorporate scans into a single visit clinic. The second respondent who checked the ‘other’ box indicated that it could be one of two answers 6–10 or 11–15. In question 2 (Figure 2), one of the respondents who checked the box ‘other’ added that in their clinic they do not have ultrasound lists, but incorporate scans into a single visit clinic. The second respondent indicated it could be one of two answers 6–10 or 0–5.

Figure 1

What is your estimate of the average number of scans per typical ultrasound scan list, with no trainees present?

Figure 2

If an inexperienced trainee is being trained in basic obstetric/gynaecology scanning skills during a ‘live’ scanning list, how many scans will typically be carried out during that list?

For question 3 (Figure 3), the respondents who chose other added: ‘The numbers are 11–15 but we have not got a robust model that has been sustainable due to reconfiguration of services’ and ‘5 in obs, 2 in gynae’. In question 4 (Figure 4), one respondent who gave the reponse ‘other’ added: ‘They need more than we offer, which is 10 lists’. The other three respondents who gave the response ‘other’ indicated that this varies depending on how quickly individuals learn and their previous experience.

Figure 3

In a typical year how many trainees do you train within your department in basic obstetric/gynaecology ultrasound scanning skills?

Figure 4

On average, how many scan lists will each trainee need to attend before they have reached a basic level of competence which will allow them to scan with confidence (initially under ‘watchful eye’ supervision)?

In question 5 (Figure 5), respondents indicated that staff in the Department would require competency assessment, continuing professional development or ongoing specialist training in obstetric/gynaecology ultrasound via ‘hands-on’ sessions (median 6–10 staff).

Figure 5

How many staff per year in your department require competency assessment, continuing professional development or ongoing specialist training in obstetric/gynaecology ultrasound via ‘hands-on’ sessions?

The survey results are summarised in Table 3. The responses to questions 1 and 2 were categorised (0–5 = category 1, 6–10 = category 2, etc.) and the paired responses were compared to determine if there is a significant difference in the number of patients that can be scanned when a trainee is present compared with a typical scan list with no trainee present. Data from one respondent was excluded because of selecting the ‘other’ box for questions 1 and 2, and clarified that this was because they don’t have ultrasound lists in their one-stop clinic. Based on the clarifying comments, it was assumed for the second respondent who checked ‘other’ that there was a 1 category difference between the numbers of patients scanned with and without trainees present.

Table 3

Results of questionnaire data

Mean number of trainees requiring training	4.7
Mean number of lists to achieve competence by traditional route	13.3
Mean number of patients fewer that can be scanned during a training list	3.8

There was a significant difference between the responses to questions 1 and 2 (Wilcoxon signed rank test) p < 0.001 at the 95% confidence level. In order to quantify this difference, it was assumed that a difference of one category corresponds to five fewer patients being scanned with trainees present. The mean number of categories difference between scanning with and without trainees was 0.7. Therefore, a difference of 0.7 × 5 = 3.8 patients per list was used for the base case analysis. Uncertainties around this assumption were explored in the sensitivity analysis.

Cost model

Base case results

The base case analysis was calculated using the best estimates of each input value (Table 4). For the data from the e-survey, the mean value was used for the base case and the upper and lower ranges were used in sensitivity analysis. For the cost of an ultrasound scan, the mean value was used for the base case and the upper and lower quartiles for the sensitivity analysis. The result therefore represents the most likely outcome based on the data available. The base case results showed that the simulator training route delivers a cost saving of £7114 per annum.

Table 4

Input values for the model

Resource	Base case	Source	Low value	High value
Number of trainees in department	4.7	e-survey (mean and range)	1	15
Number of lists to achieve competence	13.3	e-survey (mean and range)	1	30
Number of fewer patients in training list	3.8	e-survey (mean and range)	1	10
Cost of ultrasound scan	£51	NHS reference costs³ code RA23Z (ultrasound imaging less than 20 minutes) mean, upper and lower quartile	£38	£59
Lifespan of ScanTrainer	5 Years	Manufacturer	2 years	10 years
Cost of ScanTrainer	£25,000	Manufacturer	£18,000	£30,000

Sensitivity analysis results

The sensitivity analysis explored uncertainty in the values used in the base case. The impact of each parameter was explored in turn by choosing a plausible low value and a plausible high value and then re-calculating the cost analysis. The values chosen for the low and high inputs are given in Table 4 and the impact on the cost analysis is represented on a Tornado diagram (Figure 6). The ends of each bar show the result of the model for low and high values used in sensitivity analysis. In sensitivity analysis, each input is varied independently and the results calculated. The longer bars in the Tornado chart highlight the parameters that have the greatest impact on the result. Only when the bar extends to the left of the £0 on the horizontal axis, does the scenario become cost incurring.

Figure 6

Tornado chart

In almost all of the scenarios tested, the ScanTrainer remained cost saving compared with traditional training. Three factors could change the outcome from cost saving to cost incurring:

The number of trainees requiring training

The number of lists required to train them

The number fewer patients that can be scanned during a training list

These factors were further investigated to determine the point at which they became cost incurring (threshold analysis) and this is illustrated in the Tornado chart where the bars cross the £0 threshold. It was found that provided all three of the following conditions are met, the model will always show a cost-saving result:

The department has at least two trainees per year to train

It would require at least six lists to train them by the traditional method

A traditional training list has at least two fewer patients than a standard list

Discussion

The published studies found during the rapid literature review did not provide sufficient data on resource use to inform an economic analysis. Therefore, the economic analysis was based largely upon expert opinion and assumptions. A key assumption in our approach is that the two different routes to provide training deliver practitioners that are equally well trained and hence equivalent clinical outcomes for patients. A previous study has shown that performance on the ultrasound simulator can be assessed in a reliable and valid way.⁴

The list of experts was provided by MedaPhor Ltd. and may not be representative of practice across the whole of the United Kingdom. However, there was a good response rate to the survey and the sensitivity analysis addresses the uncertainties in the data. Overall, we have taken a conservative approach in building an economic case for the ScanTrainer and this has shown a potential cost saving. The threshold analysis identifies the necessary conditions for cost savings to be made and this should help centres to determine if savings are likely within their services. It remains for the assumptions in the analysis to be tested in clinical practice.

The cost savings in the model are driven by the opportunity cost of scanning fewer patients during traditional training using the existing ultrasound scanner. It therefore represents an efficiency saving, releasing the ultrasound scanner and trainer. Whether this becomes a realisable cost saving depends upon the circumstances of the organisation. For example, a clinic would be able to use the resource savings to scan additional patients, who would otherwise have gone elsewhere, reduce the waiting time for existing patients or release the trainer for other duties.

The analysis has not used probabilistic methods, which would allow all of the parameters to be varied simultaneously. However, the deterministic methods used have been applied carefully using the best available data. By presenting the conditions required for a cost saving to be made, this will enable decision-makers to make a well informed choice. The analysis has not included further potential efficiency savings and educational benefits from the use of the ScanTrainer for continuing professional development, further skills development and competency assessment of trained staff. In the questionnaire, respondents indicated that there is a need for such training. Therefore, overall the cost model is conservative and the ScanTrainer is likely to result in greater cost savings than were identified in our model.

Conclusion

Using a virtual reality training simulator for training in obstetric and gynaecology ultrasound is likely to result in resource savings compared with the traditional approach to training. The resource savings arise because fewer patients can be scanned during a training list than a standard list using the traditional approach to training.

Footnotes

DECLARATIONS

Competing interests: The authors have no conflicts of interest to declare.

Funding: This study was funded by Medaphor Ltd. Suite 16, Cardiff Medicentre, Heath Park, Cardiff CF14 4UJ Wales, UK.

Ethical approval: Not applicable

Guarantor: G Carolan-Rees

Contributorship: Grace Carolan-Rees undertook the literature search, extracted the data from the e-survey, developed the model and wrote the manuscript. Alistair Ray checked the model, inputs and commented on drafts of the manuscript.

References

Gibbs

. A proposed new clinical assessment framework for diagnostic medical ultrasound students. Ultrasound 2014; 22: 113–7.

Khangura

Konnyu

Cushman

. Evidence summaries: The evolution of a rapid review approach. Syst Rev 2012; 1: 10–10.

Department of Health. NHS Reference Costs: Financial Year 2011 to 12 (Direct Costs). See https://www.gov.uk/government/publications/nhs-reference-costs-financial-year-2011-to-2012 (last checked 5 December 2014).

Madsen ME, Konge L, Norgaard LN et al. Assessment of performance and learning curves on a virtual reality ultrasound simulator. Ultrasound Obstet Gynecol 2014 Apr 30. DOI: 10.1002/uog.13400. [Epub ahead of print].