External validation of Risk of Malignancy Index compared to IOTA Simple Rules

Abstract

Background

Mathematical predictive models for ovarian tumors have an advantage over subjective assessment due to their relative simplicity, and therefore usefulness for less experienced sonographers. It is currently unclear which predictive model is best at predicting the nature of an ovarian tumor.

Purpose

To compare the diagnostic predictive accuracy of the International Ovarian Tumour Analysis Simple Rules (IOTA SR) with Risk of Malignancy Index (RMI), to differentiate between benign and malignant ovarian tumors.

Material and Methods

A total of 202 women diagnosed with ovarian tumor(s) were included. Preoperatively, patients were examined through transvaginal ultrasonography and CA-125 (U/mL) levels were measured. RMI and IOTA SR were determined, and where possible compared to definitive histopathological diagnosis.

Results

Of the 202 women with ovarian tumors, 168 women were included in this cohort study. Of these tumors, 118 (70.2%) were benign, 17 (10.1%) were borderline, and 33 (19.7%) were malignant. The sensitivity, specificity, and area under the curve for the RMI were 72.0%, 90.7%, and 0.896, respectively. For the IOTA SR, these were 90.0%, 68.6%, and 0.793, respectively.

Conclusion

This cohort study shows that the RMI is a relatively useful diagnostic model in characterizing ovarian tumors, compared to the IOTA SR. However, due to the relatively low sensitivity of the RMI and high rate of inconclusive results of the IOTA SR, both diagnostic tests do not seem discriminative enough. Therefore, alternative diagnostic models are necessary.

Keywords

Ultrasound ovaries neoplasms pathology

Introduction

Precise preoperative risk assessment of an ovarian tumor is essential to ensure that patients with potential ovarian cancer receive appropriate treatment in a specialized oncology center, whereas patients with benign tumors can be treated in non-oncological centers. The prediction methods used for triage of patients often lead to unnecessary referrals to oncology centers, causing stress for the patient and extra costs.

Subjective assessment by an experienced sonographer combined with the use of serum markers currently seems the most effective method to preoperatively characterize ovarian tumors. Mathematical predictive algorithms have an advantage due to their relative simplicity and are therefore useful for less experienced sonographers (1 –3). The improvement caused by these prediction models may ultimately lead to decreasing overtreatment and reducing healthcare costs (3,4).

An example of a simple mathematical predictive model is the Risk of Malignancy Index (RMI), the current gold standard in the Netherlands. The RMI incorporates ultrasound score, menopausal status, and serum levels of CA-125 (3,5,6). A disadvantage of the RMI is that it has limited sensitivity since serum levels of CA-125 appear to be elevated only in 40%–50% of women with International Foundation of Gynecology and Obstetrics (FIGO) stage I ovarian cancer (7,8).

Therefore, the International Ovarian Tumour Analysis (IOTA) group developed a prediction model called “Simple Rules” (SR). This model is based on five sonographic malignant and five sonographic benign features (9). The presence or absence of these typical features can be used to classify a tumor as benign, malignant, or, in case both or none of these features are present, inconclusive. The IOTA SR model achieves both high sensitivity and specificity in recent studies, and therefore discriminates well between benign and malignant ovarian tumors than the RMI (1 –3,10 –14). However, most of the IOTA SR studies have been performed by expert sonographers.

The aim of the present cohort study was to compare the diagnostic accuracy of the RMI and IOTA SR by certified IOTA sonographers, as in daily practice, to estimate the chance of the tumor to be benign or malignant.

Material and Methods

A total of 202 women diagnosed with ovarian tumor(s) were included in the tertiary gynecological oncology center in the Netherlands between July 2015 and February 2019. A non-Wet op Medisch-wetenschappelijk Onderzoek (WMO; Medical Research Involving Human Subjects Act) declaration was obtained from the institutional ethical committee. Patients were referred from general practitioner practices or other hospitals with an ovarian tumor. All patients were preoperatively examined using transvaginal ultrasound, performed by two certified IOTA sonographers. Two different types of vaginal probes were used (V5-9 transvaginal probe (5–9 MHz) or CV1-8A (1–8 MHz); Samsung, Seoul, Republic of Korea), combined with a transabdominal probe (CA1-7A [1–7 MHz]); Samsung, Seoul, Republic of Korea), connected to a Samsung Ultrasound device (model WS80A). Serum samples were taken to measure the levels of CA-125 (U/mL) (Roche, L1633034). RMI and IOTA SR were then determined (3,10,13,15,16).

Risk of Malignancy Index (RMI)

The RMI was determined by the following calculation: U × M × serum CA-125 (U/mL) (4 –6), where the U stands for the combined results of the ultrasound with a maximum value of 3, depending on a multilocular cyst, a solid area, bilateral masses, ascites and intra-abdominal metastases, each with a value of 1 if present. The M stands for the menopausal status, in which postmenopausal women gain the value of 3 and premenopausal women the value of 1. Menopause was defined retrospectively as at least 12 months of amenorrhea or arbitrarily if age was >50 years in case of previous hysterectomy. The cut-off value for malignancy was set at RMI ≥200 (4–6).

IOTA Simple Rules (IOTA SR)

The IOTA SR were determined based on 10 echo-graphic tumor characteristics, five of them suggesting malignancy and five suggesting benignancy (3,10,15,16). If only one or more malignant characteristics were present, the tumor was classified as malignant. If only one or more benign characteristics were present, the tumor was classified as benign. If both benign and malignant characteristics occurred or were absent, the tumor was classified as inconclusive (9,11,17).

Patients underwent laparotomic or laparoscopic surgery based on tumor size or clinical presentation of the ovarian tumor, according to the Dutch Society for Obstetrics and Gynaecology guideline for ovarian masses (18). In case of RMI ≥200, frozen sections were analyzed by a pathologist to distinguish benign from malignant tumors during surgery. In case of malignancy, patients underwent subsequent staging to determine whether adjuvant chemotherapy was required. As in previous studies and for the sake of this study, borderline ovarian tumors were classified as malignant, and for the IOTA SR, ovarian tumors classified as inconclusive were considered malignant for statistical analysis (1 –4,10 –12,19 –22).

Data from patients were obtained via medical records, including age, menopausal status, definitive pathology, serum levels of CA-125, and ultrasound reports. Exclusion criteria were no calculated IOTA SR or RMI, no serum levels of CA-125 measured, and incomplete follow-up.

The definitive histopathological diagnosis of surgically removed tumors was considered the reference standard. In case of malignancy, the masses were classified according to the classification system established by FIGO (23). In case a patient’s expectant management was chosen according to the Dutch Society for Obstetrics and Gynaecology guideline for ovarian masses and later on dismissed for further treatment, the tumor was considered to be benign (18).

Statistical analysis

Statistical analysis was performed using IBM SPSS statistics, software version 25.0 (IBM Corp., Armonk, NY, USA). For RMI and IOTA SR the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated with 95% confidence intervals. To compare the diagnostic accuracy of these two models, the receiver operating characteristic (ROC) curves were constructed and corresponding areas under the curves (AUCs) were calculated. For the IOTA SR, the classification has three levels (benign, inconclusive, and malignant) and is represented by a ROC curve with two points. McNemar’s test was performed to compare the accuracy for the IOTA SR and the RMI. P < 0.05 was considered statistically significant.

Results

Of the 202 patients, three patients were excluded because no IOTA SR was calculated, three patients were excluded because no serum CA-125 was measured, 20 patients were lost to follow-up, five patients are still being followed up according to the Dutch guidelines, and three patients died during follow-up due to non-ovarian-related diseases. In total, 168 patients were included, of which 43 tumors in patients were diagnosed as benign due to completed ultrasound and level of CA-125 at follow-up. The mean age of the included women was 57.2 years (age range = 18–89 years). The mean serum level of CA-125 was 97.1 U/mL. Table 1 presents the patient baseline characteristics.

Table 1.

Patient baseline characteristics.

Baseline characteristics	Value
Mean age at surgery (years)	57.2 (18–89)
Menopausal state
Premenopausal	53 (31.5)
Postmenopausal	115 (68.5)
Definitive pathology
Benign*	118 (70.2)
Borderline	17 (10.1)
Malignant	33 (19.7)
FIGO stage I	14
FIGO stage II	4
FIGO stage III	9
FIGO stage IV	0
Other etiology	6
Ultrasound features
Multiloculated cysts	97 (57.7)
Solid areas	82 (48.1)
Bilateral lesions	24 (14.3)
Ascites	20 (11.9)
Intra-abdominal metastases	4 (2.4)
Serum level of CA-125 (U/mL)
0–99	140 (83.3)
≥100	28 (16.7)

Values are given as n (%) or median (range).

*43 tumors were considered benign due to completed follow-up.

FIGO, International Federation of Gynecology and Obstetrics.

Table 2 shows the comparison of the results from the preoperatively determined RMI and IOTA SR with the definitive histopathological diagnosis from surgically removed tumors.

Table 2.

Comparison of RMI and IOTA SR with definitive diagnosis.

	n (n = 168)	Definitive diagnosis
	n (n = 168)	Benign*	Borderline	Malignant
Nature of tumor as per RMI
RMI 1 < 200	121 (72.0)	107 (88.4)	7 (5.8)	7 (5.8)
RMI 1 ≥ 200	47 (28.0)	11 (23.4)	10 (21.3)	26 (55.3)
Nature of tumor as per IOTA SR
Benign	86 (51.2)	81 (94.2)	3 (3.5)	2 (2.3)
Inconclusive	65 (38.7)	32 (49.2)	9 (13.9)	24 (36.9)
Malignant	17 (10.1)	5 (29.4)	5 (29.4)	7 (41.2)

Values are given as n (%).

*43 tumors were considered benign due to completed follow-up.

IOTA SR, International Ovarian Tumour Analysis Simple Rules; RMI, Risk of Malignancy Index.

According to histopathological examination or follow-up according to the Dutch guideline for ovarian masses, of the included 168 tumors, 118 (70.2%) were benign, 17 (10.1%) were borderline, and 33 (19.7%) were malignant. Twenty-seven tumors were primary ovarian carcinoma, while the other six tumors were metastatic of origin. Table 3 presents the definitive histopathology of the ovarian tumors.

Table 3.

Frequencies of definitive histopathology of the ovarian tumors.

Definitive histopathology	Value
Benign tumor	75 (44.6)
Serous cystadenoma	22 (13.1)
Mucinous cystadenoma	15 (8.9)
Mature cystic teratoma	11 (6.5)
Ovarian fibroma	9 (5.4)
Endometriotic cyst	7 (4.2)
Simple epithelial cyst	3 (1.8)
Fibrothecoma	2 (1.2)
Benign Brenner tumor	2 (1.2)
No convincing histopathology, but no evidence of malignancy	2 (1.2)
Leiomyoma	1 (0.6)
Tubo-ovarian abscess	1 (0.6)
Malignant tumor	50 (29.7)
Serous adenocarcinoma	11 (6.6)
Serous borderline tumor	10 (6.0)
Endometrioid carcinoma	6 (3.6)
Metastatic carcinoma	6 (3.6)
Clear cell tumor	5 (3.0)
Mucinous borderline tumor	7 (4.2)
Mucinous adenocarcinoma	3 (1.8)
Carcinosarcoma	2 (1.2)
Other
No histopathology	43 (25.6)
Total	168 (100)

The sensitivity, specificity, PPV, and NPV of RMI and IOTA SR were determined. The sensitivity and specificity of RMI were 72.0% and 90.7%, respectively. The sensitivity and specificity of the IOTA SR were 90.0% and 68.6%, respectively (Table 4). Fig. 1 shows the ROC curve, calculated for the RMI and IOTA SR to compare the AUC of these models. The values for the AUC for RMI and IOTA SR are 0.896 and 0.793, respectively. The AUC of the RMI was significantly higher than the IOTA SR (P < 0.001).

Table 4.

Diagnostic performances of RMI and IOTA SR for characterizing ovarian tumors.

Method	Sensitivity	Specificity	PPV	NPV	AUC
RMI	72.0 (57.3–83.3)	90.7 (83.6–95.0)	76.6 (21.5–35.5)	88.4 (64.5–78.5)	0.896 (0.842–0.950)
IOTA SR	90.0 (77.4–96.3)	68.6 (59.4–76.7)	54.9 (43.5–65.8)	94.2 (34.2–56.5)	0.793 (0.722–0.865)

Values are given as % (95% CI).

AUC, area under the curve; CI, confidence interval; IOTA SR, International Ovarian Tumour Analysis Simple Rules; NPV, negative predictive value; PPV, positive predictive value; RMI, Risk of Malignancy Index.

Fig. 1.

ROC curve showing the performances of RMI and IOTA SR (three levels: benign, inconclusive, and malignant). IOTA SR - International Ovarian Tumor Analysis Simple Rules; RMI, Risk of Malignancy Index; ROC, receiver operating characteristic.

Discussion

The present study was conducted to evaluate the daily practice of preoperatively characterizing ovarian tumors. We therefore compared the diagnostic accuracy of the RMI and IOTA SR. In contrast to previously published studies, our results show that the RMI is significantly more effective at distinguishing a malignant ovarian tumor than the IOTA SR in daily practice (2,24). The diagnostic accuracy of the RMI was classified as good (AUC = 0.896), while the diagnostic accuracy of the IOTA SR was classified as just fair (AUC = 0.793).

In our cohort, the IOTA SR showed a remarkably high rate of inconclusive results (38.7%), which distorts the diagnostic accuracy of the test. Based on our results, 19% of women would have been unnecessarily referred to a gynecological oncological center because their tumor was considered inconclusive, leading to unnecessary stress and possible longer waiting times for women with a true indication for referral to a center. In daily practice, an inconclusive result will most probably be considered as an indication for referral. Therefore, as in previous studies, we classified inconclusive tumors as malignant (2,4,10,20,21,24), realizing that assuming malignancy will improve sensitivity but deteriorate specificity.

The explanation for our high inconclusive rate can be the relatively low exposure to IOTA SR ultrasounds to both IOTA certified sonographers in our clinic. Although it is said that the use of the IOTA SR is simple after following an IOTA training, the diagnostic accuracy increases considerably if the ultrasound is performed by an expert (2,24). Experience and exposure appear to be important for using the IOTA SR, resulting in referring the inconclusive results for expert consultation. Therefore, the use of the IOTA SR will probably be even less satisfactory in daily practice in non-specialized centers. Apart from the large AUC (0.896), the RMI has a relatively low sensitivity compared to the IOTA SR (72.0% and 90.0%, respectively). As described in previous studies, this may be due to the number of non-epithelial and borderline tumors (12,13). As a result, the false-negative number in the RMI group is higher, resulting in missing ovarian cancer which might lead to inadequate treatment. The diagnostic accuracy of the RMI is better than the IOTA SR in our cohort study, which differs from the results in previous studies, in which the IOTA SR is more accurate (2,24). This variation may be due to the rather limited sample size in the current cohort study compared to other studies.

The strengths of the current study were that both the RMI and the IOTA SR were obtained from all participants, allowing the results of both diagnostics tests to be compared. In addition, all ultrasounds were performed by the same two certified IOTA sonographers, keeping the variation minimal and reflecting daily practice. Furthermore, surgeon bias was ruled out by this method due to the fact that all indications for surgery were based on an RMI ≥200 or the Dutch Society for Obstetrics and Gynaecology guideline for ovarian masses (18).

However, it should be noted that the study is performed in a tertiary care center, causing a higher a priori chance of malignancy and therefore a possibility of selection bias. In addition, the study had a rather small number of included patients, so results should be interpreted with caution. It should also be noted that borderline tumors were classified as malignant for statistical analysis. This strategy has also been used in other previous studies (1 –3,11,12,22). Furthermore, if the tumor was considered to be benign according to the RMI and the Dutch Society for Obstetrics and Gynaecology guideline for ovarian masses without clinical symptoms, no surgical intervention was performed so definitive histopathology could not be obtained.

Although the RMI and IOTA SR are currently widely used to characterize ovarian tumors, several other diagnostic models have already been developed, including IOTA-LR2 and the ADNEX-model (1,2,8,15,21). It is therefore important to also validate these models in a daily practice setting.

In conclusion, this cohort study shows that the RMI is a relatively useful diagnostic model in characterizing ovarian tumors, while the IOTA SR only performs fairly. However, due to the relatively low sensitivity of the RMI and high rate of inconclusive results of the IOTA SR, both diagnostic tests do not seem discriminative enough. Therefore, alternative diagnostic models are necessary.

Footnotes

Acknowledgements

The authors acknowledge the assistance of Manita Coolen-Crooijmans and Janine van Dierendonck, for performing transvaginal and transabdominal ultrasound on all included patients.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Esmee E Mulder

Dick Schoot

Jurgen MJ Piek

References

Niemi

Saarelainen

Luukkaala

, et al. Reliability of preoperative evaluation of postmenopausal ovarian tumors. J Ovarian Res 2017; 10:15.

Meys

Kaijser

Kruitwagen

, et al. Subjective assessment versus ultrasound models to diagnose ovarian cancer: A systematic review and meta-analysis. Eur J Cancer 2016; 58:17–29.

Koneczny

Czekierdowski

Florczak

, et al. The use of sonographic subjective tumor assessment, IOTA logistic regression model 1, IOTA Simple Rules and GI-RADS system in the preoperative prediction of malignancy in women with adnexal masses. Ginekol Pol 2017; 88:647–653.

Wynants

Timmerman

Verbakel

, et al. Clinical utility of risk models to refer patients with adnexal masses to specialized oncology care: multicenter external validation using decision curve analysis. Clin Cancer Res 2017; 23:5082–5090.

CRGO. Epithelial Ovarian Cancer 2018. Available at: https://www.oncoline.nl/richtlijn/item/index.php?pagina=/richtlijn/item/pagina.php&richtlijn_id=1088

Al-Asadi

Al-Maliki

Al-Dahhhan

, et al. The accuracy of risk malignancy index in prediction of malignancy in women with adnexal mass in Basrah, Iraq. Niger J Clin Pract 2018; 21:1254–1259.

Sundar

Rick

Dowling

, et al. Refining Ovarian Cancer Test accuracy Scores (ROCkeTS): protocol for a prospective longitudinal test accuracy study to validate new risk scores in women with symptoms of suspected ovarian cancer. BMJ Open 2016; 6:e010333.

Kaijser

Sayasneh

Van Hoorde

, et al. Presurgical diagnosis of adnexal tumours using mathematical models and scoring systems: a systematic review and meta-analysis. Hum Reprod Update 2014; 20:449–462.

Timmerman

Testa

Bourne

, et al. Simple ultrasound-based rules for the diagnosis of ovarian cancer. Ultrasound Obstet Gynecol 2008; 31:681–690.

10.

Garg

Kaur

Mohi

, et al. Evaluation of IOTA Simple Ultrasound Rules to Distinguish Benign and Malignant Ovarian Tumours. J Clin Diagn Res 2017; 11:Tc06–tc9.

11.

Nunes

Ambler

Foo

, et al. Use of IOTA simple rules for diagnosis of ovarian cancer: meta-analysis. Ultrasound Obstet Gynecol 2014; 44:503–514.

12.

Campos

Sarian

Jales

, et al. Performance of the Risk of Malignancy Index for Discriminating Malignant Tumors in Women with Adnexal Masses. J Ultrasound Med 2016; 35:143–152.

13.

Ertas

Vural

Tufekci

, et al. Predictive Value of Malignancy Risk Indices for Ovarian Masses in Premenopausal and Postmenopausal Women. Asian Pac J Cancer Prev 2016; 17:2177–2183.

14.

Krascsenits

Balazs

Dudnyikova

, et al. [ Investigating the predictive value of RMI and ROMA indices in patients with ovarian tumors of uncertain dignity]. Magy Onkol 2016; 60:320–327.

15.

Coosemans

Vergote

Timmerman

, et al. Gynaecologische echografie bij adnexiële massa’s: IOTA-modellen voor dagelijks gebruik. Gunaïkeia 2016; 21:1–5.

16.

International Tumour Analysis Group. IOTA Simple Rules and SRrisk calculator to diagnose ovarian cancer. Available at: https://www.iotagroup.org/iota-models-software/iota-simple-rules-and-srrisk-calculator-diagnose-ovarian-cancer

17.

Timmerman

Ameye

Fischerova

, et al. Simple ultrasound rules to distinguish between benign and malignant adnexal masses before surgery: prospective validation by IOTA group. BMJ 2010; 341:c6839.

18.

NVOG. The enlarged ovary: Ducht Society for Obstetrics and Gynecology 2013. Available at: https://richtlijnendatabase.nl/richtlijn/het_vergrote_ovarium/het_vergrote_ovarium_-_startpagina.html#verantwoording

19.

Landolfo

Valentin

Franchi

, et al. Differences in ultrasound features of papillations in unilocular-solid adnexal cysts: a retrospective international multicenter study. Ultrasound Obstet Gynecol 2018; 52:269–278.

20.

Westwood

Ramaekers

Lang

, et al. Risk scores to guide referral decisions for people with suspected ovarian cancer in secondary care: a systematic review and cost-effectiveness analysis. Health Technol Assess Rep 2018; 22:1–264.

21.

Froyman

Wynants

Landolfo

, et al. Validation of the Performance of International Ovarian Tumor Analysis (IOTA) Methods in the Diagnosis of Early Stage Ovarian Cancer in a Non-Screening Population. Diagnostics 2017; 7:32.

22.

Tantipalakorn

Wanapirak

Khunamornpong

, et al. IOTA simple rules in differentiating between benign and malignant ovarian tumors. Asian Pac J Cancer Prev 2014; 15:5123–5126.

23.

Kurman

RJ CM

Herrington

Young

RH.

WHO Classification of Tumours of Female Reproductive Organs. 4th edn. Lyon: IARC, 2014.

24.

Auekitrungrueng

Tinnangwattana

Tantipalakorn

, et al. Comparison of the diagnostic accuracy of International Ovarian Tumor Analysis simple rules and the risk of malignancy index to discriminate between benign and malignant adnexal masses. Int J Gynecol Obstet 2019; 146:364–369.