Reliability,reproducibility,and potential pitfalls of the O-RADS scoring with non-dynamic MRI

Abstract

Background

The O-RADS scoring has been proposed to standardize the reporting of adnexal lesions using magnetic resonance imaging (MRI).

Purpose

To assess intra- and inter-observer agreement of the O-RADS scoring using non-dynamic MRI and its agreement with pathologic diagnosis, and to provide the pitfalls in the scoring based on discordant ratings.

Material and Methods

Adnexal lesions that were diagnosed using non-dynamic MRI at two centers were scored using O-RADS. Intra- and inter-observer agreements were assessed using kappa statistics. Cross-tabulations were made for intra- and inter-observer ratings and for O-RADS scores and pathological findings.

Results

Intra- and inter-observer agreements were assessed for 404 lesions in 339 patients who were admitted to center 1. Intra-observer agreement was almost perfect (97.8%, kappa = 0.963) and inter-observer agreement was substantial (83.2%, kappa = 0.730). The combined data from center 1 and center 2 included 496 patients; of them, 295 (59.5%) were operated. There was no borderline or malignant pathology for the lesions with O-RADS 1 or 2. Of those with an O-RADS score of 3, 3 (4.1%) lesions were borderline and none were malignant. The O-RADS scoring in discriminating borderline/malignant lesions from benign lesions was outstanding (area under the ROC curve 0.950, 95% CI = 0.923–0.971). Sensitivity, specificity, positive, and negative predictive values of O-RADS 4/5 lesions for borderline/malignant lesions were 96.2%, 87.1%, 72.8%, and 98.4%, respectively.

Conclusion

The O-RADS scoring using non-dynamic MRI is a reproducible method and has good discrimination for borderline/malignant lesions. Potential factors that may lead to discordant ratings are provided here.

Keywords

Magnetic resonance imaging adnexal lesion ovarian cancer O-RADS

Introduction

Ovarian cancer is the eighth most common type of cancer in women and the seventh cause of cancer-related death (1). The incidence of malignant lesions is less than 10% in premenopausal women and 15% in postmenopausal women who undergo surgery for incidental adnexal mass (2 –4). Differentiation between benign and malignant adnexal masses and accurate determination of the origin of the mass are of crucial importance in terms of conservative follow-up in benign lesions, personalized treatment in borderline tumors, such as minimally invasive surgery to preserve ovarian functions, and debulking surgery and systemic treatment decisions in malignant lesions (5 –7).

Ultrasonography (US) is the first-line imaging method used in the diagnosis of adnexal masses. However, an adnexal mass can be considered indeterminate if it does not have typical features of benign lesions in ultrasonography, and the indetermined group constitutes 5%–25% of all adnexal masses (8). Magnetic resonance imaging (MRI) has many advantages over US, has a higher diagnostic sensitivity and specificity for sonographically indeterminate lesions, and a high negative predictive value for malignancy (9 –11). In addition, MRI is superior in detecting the origin of the lesion. Approximately 10% of adnexal lesions identified on US are non-ovarian, and MRI detects the origin of the lesion with an accuracy of 93% (12). Therefore, the European Society of Urogenital Radiology recommends pelvic MRI for lesions thought to be indeterminate on US (13), and the American College of Radiology Ovarian Adnexal Reporting and Data System (O-RADS) MRI committee published a lexicon for risk stratification for adnexal lesions (10,14).

Ovarian lesions are usually assessed with dynamic contrast-enhanced MRI, which has high reproducibility and predictive value for borderline and malign lesions (15). Nevertheless, for various reasons, such as lack of software, non-dynamic examination is usually preferred in routine practice. However, there are insufficient data on the diagnostic value and reproducibility of the O-RADS scoring using non-dynamic MRI. The aim of the present study was to investigate the intra- and inter-observer agreement of the O-RADS scoring using non-dynamic MRI and its agreement with pathologic diagnosis. In addition, we provided the pitfalls in interpreting lesions with the O-RADS scoring based on our discordant ratings.

Material and Methods

The data were obtained retrospectively from the hospital database and the requirement for informed consent was waived. This study was approved by the Institutional Review Board of Bilkent City Hospital (02.11.2022 / E1-22-2995) and was performed according to the Declaration of Helsinki principles.

Study population

The flowchart of the study population and analyses are given in Supplementary Fig. 1. Patients who underwent pelvic MRI at two centers (between March 2019 and July 2022 in center 1 and between June 2021 and March 2022 in center 2) were screened for adnexal mass using the hospital databases. Patients without preoperative MRI scans were excluded from the study. The O-RADS scoring was performed by two radiologists with 10 and 15 years of experience in abdominal radiology. Data from center 1 were scored with the O-RADS system by two observers to assess the inter-observer agreement. The same lesions were rated again by one of the observers 1 month after the first assessment to assess the intra-observer agreement. The data from centers 1 and 2 were combined to assess the diagnostic value of the O-RADS scoring in predicting borderline or malignant lesions, and the agreement between the O-RADS scores and pathological findings in patients who underwent surgical treatment (n = 295). The interpretations were performed blinded to the pathologic findings. All radiological scorings were aimed to show the intrinsic diagnostic value of MRI imaging; therefore, the scorings were also blinded to the clinical and laboratory parameters.

MRI technique

MRI examinations were performed with a Signa Pioneer 3-T MRI scanner (General Electric, Milwaukee, WI, USA). The sequences and other acquisition parameters used for imaging are shown in Supplementary Table 1. All examinations were performed with non-dynamic contrast enhanced MRI. T1-weighted (T1W) images were obtained 30–45 s after contrast injections. MRI examinations that were performed without intravenous administration of contrast material were excluded. All examinations were evaluated visually without creating a time intensity curve (TIC).

Statistical analysis

Categorical variables were expressed as frequency and percentage. Continuous variables were given as mean ± standard deviation (SD). Intra- and inter-observer agreement was evaluated using unweighted kappa statistics, and linear and quadratic using weighted kappa statistics. Because the kappa statistics are affected by prevalence, prevalence and bias-adjusted kappa (PABAK) statistics were also given (16). Kappa values were assessed as follows: 0.21–40 = fair; 0.41–0.60 = moderate; 0.61–0.80 = substantial; and ≥0.81 = almost perfect agreement (16). Analyses were made using Stata 17.0 (StataCorp LP, College Station, TX, USA).

Results

Intra- and inter-observer agreement

Intra- and inter-observer agreement were assessed for 404 lesions in 339 patients (mean age = 39.6 ± 13.8 years) who were admitted to center 1. The agreement was 97.8% and 83.2% for intra- and inter-observer agreement, respectively (Table 1). The kappa and the PABAK values show that intra-observer agreement was almost perfect (0.963 and 0.972, respectively) and inter-observer agreement was substantial (0.730 and 0.789, respectively) (Table 1).

Table 1.

Intra- and inter-observer agreement for O-RADS scoring.

	Intra-observer agreement	Inter-observer agreement
Agreement (%)	97.8	83.2
Kappa (SE)
Unweighted (SE)	0.963 (0.031)	0.730 (0.303)
Linearly weighted (SE)	0.975 (0.032)	0.761 (0.032)
Quadratic weighted (SE)	0.987 (0.049)	0.782 (0.049)
PABAK (SE)
Unweighted (SE)	0.972 (0.009)	0.789 (0.023)
Linearly weighted (SE)	0.986 (0.005)	0.861 (0.018)
Quadratic weighted (SE)	0.994 (0.002)	0.906 (0.019)

PABAK, prevalence-adjusted bias-adjusted kappa; SE, standard error.

Intra- and inter-observer scorings are given in Table 2. The most common category was the O-RADS score of 2, which was observed in more than 50% of lesions. This was followed by a score of 3 (Table 2). The disagreements for the intra-observer scoring were very low (2.2%) and were among the O-RADS scores of 2, 3, and 4 (Table 2).

Table 2.

Cross-tabulations for intra- and inter-observer agreement.

O-RADS scores: Intra-observer agreement							O-RADS scores: Inter-observer agreement
Scores	1	2	3	4	5	Total	Scores	1	2	3	4	5	Total
1	46 (11.39)	0 (0.00)	0 (0.00)	0 (0.00)	0 (0.00)	46 (11.39)	1	39 (9.65)	6 (1.49)	0 (0.00)	0 (0.00)	0 (0.00)	45 (11.14)
2	0 (0.00)	234 (57.92)	2 (0.50)	0 (0.00)	0 (0.00)	236 (58.42)	2	1 (0.25)	201 (49.75)	15 (3.71)	1 (0.25)	0 (0.00)	218 (53.96)
3	0 (0.00)	4 (0.99)	70 (17.33)	2 (0.50)	0 (0.00)	76 (18.81)	3	1 (0.25)	21 (5.20)	59 (14.60)	4 (0.99)	0 (0.00)	85 (21.04)
4	0 (0.00)	0 (0.00)	1 (0.25)	30 (7.43)	0 (0.00)	31 (7.67)	4	3 (0.74)	8 (1.98)	2 (0.50)	25 (6.19)	3 (0.74)	41 (10.15)
5	0 (0.00)	0 (0.00)	0 (0.00)	0 (0.00)	15 (3.71)	15 (3.71)	5	2 (0.50)	0 (0.00)	0 (0.00)	1 (0.25)	12 (2.97)	15 (3.71)
Total	46 (11.39)	238 (58.91)	73 (18.07)	32 (7.92)	15 (3.71)	404 (100.0)	Total	46 (11.39)	236 (58.42)	76 (18.81)	31 (7.67)	15 (3.71)	404 (100.0)

Values are given as n (%). Columns and rows correspond to the scores given in the first and second assessment for intra-observer agreement, and those given by the first and second observer for inter-observer agreement.

As expected, the disagreement for inter-observer scoring was more common (16.8%) (Table 2) compared to that of intra-observer assessment. Most of the disagreements were among the O-RADS 2 to 3 categories. The most confusing lesions were those with multiseptated hemorrhagic or endometriotic cysts, which were categorized as O-RADS 3 by observer 1 and O-RADS 2 by observer 2 (Fig. 1a, b).

Fig. 1.

(a) T2-weighted and (b) contrast-enhanced T1-weighted images of a patient with endometriotic cyst scored as O-RADS 3 by both observers.

Table 3 provides possible reasons for the disagreement for each score.

Table 3.

Potential causes of discordant scoring.

O-RADS	2	3	4	5	Causes of discordant ratings
1	7				There are small differences in the measurements of the lesion size that are very close to 3 cm in premenopausal women.
1		1			The ovarian size was increased and multiple follicles were present. It was considered as multiseptated cyst by one of the observers.
1			3		Some of the lesions classified as ORADS 4 are pathologically myomas or fibromas, which were one of the reasons for inter-observer inconsistency.
1				2	Some of the lesions classified as ORADS 5 are pathologically myomas.
2		36			Some of the multiseptated hemorrhagic or endometriotic cysts were categorized as O-RADS 3 by observer 1 and O-RADS 2 by observer 2. Lesions with several cysts side-by-side were interpreted as multiseptated cysts and evaluated as O-RADS 3 by one of the raters. Hydrosalpinx may be mistakenly considered as a multiseptated cyst due to its endosalpingeal folds.
2			9		Some of the patients were misclassified as O-RADS 4 due to the contrast-enhancing component in mature cystic teratomas.
3			6		Ovarian tissue was considered as a solid component within the cystic mass and was categorized as O-RADS 4 by observer 1.
4				4	The contrast of the solid tissue in the mass compared to the myometrium was visually evaluated differently by readers.

Numbers in italics correspond to the total number of patients scored discordantly for each cell by observer 1 and observer 2 irrespective of the individual scoring of the raters.

Cross-tabulation for O-RADS and pathologic findings

The combined data from centers 1 and 2 included 561 lesions in 496 patients (mean age = 41.2 ± 14.7 years). Among these patients, 295 (59.5%) were operated on, and the percentages of the patients who underwent surgery for O-RADS 1–5 were 11.1%, 47.9%, 67.6%, 89.2%, and 92.0%, respectively. The pathology revealed that 217 (73.6%) lesions were benign, 25 (8.5%) were borderline, and 53 (17.9%) were malignant. The frequency of each diagnosis is given in Supplementary Table 2.

The cross-tabulation for the pathologic findings and O-RADS scores are given in Table 4. There was no borderline or malignant pathology for the lesions with O-RADS 1 or 2. Of those with an O-RADS score of 3, only 3 (4.1%) lesions were borderline and none were malignant. These three patients had large cystic masses with multiseptated but no solid components. These results suggest that the negative predictive value for malignant lesions is high for lesions with O-RADS 3 or below.

Table 4.

Cross-tabulation for O-RADS ratings and the pathological results.

Pathology	O-RADS ratings
Pathology	1	2	3	4	5	Total
Benign	3 (100.0)	115 (100.0)	71 (95.95)	23 (40.35)	5 (10.87)	217 (73.56)
Borderline	0	0	3 (4.05)	11 (19.30)	11 (23.91)	25 (8.47)
Malign	0	0	0	23 (40.35)	30 (65.22)	53 (17.97)
Total	3 (1.02)	115 (38.98)	74 (25.08)	57 (19.32)	46 (15.59)	295 (100.0)

Values denote are given as n (%).

Among the O-RADS score of 4 or 5 (n = 103), 28 (27.2%) were benign lesions (Supplementary Table 3). The majority of these lesions were myoma (25%) (Fig. 2a–d), mature cystic teratoma (21%) (Fig. 3a–c), or fibroma/fibrotechoma (21%) (Fig. 4a–c), and sometimes mucinous cystadenoma (7.1%) (Fig. 5a, b). The reasons for scoring these lesions as O-RADS 4 or 5 were contrast-enhanced components of mature cystic teratomas and contrast enhancement of degenerated fibroids, fibroma, fibrotechoma, and myoma. In addition, one patient had salpingitis and abscess, and one patient had ovarian cyst torsion.

Fig. 2.

(a) T2W and (b) contrast-enhanced T1W images of a patient with a pathologic diagnosis of myoma scored by both observers as O-RADS 5. (c) T2W and (d) contrast-enhanced T1W images of a patient with degenerated myoma scored by both observers as O-RADS 4. T1W, T1-weighted; T2W, T2-weighted.

Fig. 3.

(a) T2-weighted, b) non-contrast T1W, and (c) contrast-enhanced T1W images of a patient with mature cystic teratoma scored as O-RADS 4 by one of the observers. T1W, T1-weighted.

Fig. 4.

(a) T2-weighted, b) non-contrast T1W, and (c) contrast-enhanced T1W images of a patient with fibrotechoma scored as O-RADS 5 by both observers. T1W, T1-weighted.

Fig. 5.

(a) T2-weighted and (b) contrast-enhanced T1-weighted images of a patient with mucinous cystadenoma scored as O-RADS 4 by both observers.

Borderline or malignant lesions were observed in 34 (59.7%) lesions with O-RADS 4 and 41 (89.1%) lesions with O-RADS 5 (Fig. 6a–f). The area under the receiver operating characteristic (ROC) curve (AUC) for O-RADS in discriminating borderline/malignant lesions from benign lesions was 0.950 (95% confidence interval [CI] = 0.923–0.971), suggesting a very good discrimination (Supplementary Fig. 2). Sensitivity, specificity, and positive and negative predictive values (PPV/NPV) of O-RADS 4/5 lesions for borderline/malignant lesions were 96.2%, 87.1%, 72.8%, and 98.4%, respectively.

Fig. 6.

(a) T2-weighted, b) non-contrast T1W, and (c) contrast-enhanced T1W images of endometrioid carcinoma and mucinous borderline tumor (lower panels) scored as O-RADS 4 by both observers.

Discussion

Ovarian cancers have a high mortality rate among gynecological cancers. Therefore, it is important to detect these tumors at an early stage. Borderline ovarian tumors constitute 10%–20% of ovarian epithelial tumors (17). The preoperative diagnosis of lesions in this group is difficult. Although ultrasonographic assessment is the first-line and easily applicable method for diagnosing adnexal masses, it has several important limitations, such as limited imaging area and intestinal gas artifact. MRI overcomes these limitations and is therefore a superior imaging modality in the assessment of ovarian tumors.

The American College of Radiology Ovarian Adnexal Reporting and Data System (O-RADS) MRI committee published a lexicon for risk stratification for adnexal lesions (10,14), aiming to provide standardization in radiological reporting. In the present study, we assessed the reproducibility of the O-RADS scoring with non-dynamic MRI, and its diagnostic performance in discriminating borderline or malignant lesions. Moreover, based on the inter-observer discordant scores, we provided some potential pitfalls in scoring that may be helpful in the correct decision.

We found that the intra-observer agreement of O-RADS ratings was very good (97.8%, kappa = 0.963) and the inter-observer agreement was substantial (83.2%, kappa = 0.730) but not perfect. This is consistent with previous reports (18) and suggests that the O-RADS scoring using non-dynamic MRI has acceptable reproducibility, with some cautions that may lead to discordant ratings.

The highest inter-observer discordance was observed between categories 3 and 2. Some of the multiseptated hemorrhagic or endometriotic cysts were categorized as O-RADS 3 by observer 1 but O-RADS 2 by observer 2 (Fig. 1a, b). Lesions with several side-by-side cysts were interpreted as multiseptated cysts and evaluated as O-RADS 3 by one of the readers. Detailed evaluation of the lesion in different planes is important in this regard.

The size of premenopausal cystic masses (≤3 cm or >3 cm) is used as a criterion to evaluate the lesion in the O-RADS 1 or 2 category. The discordant results between the O-RADS 1 and 2 ratings were due to small differences in the measurements of the lesion size that were very close to 3 cm.

In one patient, ovarian tissue was considered to be a solid component within the cystic mass and was categorized as O-RADS 4 by observer 1.

Approximately one-quarter of the lesions with an O-RADS score of 4 or 5 had benign pathology (Supplementary Table 3). Most of these lesions were due to contrast enhancement of mature cystic teratomas (Fig. 3a–c) and degenerated fibroids, fibroma, fibrotechoma (Fig. 4a–c), and myoma (Fig. 2a–d). For mature cystic teratomas, it has been recommended that the O-RADS score should be 2 in the presence of Rokitansky nodules (19). However, the score can be overestimated due to the possibility of immature cystic teratoma. In a previous study, most of the misclassification was observed for the O-RADS 4 category (19). In the same study, Rokitansky nodules in mature cystic teratomas were classified as O-RADS 4 or 5 due to intermediate or high-risk TIC. If there is a large contrast-enhancing component with an irregular margin, such lesions might be assigned as O-RADS MRI score of 4 (20).

To have a correct classification for fibrous tumors, it should be kept in mind that the T2 signals of these tumors may not always be hypointense. In a previous study, 3/11 (27%) fibroma and fibrothecoma lesions presented dark T2/dark diffusion-weighted image (DWI) (18).

The myomas scored as O-RADS 4 or 5 were large and degenerated. Such cases probably require surgery due to the possibility of being symptomatic; therefore, the misclassification in these cases may not have a clinical implication.

Less common causes of benign lesions with O-RADS 4 or 5 were salpingitis/abscess or ovarian cyst torsion. In the article by Thomassin-Naggara et al. in which they evaluated misclassified cases (19), it was reported that pelvic inflammatory disease was a frequent cause of a false-positive result and that 11/12 cases in their series were false positives. They concluded that radiologists should be very careful in this regard and consider clinical features. American College of Radiology O-RADS MRI recommendations state that adnexal lesions should not be scored in patients presenting with acute symptoms (19). Due to the retrospective design of our study, we did not consider whether these patients had acute symptoms. We also did not exclude these cases from analysis to better emphasize image features that could lead to erroneous inferences, as Thomassin-Naggara et al. did in their article. Clinical findings of these patients along with the imaging are helpful in differential diagnosis, especially in cases such as salpingitis and ovarian torsion.

Dynamic and visual analysis of MRI in diagnosing ovarian malignancies have been compared in a subgroup of patients in the EURAD study (15). Compared with the visual analysis, dynamic contrast-enhanced MRI had a higher sensitivity (96% vs. 76%), specificity (95% vs. 76%), and overall accuracy (86% vs. 78%). In addition, the authors stated that visual analysis may not be very reliable in distinguishing between O-RADS 3 and 4, as some malignant lesions may show enhancement later than the outer myometrium (21). On the other hand, another study indicated that dynamic contrast-enhanced MRI is not superior to conventional delayed contrast-enhanced T1W imaging and DWI in the differentiation of benign and malignant adnexal mass (22). In many centers, dynamic contrast-enhanced MRI cannot be performed due to lack of time and lack of software for intensity curve analysis; therefore, non-dynamic examination is used more commonly in routine practice.

The present study shows that the overall discriminative potential of O-RADS scoring for borderline or malignant lesions with non-dynamic MRI is very high (AUC = 0.950). Thomassin-Naggara et al. assessed the diagnostic value of O-RADS 4/5 scores in diagnosing malignant lesions using dynamic MRI and found that it has a sensitivity of 93%, specificity of 91%, PPV of 71%, and NPV of 98% (12). In the present study, we assessed the same issue using non-dynamic MRI and found a very similar figure (sensitivity of 96.2%, specificity of 87.1%, PPV of 72.8%, and NPV of 98.4%). A very high NPV obtained in both studies suggests that scores less than O-RADS 4/5 are valuable in ruling out malignant lesions. In addition, the data imply that patients with O-RADS 4/5 lesions are good candidates for surgical treatment. On the other hand, as there is no borderline or malignant lesion in patients with O-RADS 1 or 2 lesions, these patients can be safely followed up. We observed that among the patients with O-RADS 3, only 3 (4.1%) patients had borderline tumors but there was no malignant tumor in this group. All three patients had large multiseptated cystic masses and no solid component. The solid component of borderline cystadenomas may not be clearly visible. The PPV for malignancy in lesions with the O-RADS 3 category has been reported as approximately 5% (12). In another study, lesions with a low-risk TIC have a PPV of 6.7% for malignancy, and most lesions found to be borderline tumors in this category (23). These findings are consistent with our study and suggest that for O-RADS 3 lesions, the patient's age and lesion size may play a role in the surgical decision.

In conclusion, the present study shows that the O-RADS scoring with non-dynamic MRI has acceptable accuracy and reproducibility and has a very good diagnostic yielding. Specifically, the cutoff value of O-RADS 4/5 has a very high NPV for borderline or malignant lesions, suggesting that scores less than O-RADS 4/5 have a high potential in ruling out borderline and malignant lesions. The reasons for discordant ratings that we presented here might be helpful for radiologists with regard to correct scoring.

Supplemental Material

sj-pdf-1-acr-10.1177_02841851241279897 - Supplemental material for Reliability, reproducibility, and potential pitfalls of the O-RADS scoring with non-dynamic MRI

Supplemental material, sj-pdf-1-acr-10.1177_02841851241279897 for Reliability, reproducibility, and potential pitfalls of the O-RADS scoring with non-dynamic MRI by Gulsum Kılıçkap, Betül Akdal Dölek, Serhat Kaya and Numan Ilteriş Çevik in Acta Radiologica

Footnotes

Data availability

Institutional permission for data sharing will be asked upon reasonable request.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Supplemental material

Supplemental material for this article is available online.

ORCID iDs

Gulsum Kılıçkap

Betül Akdal Dölek

References

International Agency for Research on Cancer. Available at: https://gco.iarc.fr/today/online-analysis-multi-bars?v=2020 (accessed 1 November 2023).

Baser

Erkilinc

Esin

, et al. Adnexal masses encountered during cesarean delivery. Int J Gynaecol Obstet 2013;123:124–126.

Masch

Daye

Lee

SI.

MR Imaging for incidental adnexal mass characterization. Magn Reson Imaging Clin N Am 2017;25:521–543.

van Nagell

Jr DePriest

, et al. Ovarian cancer screening with annual transvaginal sonography: findings of 25,000 women screened. Cancer 2007;109:1887–1896.

Woo

Kyrgiou

Bryant

, et al. Centralisation of services for gynaecological cancer. Cochrane Database Syst Rev 2012;2012:CD007945.

Engelen

Kos

Willemse

, et al. Surgery by consultant gynecologic oncologists improves survival in patients with ovarian carcinoma. Cancer 2006;106:589–598.

Froyman

Landolfo

De Cock

, et al. Risk of complications in patients with conservatively managed ovarian tumours (IOTA5): a 2-year interim analysis of a multicentre, prospective, cohort study. Lancet Oncol 2019;20:448–458.

Van Calster

Timmerman

Valentin

, et al. Triaging women with ovarian masses for surgery: observational diagnostic study to compare RCOG guidelines with an International Ovarian Tumour Analysis (IOTA) group protocol. BJOG 2012;119:662–671.

Adusumilli

Hussain

Caoili

, et al. MRI of sonographically indeterminate adnexal masses. AJR Am J Roentgenol 2006;187:732–740.

10.

Thomassin-Naggara

Toussaint

Perrot

, et al. Characterization of complex adnexal masses: value of adding perfusion- and diffusion-weighted MR imaging to conventional MR imaging. Radiology 2011;258:793–803.

11.

Thomassin-Naggara

Aubert

Rockall

, et al. Adnexal masses: development and preliminary validation of an MR imaging scoring system. Radiology 2013;267:432–443.

12.

Thomassin-Naggara

Poncelet

Jalaguier-Coudray

, et al. Ovarian-adnexal reporting data system magnetic resonance imaging (O-RADS MRI) score for risk stratification of sonographically indeterminate adnexal masses. JAMA Netw Open 2020;3:e1919896.

13.

Forstner

Thomassin-Naggara

Cunha

, et al. ESUR recommendations for MR imaging of the sonographically indeterminate adnexal mass: an update. Eur Radiol 2017;27:2248–2257.

14.

Reinhold

Rockall

Sadowski

, et al. Ovarian-adnexal reporting lexicon for MRI: a white paper of the ACR ovarian-adnexal reporting and data systems MRI committee. J Am Coll Radiol 2021;18:713–729.

15.

Wengert

Dabi

Kermarrec

, et al. O-RADS MRI classification of indeterminate adnexal lesions: time-intensity curve analysis is better than visual assessment. Radiology 2022;303:566–575.

16.

Sim

Wright

CC.

The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther 2005;85:257–268.

17.

Bourdel

Huchon

Abdel Wahab

, et al. Borderline ovarian tumors: guidelines from the French national college of obstetricians and gynecologists (CNGOF). Eur J Obstet Gynecol Reprod Biol 2021;256:492–501.

18.

Tang

Cai

, et al. Accuracy and reproducibility of the O-RADS MRI risk stratification system based on enhanced non-DCE MRI in the assessment of adnexal masses. Eur J Radiol 2023;159:110670.

19.

Thomassin-Naggara

Belghitti

Milon

, et al. O-RADS MRI score: analysis of misclassified cases in a prospective multicentric European cohort. Eur Radiol 2021;31:9588–9599.

20.

Sadowski

Maturen

Rockall

, et al. Ovary: MRI characterisation and O-RADS MRI. Br J Radiol 2021;94:20210157.

21.

Thomassin-Naggara

Razakamanantsoa

Rockall

O-RADS MRI: where are we and where we are going?

Eur Radiol 2023;33:8155–8156.

22.

Otani

Kido

Himoto

, et al. Diagnostic value of DCE-MRI for differentiating malignant adnexal masses compared with contrast-enhanced-T1WI. Magn Reson Med Sci 2022;21:599–607.

23.

Sadowski

Thomassin-Naggara

Rockall

, et al. O-RADS MRI risk stratification system: guide for assessing adnexal lesions from the ACR O-RADS committee. Radiology 2022;303:35–47.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.32 MB