Diagnostic Performance of Ultrasound-Based Risk-Stratification Systems for Thyroid Nodules: Comparison of the 2015 American Thyroid Association Guidelines with the 2016 Korean Thyroid Association/Korean Society of Thyroid Radiology and 2017 American College of Radiology Guidelines

Abstract

Purpose:

The aim of this study was to compare the diagnostic performance of ultrasound (US)-based risk-stratification systems for thyroid nodules in the 2015 American Thyroid Association (ATA) guidelines with those of the 2016 Korean Thyroid Association (KTA)/Korean Society of Thyroid Radiology (KSThR) and 2017 American College of Radiology (ACR) guidelines.

Methods:

From June 2013 to May 2015, a total of 902 consecutive thyroid nodules were enrolled in four institutions, and their US features were retrospectively reviewed and classified using the categories defined by the three guidelines. The malignancy risk of each category, as defined by all three risk-stratification systems, was calculated, and the diagnostic performance of the fine-needle aspiration (FNA) indications of the ATA guidelines were compared to those of the KTA/KSThR and ACR guidelines.

Results:

Of all nodules, 636 (70.5%) were benign and 266 (29.5%) malignant. The calculated malignancy risks for ATA categories 5, 4, 3, 2, and 1 nodule(s) were 71.7, 21.5, 2.6, 3.8, and 0%. Of all nodules, 7.6% (69/902) did not meet the ATA pattern criteria, but the malignancy risk was calculated to be 10.1% (7/69). The ATA guidelines afforded significantly higher diagnostic sensitivity (95.0%) than the ACR guidelines (80.2%; p = 0.001) but a lower specificity (38.1 vs. 68.9%; p < 0.001). On the other hand, the ATA guidelines exhibited a lower diagnostic sensitivity than the KTA/KSThR guidelines (100.0%; p = 0.07) but a higher specificity (28.2%; p < 0.001). The unnecessary FNA rate was the lowest when the ACR guidelines were used (25.8%), followed by the ATA (51.2%) and KTA/KSThR (59.4%) guidelines.

Conclusion:

The 2015 ATA guidelines afford relatively moderate sensitivity and an unnecessary FNA rate for thyroid cancer detection compared to the 2016 KTA/KSThR and 2017 ACR guidelines. US practitioners require a deep understanding of the benefits and risks of the US-based FNA criteria of different guidelines and potential impact on the diagnosis of low-risk thyroid cancers.

Introduction

Ultrasonography (US) is a primary diagnostic tool when assessing the risk of malignancy of thyroid nodules and can aid decision making when fine-needle aspiration (FNA) is contemplated (1 –3). Therefore, the clinical practice guidelines recommend US-based management of thyroid nodules (1 –3). However, as suspicious US features and the size cutoffs for FNA are not consistent among the various international guidelines, US operators may become confused when interpreting data, and effective communication with other professionals may be difficult (4,5).

Of the various guidelines, the American Thyroid Association (ATA) guidelines are widely used in practice. The revised 2015 ATA guidelines suggest a pattern-based approach for risk stratification and give the estimated risks of malignancy for each category (pattern) (1,4). Although the guidelines are effective in terms of risk stratification of thyroid nodules, the relatively complex US features of each category and the existence of a “not specified” pattern with an 18.2–19.0% malignancy risk are accepted limitations (6,7). The Korean Thyroid Association (KTA)/Korean Society of Thyroid Radiology (KSThR) proposed a simpler pattern-based approach in 2016 (3,4). This features a combination of solidity, echogenicity, and suspicious US features, and can be easily substituted for the ATA guidelines. The American College of Radiology (ACR) guidelines are a new risk-stratification system published in 2017. This system assigns points for all US features of a nodule, and the total score determines the nodule ACR Thyroid Imaging, Reporting, and Data System (TI-RADS) level (2,4). However, although the three international guidelines are widely used in different countries, few studies have compared their diagnostic performances (7). In addition, to the authors' knowledge, no study has compared the diagnostic performance of the guidelines for nodules <1 cm that are highly suspicious on US. This information would allow physicians to communicate more effectively with patients in the shared decision-making process for active surveillance.

The purpose of this study was to compare the diagnostic performance of US-based risk-stratification systems for thyroid nodules in the 2015 ATA guidelines with those of the 2016 KTA/KSThR and 2017 ACR guidelines.

Methods

This multicenter study was approved by the Institutional Review Boards of the four participating centers. Informed consent was waived for this retrospective review.

Study population

Patient data were retrospectively collected from four different hospitals (one primary medical center and three tertiary hospitals). From June 2013 to May 2015, a total of 1109 thyroid nodules (>5 mm in diameter) in 928 consecutive patients who had undergone thyroid US were initially enrolled. Of the 1109 nodules, 198 nodules of 169 patients were excluded because final diagnoses were not available, or the US characteristics could not be analyzed in nine nodules that were completely calcified. Therefore, a total of 902 nodules in 750 patients were finally included (594 females and 156 males; M _age = 49.2 years; range 9–81 years). The data set was obtained from the KSThR database (SOMARTUS), which was used to validate the K-TI-RADS. The patients/nodules evaluated here include those discussed in a prior publication (8).

Final diagnoses were determined by the cytopathologic results of the Bethesda system, and surgical findings. The cytological results of non-diagnostic lesions and lesions of indeterminate significance (atypia of undetermined significance or only a suspicion of follicular neoplasm and malignancy) were excluded.

US examination and image analysis

All US examinations were performed using a 10–16 MHz linear probe and real-time US systems (EUB-7500, Hitachi Medical Systems, Tokyo, Japan; iU22 and HDI-5000, Philips Healthcare, Bothell, WA; Aplio SSA-770A, Toshiba Medical Systems Corp., Otawara-shi, Japan; Accuvix XG, Samsung Medison, Seoul, Korea). Five board-certified radiologists (D.G.N., W.J.M., Y.H.L., N.C., and E.J.H.) in the four different hospitals who specialized in thyroid imaging (with 8–20 years of clinical experience with thyroid US) performed all examinations. Each radiologist assessed the US features of all thyroid nodules for their own patients, blinded to FNA data and the final diagnoses.

Before starting the data collection, training sessions were held to establish a baseline consensus on the evaluation of US features by the five radiologists. They evaluated images of 50 masses not included in this study and assessed the US features during a consensus meeting. Internal nodule contents were categorized as solid (uniformly solid or nearly so), predominantly solid (<50% cystic), predominantly cystic (>50% cystic), and cystic (no obvious solid content). In partially cystic nodules, the configuration of the solid areas was categorized as concentric or eccentric. Nodule echogenicity was categorized as markedly hypoechogenic, hypoechogenic, isoechogenic, and hyperechogenic by reference to the predominant echogenicity compared to that of normal thyroid tissue and the anterior strap muscles. Calcification was classified as microcalcification, macrocalcification, or rim calcification. In rim-calcified nodules, the presence or absence of an extrusive soft-tissue component was noted to allow analysis using the ATA guidelines. Nodules with disrupted rim calcifications and small protrusions of soft tissue around the calcifications were considered positive for this feature. A spongiform appearance was defined as an isoechoic pattern with microcystic changes in >50% of the nodular volume. The comet-tail artifact was defined as intracystic echogenic foci accompanied by reverberation artifacts. Extrathyroidal extension status was not evaluated in this study because of limitations imposed by the retrospective design and the absence of standardized US criteria for diagnosis of such extensions.

Statistical analyses

All nodules were retrospectively classified using the US criteria of three international guidelines (Supplementary Tables S1 and S2; Supplementary Data are available online at www.liebertpub.com/thy). The malignancy risks and frequencies associated with each category were calculated as percentages. The associations between all categories and the final diagnoses were evaluated with the aid of the Cochran–Armitage test for trend.

All nodules were dichotomized into two groups: those for which US-guided FNA was or was not indicated by the FNA criteria (i.e., sonographic features and nodule size; Supplementary Tables S1 and S2). The diagnostic performances of all criteria in terms of thyroid cancer were evaluated. Sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and accuracy of each set of guidelines were calculated, and these values were compared using the McNemar test. The potentially unnecessary FNA rate for the diagnosis of thyroid cancer was defined as the number of benign nodules among FNA-required nodules in the entire nodules, and calculated as percentages. The results were compared using the McNemar test.

All statistical analyses were performed with IBM SPSS Statistics for Windows v23 for Windows (IBM Corp., Armonk, NY) and R v3.4.1 for Windows (R Development Core Team, Vienna, Austria). A p-value of <0.05 was taken to indicate a significant difference.

Results

The mean nodule diameter was 1.5 ± 1.1 cm. The final diagnoses of 902 nodules were 636 (70.5%) benign and 266 (29.5%) malignant nodules. Final diagnoses were determined via surgical resection in 191 (71.8%) of 266 malignant nodules, which featured 186 papillary thyroid carcinomas (PTCs), including 24 of the follicular variant and five follicular carcinomas. Thirty-six benign nodules confirmed by surgery included 31 nodular hyperplasias, four follicular adenomas, and one thyroiditis. All 75 malignant nodules diagnosed via FNA or core-needle biopsy were PTCs.

Frequencies and malignancy risks by guideline categories

Table 1 and Supplementary Table S3 list the frequencies and malignancy risks of all categories of the three guidelines. For the 2015 ATA guidelines, all categories were generally within the ranges of the suggested malignancy risks, except the “very low suspicion” category. The malignancy risk of this category was slightly higher (3.8%; 6/157) than the suggested risk (<3%). Of all nodules, 7.6% (69/902) did not meet the criteria for any pattern, but the malignancy risk was 10.1% (7/69).

Table 1.

Frequencies and Malignancy Risks of Tumors of the Various Categories of the Three Guidelines

Guidelines	Benign nodules (n = 636)	Malignant nodules (n = 266)	Total (n = 902)	Suggested risk of malignancy (%)	Calculated risk of malignancy (%)	p-Value
ATA 2015						<0.001
High suspicion	85	215	300 (33.3)	>70–90	71.7 [66.6–76.8]
Intermediate suspicion	117	32	149 (16.5)	10–20	21.5 [14.9–28.1]
Low suspicion	221	6	227 (25.2)	5–10	2.6 [0.6–4.7]
Very low suspicion	151	6	157 (17.4)	<3	3.8 [0.8–6.8]
Benign	0	0	0 (0.0)	<1	0
Not specified	62	7	69 (7.6)	—	10.1 [3.0–17.3]
ACR 2017						<0.001
Highly suspicious	84	202	286 (31.7)	>20	70.6 [65.4–0.75.9]
Moderately suspicious	213	53	266 (29.5)	5–20	19.9 [15.1–24.7]
Mildly suspicious	210	7	217 (24.1)	5	3.2 [0.9–5.6]
Not suspicious	105	4	109 (12.1)	<2	3.7 [0.1–7.2]
Benign	24	0	24 (2.7)	<2	0
KTA/KSThR 2016						<0.001
High suspicion	76	210	286 (31.7)	>60	73.4 [68.3–78.5]
Intermediate suspicion	187	44	231 (25.6)	15–50	19.0 [14.0–24.1]
Low suspicion	328	12	340 (37.7)	3–15	3.5 [1.6–5.5]
Benign	45	0	45 (5.0)	<3	0.0

Data in parentheses are percentages; data in square brackets are confidence intervals.

ACR, American College of Radiology; ATA, American Thyroid Association; KTA/KSThR, Korean Thyroid Association/ Korean Society of Thyroid Radiology.

In terms of the 2017 ACR guidelines, nodules in the “not suspicious” category had a higher malignancy risk (3.7%; 4/109) than the suggested risk (<2%). Of a total of 109 nodules in the “not suspicious” category, 5 and 104, respectively, were matched to the “low suspicion” and “very low suspicion” categories of the ATA guidelines, and 91 and 18, respectively, to the “low suspicion” and “benign” categories of the KTA/KSThR guidelines. All predicted malignancy probabilities tended to rise through the higher risk categories of all guidelines (p < 0.001).

Diagnostic performance in terms of detecting thyroid cancer

Table 2 lists the diagnostic performances of FNA indications for prediction of thyroid malignancies in nodules ≥1 cm in diameter. The ACR guidelines afforded the lowest sensitivity and NPV (80.2% and 94.4%, respectively), followed by (in increasing order) the ATA (95.0% and 97.4%, respectively) and the KTA/KSThR (100.0% and 100.0%, respectively) guidelines. When nodules with an unclassified pattern in the ATA guidelines were considered as an intermediate suspicion group, the sensitivity of the ATA guidelines increased to 99.0%. The ATA guidelines afforded a significantly higher diagnostic sensitivity than the ACR guidelines (p = 0.001) but a lower sensitivity than the KTA/KSThR guidelines (p = 0.07).

Table 2.

Comparison of the Diagnostic Performances of Various Guidelines in Terms of Detecting Malignant Thyroid Nodules (≥1 cm)

Guidelines	Sensitivity	Specificity	PPV	NPV	Accuracy
All nodules (n = 586)
ATA	95.0% (96/101)	38.1% (185/485)	24.2% (96/396)	97.4% (185/190)	48.0% (281/586)
ACR	80.2% (81/101)	68.9% (334/485)	34.9% (81/232)	94.4% (334/354)	70.8% (415/586)
KTA/KSThR	100.0% (101/101)	28.2% (137/485)	22.5% (101/449)	100.0% (137/137)	40.6% (238/586)

PPV, positive predictive value; NPV, negative predictive value.

Table 3 lists the diagnostic performances of FNA indications for nodules with highly suspicious US features in terms of predicting thyroid microcarcinomas. The PPV and accuracy of the ATA guidelines (74.1% and 77.2%, respectively) were similar to those of the ACR (73.5% and 75.3%, respectively) and KTA/KSThR (74.9% and 77.8%, respectively) guidelines.

Table 3.

Comparison of the Diagnostic Performances of Various Guidelines in Terms of Detecting Malignant Thyroid Nodules (<1 cm)

Guidelines	Sensitivity	Specificity	PPV	NPV	Accuracy
High suspicion (n = 316)
ATA	86.7% (143/165)	66.9% (101/151)	74.1% (143/193)	82.1% (101/123)	77.2% (244/316)
ACR	82.4% (136/165)	67.5% (102/151)	73.5% (136/185)	77.9% (102/131)	75.3% (238/316)
KTA/KSThR	86.7% (143/165)	68.2% (103/151)	74.9% (143/191)	82.4% (103/125)	77.8% (246/316)

Unnecessary FNA rates

Table 4 compares the unnecessary FNA rates. Of the three guidelines, the rate was the lowest when the ACR guidelines were adopted (25.8%), followed by the ATA (51.2%) and KTA/KSThR guidelines (59.4%). The ATA guidelines were associated with higher unnecessary FNA rates than the ACR guidelines but lower rates than the KTA/KSThR guidelines (p < 0.001).

Table 4.

Comparison of Unnecessary FNA Rates for the Diagnosis of Thyroid Cancer (≥ 1cm)

Guidelines	No. of nodules indicated for FNAs	No. of malignant nodules among the FNA nodules	No. of benign nodules among the FNA nodules (A)	Unnecessary FNA rate (A/586)
ATA	396	96 (24.2)	300 (75.8)	51.2%
High suspicion (n = 107)	107	72	35
Intermediate suspicion (n = 92)	92	18	74
Low suspicion (n = 198)	126	3	123
Very low suspicion (n = 136)	71	3	68
Benign (n = 0)	0	0	0
Not specified (n = 53)	0	0	0
ACR	232	81 (34.9)	151 (65.1)	25.8%
Highly suspicious (n = 101)	101	66	35
Moderately suspicious (n = 180)	93	14	79
Mildly suspicious (n = 189)	38	1	37
Not suspicious (n = 93)	0	0	0
Benign (n = 23)	0	0	0
KTA/KSThR	449	101 (22.5)	348 (77.5)	59.4%
High suspicion (n = 95)	95	67	28
Intermediate suspicion (n = 156)	156	27	129
Low suspicion (n = 293)	198	7	191
Benign (n = 42)	0	0	0

Data in parentheses are percentages.

FNA, fine-needle aspiration.

Discussion

The diagnostic performance of the 2015 ATA guidelines was compared to those of the 2016 KTA/KSThR and 2017 ACR guidelines. The ATA guidelines afforded a relatively moderate diagnostic sensitivity (95.0%), specificity (38.1%), and unnecessary FNA rate (51.2%) compared to the other guidelines. Of all nodules, 7.6% did not meet the criteria for any pattern of the ATA guidelines, but the malignancy risk was 10.1%. When an unclassified pattern was considered to arouse intermediate suspicion, the diagnostic sensitivity increased to 99.0%, with 28.0% specificity and a 59.6% unnecessary FNA rate.

US-based risk-stratification systems play essential roles when considering whether thyroid nodules should be subjected to FNA (1 –4). However, as the US features of each category and the size cutoff for FNA are not consistent among the guidelines, a deep understanding of the benefits and risks of the US-based FNA criteria of different guidelines is required for US practitioners to optimize patient management (7). In the time since 2015, various international societies have revised their guidelines. Thus, the most recent risk-stratification systems were compared in the present study. First, the malignancy risks were validated and compared by the categories of all three guidelines. It was a multicenter geographic external validation in terms of the malignant risks in each category. These were generally well matched in terms of suggested malignancy risk range. However, the malignancy risk of the “very low suspicion” category of the 2015 ATA guidelines exhibited a slightly higher malignancy risk (3.8%; 6/157) than suggested. As previously found by Yoon et al. and Ha et al. (6,7), 7.6% of nodules did not meet the criteria of any ATA pattern (i.e., isoechoic nodules with irregular margins, microcalcifications, and a taller-than-wide shape), and their malignancy risk was calculated to be 10.1% in the present study. Therefore, such nodules should be interpreted as being of intermediate suspicion in terms of malignancy risk. In the 2017 ACR guidelines, the malignancy risk of the “not suspicious” category (i.e., partially cystic and isohyperechoic nodules without calcifications, taller-than-wide, lobulated or irregular margins) was higher (3.7%) than the suggested malignancy risk, as reported earlier by Ha et al., who found that the actual risk of malignancy was 4.4% (7). The mean diameter of malignant tumors showing these US patterns was 27.3 mm (range 17–34 mm) in this study. As the US patterns of such nodules are not highly suggestive of benign status (i.e., the nodules are not simple cysts or spongiform in appearance), it is considered that FNA may be selectively indicated for such nodules if they are large.

Regarding the diagnostic performance of the three guidelines, significant differences were found between the ACR and the ATA and KTA/KSThR guidelines. The ACR guidelines were significantly less sensitive (80.2%) than the other two guidelines (95.0–100.0%). When the ATA “not specified” pattern was considered to arouse an intermediate level of suspicion, the sensitivities of the ATA and KTA/KSThR guidelines increased to 99.0–100.0%. On the other hand, the ACR guidelines afforded a significantly higher specificity (68.9%) than the other guidelines (28.2–38.1%). It is probably because of differences in the management guidelines for thyroid nodules 1.0–2.5 cm in diameter without highly suspicious US patterns. The ACR guidelines propose higher cutoffs for mildly (1.5 cm) and moderately (2.5 cm) suspicious nodules than do the ATA and KTA/KSThR guidelines (1.0 and 1.5 cm, respectively). The clinical significance of delayed diagnosis of thyroid cancers that meet these US criteria is not well known. Although Nam et al. reported that PTCs that did not meet the malignant US criteria exhibited better prognosis than PTCs that did meet the criteria, further studies are required to confirm that patients with PTCs without suspicious US features have tumors of lower stage and good outcomes in terms of the absence of distant metastasis or recurrence (9). Molecular testing may be a useful prognostic marker for thyroid cancer. However, it requires biopsy of a nodule.

Turning to the unnecessary FNA rates, the KTA/KSThR guidelines were associated with the highest such rate, and the ACR guidelines were associated with the lowest rate. The sensitivity of the KTA/KSThR guidelines for nodules >1 cm was 100.0%, higher than previously reported, in the present study (8). It is probably because of the higher prevalence of PTC, especially the classic type, in the study population. The low size criterion of low suspicion nodules for FNA may be a reasonable cause for the highest sensitivity of the KTA/KSThR guidelines. When size criterion of low suspicion nodules for FNA was increased from 1.5 to 2.0 cm in the KTA/KSThR guidelines, the diagnostic specificity increased from 28.2% to 44.1%, the unnecessary FNA rate decreased from 59.4% to 46.2%, and the sensitivity from 100.0% to 97.0%. It is potentially important to analyze the actual impact of these differences among the guidelines on the diagnosis of thyroid cancers that will benefit from surgery or possibly further treatment in the future. Recently, the management paradigm for thyroid nodules has become more conservative, thus minimizing FNA. The current guidelines suggest active surveillance of subcentimeter nodules that are highly suspicious on US, rather than US-guided FNA (1 –3). This is based on the fact that active surveillance, instead of immediate surgery, can be chosen for adult patients with low-risk papillary thyroid microcarcinomas (10,11). However, controversy exists around whether to perform FNA on subcentimeter nodules that exhibit highly suspicious US features because some patients with benign nodules mimicking the US features of malignancy may undergo unnecessary active surveillance. A recent study reported that the risk of malignancy ranged from 77.4% to 82.8% among suspicious thyroid nodules <1 cm in diameter on US (12). This study found that the FNA criteria of the ACR, ATA, and KTA/KSThR guidelines afforded similar diagnostic PPVs and accuracies for nodules with suspicious US features (approximately 74% and 77%, respectively). Knowledge of the diagnostic performance of US-based FNA criteria for nodules <1 cm that are highly suspicious on US would allow physicians to communicate more effectively with patients in the shared decision-making process.

This study had several limitations. First, only thyroid nodules that had undergone US-guided FNA were included, which was usually performed when suspicious US features were noted or on the largest nodule when no suspicious feature was detected. Therefore, selection bias may be in play. Second, the malignancy rate was high (29.5%), probably because three of the participating institutions were tertiary referral centers dealing with patients with more serious disease. The diagnostic performances of the guidelines may have been affected by this high malignancy rate. A high percentage of PTCs (97.4%) with a relatively low percentage of follicular carcinomas (2.6%) may also increase the diagnostic performances of the three guidelines. Third, the final diagnoses of benign nodules were based on both cytopathologic data and surgical histology. This may cause false-negative and false-positive results. The cytological results of indeterminate significance were excluded without surgical confirmation. This may have influenced a small proportion of follicular thyroid cancers or follicular variant PTCs. Fourth, extrathyroidal extension status was not evaluated because of limitations by the retrospective design, and completely calcified nodules were excluded in this study. They may affect the diagnostic performances of the ATA and ACR guidelines. Fifth, the pathological classification criteria of “noninvasive follicular thyroid neoplasm with papillary-like nuclear features” were not defined at the time of data collection and were not evaluated in this study. Sixth, the study did not evaluate the variation in interpretation among the readers. Seventh, the indications for FNA made on criteria in this study have differed from the recent guidelines. Since FNA was generally indicated for all nodules ≥1 cm, except for typical benign nodules, it is considered that the results may be more reliable to evaluate the diagnostic performance and unnecessary FNA rates.

In conclusion, the 2015 ATA guidelines afford a relatively moderate sensitivity and unnecessary FNA rates for detecting thyroid cancer compared to the 2016 KTA/KSThR and 2017 ACR guidelines. US practitioners require a deep understanding of the benefits and risks of the US-based FNA criteria of different guidelines and potential impact on the diagnosis of low risk thyroid cancers.

Footnotes

Author Disclosure Statement

There is no conflict of interest to be disclosed for any of the authors regarding this manuscript. No source of funding was received for this work.

References

Haugen

, Alexander

, Bible

, Doherty

, Mandel

, Nikiforov

, Pacini

, Randolph

, Sawka

, Schlumberger

, Schuff

, Sherman

, Sosa

, Steward

, Tuttle

, Wartofsky

. 2016. 2015 American Thyroid Association management guidelines for adult patients with thyroid nodules and differentiated thyroid cancer: the American Thyroid Association Guidelines Task Force on Thyroid Nodules and Differentiated Thyroid Cancer. Thyroid, 26:1–133.

Tessler

, Middleton

, Grant

, Hoang

, Berland

, Teefey

, Cronan

, Beland

, Desser

, Frates

, Hammers

, Hamper

, Langer

, Reading

, Scoutt

, Stavros

. 2017. ACR Thyroid Imaging, Reporting and Data System (TI-RADS): White Paper of the ACR TI-RADS Committee. J Am Coll Radiol, 14:587–595.

Shin

, Baek

, Chung

, Ha

, Kim

, Lee

, Lim

, Moon

, Na

, Park

, Choi

, Hahn

, Jeon

, Jung

, Kim

, Kwak

, Lee

, Park

, Sung

; Korean Society of Thyroid Radiology and Korean Society of Radiology. 2016. Ultrasonography diagnosis and imaging-based management of thyroid nodules: revised Korean Society of Thyroid Radiology consensus statement and recommendations. Korean J Radiol, 17:370–395.

, Baek

, Na

. 2017. Risk stratification of thyroid nodules on ultrasonography: current status and perspectives. Thyroid, 27:1463–1468.

Yoon

, Han

, Kim

, Moon

, Kwak

. 2017. Diagnosis and management of small thyroid nodules: a comparative study with six guidelines for thyroid nodules. Radiology, 283:560–569.

Yoon

, Lee

, Kim

, Moon

, Kwak

. 2016. Malignancy risk stratification of thyroid nodules: comparison between the Thyroid Imaging Reporting and Data System and the 2014 American Thyroid Association management guidelines. Radiology, 278:917–924.

, Na

, Baek

, Sung

, Kim

, Kang

. 2018. Ultrasound fine needle aspiration biopsy for thyroid malignancy: diagnostic performance of seven society guidelines applied to 2000. Radiology, 287:893–900.

, Moon

, Na

, Lee

, Choi

, Kim

. 2016. A multicenter prospective validation study for the Korean Thyroid Imaging Reporting and Data System in patients with thyroid nodules. Korean J Radiol, 17:811–821.

Nam

, Shin

, Han

, Ko

, Hahn

, Chung

. 2013. Preoperative ultrasonographic features of papillary thyroid carcinoma predict biological behavior. J Clin Endocrinol Metab, 98:1476–1482.

10.

Ito

, Oda

, Miyauchi

. 2016. Insights and clinical questions about the active surveillance of low-risk papillary thyroid microcarcinomas [Review]. Endocr J, 63:323–328.

11.

Oda

, Miyauchi

, Ito

, Yoshioka

, Nakayama

, Sasai

, Masuoka

, Yabuta

, Fukushima

, Higashiyama

, Kihara

, Kobayashi

, Miya

. 2016. Incidences of unfavorable events in the management of low-risk papillary microcarcinoma of the thyroid by active surveillance versus immediate surgery. Thyroid, 26:150–155.

12.

, Kim

, Baek

. 2017. Detection of malignancy among suspicious thyroid nodules <1 cm on ultrasound with various thyroid image reporting and data systems. Thyroid, 27:1307–1315.