Nomogram Based on Shear-Wave Elastography Radiomics Can Improve Preoperative Cervical Lymph Node Staging for Papillary Thyroid Carcinoma

Abstract

Background:

Accurate preoperative prediction of cervical lymph node (LN) metastasis in patients with papillary thyroid carcinoma (PTC) provides a basis for surgical decision-making and the extent of tumor resection. This study aimed to develop and validate an ultrasound radiomics nomogram for the preoperative assessment of LN status.

Methods:

Data from 147 PTC patients at the Wuhan Tongji Hospital and 90 cases at the Hunan Provincial Tumor Hospital between January 2017 and September 2019 were included in our study. They were grouped as the training and external validation set. Radiomics features were extracted from shear-wave elastography (SWE) images and corresponding B-mode ultrasound (BMUS) images. Then, the minimum redundancy maximum relevance algorithm and the least absolute shrinkage and selection operator regression were used to select LN status-related features and construct the SWE and BMUS radiomics score (Rad-score). Multivariate logistic regression was performed using the two radiomics scores together with clinical data, and a nomogram was subsequently developed. The performance of the nomogram was assessed with respect to discrimination, calibration, and clinical usefulness in the training and external validation set.

Results:

Both the SWE and BMUS Rad-scores were significantly higher in patients with cervical LN metastasis. Multivariate analysis indicated that the SWE Rad-scores, multifocality, and ultrasound (US)-reported LN status were independent risk factors associated with LN status. The radiomics nomogram, which incorporated the three variables, showed good calibration and discrimination in the training set (area under the receiver operator characteristic curve [AUC] 0.851 [CI 0.791–0.912]) and the validation set (AUC 0.832 [CI 0.749–0.916]). The significantly improved net reclassification improvement and index-integrated discrimination improvement demonstrated that SWE radiomics signature was a very useful marker to predict the LN metastasis in PTC. Decision curve analysis indicated that the SWE radiomics nomogram was clinically useful. Furthermore, the nomogram also showed favorable discriminatory efficacy in the US-reported LN-negative (cN0) subgroup (AUC 0.812 [CI 0.745–0.860]).

Conclusions:

The presented radiomics nomogram, which is based on the SWE radiomics signature, shows a favorable predictive value for LN staging in patients with PTC.

Introduction

Papillary thyroid carcinoma (PTC) is the most common type of primary thyroid malignancy, with rapidly increasing incidence, but a stable high survival rate, worldwide (1,2). Although PTC has an indolent clinical course, metastasis remains an important issue. PTC is a lymphotropic tumor where cervical lymph node (LN) metastasis is present at diagnosis in 20–90% of patients (3). Recent trends in the surgical treatment of PTC have changed to more individualized and conservative therapies, especially for small unifocal (<1 cm) tumors without extrathyroidal extension and LN metastasis (4). The revised American Thyroid Association (ATA) guidelines recommend that lobectomy alone can be used safely in patients with no LN metastasis (4). However, it remains difficult for clinicians to identify LN metastasis in clinical practice. Since cervical LN metastasis is an important risk factor for recurrence (5 –11) and mortality (10,12 –14) in PTC patients, its accurate preoperative identification will influence decisions about the extent of surgery.

B-mode ultrasound (BMUS) is the first-line noninvasive imaging method for preoperative assessment of cervical LN status for PTC, with high reported specificity (85.0–97.4%) but relative low sensitivity (36.7–61.0%) (15 –18). Recently, several studies have explored the clinical utility of shear-wave elastography (SWE) in detecting tumor sonographic features associated with cervical LN metastasis (19 –22). The results demonstrated that the quantitative elasticity index on preoperative SWE imaging could be useful for predicting LN metastasis.

Radiomics is the process of high-throughput mining of quantitative image features from medical imaging, which enables data to be extracted and applied in a clinical-decision support system to improve the diagnostic, predictive, and prognostic accuracy (23). Radiomics-derived data, when combined with other pertinent clinicopathological features, can produce accurate and robust evidence-based decision-making systems (24). Recently, radiomics characteristics of preoperative ultrasound (US) images have been shown to have potential for predicting LN metastasis for PTC (25,26).

Nomogram is an individualized and evidence-based graphic tool for assessing the probability of a clinical event. It has been shown that nomograms incorporating clinical risk factors, such as age, sex, number of positive LNs, and tumor size, can be helpful in predicting LN metastasis for PTC, with area under the receiver operator characteristic (ROC) curve (AUC) ranging from 0.711 to 0.834 (3,27 –29). To the best of our knowledge, there is no report that has determined whether a nomogram involving SWE radiomics features would enable superior prediction of cervical LN metastasis for PTC. Therefore, this study aimed to develop and validate a nomogram that incorporated the SWE radiomics as well as clinicopathological risk factors for individual preoperative prediction of cervical LN metastasis in PTC.

Materials and Methods

Patients

Between January 2017 and September 2019, consecutive patients with thyroid nodules in hospital no. 1 (Tongji Hospital of Huazhong University of Science and Technology, Wuhan, China; training cohort) and hospital no. 2 (Hunan Provincial Tumor Hospital, Changsha, China; validation cohort) were included. This retrospective study (clinical trial ChiCTR1900026179) was approved by the Institutional Review Board of the two hospitals, and informed consent was waived.

The study inclusion criteria were as follows: (i) surgical resection was performed for the target tumor; (ii) the tumor was pathologically proven PTC; (iii) SWE image of the tumor displayed with a BMUS image in split-screen mode was performed within two weeks before surgery; (iv) the SWE images of the target tumor in the longest axis cross section were available; and (v) cervical LN dissection was performed and pathologically examined. The exclusion criteria included the following: (i) the pathological result of the surgical specimens was uncertain; (ii) the patient had undergone preoperative radiofrequency ablation, radiotherapy, or chemotherapy; (iii) the target tumor was unclear on US images due to artifacts; and (iv) concomitant other malignancies.

The patients were divided into two groups (pN0 and pN1) according to the pathologic results of LN status after cervical LN dissection. In total, 147 patients were identified and comprised the training cohort (Wuhan cohort): 42 males and 105 females; mean age, 41.59 ± 9.68 years; range, 22 to 67 years. An independent external validation cohort of 90 consecutive patients (Changsha cohort, 16 males and 74 females; mean age, 43.14 ± 10.76 years; range, 23–66 years) was included using the same criteria.

Clinical information

Baseline clinicopathological data, including age, sex, and nodule pathology, were derived from medical records, and dates of US imaging were also recorded. The cutoff value for age was 45- and 55-year olds separately according to the 7th and 8th American Joint Committee on Cancer (AJCC) staging systems.

US image acquisition and reported LN status

BMUS and SWE images were acquired with a Supersonic Aixplorer system (SuperSonic Imagine, Aix en Provence, France) using a 5–14 MHz linear transducer by 2 radiologists with more than 10 years of experience. For the target tumor, the largest diameter was measured as tumor size on the BMUS image. The region of interest on the SWE image was set to include the whole thyroid lesion and adjacent normal parenchyma, and a Q-Box was placed over the stiffest part of the lesion, as assessed by visual inspection. For mixed cystic and solid nodules, the Q-Box was placed over the solid part of the lesion. More detailed descriptions on SWE are given in Supplementary Appendix SA1. We applied the American College of Radiology Thyroid Imaging, Reporting and Data System, which provides guidance regarding management of thyroid nodules on the basis of their BMUS appearance, for risk stratification (30). The results of SWE and color Doppler acted as an auxiliary tool for further risk evaluation based on BMUS features. Fine-needle aspiration (FNA) biopsy was performed for suspected nodules before surgery, and the patients were handled based on the recommendations of the 2015 ATA management guidelines for adult patients with thyroid nodules and differentiated thyroid cancer according to various cytologic categories (31).

The US-reported LN status was documented by the radiologists, and images containing important features of LNs were stored into the Picture Archiving and Communication Systems. It was retrospectively reviewed and verified by 2 radiologists (A.J.Y and S.Y.Y) with over 10 years of experience. Metastatic LNs were deemed present by US when at least one of the following five criteria was met: (i) focal or diffuse hyperechogenicity, (ii) micro- or macrocalcification, (iii) cystic change, (iv) abnormal vascular pattern (a chaotic or peripheral vascular pattern), or (v) a round shape (long/transverse diameter ratio <1.5) (15,17,32). One or more nodes that met at least one of the five US criteria would be considered positive, regardless of the site of the LN. Preoperative biopsy was performed for high-suspicion nodes in all cases.

Surgical technique

All patients diagnosed with thyroid cancer underwent total thyroidectomy with routine bilateral central neck dissection since 2008 in the two centers, regardless of disease stage or tumor size. Additional lateral LN dissection was performed in patients who had clinically suspicious lateral LN metastasis confirmed by preoperative FNA or an intraoperative frozen biopsy.

Region of interest segmentation and radiomics feature extraction

Region of interest (ROI) was manually delineated on the BMUS image of the largest cross section using an open-source software (ITK-SNAP 3.8.0). All the manual segmentations were conducted by two radiologists and one radiologist (twice) with more than five years of experience in thyroid oncologic imaging who were blinded from the final LN status (for interobserver and intraobserver reproducibility evaluation). Because the boundary of the tumor in SWE image was indefinite, the ROI of BMUS image was directly applied as a template to the corresponding SWE image. Then the SWE image was segmented by an adaptive threshold algorithm automatically followed by a morphologic closing process. Since SWE image contained an SWE color elasticity layer and the corresponding B-model image, methods reported in the previous study were applied to extract the SWE information and build a purified grayscale SWE elasticity image (33). SWE and corresponding BMUS images were then exported as DICOM format. The textural, morphological, intensity, law, and wavelet features were extracted automatically by using an open-source software (Pyradiomics) (34). The interclass correlation coefficient (ICC) was used to assess the interobserver and intraobserver agreement of the feature extraction. An ICC that was greater than 0.80 was considered excellent.

LN status-related feature selection and radiomics signature building

Spearman's correlation coefficient was applied to calculate the relevance and redundancy of the features. Redundant features with a Spearman's correlation coefficient ≥0.8 were eliminated. Then, the minimum redundancy maximum relevance (mRMR) algorithm was used to choose the remaining features. The least absolute shrinkage and selection operator (LASSO) logistic regression method using 10-fold cross-validation was applied after mRMR to select the most useful predictive LN status-related features from the training data set (35). Radiomics score (Rad-score) was generated using a linear combination of the selected features weighted by the LASSO algorithm. The formula for the SWE and BMUS Rad-scores was built using the respective selected features. Then the potential association of the Rad-scores with LN status was assessed in the training and validation cohort using a Mann–Whitney U test.

Development of US radiomics nomogram

Univariate analyses, either Student's independent test (continuous variables) or a χ² test (categorical variables), were used to identify the clinicopathological risk factor associated with cervical LN metastasis. A multivariate logistic regression analysis incorporating the Rad-scores and the independent clinical variables was performed, using the backward step-down selection procedure with a liberal p < 0.05 as the retention criteria to select the final predictors for cervical LN metastasis. Then an US radiomics nomogram was developed based on the multivariate analysis in the training cohort. For comparison, a clinical prediction model was developed using the independent clinical risk factors alone.

Performance of the US radiomics nomogram

Calibration of the US radiomics nomogram was evaluated using the calibration curve and Hosmer–Lemeshow test (a nonsignificant test statistic implies that the model calibrates perfectly) (36). The discrimination performance of the nomogram was evaluated using the AUC. Then the performance of the nomogram was tested in the external validation cohort by calibration curve and AUC.

Clinical utility of the US radiomics nomogram

Decision curve analysis (DCA) was conducted to determine the clinical usefulness of the US radiomics nomogram by quantifying the net benefits at different threshold probabilities in the combined training and validation set (37). The improvement in the predictive accuracy of the US radiomics nomogram was evaluated by the index-integrated discrimination improvement (IDI) and the net reclassification improvement (NRI).

For clinical use, the nomogram-predicted probability (defined as Nomo-score in this study) of each patient was calculated according to the nomogram algorithm. Then the optimal cutoff value was determined by maximizing the Youden index. Performance of the optimal cutoff value of the Nomo-score was assessed by the ROC, as well as sensitivity, specificity, predictive values, and likelihood ratios.

Statistical analyses

The Delong test was used to compare different AUC. Statistical analyses were conducted with R software 3.6.1 and SPSS19.0 software (SPSS, Inc., Chicago, IL). All the statistical significance levels were two sided, with p-value less than 0.05. The packages of R3.6.1 that were used (Supplementary Appendix Table SA1) and detailed descriptions of the LASSO and DCA algorithm are provided in the Supplement Data (Supplementary Appendix SA2).

Results

Clinical characteristics

The study flowchart is shown in Figure 1. Patients' clinical and pathological characteristics in the training and validation cohorts are summarized in Tables 1 and 2. Except for nodular goiter and US-reported LN status, there were no differences in the other clinicopathological characteristics between the two cohorts. LN metastasis patients (pN1) accounted for 51.0% (75/147) and 42.2% (38/90) of the training and validation cohorts, respectively, and there were no significant differences between them (p = 0.188). In total, 18.5% (23/124) of the patients without LN metastasis (pN0) were overstaged and 51.3% (58/113) of the patients with LN metastasis (pN1) were understaged according to US-reported LN status in our study.

FIG. 1.

US radiomics workflow and study flowchart. BMUS, B-model ultrasound; LASSO, least absolute shrinkage and selection operator; mRMR, minimum redundancy maximum relevance; ROI, region of interest; SWE, shear-wave elastography; US, ultrasound.

Table 1.

Clinical Characteristics of Patients in the Training and Validation Cohorts

Characteristic	Training cohort (n = 147)	Validation cohort (n = 90)	p
Age, mean ± SD, years	41.59 ± 9.68	43.14 ± 10.76	0.250
Age (years)			0.140
>55	11 (7.5)	12 (13.3)
≤55	136 (92.5)	78 (86.7)
Age (years)			0.376
>45	52 (35.4)	37 (41.1)
≤45	95 (64.6)	53 (58.9)
LN metastasis			0.188
Positive	75 (51.0)	38 (42.2)
Negative	72 (49.0)	52 (57.8)
Sex			0.061
Male	42 (28.6)	16 (17.8)
Female	105 (71.4)	74 (82.2)
Primary site			0.727
Right lobe	69 (46.9)	47 (52.2)
Left lobe	74 (50.3)	41 (45.6)
Isthmus	4 (2.7)	2 (2.2)
Tumor location			0.872
Subcapsular thyroid	62 (42.2)	37 (41.1)
Intrathyroidal	85 (57.8)	53 (58.9)
Tumor size, mean ± SD, mm	12.46 ± 7.40	11.64 ± 5.77	0.347
Multifocality^a			0.779
Positive	32 (21.8)	21 (23.3)
Negative	115 (78.2)	69 (76.7)
US-reported LN status			<0.001
Positive	32 (21.8)	46 (51.1)
Negative	115 (78.2)	44 (48.9)
TI-RADS level			0.059
TR 4	79 (54)	37 (41)
TR 5	68 (46)	53 (59)
Composition			0.857
Solid	138 (94)	85 (94)
Mixed cystic and solid	9 (6)	5 (6)
Hashimoto thyroiditis			0.127
Positive	12 (8.2)	13 (14.4)
Negative^b	135 (91.8)	77 (85.6)
Nodular goiter			0.014
Positive	31 (21.1)	1 (2.2)
Negative	116 (78.9)	44 (97.8)
BMUS Rad-score, median (interquartile range)	−0.38 (−0.49 to −0.30)	−0.37 (−0.45 to −0.24)	0.206
SWE Rad-score, median (interquartile range)	0.07 (−0.68 to 0.54)	−0.15 (−0.84 to 0.37)	0.249

Data are number of patients and percentage if not specified.

Multifocality refers to more than one tumor site in the thyroid, which was evaluated by preoperative biopsy and confirmed by pathological examination of resection specimens.

Determined based on pathology.

BMUS, B-model ultrasound; LN, lymph node; Rad-score, radiomics score; SD, standard deviation; SWE, shear-wave elastography; TI-RADS, Thyroid Imaging, Reporting and Data System; US, ultrasound.

Table 2.

Pathological Characteristics of Study Cohort

	Training cohort (n = 147)	Validation cohort (n = 90)
Pathologic T stage, no. (%)
pT1	114 (78%)	80 (89%)
pT2	22 (15%)	6 (7%)
pT3	2 (1%)	3 (3%)
pT4	9 (6%)	1 (1%)
Pathologic N stage, no. (%)
pN0	72 (49%)	52 (58%)
pN1a	48 (33%)	24 (27%)
pN1b	27 (18%)	14 (15%)
Number of LNs removed
Central
Median (interquartile range)	9 (5–13)	7 (5–8)
Lateral
Median (interquartile range)^a	17 (6–23)	17 (3–21)
Number of positive LNs
Central
Median (interquartile range)^a	5 (3–8)	4 (3–7)
Lateral
Median (interquartile range)	4 (2–7)	3 (2–6)

Bilateral.

Establishment of US radiomics signature

One image per nodule was used for analysis. For patients with more than one nodule, only the image of the largest one was applied. Three hundred ten imaging features were extracted from each BMUS image. The BMUS features were reduced to two LN status-related features after mRMR and LASSO algorithm in the training cohort (Supplementary Appendix Fig. SA1A, B). Likewise, the 209 SWE features were reduced to four risk predictors by mRMR algorithm and LASSO regression in the training cohort (Supplementary Appendix Fig. SA1C, D). Favorable interobserver and intraobserver reproducibility of feature extraction was achieved, with intraobserver ICCs ranging from 0.823 to 0.901 and the interobserver ICCs ranging from 0.716 to 0.930. The BMUS and SWE Rad-score calculation formulas are presented in Supplementary Appendix SA3. The BMUS and SWE Rad-scores were much higher in the pN1 group in both the training and validation sets than that in the pN0 group (Table 3).

Table 3.

Demographic and Clinicopathological Characteristics of Patients by Lymph Node Status

Characteristic	Training cohort no. (%)			Validation cohort no. (%)
Characteristic	LN metastasis (+)	LN metastasis (−)	p	LN metastasis (+)	LN metastasis (−)	p
Age (mean ± SD, years)	41.32 ± 10.18	41.86 ± 9.19	0.736	39.50 ± 10.33	45.81 ± 10.38	0.005
Age (years)			0.384			0.194
>55	7 (9.3)	4 (5.6)		3 (7.9)	9 (17.3)
≤55	68 (90.7)	68 (94.4)		35 (92.1)	43 (82.7)
Age (years)			0.383			0.004
>45	24 (32.0)	28 (38.9)		9 (18.2)	28 (58.8)
≤45	51 (68.0)	44 (61.1)		29 (81.8)	24 (41.2)
Sex			0.095			0.018
Male	26 (34.7)	16 (22.2)		11 (28.9)	5 (9.6)
Female	49 (65.3	56 (77.8)		27 (71.1)	47 (90.4)
Primary site			0.138			0.729
Right lobe	34 (45.3)	35 (48.6)		18 (47.4)	29 (55.8)
Left lobe	37 (49.3)	37 (51.4)		19 (50.0)	22 (42.3)
Isthmus	4 (5.3)	0 (0.0)		1 (2.6)	1 (1.9)
Tumor location			0.073			0.143
Subcapsular	37 (49.3)	25 (34.7)		19 (50.0)	18 (34.6)
Intrathyroidal	38 (50.7)	47 (65.3)		19 (50.0)	34 (65.4)
Tumor size, mean ± SD, mm	15.15 ± 7.95	9.65 ± 5.59	<0.001	13.82 ± 6.91	10.06 ± 4.16	0.004
Multifocality			0.008			0.114
Positive	23 (30.7)	9 (12.5)		12 (31.6)	9 (17.3)
Negative	52 (69.3)	63 (87.5)		26 (68.4)	43 (82.7)
US-reported LN status			<0.001			0.019
Positive	28 (37.3)	4 (5.6)		9 (81.8)	14 (41.2)
Negative	47 (62.7)	68 (94.4)		2 (18.2)	20 (58.8)
TI-RADS level			0.117			0.255
TR 4	38 (48.1)	24 (35.3)		13	24
TR 5	41 (51.9)	44 (64.7)		25	28
Composition			0.097			0.078
Solid	68 (90.7)	70 (97.2)		34 (89.5)	51 (98.1)
Mixed cystic and solid	7 (9.3)	2 (2.8)		4 (10.5)	1 (1.9)
Hashimoto thyroiditis			0.597			0.366
Positive	7 (9.3)	5 (6.9)		4 (10.5)	9 (17.3)
Negative	68 (90.7)	67 (93.1)		34 (89.5)	43 (82.7)
Nodular goiter			0.632			0.019
Positive	17 (22.7)	14 (19.4)		7 (18.4)	1 (1.9)
Negative	58 (77.3)	58 (80.6)		31 (81.6)	51 (98.1)
BMUS Rad-score, median (interquartile range)	0.16 (0.02 to 0.30)	−0.04 (−0.21 to 0.08)	<0.001	−0.25 (−0.38 to 0.13)	−0.41 (−0.47 to −0.29)	<0.001
SWE Rad-score, median (interquartile range)	0.49 (0 to 0.97)	−0.50 (−1.35 to 0.19)	<0.001	0.26 (−0.29 to 0.80)	−0.51 (−0.38 to 0.08)	<0.001

Development and performance of the radiomics nomogram

The SWE radiomics signature, multifocality, and US-reported LN status were identified as independent predictors of LN metastasis in PTC patients by a multivariate logistic regression model (Table 4). An SWE radiomics nomogram incorporating these three predictors was constructed (Fig. 2A). Figure 2B shows the calibration curve of the nomogram. The calibration curve and Hosmer–Lemeshow test statistic (p = 0.226) showed good calibration in the training cohort. An AUC of 0.851 [CI 0.791–0.912] also showed good discrimination by the nomogram (Table 4). The favorable calibration of the SWE radiomics nomogram was also confirmed in the validation set (Fig. 2C). The Hosmer–Lemeshow test yielded a p-value of 0.225, and the AUC of the validation set was 0.832 [CI 0.749–0.916]. Thus, our nomogram performed well in both the training and external validation sets.

FIG. 2.

SWE radiomics nomogram for the preoperative estimation of cervical LN metastasis. (A) Nomogram to estimate the risk of LN metastasis preoperatively in PTC. To use the nomogram, find the position of each variable on the corresponding axis, draw a vertical line to the points axis for the number of points, add the points from all of the variables, and draw a line from the total points' axis to determine the LN metastasis probabilities at the lower line of the nomogram. (B, C) Calibration curves of the SWE radiomics nomogram in the training (B) and validation (C) set. Calibration curves depict the calibration of SWE radiomics model in terms of the agreement between the predicted probabilities of LN metastasis and observed outcomes of LN metastasis. The dotted blue line represents an ideal prediction, and the dotted red line represents the predictive ability of the nomogram. The closer the dotted red line fit is to the dotted blue line, the better the predictive accuracy of the nomogram is. LN, lymph node; PTC, papillary thyroid carcinoma.

Table 4.

Risk Factors for Cervical Lymph Node Metastasis Based on Preoperative Data in the Training Cohort

Intercept and variable	Clinical model			SWE radiomics nomogram
Intercept and variable	β	Odds ratio [CI]	p	β	Odds ratio [CI]	p
Intercept	−1.943			−0.550
Tumor size	0.113	1.119 [1.051–1.192]	<0.001	NA	NA	NA
Multifocality	1.330	3.780 [1.446–9.883]	0.007	1.446	4.245 [1.386–13.003]	0.011
US-reported LN status	2.058	7.831 [2.437–25.167]	0.001	1.469	4.344 [1.235–15.286]	0.022
BMUS Rad-score	NA	NA	NA	NA	NA	NA
SWE Rad-score	NA	NA	NA	1.372	3.943 [2.258–6.884]	<0.001

AUC						p^a
Training cohort	0.800 [0.729–0.871]			0.851 [0.791–0.912]		0.034
Validation cohort	0.783 [0.689–0.876]			0.832 [0.749–0.916]		0.048

p value represents difference of AUC between SWE radiomics nomogram and clinical model.

AUC, area under the receiver operator characteristic curve; NA, not available; PTC, papillary thyroid carcinoma.

Comparison of the SWE radiomics nomogram with the clinical model

The predictive performance of the SWE radiomics nomogram was superior to that of the clinical model in both the training set (AUC 0.851 vs. 0.800, p = 0.034) and validation set (AUC 0.832 vs. 0.783, p = 0.048) (Table 4). The DCA for the SWE radiomics nomogram is presented in Supplementary Appendix Figure SA2. The DCA demonstrated that the SWE radiomics nomogram had a higher overall net benefit than the clinical model and US-reported LN status alone when the threshold probability for a clinician or a patient ranges from 0 to 0.83, and the nomogram was more beneficial than either the treat-all or the treat-none strategy.

The utilization of the SWE Rad-score significantly improved the predictive value for cervical LN metastasis in terms of NRI and IDI compared with the prediction model incorporating only the independent clinical risk factors (Table 5).

Table 5.

Evaluation of the Shear-Wave Elastography Radiomics Signature with Respect to NRI and IDI

Characteristic	Training cohort			Validation cohort
Characteristic	Categorical NRI [CI]	Continuous NRI [CI]	IDI [CI]	Categorical NRI [CI]	Continuous NRI [CI]	IDI [CI]
SWE radiomics nomogram vs. clinical model	0.547 [0.332–0.762]	0.607 [0.301–0.912]	0.103 [0.054–0.151]	0.443 [0.147–0.39]	0.666 [0.279–1.053]	0.145 [0.068–0.222]
p	<0.0001	<0.0001	<0.0001	0.0034	0.0007	0.0002

IDI, index-integrated discrimination improvement; NRI, net reclassification improvement.

Predicting LN metastasis based on the Nomo-score

The optimal cutoff value of the Nomo-score was determined to be 0.574. The discrimination and AUC for differentiating the presence of LN metastasis were 0.851 (0.791–0.912) in the training cohort (Fig. 3A, B) and 0.832 (0.749–0.916) in the validation cohort, respectively (Fig. 3C, D). The performance of the optimal cutoff value of the Nomo-score is summarized in Table 6.

FIG. 3.

Diagnostic accuracy of the SWE radiomics nomogram for the estimation of LN metastasis in patients with PTC in the training and validation cohorts. In (A, C), the use of a cutoff value of 0.574 of the Nomo-score performed well for the differential diagnosis of pN1 from pN0 in the training (A) and validation (C) cohorts. In (B, D), ROCs show good diagnostic performance of the nomogram in the training and validation cohorts. ROC, receiver operator characteristic.

Table 6.

Performance of the Prediction Nomo-Score for Estimating the Risk of Lymph Node Metastasis

Variable	Value [CI]
Variable	Training cohort (147)	Validation cohort (90)	Combined cohort (237)
Cutoff value	0.574	0.574	0.574
AUC	0.851 [0.791–0.912]	0.832 [0.749–0.916]	0.839 [0.789–0.889]
Sensitivity, %	80.67 [69.56–89.76]	86.84 [72.67–94.25]	84.34 [75.58–91.50]
Specificity, %	87.50 [77.92–93.28]	73.08 [59.75–83.23]	82.26 [74.60–87.98]
Positive predictive value, %	85.48 [74.66–92.17]	70.21 [56.02–81.35]	79.25 [70.57–85.88]
Negative predictive value, %	74.12 [63.91–82.24]	88.37 [75.52–94.93]	77.86 [70.02–84.12]
Positive likelihood ratio	5.65 [4.48–7.14]	3.23 [2.78–3.74]	4.19 [3.80–4.62]
Negative likelihood ratio	0.34 [0.31–0.37]	0.18 [0.12–0.27]	0.31 [0.29–0.34]
Diagnostic accuracy, %	78.91 [71.62–84.73]	78.89 [69.37–86.05]	78.48 [72.82–83.24]

We further assessed the discriminatory ability of the SWE radiomics nomogram in all the 237 patients and in the US-reported LN-negative (cN0) subgroup (n = 159). The patients were divided into low- and high-risk subsets based on the optimal cutoff value of the Nomo-score (0.574). Notably, the high-risk group had a greater proportion of cervical LN metastasis in all patients (Fig. 4A) and in the cN0 subcohort (Fig. 4C). Figure 4B shows ROC analyses comparing the discriminatory performance of the nomogram to those of the clinical model and the US-reported LN status alone for all 237 patients. The SWE radiomics nomogram yielded the greatest AUC of 0.839 [CI 0.789–0.889]. The nomogram also showed favorable discriminatory efficacy in the cN0 subset (AUC 0.812 [CI 0.745–0.860]; Fig. 4D).

FIG. 4.

Performance of the nomogram in all 237 patients and in the cN0 subgroup (n = 159). The left panels show the risk-classification performance of the nomogram. The right panels present the ROC analyses for the nomogram. (A, B) All 237 patients. (C, D) cN0 subgroup.

Discussion

LN metastasis is one of the most important clinical features associated with local recurrence and distant metastasis (38). The overall survival and disease-free survival are negatively impacted by cervical LN metastasis (39). Previous reports revealed that cervical LN metastasis is associated with a higher rate of distant metastasis, 11.2-fold (40), and disease-related mortality, 3-fold (39). Therefore, accurate preoperative prediction of LN status in patients with PTC is important for clinical decision-making.

In the 2015 ATA guidelines (31), either total thyroidectomy or lobectomy is considered to be a reasonable surgical strategy for PTC measuring 1–4 cm without extrathyroidal extension or clinically apparent LN metastasis on preoperative examination or imaging. However, total thyroidectomy and prophylactic cervical LN dissection have potential for more harm, which may increase the risk of hypoparathyroidism and nerve injury. For properly selected patients, the recurrence rates may be as low as 4% if treated with lobectomy alone. The accurate preoperative identification of cervical LN metastasis could improve the surgical outcomes and quality of life in patients with low-risk PTC. Nevertheless, it is difficult to identify patients who will benefit from lobectomy without prophylactic cervical LN dissection.

Preoperative neck US for cervical LNs plays an important role in deciding the extent of surgical resection, especially for determining the need for LN dissection. Unfortunately, the detection rate of cervical LN metastasis by US is unsatisfactory, especially for central neck LN metastasis (41). In our data, 113 tumors had LN metastasis, however, only 48.7% (55/113) of them were correctly reported as LN positive by US. Many previous studies have explored the association between cervical LN metastasis and tumor ultrasonic features. Guo et al. found that tumor size, echogenicity, and calcification were significantly associated with LN status (42). In addition, “wider-than-tall” shape and extrathyroidal extension were also indicators of LN metastasis (43). Although the US characteristics mentioned above are encouraging, the diagnostic accuracy is significantly affected by the expertise of the operator.

Radiomics has recently attracted the interest of many researchers for predicting the LN metastasis with quantitative medical image features. A nomogram incorporating CT radiomics, carcinoembryonic antigen status (CEA), and CT-reported LN status reached a C-index of 0.736 for the preoperative prediction of LN metastasis in patients with colorectal cancer (24). Wu et al. demonstrated that nomogram based on the CT radiomics signature showed a favorable predictive value for LN metastasis in bladder cancer (44). Recently, a nomogram that incorporated radiomics signature from contrast-enhanced CT images showed promise for the preoperative prediction of cervical LN metastasis in patients with PTC with an AUC of 0.867. However, using iodinated contrast agents for contrast-enhanced CT may delay radioactive iodine therapy in patients with PTC (45). Moreover, contrast-enhanced CT is costly and time-consuming, and has a risk of contrast allergy, which is inconvenient for routine screening.

Liu et al. proposed a radiomics method based on preoperative BMUS images for staging LN status, which demonstrated the feasibility of applying a US radiomics analysis in patients with PTC (25). Thus, we attempted to develop a US-based radiomics nomogram for the preoperative prediction of LN metastasis in patients with PTC. Multimodal US technology, containing different aspects of anatomical and biological information about a tumor, has been applied conjointly for the diagnosis of thyroid tumors in clinical practice. Here, we investigated the value of SWE combined with BMUS for the prediction of LN metastasis using radiomics analysis. Interestingly, for univariate analysis, the BMUS and SWE radiomics signatures were both significantly associated with LN status. However, the BMUS Rad-score was not included in the final nomogram. We found that the strong discriminatory power of the SWE Rad-score diminished the value of BMUS radiomics signature in the final multivariate logistic regression analysis.

SWE is an elastographic technique used to evaluate the tissue hardness quantitatively. Several studies have shown that a higher elasticity index on SWE can be a sign of cervical LN metastasis for PTC (19,21). Radiomics analysis of SWE has been shown useful to distinguish benign and malignant thyroid nodules (33). However, to the best of our knowledge, no studies have investigated the association between the SWE radiomics features and LN status in PTC. In our study, the SWE radiomics signature showed favorable discrimination for LN status, yielding an AUC of 0.829 across all 237 patients. Our results demonstrated that radiomics analysis using SWE to predict cervical LN metastasis in patients with PTC can attain a satisfactory discriminatory efficacy. However, the relationship between thyroid cancer stiffness and the likelihood of LN metastasis remains to be elucidated. Many investigators have demonstrated that extracellular matrix crosslinking is an important ingredient of cancer cell biology and related to tissue stiffening in tissue fibrosis (46,47). Tissue fibrosis influences tumor progression by regulating soluble factors that trigger inflammation and angiogenesis and induce cell growth and invasion (48). Therefore, it was speculated that tumor progression with cell proliferation and fibrosis could affect both tumor stiffness and aggressiveness, including the development of regional LN metastasis (19).

We considered clinicopathological risk factors. A multivariate logistic regression analysis indicated that multifocality and US-reported LN status were significant predictive variables distinct from the SWE radiomics signature. To provide an easy-to-use tool for clinical use, we developed a radiomics nomogram based on the multivariate logistic regression analysis. Our nomogram exhibited good discrimination and calibration in the training and validation sets. The AUC of the nomogram was 0.851 in the training set and achieved greater predictive efficacy than the prediction model involving the clinical risk factors alone. The addition of SWE radiomics features to the clinical model significantly improved the NRI and IDI, and implied that the SWE radiomics signature could be a very useful new marker for staging LN status in PTC. DCA demonstrated that the SWE radiomics nomogram can improve patient LN staging preoperatively.

When categorized into low- and high-risk subcohorts according to the optimal cutoff value of the Nomo-score, the high-risk set has a significantly greater probability of cervical LN metastasis. Notably, our nomogram showed good discriminatory ability in the cN0 patients. PTC diagnosed as cN0 are considered to be at low risk of cervical LN metastasis. However, some cN0 patients may be understaged and actually harbor LN metastasis. Encouragingly, our nomogram showed good discrimination among these patients. Therefore, our SWE radiomics nomogram may serve as a reliable predictive tool for cervical LN metastases in patients with PTC.

For clinical use of the nomogram, we summarized the sensitivity, specificity, positive predictive value, as well as negative predictive value in assessing the risk of LN metastasis. We show that patients with a Nomo-score of 0.574 or more are the subset of high-risk PTC likely to have cervical LN metastasis (positive predictive value, 79.25%). Thus, this subgroup of PTC patients may benefit from prophylactic cervical LN dissection. Since the current nomogram was constructed regardless of the location of the metastatic LNs, it is appropriate to recommend that a high-risk score should prompt a central LN dissection, leaving the decision to perform a lateral LN dissection to be based on intraoperative findings.

Some limitations of our study should be acknowledged. First, some bias may inevitably exist and affect our analysis because it was a retrospective study. Prospective multicenter validation using a larger group of patients is needed to acquire high-level evidence for further clinical application. Second, since we performed routine bilateral central neck dissection for all patients, but additional lateral LN dissection was conducted based on preoperative FNA or an intraoperative frozen biopsy, the different strategy for LN dissection might be another source of bias. Third, although our nomogram showed favorable discriminatory ability in regard to negative predictive value and positive predictive value in the training and validation cohorts, physicians might still encounter a dilemma in the case that nonaggressive-appearing primary tumors (e.g., no gross extrathyroidal extension) with cN0 PTCs were categorized as high risk of LN metastasis based on the nomogram and vice versa. In a situation such as this, intraoperative evaluation of minimal or gross extrathyroidal extension might provide additional information for decision-making. Fourth, due to the lack of data on the size of the pathologically proven LNs, we could not determine if the nomogram was capable of detecting small as well as large nodal metastases, which necessitates a prospective cohort study in the future. In addition, genomic characteristics were not incorporated into our nomogram. In recent years, for preoperative nodal staging, the BRAF^V600E mutation in patients with PTC has been associated with a higher rate of LN metastasis (49,50). Radiogenomics consisting of a radiomics signature and gene marker may improve the ability for preoperative LN staging in patients with PTC.

In conclusion, this study presents a noninvasive predictive tool that incorporates both the SWE radiomics signature and clinical risk factors, and shows favorable predictive accuracy in predicting preoperative cervical LN metastasis in patients with PTC.

Footnotes

Author Disclosure Statement

No competing financial interests exist.

Funding Information

This research was supported by a grant from Wuhan Tongji Hospital (no. 2017A002), a grant from Wuhan Science and Technology Bureau (no. 2017060201010181), Key project supported by the Xinjiang Construction Corps (2019DB012), and Project supported by the Health Commission of Hubei province (WJ2019H227).

Supplementary Material

Supplementary Data

Supplementary Appendix SA1

Supplementary Figure S1

Supplementary Appendix SA2

Supplementary Appendix SA3

Supplementary Appendix Table SA1

Supplementary Appendix Figure SA1

Supplementary Appendix Figure SA2

References

Miller

, Nogueira

, Mariotto

, Rowland

, Yabroff

, Alfano

, Jemal

, Kramer

, Siegel

. 2019. Cancer treatment and survivorship statistics, 2019. CA Cancer J Clin, 69:363–385.

Siegel

, Ma

, Zou

, Jemal

. 2014. Cancer statistics, 2014. CA Cancer J Clin, 64:9–29.

Kim

, Chai

, Park

, Woo

, Lee

, Choe

, Kim

. 2017. Nomogram for predicting central node metastasis in papillary thyroid carcinoma. J Surg Oncol, 115:266–272.

Cooper

, Doherty

, Haugen

, Kloos

, Lee

, Mandel

, Mazzaferri

, McIver

, Pacini

, Schlumberger

, Sherman

, Steward

, Tuttle

. 2009. Revised American Thyroid Association management guidelines for patients with thyroid nodules and differentiated thyroid cancer. Thyroid, 19:1167–1214.

Suh

, Kwon

, Kim

, Choi

, Lee

, Park

, Youn

. 2015. Factors affecting the locoregional recurrence of conventional papillary thyroid carcinoma after surgery: a retrospective analysis of 3381 patients. Ann Surg Oncol, 22:3543–3549.

Kim

, Hong

, Kim

, Gong

, Ryu

, Kim

, Yun

, Shong

. 2008. Prognostic parameters for recurrence of papillary thyroid microcarcinoma. BMC Cancer, 8:296.

Baek

, Jung

, Kang

, Kwon

, Woo

, Cho

, Chung

. 2010. Clinical risk factors associated with cervical lymph node recurrence in papillary thyroid carcinoma. Thyroid, 20:147–152.

Ito

, Kudo

, Kobayashi

, Miya

, Ichihara

, Miyauchi

. 2012. Prognostic factors for recurrence of papillary thyroid carcinoma in the lymph nodes, lung, and bone: analysis of 5,768 patients with average 10-year follow-up. World J Surg, 36:1274–1278.

Kruijff

, Petersen

, Chen

, Aniss

, Clifton-Bligh

, Sidhu

, Delbridge

, Gill

, Learoyd

, Sywak

. 2014. Patterns of structural recurrence in papillary thyroid cancer. World J Surg, 38:653–659.

10.

Grogan

, Kaplan

, Cao

, Weiss

, Degroot

, Simon

, Embia

, Angelos

, Kaplan

, Schechter

. 2013. A study of recurrence and death from papillary thyroid cancer with 27 years of median follow-up. Surgery, 154:1436–1447.

11.

Schneider

, Mazeh

, Chen

, Sippel

. 2013. Lymph node ratio predicts recurrence in papillary thyroid cancer. Oncologist, 18:157–162.

12.

Podnos

, Smith

, Wagman

, Ellenhorn

. 2005. The implication of lymph node metastasis on survival in patients with well-differentiated thyroid cancer. Am Surg, 71:731–734.

13.

Zaydfudim

, Feurer

, Griffin

, Phay

. 2008. The impact of lymph node involvement on survival in patients with papillary and follicular thyroid carcinoma. Surgery, 144:1070–1078.

14.

Adam

, Pura

, Goffredo

, Dinan

, Reed

, Scheri

, Hyslop

, Roman

, Sosa

. 2015. Presence and number of lymph node metastases are associated with compromised survival for patients younger than age 45 years with papillary thyroid cancer. J Clin Oncol, 33:2370–2375.

15.

Kim

, Park

, Son

, Kim

, Jeon

, Na

. 2008. Preoperative diagnosis of cervical metastatic lymph nodes in papillary thyroid carcinoma: comparison of ultrasound, computed tomography, and combined ultrasound with computed tomography. Thyroid, 18:411–418.

16.

Jeong

, Baek

, Son

, Choi

, Kim

, Ko

, Chung

, Baek

. 2006. Integrated 18F-FDG PET/CT for the initial evaluation of cervical node level of patients with papillary thyroid carcinoma: comparison with ultrasound and contrast-enhanced CT. Clin Endocrinol (Oxf), 65:402–407.

17.

Choi

, Kim

, Kwak

, Kim

, Chang

, Kim

. 2009. Preoperative staging of papillary thyroid carcinoma: comparison of ultrasound imaging and CT. AJR Am J Roentgenol, 193:871–878.

18.

Roh

, Park

, Kim

, Song

. 2009. Use of preoperative ultrasonography as guidance for neck dissection in patients with papillary thyroid carcinoma. J Surg Oncol, 99:28–31.

19.

Park

, Kim

, Son

, Youk

. 2016. Shear-wave elastography for papillary thyroid carcinoma can improve prediction of cervical lymph node metastasis. Ann Surg Oncol, 23:722–729.

20.

Park

, Kim

, Son

, Youk

, Park

. 2013. Quantitative shear wave elastography as a prognostic implication of papillary thyroid carcinoma (PTC): elasticity index can predict extrathyroidal extension (ETE). Ann Surg Oncol, 20:2765–2771.

21.

Jung

, Kim

, Son

, Youk

, Park

. 2015. Shear wave elastography in evaluation of cervical lymph node metastasis of papillary thyroid carcinoma: elasticity index as a prognostic implication. Ann Surg Oncol, 22:111–116.

22.

Cosgrove

, Barr

, Bojunga

, Cantisani

, Chammas

, Dighe

, Vinayak

, Xu

, Dietrich

. 2017. WFUMB guidelines and recommendations on the clinical use of ultrasound elastography: part 4. Thyroid. Ultrasound Med Biol, 43:4–26.

23.

Lambin

, Leijenaar

, Deist

, Peerlings

, de Jong

, van Timmeren

, Sanduleanu

, Larue

, Even

, Jochems

, van Wijk

, Woodruff

, van Soest

, Lustberg

, Roelofs

, van Elmpt

, Dekker

, Mottaghy

, Wildberger

, Walsh

. 2017. Radiomics: the bridge between medical imaging and personalized medicine. Nat Rev Clin Oncol, 14:749–762.

24.

Huang

, Liang

, He

, Tian

, Liang

, Chen

, Ma

, Liu

. 2016. Development and validation of a radiomics nomogram for preoperative prediction of lymph node metastasis in colorectal cancer. J Clin Oncol, 34:2157–2164.

25.

Liu

, Zhou

, Yu

, Guo

, Wang

, Zhou

, Chang

. 2019. Prediction of lymph node metastasis in patients with papillary thyroid carcinoma: a radiomics method based on preoperative ultrasound images. Technol Cancer Res Treat, 18:1–13.

26.

Liu

, Ge

, Yu

, Guo

, Wang

, Cui

. 2018. Comparison of the application of B-mode and strain elastography ultrasound in the estimation of lymph node metastasis of papillary thyroid carcinoma based on a radiomics approach. Int J Comput Assist Radiol Surg, 13:1617–1627.

27.

Wang

, Guan

, Xiang

. 2018. Nomogram for predicting central lymph node metastasis in papillary thyroid microcarcinoma: a retrospective cohort study of 8668 patients. Int J Surg, 55:98–102.

28.

Hei

, Song

, Qin

. 2016. A nomogram predicting contralateral central neck lymph node metastasis for papillary thyroid carcinoma. J Surg Oncol, 114:703–707.

29.

Thompson

, Turner

, Hayen

, Aniss

, Jalaty

, Learoyd

, Sidhu

, Delbridge

, Yeh

, Clifton-Bligh

, Sywak

. 2014. A preoperative nomogram for the prediction of ipsilateral central compartment lymph node metastases in papillary thyroid cancer. Thyroid, 24:675–682.

30.

Tessler

, Middleton

, Grant

, Hoang

, Berland

, Teefey

, Cronan

, Beland

, Desser

, Frates

, Hammers

, Hamper

, Langer

, Reading

, Scoutt

, Stavros

. 2017. ACR thyroid imaging, reporting and data system (TI-RADS): white paper of the ACR TI-RADS committee. J Am Coll Radiol, 14:587–595.

31.

Haugen

, Alexander

, Bible

, Doherty

, Mandel

, Nikiforov

, Pacini

, Randolph

, Sawka

, Schlumberger

, Schuff

, Sherman

, Sosa

, Steward

, Tuttle

, Wartofsky

. 2016. 2015 American Thyroid Association Management Guidelines for adult patients with thyroid nodules and differentiated thyroid cancer: the American Thyroid Association Guidelines task force on thyroid nodules and differentiated thyroid cancer. Thyroid, 26:1–133.

32.

, Zhong

, Dong

, Fang

, Dai

, Leng

, Zhang

, Sun

, Tian

, Zheng

, Jin

. 2019. Radiomic analysis for preoperative prediction of cervical lymph node metastasis in patients with papillary thyroid carcinoma. Eur J Radiol, 118:231–238.

33.

Bhatia

KSS

, Lam

ACL

, Pang

SWA

, Wang

, Ahuja

. 2016. Feasibility study of texture analysis using ultrasound shear wave elastography to predict malignancy in thyroid nodules. Ultrasound Med Biol, 42:1671–1680.

34.

van Griethuysen

, Fedorov

, Parmar

, Hosny

, Aucoin

, Narayan

, Beets-Tan

, Fillion-Robin

, Pieper

, Aerts

. 2017. Computational radiomics system to decode the radiographic phenotype. Cancer Res, 77:e104–e107.

35.

Sauerbrei

, Royston

, Binder

. 2007. Selection of important variables and determination of functional form for continuous predictors in multivariable model building. Stat Med, 26:5512–5528.

36.

Kramer

, Zimmerman

. 2007. Assessing the calibration of mortality benchmarks in critical care: the Hosmer-Lemeshow test revisited. Crit Care Med, 35:2052–2056.

37.

Fitzgerald

, Saville

, Lewis

. 2015. Decision curve analysis. JAMA, 313:409–410.

38.

Zhao

, Ming

, Liu

, Shi

, Xu

, Nie

, Huang

. 2013. Multifocality and total tumor diameter predict central neck lymph node metastases in papillary thyroid microcarcinoma. Ann Surg Oncol, 20:746–752.

39.

Lundgren

, Hall

, Dickman

, Zedenius

. 2006. Clinically significant prognostic factors for differentiated thyroid carcinoma. Cancer, 106:524–531.

40.

Chow

, Law

SCK

, Chan

JKC

, Au

, Yau

, Lau

. 2003. Papillary microcarcinoma of the thyroid?Prognostic significance of lymph node metastasis and multifocality. Cancer, 98:31–40.

41.

, Yao

, Zhou

, Chen

, Zhan

. 2019. Clinical characteristics and ultrasonographic features for predicting central lymph node metastasis in clinically node-negative papillary thyroid carcinoma without capsule invasion. Head Neck, 41:3984–3991.

42.

Guo

, Ma

, Yao

, Wu

, Deng

, Zhu

, Luo

, Tang

. 2019. Role of ultrasonographic features and quantified BRAFV600E mutation in lymph node metastasis in Chinese patients with papillary thyroid carcinoma. Sci Rep, 9:1–9.

43.

Wang

, Cheng

, Wen

, Li

, Jing

, Nie

. 2014. Shorter distance between the nodule and capsule has greater risk of cervical lymph node metastasis in papillary thyroid carcinoma. Asian Pac J Cancer Prev, 15:855–860.

44.

, Zheng

, Li

, Yu

, Shi

, Xie

, Liu

, Su

, Huang

, Lin

. 2017. A radiomics nomogram for the preoperative prediction of lymph node metastasis in bladder cancer. Clin Cancer Res, 23:6904–6911.

45.

Amdur

, Mazzaferri

. 2005. Intravenous iodinated contrast effects iodine 511 uptake for months. In Amdur

, Mazzaferri

(eds) Essentials of Thyroid Cancer Management, 1st edition. Springer, New York, pp 211–213.

46.

, Weaver

, Werb

. 2012. The extracellular matrix: a dynamic niche in cancer progression. J Cell Biol, 196:395–406.

47.

Levental

, Yu

, Kass

, Lakins

, Egeblad

, Erler

, Fong

, Csiszar

, Giaccia

, Weninger

, Yamauchi

, Gasser

, Weaver

. 2009. Matrix crosslinking forces tumor progression by enhancing integrin signaling. Cell, 139:891–906.

48.

Bierie

, Chung

, Parker

, Stover

, Cheng

, Chytil

, Aakre

, Shyr

, Moses

. 2009. Abrogation of TGF-beta signaling enhances chemokine production and correlates with prognosis in human breast cancer. J Clin Invest, 119:1571–1582.

49.

Xing

. 2007. BRAF mutation in papillary thyroid cancer: pathogenic role, molecular bases, and clinical implications. Endocr Rev, 28:742–762.

50.

Chen

, Li

, Zhao

, Wang

, Li

, Wei

, Ji

, Xu

. 2018. Conventional ultrasound, immunohistochemical factors and BRAFV600E mutation in predicting central cervical lymph node metastasis of papillary thyroid carcinoma. Ultrasound Med Biol, 44:2296–2306.