Machine Learning–Assisted System for Thyroid Nodule Diagnosis

Abstract

Background:

Ultrasound (US) examination is helpful in the differential diagnosis of thyroid nodules (malignant vs. benign), but its accuracy relies heavily on examiner experience. Therefore, the aim of this study was to develop a less subjective diagnostic model aided by machine learning.

Methods:

A total of 2064 thyroid nodules (2032 patients, 695 male; M _age = 45.25 ± 13.49 years) met all of the following inclusion criteria: (i) hemi- or total thyroidectomy, (ii) maximum nodule diameter 2.5 cm, (iii) examination by conventional US and real-time elastography within one month before surgery, and (iv) no previous thyroid surgery or percutaneous thermotherapy. Models were developed using 60% of randomly selected samples based on nine commonly used algorithms, and validated using the remaining 40% of cases. All models function with a validation data set that has a pretest probability of malignancy of 10%. The models were refined with machine learning that consisted of 1000 repetitions of derivatization and validation, and compared to diagnosis by an experienced radiologist. Sensitivity, specificity, accuracy, and area under the curve (AUC) were calculated.

Results:

A random forest algorithm led to the best diagnostic model, which performed better than radiologist diagnosis based on conventional US only (AUC = 0.924 [confidence interval (CI) 0.895–0.953] vs. 0.834 [CI 0.815–0.853]) and based on both conventional US and real-time elastography (AUC = 0.938 [CI 0.914–0.961] vs. 0.843 [CI 0.829–0.857]).

Conclusions:

Machine-learning algorithms based on US examinations, particularly the random forest classifier, may diagnose malignant thyroid nodules better than radiologists.

Introduction

The incidence of thyroid cancer has increased rapidly in recent years due to the more frequent use of imaging methods such as ultrasound (US) of the neck and fine-needle aspiration (FNA) of thyroid nodules (1,2). Approximately 20–67% of the general population is estimated to have thyroid nodules that cannot be detected by palpation (3). By 2030, thyroid cancer may become the second leading cancer diagnosis in women and ninth leading cancer diagnosis in men (4). Nevertheless, only 5–15% of thyroid nodules are malignant (5). Therefore, the major clinical challenge is to differentiate reliably those nodules that are malignant and need to be treated surgically from the majority of nodules that are benign and do not require surgery. This highlights the importance of diagnostic methods that can differentiate benign and malignant nodules.

Conventional US is the preferred imaging method for diagnosing thyroid diseases because it is noninvasive, convenient, and inexpensive, and it does not expose the patient to damaging radiation. However, US features show diagnostic sensitivity of only 27–63% for detecting nodule malignancy (6). This is due to the complex structure of thyroid nodules and the many features common to benign and malignant nodules. As a result, US for diagnosing malignant nodules remains highly subjective and strongly dependent on clinician experience. Efforts have been made to supplement US with assessment of nodule hardness using shear wave elastography (SWE) or real-time elastography (RTE) (7), but the added value of these techniques is controversial.

Among the possibilities available, US-guided FNA biopsy remains the most effective and practical test to determine whether a nodule is malignant or whether surgery is required for definitive diagnosis (8). However, FNA biopsy is ineffective in absolute terms. Its sensitivity ranges from 54% to 90%, its specificity ranges from 60% to 98% (9), and it fails to provide definitive diagnosis in 20–30% of patients whose cancer risk can be as high 75% (10), with repeat FNA yielding indeterminate results in 20–28% of cases (11,12). In addition, biopsying all patients with thyroid nodules is too burdensome. Improved diagnostic methods are needed for identifying malignant thyroid nodules that would be less subjective and more robust than conventional US and that would reduce the need for biopsy or diagnostic surgery.

It may be possible to develop a superior noninvasive diagnostic method by taking advantage of the ability of artificial intelligence to “learn” from clinical data sets and predict diagnoses, in some cases more accurately than human experts (13). Here, a machine learning classifier was developed and validated for identifying thyroid nodules that are likely to be malignant and that should therefore be biopsied. This system may reduce the risk of unnecessary diagnostic hemithyroidectomy for benign conditions, benefit further management of patients, and reduce health-care costs as well as psychological burden and risk for patients.

Methods

Ethics

This retrospective study was approved by the Ethics Committee of the First Affiliated Hospital of Jinan University, which waived the requirement for informed patient consent because data for all subjects were anonymized.

Patients and nodules

A total of 4765 consecutive patients with 4989 thyroid nodules who underwent hemi- or total thyroidectomy at the authors' institution between April 2011 and June 2016 were screened for inclusion in this retrospective study. Patients were included if they met the following inclusion criteria: (i) greatest nodule diameter ≤2.5 cm, (ii) no previous surgery or US-guided percutaneous thermotherapy, and (iii) conventional US examination and RTE within one month before thyroidectomy. RTE is typically not performed at Guangdong Provincial People's Hospital if the nodules are close to the trachea, if the nodules protrude above the surface of the thyroid gland, or if the RTE pressure indicator shows <3 or >4. Patients were excluded from the study if (i) they had coalescent thyroid lesions that were not clearly distinguishable or (ii) pathology provided ambiguous diagnostic findings for their nodules. In addition, duplicate nodules, defined as those sharing the same features by US and RTE, were removed from the data set to satisfy the requirement for machine learning that all observations be independent of one another (14).

Conventional US assessment of thyroid nodules

All nodules were assessed using HI Vision 900, HI Vision Ascendus, and HI Vision Preirus color US units from Hitachi (Tokyo, Japan) equipped with 6–13 MHz linear probes. All patients were examined in the supine position, with the neck extended and a small pillow under the shoulders for better exposure of the lower thyroid margins. Scans of both thyroid lobes and isthmus were obtained in both transverse and longitudinal planes. Longitudinal and transverse images of the thyroid were obtained according to American College of Radiology accreditation standards, and images were recorded on PACS workstations (Landwind Healthcare, Shenhen, China). Thyroid radiologists with more than eight years of clinical experience performed and evaluated the US thyroid data. The following features were measured for each nodule: size, margin, shape, aspect ratio, capsule, hypoechoic halo, internal composition, echogenicity, presence of calcifications, vascularity on color Doppler, and cervical lymph node status. Static US features were reviewed by an independent radiologist with 17 years of experience in thyroid imaging. Nodule length corresponded to the long-axis measurement on a longitudinal scan, while width and thickness corresponded to the short-axis measurements on a transverse scan. Nodule size was divided into four categories according to maximum diameter in cm (≤0.5, 0.5–1.0, 1.0–2.0, 2.0–2.5). Shape was classified as regular or irregular. The aspect ratio (height divided by width on transverse views) was classified as ≤1 or >1. Internal composition was categorized in terms of the ratio of the cystic portion to the solid portion as solid (≤10% of the cystic portion), mixed (>10% and ≤90% of the cystic portion), or cystic (>90% of the cystic portion). Margins were classified as well-defined (smooth, microlobulated or spiculated) or ill-defined. Nodules were categorized as an-, iso-, hypo-, or hyperechoic based on healthy thyroid tissue as reference. The calcification pattern was categorized as microcalcification, which was defined as punctate echogenic foci of ≤2 mm with or without acoustic shadowing, and/or as macro-calcification, which was defined as a hyperechoic structure of >2 mm with acoustic shadowing. Vascularity was assessed as the presence or absence of blood flow on color Doppler. Hypoechoic halo was categorized as usual or unusual. In the latter case, the lesion was surrounded by an irregular or thick hypoechoic halo. Thyroid capsule was categorized as complete or incomplete. In the latter case, tumoral tissue extends beyond the contours of the thyroid gland and invades into adjacent structures. Cervical lymph node status was assigned as normal or abnormal based on US findings of calcification or cystic changes. Cervical lymph nodes (both central and lateral nodes) were considered enlarged when the aspect ratio was >0.5 without calcifications or cystic changes.

RTE assessment of thyroid nodules

RTE was usually performed by the same radiologists who performed conventional US, and RTE results were retrospectively reviewed by a radiologist with 17 years of experience. RTE results were considered to be reliable when they were obtained in the following way (15): (i) light compression was repeatedly applied at the skin above the targeted thyroid nodule; (ii) a square region of interest was defined with the targeted nodule in the center, the superior margin was set to include subcutaneous fat, and the inferior margin was set to include the longus colli muscle; and (iii) the pressure release index on the RTE instrument was maintained at 3–4. Elastic images that clearly reflected the hardness of tissues in each layer were considered of sufficient quality, and elastic images were captured during 3–5 seconds. Images were displayed in split-screen mode, with the conventional images on the right and the translucent color-scale RTE images superimposed on the conventional image on the left.

Each pixel of the elasticity image was colored on a 256-color scale according to the extent of strain. The scale ranged from red, indicating greatest strain (softest tissue), to blue, indicating no strain (hardest tissue). Lesion hardness was then assigned to one of five grades based on the Asteria criteria (16): grade 0, the lesion contained blue, green, and red areas, indicating predominantly cystic composition with a typical “mosaic” sign; grade I, the lesion area was uniformly green; grade II, >50% of the lesion area was green; grade III, 50–90% of the lesion area was blue; or grade IV, >90% of the lesion area was blue. Nodules assigned to grades 0–II were considered benign; those assigned to grades III–IV were considered malignant.

Machine-learning training

For each nodule, data were collected concerning 11 features in conventional US and one feature in RTE (Fig. 1). The ground truth was labeled at lesion level. Note that since the number of features in the data set is much smaller than the number of observations, the variable selection step was not considered, which is commonly used in high-dimensional data analysis for subsequent statistical analysis. Data set 1 covered the 11 US features; data set 2 covered these features as well as the RTE feature. The two data sets were entered into nine machine-learning algorithms: L²-logistic regression, linear discriminant analysis, random forest, kernel support vector machines, adaptive boosting, k-nearest neighborhood, neural network, naive Bayesian, and convolutional neural network (see Supplementary Methods for detailed descriptions). Using each of these algorithms, data were analyzed 1000 times after randomly dividing the entire sample of 2064 nodules into a training set and a validation set in a 3:2 ratio. Note that the performance of all the aforementioned methods relies on the choice of tuning parameters. The methods are trained via 10-fold cross-validation by using the training set and are evaluated by using the validation set. For example, the tuning parameter in k-SVM is determined by 10-fold cross-validation via a grid search, where the grid is set as {0.1 + 0.1 s: s = 0, …, 20}. For the RF method, the “tuneRF” function in R package “randomForest” was used, with the number of trees set to 500, and the number of variables randomly sampled was tuned as candidates at each split. For the k-NN method, the “trainControl” function in R package “caret” was used via a 10-fold cross-validation to choose the best k and so on. None of the authors have any affiliation with the artificial intelligence machine-learning algorithms tested.

FIG. 1.

Study work flow.

Machine-learning validation and benchmarking

The nine machine-learning algorithms function with a validation data set that has a pretest probability of malignancy of 10%. The ability of the algorithms to diagnose thyroid nodules correctly as malignant or benign (based on pathology diagnosis as the reference standard) was compared in terms of area under the curve (AUC; Fig. 1). The best-performing algorithm was then compared to diagnosis by an independent radiologist as follows. Using a fixed random number seed to generate a subset of the data for this validation (model vs. radiologist) (17), the diagnostic performance of the best machine-learning algorithm was compared to diagnosis by a blinded radiologist based on (i) US features only or (ii) US and RTE features.

Statistical analysis

Data were analyzed using R v3.2.3 (R Foundation for Statistical Computing, Vienna, Austria), and the following R packages were used for each machine-learning algorithm (18): LibLineaR for L²-logistic regression, MASS for linear discriminant analysis, randomForest for random forest, e1071 for kernel support vector machines, adabag for adaptive boosting, knn for k-nearest neighborhood, nnet for neural network, e1071 for naive Bayesian, and mxnet for convolutional neural network. Differences in conventional US and RTE features between benign and malignant thyroid nodules were assessed for significance using the chi-square test. The significance level was set at p < 0.05 with Bonferroni correction.

Results

Nodule characteristics

A total of 2064 nodules from 2032 patients (695 men) with a mean age of 45.25 ± 13.49 year (range 9–86 years) were included in the data analysis (Fig. 2). Of the 2064 nodules, 750 (36.3%) were found by pathology to be malignant and 1314 (63.7%) were found to be benign. Features of these nodules in conventional US and RTE are summarized in Table 1.

FIG. 2.

Flow diagram summarizing inclusion of thyroid nodules.

Table 1.

Comparison of Ultrasound and RTE Features of Benign and Malignant Thyroid Nodules

Feature	Benign nodules (n = 1314)	Malignant nodules (n = 750)	p
Size (cm)
≤0.5	44 (3.3)	53 (7.1)	<0.001
0.5–1.0	361 (27.5)	292 (38.9)
1.0–2.0	682 (51.9)	286 (38.1)
2.0–2.5	227 (17.3)	119 (15.9)
Echogenicity
Anechoic	31 (2.4)	1 (0.1)	<0.001
Isoechoic	648 (49.3)	133 (17.7)
Hypoechoic	512 (39.0)	603 (80.4)
Hyperechoic	123 (9.4)	13 (1.7)
Margins
Well-defined	953 (72.5)	191 (25.5)	<0.001
Ill-defined	361 (27.5)	559 (74.5)	<0.001
Internal composition
Cystic	31 (2.4)	0	<0.001
Solid	1085 (82.6)	711 (94.8)
Mixed	198 (15.1)	39 (5.2)
Shape
Regular	928 (70.6)	141 (18.8)	<0.001
Irregular	386 (29.4)	609 (81.2)	<0.001
Aspect ratio
≤1	1160 (88.3)	406 (54.1)	<0.001
>1	154 (11.7)	344 (45.9)	<0.001
Vascularity
Absent	1222 (93.0)	620 (82.7)	<0.001
Present	92 (7.0)	130 (17.3)	<0.001
Capsule
Complete	1302 (99.1)	596 (79.5)	<0.001
Incomplete	12 (0.9)	154 (20.5)	<0.001
Hypoechoic halo
Absent	1002 (76.3)	703 (93.7)	<0.001
Usual	260 (19.8)	11 (1.5)
Unusual	52 (4.0)	36 (4.8)
Cervical lymph nodes
Absent	1267 (96.4)	507 (67.7)	<0.001
Enlarged	47 (3.6)	120 (16)
Calcifications or cystic changes	0	122 (16.3)
Calcifications
Absent	927 (70.5)	175 (23.3)	<0.001
Micro	131 (10.0)	453 (60.4)
Macro	174 (13.2)	68 (9.1)
Micro + macro	82 (6.2)	54 (7.2)
Real-time elastography grade
0	31 (2.4)	0	<0.001
I	281 (21.4)	9 (1.2)
II	795 (60.5)	128 (17.1)
III	179 (13.6)	527 (70.3)
IV	28 (2.1)	86 (11.5)

RTE, real-time elastography.

Selection of machine-learning model

The diagnostic performance of the nine machine-learning algorithms based solely on US features (data set 1) is shown in Table 2. In the validation data set, the random forest classifier demonstrated the highest values for AUC (0.924 [confidence interval (CI) 0.895–0.953]), sensitivity (88.1% [CI 81.3–95.0%]), specificity (84.2% [CI 80.9–87.5%]), and accuracy (84.6% [CI 81.6–87.5%]). Similarly, the random forest algorithm showed the best performance when the diagnosis was based on US and RTE features (data set 2, Table 3), showing the highest values for AUC (0.938 [CI 0.914–0.961]) and accuracy (85.7% [CI 83.6–87.8%]). The random forest classifier was second in sensitivity (89.1% [CI 82.5–95.8%]) to k-SVM (91.4% [CI 85.2–97.5%]) and also second in specificity (85.3% [CI 83.0–87.6%]) after AdaBoost (85.8% [CI 78.0–93.5%]). Therefore, the random forest algorithm was chosen as the best and compared to diagnosis by an experienced radiologist.

Table 2.

Comparison of the Performance of Nine Classifiers Based on Ultrasound Features Assuming a Pretest Probability of Malignancy of 10% in the Validation Data Set

Classifier	Training data set (n = 1238)				Validation data set (n = 826)
Classifier	AUC	Sensitivity	Specificity	Accuracy	AUC	Sensitivity	Specificity	Accuracy
RF	0.986 [0.968–1.000]	94.2 [92.2–96.2]	94.6 [90.5–98.7]	94.4 [91.6–97.2]	0.924 [0.895–0.953]	88.1 [81.3–95.0]	84.2 [80.9–87.5]	84.6 [81.6–87.5]
k-SVM	0.947 [0.937–0.956]	90.0 [88.4–91.7]	86.8 [84.4–89.2]	88.6 [87.2–89.9]	0.917 [0.887–0.946]	86.1 [78.7–93.4]	82.7 [79.6–85.8]	83.0 [80.4–85.7]
k-NN	0.941 [0.924–0.958]	86.5 [84.5–88.4]	85.6 [82.3–88.8]	86.1 [84.1–88.0]	0.897 [0.863–0.932]	84.0 [76.2–91.9]	83.1 [80.1–86.1]	83.2 [80.5–85.8]
L²-LOG	0.884 [0.872–0.896]	83.2 [81.4–84.9]	78.9 [76.8–81.1]	81.2 [79.8–82.7]	0.878 [0.840–0.915]	82.3 [74.0–90.5]	78.4 [75.4–81.5]	78.8 [76.1–81.6]
LDA	0.876 [0.865–0.887]	83.8 [82.3–85.3]	77.7 [75.4–79.9]	81.0 [79.5–82.4]	0.870 [0.833–0.907]	83.1 [75.1–91.1]	77.4 [74.7–80.0]	77.9 [75.5–80.4]
NB	0.878 [0.867–0.889]	76.5 [71.5–81.4]	83.4 [79.2–87.6]	79.7 [77.9–81.4]	0.875 [0.836–0.914]	76.0 [66.1–85.9]	83.0 [78.4–87.7]	82.3 [78.4–86.2]
Nnet	0.872 [0.833–0.912]	86.0 [78.6–93.3]	79.7 [70.5–88.9]	83.1 [80.3–85.8]	0.859 [0.798–0.920]	84.3 [73.4–95.2]	78.2 [68.7–87.6]	78.8 [71.0–86.6]
AdaBoost	0.999 [0.998–1.000]	98.5 [97.3–99.6]	97.9 [96.7–99.0]	98.1 [97.2–99.0]	0.889 [0.854–0.924]	80.6 [77.3–83.9]	82.8 [74.9–90.7]	80.9 [78.0–83.7]
CNN	0.892 [0.868–0.916]	84.6 [72.6–96.5]	77.7 [65.6–89.8]	81.4 [78.5–84.3]	0.885 [0.845–0.925]	83.8 [69.7–97.9]	77.1 [64.8–89.3]	77.8 [67.8–87.7]

Results are the mean [confidence interval] calculated relative to the pathology-determined diagnosis as the reference.

AUC, area under the curve; US, ultrasound; L²-LOG, L²-logistic regression; LDA, linear discriminant analysis, RF, random forest; k-SVM, kernel support vector machines; AdaBoost, adaptive boosting; Nnet, neural network; k-NN, k-nearest neighborhood; NB, naive Bayesian; CNN, convolutional neural network.

Table 3.

Comparison of the Performance of Nine Classifiers Based on Ultrasound and RTE Features Assuming a Pretest Probability of Malignancy of 10% in the Validation Data Set

Classifier	Training data set (n = 1238)				Validation data set (n = 826)
Classifier	AUC	Sensitivity	Specificity	Accuracy	AUC	Sensitivity	Specificity	Accuracy
RF	0.993 [0.980–1.000]	96.9 [95.1–98.7]	95.4 [91.5–99.2]	96.2 [93.5–98.8]	0.938 [0.914–0.961]	89.1 [82.5–95.8]	85.3 [83.0–87.6]	85.7 [83.6–87.8]
k-SVM	0.958 [0.948–0.967]	92.4 [90.9–93.9]	88.8 [86.6–90.9]	90.7 [89.3–92.1]	0.931 [0.908–0.955]	91.4 [85.2–97.5]	84.7 [82.2–87.1]	85.4 [83.2–87.5]
k-NN	0.951 [0.931–0.971]	88.4 [86.0–90.9]	86.0 [82.3–89.7]	87.3 [84.6–90.0]	0.907 [0.875–0.938]	85.4 [77.6–93.2]	83.4 [80.5–86.2]	83.6 [81.1–86.1]
L²-LOG	0.911 [0.900–0.921]	86.4 [85.3–87.5]	85.4 [83.3–87.5]	86.0 [84.7–87.2]	0.905 [0.872–0.937]	85.7 [78.5–93.0]	84.9 [82.4–87.3]	84.9 [82.7–87.1]
LDA	0.904 [0.894–0.914]	86.2 [85.2–87.2]	83.6 [81.1–86.1]	85.0 [83.7–86.3]	0.899 [0.867–0.931]	85.8 [78.6–93.0]	83.2 [79.9–86.4]	83.4 [80.5–86.4]
NB	0.893 [0.882–0.903]	81.5 [78.8–84.3]	85.0 [82.9–87.2]	83.2 [81.6–84.7]	0.890 [0.855–0.926]	81.3 [73.0–89.6]	84.9 [82.6–87.2]	84.5 [82.4–86.6]
Nnet	0.892 [0.856–0.927]	89.3 [84.5–94.1]	85.5 [80.1–90.8]	87.5 [85.1–90.0]	0.873 [0.815–0.932]	87.3 [79.6–95.0]	83.5 [77.8–89.2]	83.9 [79.1–88.8]
AdaBoost	0.999 [0.999–1.000]	99.4 [98.6–100]	99.2 [98.6–99.9]	99.3 [87.5–99.8]	0.911 [0.881–0.942]	83.7 [80.9–86.6]	85.8 [78.0–93.5]	83.9 [81.4–86.4]
CNN	0.910 [0.852–0.968]	86.2 [77.5–94.9]	83.2 [72.5–93.9]	84.9 [82.5–87.3]	0.903 [0.837–0.968]	85.6 [74.4–96.7]	82.5 [71.6–93.5]	82.9 [74.7–91.1]

Results are the mean [confidence interval] calculated relative to the pathology-determined diagnosis as the reference.

Factors affecting diagnostic performance of the random forest algorithm

The diagnostic performance of the random forest machine-learning algorithm based only on conventional US depended to the greatest extent on the following features: calcification pattern, hypoechoic halo, cervical lymph node status, shape, and vascularity on color Doppler (Fig. 3A). All these features together with the RTE feature were the most important factors in the diagnostic performance of the random forest algorithm based on the combination of US and RTE data (Fig. 3B).

FIG. 3.

Relative importance of each input feature for the diagnostic performance of the random forest model. (A) Diagnosis based only on conventional ultrasound (US) features. (B) Diagnosis based on the combination of US features and real-time elastography.

Added value of RTE for diagnosis based on the random forest algorithm or a radiologist

Adding RTE to conventional US resulted in better differentiation of malignant and benign thyroid nodules than conventional US alone (Table 4). This was true for diagnosis performed by a radiologist, as measured in terms of AUC (0.843 [CI 0.829–0.857] vs. 0.834 [CI 0.815–0.853]), sensitivity (85.0% [CI 83.0–87.0%] vs. 83.5% [CI 80.0–86.6%]), specificity (83.6% [CI 81.8–85.4%] vs. 83.2% [CI 80.9–85.5%]), and accuracy (84.1% [CI 82.7–85.5%] vs. 83.3% [CI 81.5–85.2%]). It was also true for diagnosis performed by the random forest algorithm, as measured in terms of AUC (0.938 [CI 0.914–0.961] vs. 0.924 [CI 0.895–0.953]), sensitivity (89.1% [CI 82.5–95.8%] vs. 88.1% [CI 81.3–95.0%]), specificity (85.3% [CI 83.0–87.6%] vs. 84.2% [CI 80.9–87.5%]), and accuracy (85.7% [CI 83.6–87.8%] vs. 84.6% [CI 81.6–87.5%]).

Table 4.

Comparison of the Performance of the Random Forest Classifier with Diagnosis by an Experienced Radiologist

Comparison	AUC	Sensitivity	Specificity	Accuracy
RF classifier based on US features	0.924 [0.895–0.953]	88.1 [81.3–95.0]	84.2 [80.9–87.5]	84.6 [81.6–87.5]
Radiologist diagnosis based on US features	0.834 [0.815–0.853]	83.5 [80.0–86.6]	83.2 [80.9–85.5]	83.3 [81.5–85.2]
RF classifier based on US-RTE features	0.938 [0.914–0.961]	89.1 [82.5–95.8]	85.3 [83.0–87.6]	85.7 [83.6–87.8]
Radiologist diagnosis based on US-RTE features	0.843 [0.829–0.857]	85.0 [83.0–87.0]	83.6 [81.8–85.4]	84.1 [82.7–85.5]

Results are the mean [confidence interval] calculated relative to the pathology-determined diagnosis as the reference.

Online diagnostic system for differentiating malignant and benign thyroid nodules

The random forest diagnostic model described here is freely available online.

Comparison of diagnostic performance by the random forest algorithm and a radiologist

When the diagnosis was based only on conventional US, the random forest classifier performed better than an experienced radiologist (Table 4), as measured in terms of AUC (0.924 [CI 0.895–0.953] vs. 0.834 [CI 0.815–0.853]), sensitivity (88.1% [CI 81.3–95.0%] vs. 83.5% [CI 80.0–86.6%]), specificity (84.2% [CI 80.9–87.5%] vs. 83.2% [CI 80.9–85.5%]), and accuracy (84.6% [CI 81.6–87.5%] vs. 83.3% [CI 81.5–85.2%]).

When diagnosis was based on the combination of conventional US and RTE features, the random forest algorithm again outperformed diagnosis by a radiologist, as measured in terms of AUC (0.938 [CI 0.914–0.961] vs. 0.843 [CI 0.829–0.857]), sensitivity (89.1% [CI 82.5–95.8%] vs. 85.0% [CI 83.0–87.0%]), specificity (85.3% [CI 83.0–87.6%] vs. 83.6% [CI 81.8–85.4]), and accuracy (85.7% [CI 83.6–87.8%] vs. 84.1% [CI 82.7–85.5%]).

Discussion

Clinical and pathology data on 2064 thyroid nodules were retrospectively analyzed, and the data were used to train nine popular artificial intelligence-based machine-learning classifiers in an effort to identify which one would best differentiate between malignant and benign nodules. It was found that the random forest algorithm performed best and, in fact, outperformed diagnosis by a radiologist based on US alone and a combination of US and RTE. It is further shown that adding an RTE feature to conventional US features improved the diagnostic performance of the random forest classifier and the radiologist. These results demonstrate the feasibility of strengthening diagnosis of malignant thyroid nodules using machine learning. They also led to an online diagnostic tool being created that can guide diagnosis and clinical decision making.

Conventional US has proven quite useful for detecting clinically significant nodules that escape detection by physical examination. In fact, several studies recommend thyroid US for patients with suspected thyroid nodules. At the same time, this imaging approach is far from perfect. Although several conventional US features have been associated with increased risk of malignancies, no single US feature on its own can reliably differentiate malignant nodules from benign ones. A meta-analysis has shown that the sensitivity of most US features to thyroid nodule malignancy ranges from 27% to 63% (6). Four features—microcalcifications, central vascularization, irregular margins, and taller-than-wide shape—showed higher specificities of 78.0–96.6% (19). The 2015 American Thyroid Association management guidelines indicate that features with the highest specificities (median >90%) for thyroid cancer include microcalcifications, irregular margins, and an aspect ratio >1 (20). Central vascularization may not be an important feature for the diagnosis of malignancy. Up to 55% of benign nodules are hypoechoic compared to the thyroid parenchyma, making nodule hypoechogenicity less specific (20). In fact, most benign nodules with a size <1 cm usually appear hypoechoic (20). In this study, echogenicity played a relatively minor role because 39.0% of the benign nodules were also hypoechoic. Searching for such patterns is difficult for humans and dependent on their experience, leading several groups to apply machine-learning approaches such as extreme machine learning, deep convolutional neural networks, and artificial neural networks to thyroid nodule diagnosis (21,22). Two previous studies did not systematically explore which machine-learning classifier algorithms work the best (21,22). In the present study, comparison of nine algorithms identified the random forest classifier as the best. This classifier includes many decision trees, avoids high variance or bias in prediction, and is quite simple to implement efficiently and accurately.

RTE can differentiate malignant from benign thyroid nodules because malignant nodules are harder than the surrounding adjacent parenchyma (23 –27). Several meta-analyses suggest that RTE is superior to SWE for this purpose (28,29). Some studies, most of which involve small case series and few US features, have reported that RTE can complement US to improve diagnostic accuracy (30,31), whereas one study of 703 thyroid nodules found that using RTE led to lower diagnostic accuracy than using US alone (15). As a result, the added value with RTE remains under debate (32). The present much larger study provides strong evidence that RTE improves the performance of not only the experienced radiologist but also the random forest machine-learning algorithm.

The present work possesses advantages over several studies that have attempted to differentiate malignant and benign thyroid nodules using computer-aided diagnosis (CAD) systems on the basis of conventional US features (33 –35). Those CAD systems classify US features differently from the way radiologists do, whereas the present system examines the same features as radiologists. The present system examines a larger array of US features than previous CAD studies, and it incorporates complementary information from RTE. The machine learning–based tool for diagnosis of thyroid nodules described in this study was trained using what appears to be the largest data set so far.

Despite these strengths, the present study also has limitations. First, 705/750 (94%) thyroid malignancies in the data set were classical papillary thyroid carcinomas, which differ in US features from follicular carcinoma, medullary carcinoma, and lymphoma. This highlights the need to verify and extend the diagnostic model by incorporating more training data from various types of thyroid malignancies. Second, only nodules for which definitive pathology findings of malignant or benign were available were included. As a result, the diagnostic model still needs to be validated for nodules giving indeterminate cytology results and therefore characterized as “atypia of undetermined significance” or “follicular lesion of undetermined significance.” Third, thyroid radiologists reviewed static US images instead of dynamic images. Reviewing images dynamically or in real time, the reported US features would be more objective and reliable. At the Guangdong Provincial People's Hospital, radiologists with more than eight years of clinical experience in thyroid imaging performed the US in real time and described the key US features in the reports. To reduce potential bias of retrospective nature, the radiologists also checked the US reports. Fourth, we could not evaluate and compare the performance of machine-learning algorithms for each cancer subtype because most malignant nodules were papillary carcinomas. However, other subtypes such as follicular, Hürthle cell, or anaplastic carcinomas also have some different US features compared to benign thyroid lesions, and the algorithm was effective in the differentiation of these entities from benign thyroid lesions. Nevertheless, the diagnostic performance of the algorithm may be less efficient when applied to medullary carcinoma than other subtypes because it shares more US features with benign thyroid lesions. Finally, the data in the study came from a single center, which means that the system needs to be validated in other clinical settings.

In summary, the present results demonstrate that machine learning can lead to more reliable differentiation of benign and malignant nodules than what can be achieved by radiologists on the basis of US alone or combined with RTE.

Footnotes

Acknowledgments

This research was supported by a grant of the National Scientific Foundation of China (81571664, 81871323, 81801665), the Science and Technology Planning Project of Guangdong Province (2014A020212244, 2016A020216020), the Scientific Research General Project of Guangzhou Science Technology and Innovation Commission (201605110912158), the China Postdoctoral Science Foundation (2016M600145), and the Guangdong Grand Science and Technology Special Project (2015B010106008).

Author Disclosure Statement

No competing financial interests exist.

Supplementary Material

Supplementary Methods

Supplementary Figure S1

References

Ahn

, Kim

, Welch

. 2014. Korea's thyroid-cancer “epidemic”—screening and overdiagnosis. N Engl J Med, 371:1765–1767.

Raue

, Frank-Raue

. 2016. Thyroid cancer: risk-stratified management and individualized therapy. Clin Cancer Res, 22:5012–5021.

Gharib

, Papini

, Paschke

, Duick

, Valcavi

, Hegedüs

, Vitti

; AACE/AME/ETA Task Force on Thyroid Nodules. 2010. American Association of Clinical Endocrinologists, Associazione Medici Endocrinologi, and European Thyroid Association medical guidelines for clinical practice for the diagnosis and management of thyroid nodules: executive summary of recommendations. J Endocrinol Invest, 33:51–56.

Rahib

, Smith

, Aizenberg

, Rosenzweig

, Fleshman

, Matrisian

. 2014. Projecting cancer incidence and deaths to 2030: the unexpected burden of thyroid, liver, and pancreas cancers in the United States. Cancer Res, 74:2913–2921.

Burman

, Wartofsky

. 2015. Clinical practice. Thyroid nodules. N Engl J Med, 373:2347–2356.

Remonti

, Kramer

, Leitao

, Pinto

, Gross

. 2015. Thyroid ultrasound features and risk of carcinoma: a systematic review and meta-analysis of observational studies. Thyroid, 25:538–550.

Shuzhen

. 2012. Comparison analysis between conventional ultrasonography and ultrasound elastography of thyroid nodules. Eur J Radiol, 81:1806–1811.

Hang

, Hsu

, Lai

. 2017. Thyroid fine-needle aspiration in Taiwan: the history and current practice. J Pathol Transl Med, 51:560–564.

Magri

, Chytiris

, Zerbini

, Capelli

, Gaiti

, Carbone

, Fonte

, Malovini

, Rotondi

, Bellazzi

, Chiovato

. 2015. Maximal stiffness evaluation by real-time ultrasound elastography, an improved tool for the differential diagnosis of thyroid nodules. Endocr Pract, 21:474–481.

10.

Theoharis

, Schofield

, Hammers

, Udelsman

, Chhieng

. 2009. The Bethesda thyroid fine-needle aspiration classification system: year 1 at an academic institution. Thyroid, 19:1215–1223.

11.

Misiakos

, Margari

, Meristoudis

, Machairas

, Schizas

, Petropoulos

, Spathis

, Karakitsos

, Machairas

. 2016. Cytopathologic diagnosis of fine needle aspiration biopsies of thyroid nodules. World J Clin Cases, 4:38–48.

12.

VanderLaan

, Marqusee

, Krane

. 2011. Clinical outcome for atypia of undetermined significance in thyroid fine-needle aspirations: should repeated FNA be the preferred initial approach?. Am J Clin Pathol, 135:770–775.

13.

Oberije

, Nalbantov

, Dekker

, Boersma

, Borger

, Reymen

, van Baardwijk

, Wanders

, De Ruysscher

, Steyerberg

, Dingemans

, Lambin

. 2014. A prospective study comparing the predictions of doctors versus models for treatment outcome of lung cancer patients: a step toward individualized care and shared decision making. Radiother Oncol, 112:37–43.

14.

Van den Broeck

, Cunningham

, Eeckels

, Herbst

. 2005. Data cleaning: detecting, diagnosing, and editing data abnormalities. PLoS Med, 2:e267.

15.

Moon

, Sung

, Kim

, Yoon

, Youk

, Kwak

. 2012. Diagnostic performance of gray-scale US and elastography in solid thyroid nodules. Radiology, 262:1002–1013.

16.

Asteria

, Giovanardi

, Pizzocaro

, Cozzaglio

, Morabito

, Somalvico

. 2008. US-elastography in the differential diagnosis of benign and malignant thyroid nodules. Thyroid, 18:523-531.

17.

Knuth DE 1998 The Art of Computer Programming, Volume 2: Seminumerical Algorithms. Third edition. Addison-Wesley, Boston, MA.

18.

Hastie

, Tibshirani

, Friedman

. 2009. The Elements of Statistical Learning. Second edition. Springer, New York, NY.

19.

Frates

, Benson

, Charboneau

, Cibas

, Clark

, Coleman

, Cronan

, Doubilet

, Evans

, Goellner

, Hay

, Hertzberg

, Intenzo

, Jeffrey

, Langer

, Larsen

, Mandel

, Middleton

, Reading

, Sherman

, Tessler

. 2005. Management of thyroid nodules detected at US: Society of Radiologists in Ultrasound consensus conference statement. Radiology, 237:794–800.

20.

Frates

, Benson

, Charboneau

, Cibas

, Clark

, Coleman

, Cronan

, Doubilet

, Evans

, Goellner

, Hay

, Hertzberg

, Intenzo

, Jeffrey

, Langer

, Larsen

, Mandel

, Middleton

, Reading

, Sherman

, Tessler

. 2016. 2015 American Thyroid Association management guidelines for adult patients with thyroid nodules and differentiated thyroid cancer: the American Thyroid Association Guidelines Task Force on Thyroid Nodules and Differentiated Thyroid Cancer. Thyroid, 26:1–133.

21.

Xia

, Chen

, Li

, Zhou

, Chen

, Cai

, Fang

, Zhou

. 2017. Ultrasound-based differentiation of malignant and benign thyroid nodules: an extreme learning machine approach. Comput Methods Programs Biomed, 147:37–49.

22.

Zhu

, Ye

, Luo

, Su

, Wei

, Zhang

, Wei

, Zou

. 2013. A model to discriminate malignant from benign thyroid nodules using artificial neural network. PLoS One, 8:e82211.

23.

Hong

, Choi

, Cheon

, Kim

, Kang

. 2018. Accurate measurements of liver stiffness using shear wave elastography in children and young adults and the role of the stability index. Ultrasonography, 37:226–232.

24.

Youk

, Son

, Han

, Gweon

, Kim

. 2018. Performance of shear-wave elastography for breast masses using different region-of-interest (ROI) settings. Acta Radiol, 59:789–797.

25.

Mancini

, Salomone Megna

, Ragucci

, De Luca

, Marino Marsilia

, Nardone

, Coccoli

, Prinster

, Mannelli

, Vergara

, Monti

, Liuzzi

, Incoronato

. 2017. Reproducibility of shear wave elastography (SWE) in patients with chronic liver disease. PLoS One, 12:e185391.

26.

Bardet

, Ciappuccini

, Pellot-Barakat

, Monpeyssen

, Michels

, Tissier

, Blanchard

, Menegaux

, de Raucourt

, Lefort

, Reznik

, Rouxel

, Heutte

, Brenac

, Leconte

, Buffet

, Clarisse

, Leenhardt

. 2017. Shear wave elastography in thyroid nodules with indeterminate cytology: results of a prospective bicentric study. Thyroid, 27:1441–1449.

27.

Wang

, Dan

, Li

, Hu

. 2010. Differential diagnosis of small single solid thyroid nodules using real-time ultrasound elastography. J Int Med Res, 38:466–472.

28.

Tian

, Hao

, Gao

, Jiang

, Zhang

, Guo

, Luo

. 2015. Comparison of diagnostic accuracy of real-time elastography and shear wave elastography in differentiation malignant from benign thyroid nodules. Medicine (Baltimore), 94:e2312.

29.

Tian

, Hao

, Gao

, Jiang

, Zhang

, Guo

, Yan

, Luo

. 2016. Comparing the diagnostic accuracy of RTE and SWE in differentiating malignant thyroid nodules from benign ones: a meta-analysis. Cell Physiol Biochem, 39:2451–2463.

30.

Trimboli

, Guglielmi

, Monti

, Misischi

, Graziano

, Nasrollah

, Amendola

, Morgante

, Deiana

, Valabrega

, Toscano

, Papini

. 2012. Ultrasound sensitivity for thyroid malignancy is increased by real-time elastography: a prospective multicenter study. J Clin Endocrinol Metab, 97:4524–4530.

31.

Ragazzoni

, Deandrea

, Mormile

, Ramunni

, Garino

, Magliona

, Motta

, Torchio

, Garberoglio

, Limone

. 2012. High diagnostic accuracy and interobserver reliability of real-time elastography in the evaluation of thyroid nodules. Ultrasound Med Biol, 38:1154–1162.

32.

Kwak

, Kim

. 2014. Ultrasound elastography for thyroid nodules: recent advances. Ultrasonography, 33:75–82.

33.

Choi

, Baek

, Park

, Shim

, Kim

, Shong

, Lee

. 2017. A computer-aided diagnosis system using artificial intelligence for the diagnosis and characterization of thyroid nodules on ultrasound: initial clinical assessment. Thyroid, 27:546–552.

34.

, Ouyang

, Chen

, Liu

. 2012. A computer aided diagnosis system for thyroid disease using extreme learning machine. J Med Syst, 36:3327–3337.

35.

Chang

, Paul

, Kim

, Baek

, Choi

, Ha

, Lee

, Shin

, Kim

. 2016. Computer-aided diagnosis for classifying benign versus malignant thyroid nodules based on ultrasound images: a comparison with radiologist-based assessments. Med Phys, 43:554.