A Preliminary Evaluation of the Diagnostic Performance of a Smartphone-Based Machine Learning-Assisted System for Evaluation of Clinical Activity Score in Digital Images of Thyroid-Associated Orbitopathy

Abstract

Background:

We previously developed a machine learning (ML)-assisted system for predicting the clinical activity score (CAS) in thyroid-associated orbitopathy (TAO) using digital facial images taken by a digital single-lens reflex camera in a studio setting. In this study, we aimed to apply this system to smartphones and detect active TAO (CAS ≥3) using facial images captured by smartphone cameras. We evaluated the performance of our system on various smartphone models and compared it with the performance of ophthalmologists with varying clinical experience.

Methods:

We applied the preexisting ML architecture to classify photos taken with smartphones (Galaxy S21 Ultra, iPhone 12 pro, iPhone 11, iPhone SE 2020, Galaxy M20, and Galaxy A21S). The performance was evaluated with smartphone-captured images from 100 patients with TAO. Three ophthalmology residents, three general ophthalmologists with <5 years of clinical experience, and three oculoplastic specialists independently interpreted the same set of images taken under a studio environment and compared their results with those generated by the smartphone-based ML-assisted system. Reference CAS was determined by a consensus of three oculoplastic specialists.

Results:

Active TAO (CAS ≥3) was identified in 28 patients. Smartphone model used in capturing facial images influenced active TAO detection performance (F1 score 0.59–0.72). The smartphone-based system showed 74.5% sensitivity, 84.8% specificity, and F1 score 0.70 on top three smartphones. On images from all six smartphones, average sensitivity, specificity, and F1 score were 71.4%, 81.6%, and 0.66, respectively. Ophthalmology residents’ values were 69.1%, 55.1%, and 0.46. General ophthalmologists’ values were 61.9%, 79.6%, and 0.55. Oculoplastic specialists’ values were 73.8%, 90.7%, and 0.75. This smartphone-based ML-assisted system predicted CAS within 1 point of reference CAS in 90.7% using facial images from smartphones.

Conclusions:

Our smartphone-based ML-assisted system shows reasonable accuracy in detecting active TAO, comparable with oculoplastic specialists and outperforming residents and general ophthalmologists. It may enable reliable self-monitoring for disease activity, but confirmatory research is needed for clinical application.

Introduction

Thyroid-associated orbitopathy (TAO) is the predominant extrathyroidal manifestation of autoimmune thyroid disorders, such as Graves’ disease and autoimmune thyroiditis. Manifestations of TAO are clinically evident in ∼30–50% of Graves’ disease patients and occur less frequently (0.1–0.3%) in patients with chronic autoimmune thyroiditis. The management of TAO depends on its clinical activity and severity, determined by standardized criteria. It primarily involves immunosuppressive treatment, including intravenous glucocorticoids, for patients with inflammatory active TAO of moderate to severe seterity.¹ However, this treatment is less effective for those with fibrous orbital changes because of chronic inflammation, highlighting the importance of timely intervention during the active phase to prevent serious complications.^2,3 Early identification of individuals at risk of active TAO and immediate initiation of suitable treatment are crucial for enhancing prognosis.^1
–3

The most reliable scoring system for assessing TAO activity is the clinical activity score (CAS), which consists of seven binary (yes/no) items.¹ TAO is considered active if the score is ≥3.⁴ Each CAS item pertains to inflammatory symptoms and signs of TAO, including five signs of orbital redness and swelling, as well as two symptoms of orbital pain.⁴ While CAS is straightforward to use, its results may vary depending on the evaluator, necessitating evaluation by an experienced ophthalmologist for accuracy.⁵ In a prior study, we developed a machine learning (ML)-assisted system to predict CAS using digital facial images captured with a digital single-lens reflex (DSLR) camera in a controlled studio setting.⁶ This system was specifically designed to emulate the process of evaluating the inflammatory activity of TAO performed by ophthalmologists and demonstrated clinically reasonable performance. However, for practical application on smart devices, it is crucial to determine whether the CAS prediction performance remains effective when using images taken by users themselves with the built-in cameras of their smartphones.

In this study, we optimized a preexisting ML-assisted system evaluating CAS with digital facial images captured in studio environment for detecting active TAO using selfie images taken with a smartphone camera and assessed the performance of this system. Furthermore, we compared the performance of this smartphone-based ML-assisted system with the accuracy of assessments made by ophthalmologists with varying levels of clinical experience.

Methods

Study participants and facial images

A single-center, prospective cohort study was conducted at the ophthalmology department’s outpatient clinic of Seoul National University Bundang Hospital (SNUBH). A total of 100 patients with TAO, aged 18 and above, who visited the outpatient clinic during the period from December 2021 to July 2022, were sequentially enrolled in the study after providing their consent to participate in the research. Digital facial images of participants were captured from a distance of 1 m using a DSLR camera (Canon EOS 80D, resolution 2976 × 1984). Participants were directed to keep their heads still with their eyes in the primary position, and the positioning of the head was assessed to ensure there was no noticeable tilt or deviation in the chin-up or chin-down position. After conducting photo sessions in a studio environment, participants were provided with six types of smartphones (Galaxy S21 Ultra, Galaxy A21S, and Galaxy M20 Samsung Electronics Co., Ltd.; iPhone 12 pro, iPhone 11, and iPhone SE 2020, Apple Inc.) installed with a specialized capture module and asked to take two selfies of their own faces for each device. This capture module was designed to provide guidance on maintaining a consistent angle and distance during facial photography (Fig. 1). When the predefined conditions were met, the capture module automatically triggered the photo capture process. The provided guidance and automated capture feature ensured that facial photos were taken under controlled and consistent conditions.

FIG. 1.

A capture module built in the specialized research app equipped in each smartphone. The facial image included in the figure is a digitally generated representation.

The research received approval from the Institutional Review Board of SNUBH (IRB# B-2110-715-304) and was registered on ClinicalTrials.gov (trial registration number NCT05731154). All procedures adhered to applicable guidelines and regulations, and all images and clinical data underwent anonymization before being transferred from the hospital premises.

CAS scoring

Three ophthalmology residents, three general ophthalmologists with <5 years of clinical experience, and three oculoplastic specialists with >15 years of experience participated as evaluators. They assessed facial images from 100 TAO patients, captured with a DSLR camera in a controlled studio environment. Assessment of the CAS followed criteria advised by the European Group on Graves’ Orbitopathy (EUGOGO).^4,7 CAS includes seven binary elements, which inquire about spontaneous retrobulbar pain, pain during upward and downward gaze, redness of the eyelids, redness of the conjunctiva, swelling of the eyelids, inflammation of the caruncle and/or plica, and conjunctival edema. The overall score, ranging from 0 to 7, is determined by adding up scores from all items, with active TAO identified when the CAS is ≥3.⁸ Evaluators assessed patients’ facial images and scored the five items concerning orbital swelling and redness by consulting a standard photographic color atlas.⁹ The data regarding the two items concerning orbital pain were extracted from the patients’ medical records. Clinical information, including symptoms of orbital pain, was accessible to the evaluators, but the reference CAS was not provided to them.

The reference values for evaluating the performance of each evaluator and the ML-assisted system were established using consensus results from three oculoplastic specialists. If the three specialists provided a consistent score for the five items related to signs of orbital swelling and redness, that result was used as is. In cases of inconsistent scores, the specialists engaged in a discussion until a consensus was reached. The most ideal CAS evaluation method involves the consensus results of at least three highly experienced oculoplastic specialists through in-person evaluations, which can be considered as the gold standard. However, in this study, ophthalmologists did not directly examine the patients but instead interpreted photos to calculate CAS. The three oculoplastic specialists who participated in this study belonged to different institutions, making it impossible to simultaneously examine the same patient during the research process. As an alternative, the gold standard that could be chosen was either the evaluation by three oculoplastic specialists using photos taken in a studio setting or selecting the CAS results evaluated in person by one oculoplastic specialist who participated in patient care. In the development stages of our artificial intelligence (AI) model, we chose the results evaluated by three oculoplastic specialists through photo assessments as the gold standard for training. This decision was made because the interobserver variability of CAS was reported to be high,⁵ and we prioritized “the consensus of three experts” over “in-person evaluation” during the development of our AI model. Despite this, we also presented the performance of the AI model when the reference standard was the results of in-person CAS evaluation by one oculoplastic specialist.

Development of the ML-assisted system

In a prior study, we introduced an ML system aimed at replicating the process used by an ophthalmologist to calculate the CAS and diagnose active TAO.⁶ This process involves three main steps: (1) evaluating the presence or absence of each of the five indicators of orbital swelling and redness in each eye, (2) determining if the patient has the sign based on the result in each eye (whether at least one eye shows the sign), and (3) calculating the CAS and determining the clinical activity of TAO. Our ML-assisted system’s structure includes components corresponding to these processes, along with an image preprocessing module to extract the region of interest from the provided facial image.⁶ The system takes the patient’s digital facial images and responses to two questions regarding orbital pain as inputs, and it provides the activity status of the patient’s TAO as output.

We optimized our previously developed ML-assisted system for predicting TAO activity from facial photos captured with smartphones by diversifying our training dataset. The dataset was divided into four distinct groups: (1) the original DSLR photos used as the training dataset in our previous study,⁶ (2) white-balanced versions of the original DSLR photos, (3) synthetic images generated by the CycleGAN using the original DSLR photos as input, and (4) synthetic images created by the CycleGAN using the white-balanced DSLR photos as input. The CycleGAN generative AI model transformed DSLR photos to resemble images taken with various models of smartphones. Training with this diverse dataset enabled our AI model to be exposed to a wide array of image characteristics and optimized for the task of detecting active TAO using facial photos captured by a smartphone’s camera.

The experiments were carried out on a computer operating Ubuntu 20.04.5 LTS, which was equipped with an AMD Ryzen Threadripper 5995wx 64-core processor CPU, 512 GB of RAM, and four NVIDIA GeForce RTX 3090 GPUs.

Biochemical measurements

The serum levels of free thyroxine (fT4) and thyrotropin (TSH) were assessed using immunoradiometric assays from DiaSorin S.p.A. for fT4 and CIS Bio International for TSH. The normal reference ranges for fT4 and TSH were 0.89–1.79 ng/dL and 0.3–4.0 mIU/L, respectively. Patients were also tested for the presence of anti-TSH receptor antibodies through a radioimmunoassay provided by Cis Bio International, and the cutoff for considering it positive was >1.0 U/mL.

Statistical analysis

Data with a normal distribution are presented as mean ± standard deviation, while those with a non-normal distribution are presented as median (interquartile range). The ML-assisted system’s sensitivity and specificity in identifying active TAO were calculated using the reference diagnoses provided by the three oculoplastic specialists. The F1 score is computed as the harmonic mean of the sensitivity and positive predictive value when assessing the performance of the ML-assisted system and evaluators. The diagnostic values of each evaluator were analyzed using the test set including facial images from 100 TAO patients, which was taken by a DSLR camera in a studio environment. The diagnostic values of the smartphone-based ML-assisted system were analyzed using the test set including facial images from 100 TAO patients, which were taken by smartphones.

Results

Participant characteristics

Table 1 displays the baseline characteristics of the 100 participants enrolled in this study. The average age was 46.1 ± 12.4 years, and 80 out of 100 were female. Anti-TSH receptor antibody was detected (>1.0 U/mL) in 84.0% of the patients. The mean CAS was 1.9 ± 1.3, with the most prevalent inflammatory symptom or sign being eyelid swelling. Active TAO (CAS ≥3) was identified in 28 patients (28.0%), and highly active TAO (CAS ≥5) was identified in 5 patients (5.0%).

Table 1.

Baseline Characteristics

Age (Year)	46.1 ± 12.4
Sex (Male/Female)	20/80
BMI (kg/m²)	23.3 ± 3.9
Thyroid Function Test
fT4 (ng/dL)	1.32 ± 0.45
TSH (mIU/L)	0.12 (1.49)
Anti-TSH receptor antibody	4.64 (7.91)
CAS (total score)*	1.93 ± 1.28
Spontaneous retrobulbar pain (Yes/No)	9/91
Pain on attempt up and down gaze (Yes/No)	15/85
Redness of eyelids (Yes/No)	27/73
Redness of conjunctiva (Yes/No)	34/66
Swelling of eyelids (Yes/No)	86/14
Inflammation of the caruncle and/or plica (Yes/No)	12/88
Conjunctival edema (Yes/No)	10/90

Data are expressed as mean ± SD or median (interquartile range).

*Consensus results of three oculoplastic specialists who assessed facial photographs taken in a studio environment.

BMI, body mass index; CAS, clinical activity score; fT4, free thyroxine; SD, standard deviation; TSH, thyrotropin.

Diagnostic performance of the smartphone-based ML-assisted system for detecting active TAO

Table 2 shows the performance of the smartphone-based ML-assisted system detecting active TAO when analyzing photos taken with each smartphone. It appears that higher-tier smartphone models exhibit better diagnostic performance. The performance of top-performing smartphone model demonstrated 76.8% sensitivity, 85.9% specificity, and an F1 score of 0.72. When using facial images from the three top performing smartphones, our ML-assisted system exhibited 74.5% of sensitivity, 84.8% of specificity, and an F1 score of 0.70. On facial images from all six smartphones, sensitivity, specificity, and F1 score of the system for detecting active TAO were 71.4%, 81.6%, and 0.66, respectively.

Table 2.

Diagnostic Accuracy of Smartphone-Based ML-Assisted System for Active TAO (CAS ≥3) Using Facial Images from Various Smartphones with Consensus Results of Photo-Based CAS Evaluation as Reference

	Sensitivity, % [CI]	Specificity, % [CI]	F1 score [CI]
Galaxy S21 Ultra	76.8 [63.0–88.9]	85.9 [79.4–91.6]	0.72 [0.60–0.82]
iPhone 12 pro	73.2 [58.7–87.0]	85.0 [77.5–91.7]	0.69 [0.57–0.81]
iPhone 11	73.6 [57.5–88.4]	83.6 [76.2–90.2]	0.68 [0.53–0.79]
iPhone SE 2020	64.3 [50.0–79.6]	87.2 [80.8–92.6]	0.65 [0.52–0.76]
Galaxy M20	70.4 [55.0–85.2]	79.6 [71.3–87.2]	0.63 [0.49–0.75]
Galaxy A21S	70.4 [55.3–85.2]	74.1 [65.7–81.9]	0.59 [0.45–0.72]
Average of top 3 smartphones	74.5 [61.5–86.2]	84.8 [78.9–90.4]	0.70 [0.58–0.80]
Average of all smartphones	71.4 [59.5–82.9]	82.6 [76.8–87.9]	0.66 [0.55–0.75]

CI, confidence interval; ML, machine learning; TAO, thyroid-associated orbitopathy.

Table 3 demonstrates the performance of the system detecting active TAO when the reference standard is the results of in-person CAS evaluation by one oculoplastic specialist. The ML-assisted system showed 72.8% of sensitivity, 77.6% of specificity, and an F1 score of 0.54 when using facial images from the three top performing smartphones.

Table 3.

Diagnostic Accuracy of Smartphone-Based ML-Assisted System for Active TAO (CAS ≥3) Using Facial Images from Various Smartphones with In-Person CAS Evaluation as Reference

	Sensitivity, % [CI]	Specificity, % [CI]	F1 score [CI]
Galaxy S21 Ultra	70.3 [51.2–86.2]	77.0 [69.0–84.3]	0.52 [0.36–0.65]
iPhone 12 pro	67.6 [46.3–87.0]	76.7 [67.9–84.7]	0.51 [0.33–0.66]
iPhone 11	80.6 [63.6–94.6]	79.0 [70.5–86.5]	0.59 [0.44–0.72]
iPhone SE 2020	62.2 [40.0–81.0]	80.6 [73.4–87.3]	0.51 [0.32–0.65]
Galaxy M20	70.3 [50.0–86.4]	74.2 [65.8–81.7]	0.50 [0.33–0.64]
Galaxy A21S	78.4 [59.5–93.8]	71.3 [62.0–79.0]	0.52 [0.36–0.65]
Average of top 3 smartphones	72.8 [55.3–86.9]	77.6 [70.0–84.4]	0.54 [0.39–0.67]
Average of all smartphones	71.5 [55.0–84.9]	76.5 [69.7–82.7]	0.52 [0.37–0.65]

Diagnostic performance of ophthalmologists with varying clinical experience

The performance of ophthalmologists with varying clinical experience for diagnosing active TAO when they interpreted facial images taken by a DSLR camera in a studio environment was demonstrated in Table 4. It was observed that more experienced ophthalmologists demonstrated better diagnostic performance. When comparing the diagnostic results of each oculoplastic specialist to the consensus reference, the average sensitivity was 73.8%, specificity was 90.7%, and the F1 score was 0.75. General ophthalmologists with <5 years of experience showed 61.9% of sensitivity, 79.6% of specificity, and an F1 score of 0.55. For ophthalmology residents, values were 69.1%, 55.1%, and 0.46.

Table 4.

Diagnostic Accuracy of Ophthalmologists with Varying Clinical Experience for Active TAO (CAS ≥3) Using Consensus Results of Photo-Based CAS Evaluation as Reference

	Sensitivity, % [CI]	Specificity, % [CI]	F1 score [CI]
Resident 1	85.7 [72.4–96.8]	48.6 [37.3–59.7]	0.54 [0.41–0.65]
Resident 2	100.0 [100.0–100.0]	16.7 [8.5–25.4]	0.48 [0.37–0.60]
Resident 3	21.4 [7.4–37.5]	100.0 [100.0–100.0]	0.35 [0.14–0.55]
Average of residents	69.1 [61.9–76.2]	55.1 [49.7–60.4]	0.46 [0.34–0.55]
General ophthalmologist 1	28.6 [12.5–46.2]	98.6 [94.8–100.0]	0.43 [0.22–0.62]
General ophthalmologist 2	60.7 [42.3–78.3]	84.7 [75.3–92.2]	0.61 [0.44–0.74]
General ophthalmologist 3	96.4 [88.0–100.0]	55.6 [43.6–67.6]	0.62 [0.49–0.73]
Average of general ophthalmologists	61.9 [50.8–72.8]	79.6 [73.5–85.1]	0.55 [0.42–0.65]
Oculoplastic specialist 1	85.7 [71.4–97.1]	95.8 [91.4–100.0]	0.87 [0.77–0.96]
Oculoplastic specialist 2	82.1 [66.7–95.3]	76.4 [65.8–86.2]	0.68 [0.54–0.80]
Oculoplastic specialist 3	53.6 [35.0–72.4]	100.0 [100.0–100.0]	0.70 [0.52–0.84]
Average of oculoplastic specialists	73.8 [64.4–83.3]	90.7 [86.8–94.8]	0.75 [0.66–0.82]

Table 5 shows the performance of ophthalmologists when comparing their diagnostic results with the results of in-person CAS evaluation by one oculoplastic specialist. The average sensitivity was 68.4%, specificity was 82.3%, and the F1 score was 0.59 in three oculoplastic specialists. General ophthalmologists with <5 years of experience showed 63.2% of sensitivity, 77.0% of specificity, and an F1 score of 0.49. For ophthalmology residents, values were 68.4%, 52.7%, and 0.38.

Table 5.

Diagnostic Accuracy of Ophthalmologists with Varying Clinical Experience for Active TAO (CAS ≥3) Using In-Person CAS Evaluation as Reference

	Sensitivity, % [CI]	Specificity, % [CI]	F1 score [CI]
Resident 1	78.9 [58.8–95.0]	44.4 [34.2–54.7]	0.38 [0.25–0.51]
Resident 2	100.0 [100.0–100.0]	14.8 [7.5–22.5]	0.36 [0.25–0.46]
Resident 3	26.3 [8.3–47.1]	98.8 [96.2–100.0]	0.40 [0.15–0.62]
Average of residents	68.4 [57.8–78.4]	52.7 [47.6–57.5]	0.38 [0.26–0.48]
General ophthalmologist 1	31.6 [11.8–53.4]	96.3 [91.7–100]	0.43 [0.18–0.65]
General ophthalmologist 2	68.4 [47.6–88.9]	84.0 [75.3–91.0]	0.58 [0.40–0.74]
General ophthalmologist 3	89.5 [75.0–100]	50.6 [39.4–61.4]	0.45 [0.32–0.58]
Average of general ophthalmologists	63.2 [47.6–77.8]	77.0 [70.7–82.4]	0.49 [0.34–0.61]
Oculoplastic specialist 1	73.7 [52.4–92.9]	84.0 [76.8–92.2]	0.61 [0.44–0.77]
Oculoplastic specialist 2	68.4 [45.4–87.5]	66.7 [57.5–77.6]	0.44 [0.28–0.60]
Oculoplastic specialist 3	63.2 [40.0–84.2]	96.3 [91.7–100.0]	0.71 [0.50–0.56]
Average of oculoplastic specialists	68.4 [50.9–84.1]	82.3 [76.9–88.4]	0.59 [0.44–0.71]

Prediction of CAS

Tables 6 and 7 display the CAS prediction performance of smartphone-based ML-assisted model when using consensus results of photo-based CAS evaluation and in-person CAS evaluation as references, respectively. Using 1177 facial photos captured with all six smartphones, our ML-assisted system accurately predicted CAS within 1 point of the reference CAS in 90.7% with the consensus results of photo-based CAS evaluation serving as the reference (Table 6). When considering the results using in-person CAS evaluation as a reference, our system demonstrated a CAS prediction accuracy within 1 point of the reference CAS in 82.9% (Table 7).

Table 6.

Prediction of the CAS by the Smartphone-Based ML-Assisted Systems Using Consensus Results of Photo-Based CAS Evaluation as Reference

	Predicted CAS
		0	1	2	3	4	5	6	7
Reference CAS
	0	47	50	10	0	0	0	0	0
	1	62	163	130	35	5	1	0	0
	2	5	90	143	90	17	0	0	0
	3	0	12	79	85	35	10	1	0
	4	0	0	2	21	18	7	0	0
	5	0	0	1	10	21	13	2	0
	6	0	0	0	0	0	0	0	0
	7	0	0	0	0	0	0	11	1

All of 1177 test results from the smartphone-based ML-assisted systems using facial images captured by all 6 smartphones were presented. The predicted CAS was generated using the ML-assisted system, and the reference CAS was the consensus results of photo-based CAS evaluation from three oculoplastic specialists.

Table 7.

Prediction of the CAS by the Smartphone-Based ML-Assisted Systems Using In-Person CAS Evaluation as Reference

	Predicted CAS
		0	1	2	3	4	5	6	7
Reference CAS
	0	26	35	25	9	0	0	0	0
	1	77	200	129	59	10	2	0	0
	2	5	71	163	85	45	14	1	0
	3	6	9	37	65	17	4	0	0
	4	0	0	11	23	22	3	0	0
	5	0	0	0	0	2	8	13	1
	6	0	0	0	0	0	0	0	0
	7	0	0	0	0	0	0	0	0

All of 1177 test results from the smartphone-based ML-assisted systems using facial images captured by all 6 smartphones were presented. The predicted CAS was generated using the ML-assisted system, and the reference CAS was the results of in-person CAS evaluation from one oculoplastic specialist.

Reproducibility of the smartphone-based ML-assisted system

In this study, two photos were captured for each smartphone model. The average difference in CAS between the two photos was 0.48, and the diagnostic concordance for active TAO was 82.59%.

Quality analysis of the images from various smartphones

We used a face quality assessment algorithm (FQAA)¹⁰ to assess the quality of photos taken with each smartphone model and investigate the relationship between the quality of these photos and the performance of our ML-assisted system in detecting active TAO. Image quality was quantified as the average Mahalanobis distance between the parameter distributions, including lighting symmetry, pose symmetry, brightness, and contrast, in the original DSLR photos used in the previous study⁶ and those in images captured by each smartphone. We found that as this Mahalanobis distance of photos taken with each smartphone model decreases, the ML-assisted system analyzing photos from that particular smartphone model exhibits a higher F1 score in diagnostic performance (Table 8).

Table 8.

Quality of Facial Images Taken by Each Smartphone Model and Diagnostic Performance of Smartphone-Based ML-Assisted System Using Facial Images from Each Smartphone Model

	Mahalanobis distance	F1 score
Galaxy S21 Ultra	2.08	0.72
iPhone 12 pro	1.84	0.69
iPhone 11	2.24	0.68
iPhone SE 2020	2.29	0.65
Galaxy M20	3.55	0.63
Galaxy A21S	3.48	0.59

Mahalanobis distance was calculated between the parameter distributions, including lighting symmetry, pose symmetry, brightness, and contrast, in the original DSLR photos used in the previous study.⁶

DSLR, digital single-lens reflex.

Discussion

We demonstrated a smartphone-based ML-assisted system for evaluating CAS and detecting active TAO using selfie images captured by a smartphone camera. The accuracy of this solution surpassed that of ophthalmology residents and general ophthalmologists with <5 years of experience, and it was comparable with oculoplastic specialists with >15 years of expertise in TAO. Through this research, we have illustrated the feasibility of embedding the previously proposed ML-assisted diagnostic solution for active TAO onto smartphones, making it accessible to users. This finding serves as crucial evidence for the practical clinical application of this solution in the management of TAO.

The treatment approach for TAO is determined by evaluating its activity, severity, and duration. This is because of the reduced effectiveness of anti-inflammatory or immunosuppressive treatment in cases of inactive disease or disease lasting for ≥18 months and the treatment’s risks outweighing its benefits in patients with mild disease.^1,11,12 As a result, the primary focus of TAO treatment is on moderate to severe cases in the active phase. To ensure proper management, it is essential to detect patients with active TAO during their hyperthyroidism treatment. Consequently, endocrinologists play a pivotal role in diagnosing the condition, initiating initial treatment, and determining which patients should be referred to specialized care. Therefore, the recent joint Consensus Statement by the American and European Thyroid Association also stipulates that endocrinologists should be familiar with the basic elements of TAO assessment, enabling them to evaluate the activity and severity of TAO.¹³ Our ML-assisted system holds the potential to aid endocrinologists and general physicians involved in the management of Graves’ disease. Using smartphones without the need for additional medical equipment in clinical settings, our system can assess TAO activity by capturing facial photographs of patients. Moreover, this solution can be directly installed on patients’ personal smartphones, enabling them to continually assess their TAO activity even outside the clinic. This has the potential to reshape the paradigm of TAO management, providing a valuable tool for health care professionals and empowering patients in monitoring their condition. However, it is important to clarify that this ML-assisted system, validated for detecting active TAO in patients already diagnosed with TAO, should not be considered a diagnostic tool for TAO itself. Furthermore, given that the decision to initiate anti-inflammatory treatment in TAO should be based on both the activity and the severity of the condition, using the results obtained from our ML-assisted system as the sole criterion for initiating anti-inflammatory treatment without the assessment of an ophthalmologist would be inappropriate.

Our smartphone-based ML-assisted system determines the presence of active TAO based on 7-point CAS. It assesses five ocular inflammatory signs using selfie images captured by a smartphone and calculates a composite score incorporating two ocular pain symptoms input directly by the user. The 7-point CAS, which has been endorsed by both the EUGOGO and American Thyroid Association, serves as the most widely employed tool for measuring TAO activity.^11,13,14 In this system, a score of 3 or higher indicates the presence of active TAO.⁸ Nevertheless, recent studies have highlighted significant interobserver variability in the application of the CAS, implying that different observers may produce disparate results.⁵ We also demonstrated variability in CAS assessments based on the experience of practitioners. Given these challenges, our ML-assisted system can alleviate the limitations related to the consistency of conventional CAS evaluations. This is because our system emulates the assessments of three oculoplastic specialists with >15 years of experience, ensuring consistent performance.

In this study, a smartphone-based ML-assisted system analyzed photos taken with various smartphone cameras and evaluated the presence of active TAO. As the ML algorithm applied is consistent across smartphone models, the variation in active TAO detection performance among them is likely attributed to differences in photo quality. To address this, photos taken with each smartphone were assessed for quality using the FQAA¹⁰ based on four evaluation criteria: lighting symmetry, pose symmetry, brightness, and contrast. Results indicated that facial images captured by the top three smartphone cameras, which exhibited excellent predictive performance for active TAO, demonstrated a quality similar to that of facial images taken by a DSLR camera in a studio environment. These findings seem to reflect the fact that the ML-assisted system was developed based on results interpreted by oculoplastic specialists using facial images captured by a DSLR camera in a studio setting. These insights can serve as a foundation for establishing criteria for smartphone camera modules to which the ML-assisted system will be applied. In addition, it indicates the possibility of developing an AI module to improve image quality for analysis, further enhancing the system’s applicability.

This study holds the distinctive strength of being the first to apply a facial image-based AI model for evaluating CAS and detecting active TAO to user-accessible smart devices. While several studies have employed AI in TAO assessment thus far, most have used orbital images like CT and MRI,^{15

–22} restricting their application to the interpretation of these specific imaging modalities. To the best of our knowledge, there have been two published studies by Karlin et al.²³ and Huang et al.²⁴ that developed AI models for diagnosing or evaluating TAO using facial images. The former focused on diagnosing the presence of TAO based on facial disfiguration, while the latter assessed ocular dyskinesia and other specific signs of TAO. In comparison to the AI models proposed in these two studies, our AI model excels in assessing the inflammatory activity of TAO through CAS, enabling the early detection of active TAO.⁶ Moreover, this study validates the immediate suitability of our AI model for clinical use with suitable-performance smartphones.

This study also has some limitations that should be considered when interpreting the results. First, the usage of photo-based CAS evaluation by three oculoplastic specialists as the reference standard raises concerns about whether this system can have practical value in real-world clinical settings. In the development stages of our AI model, we chose the results evaluated by three oculoplastic specialists through photo assessments as the gold standard for training. As a result, the performance of our AI model is expected to be superior when the consensus results evaluated by three individuals through photo assessments are used as references. Evaluating the performance of our AI model using the results assessed by one expert during in-person examinations as references yielded a lower F1 score than when using the consensus results of three photo assessments as references. While it still demonstrated better performance than assessments made by ophthalmologists with <5 years of experience, the development and performance validation of the AI model using in-person CAS evaluation as the reference standard are crucial for enhancing the clinical value of this system in the future. Second, a small sample size of only 28 patients with active TAO is a major limitation. The sample size for this study was set to include the maximum number of patients recruitable during the research period. However, further studies are needed to include a larger number of patients with active TAO. Third, we analyzed facial images captured by six different smartphone models, but we only used products from two manufacturers. Future analyses should include a diverse range of photos from various smartphone models by different manufacturers to establish quality criteria for photos suitable for analysis. In addition, the observed variation in CAS assessment performance based on the smartphone models used for capturing facial images suggests the need for future research to evaluate performance using a variety of smartphone models and provide criteria for suitable photo quality for analysis. Fourth, our system tends to underscore the CAS in patients with highly active TAO. The higher the CAS in a patient, the more likely they are to exhibit inflammatory signs with lower prevalence. As our system is optimized for high specificity in low-prevalence CAS items, false-negative results for these items may result in underscoring in patients with high CAS. However, this tuning contributes to a more robust performance in detecting active TAO. Finally, this study was conducted exclusively in the Korean population, and given the nature of a solution that analyzes facial images, there is a high likelihood of interracial differences. Therefore, we are currently conducting subsequent studies in diverse ethnic groups.

In conclusion, this study demonstrated the feasibility of applying an ML-assisted system based on facial images and grounded in CAS for detecting active TAO on smartphones. Using smartphone-captured images, the system exhibited diagnostic performance comparable with those of oculoplastic specialists. This system may not only offer the convenience of usage within medical clinics by health care professionals but also empower TAO patients to monitor their own disease activity. However, it is crucial to emphasize that the present findings are preliminary, and future studies using a reference standard that enhances clinical relevance will be essential to provide a more robust foundation for the practical application of our ML-assisted system.

Footnotes

Acknowledgments

Authors thanks to clinical research coordinator Ji Yeon Kim at Seoul National University Bundang Hospital for her invaluable assistance for this research. IT Development Team of THYROSCOPE INC. is acknowledged for the development of the specialized research app.

Authors’ Contributions

K.S.: Writing—original draft (equal), analyzing and interpreting the data (equal), methodology (lead). H.C.: Writing—reviewing and editing (equal), analyzing and interpreting the data (equal). M.J.L.: Writing—review and editing (equal), analyzing and interpreting the data (equal). J.K.: Analyzing and interpreting the data (supporting), methodology (supporting). G.M.L.: Analyzing and interpreting the data (equal). S.K.: Analyzing and interpreting the data (equal). J.H.K.: Analyzing and interpreting the data (equal). R.O.: Analyzing and interpreting the data (equal). J.P.: Analyzing and interpreting the data (equal). S.M.L.: Analyzing and interpreting the data (equal). J.P.: Conceptualization (supporting), analyzing and interpreting the data (supporting). N.K.: Conceptualization (equal), collecting the data (lead); writing—reviewing and editing (equal). J.H.M.: Conceptualization (equal), collecting the data (supporting), writing—original draft (equal).

Author Disclosure Statement

THYROSCOPE Inc. has a patent for the ML-assisted system to predict the clinical activity of TAO. J.H.M. and J.P.: A stock owner of THYROSCOPE Inc. H.C., M.J.L., G.M.L., S.K., J.H.K., R.O., J.P., S.M.L., and N.K.: A member of the medical advisory board of THYROSCOPE Inc. K.S. and J.K.: No competing interests.

Funding Information

This research was supported by a grant of the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (Grant No. HI22C1078); a grant of the Tech Incubator Program for Startup through Korea Business Angels Association, funded by the Ministry of SMEs and Startups, Republic of Korea (Grant No. S3286890); and a grant of the Technological Innovation R&D Program, funded by the Ministry of SMEs and Startups, Republic of Korea (Grant No. RS-2023–00281098).

References

Bartalena

, Kahaly

, Baldeschi

, et al. The 2021 European Group on Graves’ orbitopathy (EUGOGO) clinical practice guidelines for the medical management of Graves’ orbitopathy. Eur J Endocrinol, 2021; 185(4):G43–G67; doi: 10.1530/EJE-21-0479

Gontarz-Nowak

, Szychlinska

, Matuszewski

, et al. Current knowledge on Graves’ orbitopathy. J Clin Med, 2020; 10(1):16; doi: 10.3390/jcm10010016

Dolman

. Evaluating Graves’ orbitopathy. Best Pract Res Clin Endocrinol Metab, 2012; 26(3):229–248; doi: 10.1016/j.beem.2011.11.007

Bartalena

, Baldeschi

, Dickinson

, et al. Consensus statement of the European group on Graves’ orbitopathy (EUGOGO) on management of Graves’ orbitopathy. Thyroid, 2008; 18(3):333–346; doi: 10.1089/thy.2007.0315

Perros

, Zarkovic

, Pearce

, et al. Inter-observer variability of clinical activity score: Assessments in patients with thyroid eye disease. Am J Ophthalmol, 2023; 252:94–100; doi: 10.1016/j.ajo.2023.03.027

Moon

, Shin

, Lee

, et al. Machine learning-assisted system using digital facial images to predict the clinical activity score in thyroid-associated orbitopathy. Sci Rep, 2022; 12(1):22085; doi: 10.1038/s41598-022-25887-8

Wiersinga

, Perros

, Kahaly

, et al. Clinical assessment of patients with Graves’ orbitopathy: The European Group on Graves’ Orbitopathy recommendations to generalists, specialists and clinical researchers. Eur J Endocrinol, 2006; 155(3):387–389; doi: 10.1530/eje.1.02230

Bartalena

, Baldeschi

, Dickinson

, et al. Consensus statement of the European Group on Graves’ orbitopathy (EUGOGO) on management of GO. Eur J Endocrinol, 2008; 158(3):273–285; doi: 10.1530/EJE-07-0666

Dickinson

, Perros

. Controversies in the clinical evaluation of active thyroid-associated orbitopathy: Use of a detailed protocol with comparative photographs for objective assessment. Clin Endocrinol (Oxf), 2001; 55(3):283–303; doi: 10.1046/j.1365-2265.2001.01349.x

10.

ISO/IEC JTC 1/SC 37 Biometrics. ISO/IEC TR 29794-5:2010 Information technology—Biometric sample quality—Part 5: Face image data. International Organization for Standardization: 2010.

11.

Bartalena

, Baldeschi

, Boboridis

, et al. The 2016 European Thyroid Association/European Group on Graves’ Orbitopathy Guidelines for the Management of Graves’ Orbitopathy. Eur Thyroid J, 2016; 5(1):9–26; doi: 10.1159/000443828

12.

Terwee

, Prummel

, Gerding

, et al. Measuring disease activity to predict therapeutic outcome in Graves’ ophthalmopathy. Clin Endocrinol (Oxf), 2005; 62(2):145–155; doi: 10.1111/j.1365-2265.2005.02186.x

13.

Burch

, Perros

, Bednarczuk

, et al. Management of thyroid eye disease: A consensus statement by the American Thyroid Association and the European Thyroid Association. Thyroid, 2022; 32(12):1439–1470; doi: 10.1089/thy.2022.0251

14.

Ross

, Burch

, Cooper

, et al. 2016 American Thyroid Association Guidelines for Diagnosis and Management of Hyperthyroidism and Other Causes of Thyrotoxicosis. Thyroid, 2016; 26(10):1343–1421; doi: 10.1089/thy.2016.0229

15.

Lin

, Song

, Li

, et al. Detection of active and inactive phases of thyroid-associated ophthalmopathy using deep convolutional neural network. BMC Ophthalmol, 2021; 21(1):39; doi: 10.1186/s12886-020-01783-5

16.

Song

, Liu

, Li

, et al. Artificial intelligence CT screening model for thyroid-associated ophthalmopathy and tests under clinical conditions. Int J Comput Assist Radiol Surg, 2021; 16(2):323–330; doi: 10.1007/s11548-020-02281-1

17.

Hanai

, Tabuchi

, Nagasato

, et al. Automated detection of enlarged extraocular muscle in Graves’ ophthalmopathy with computed tomography and deep neural network. Sci Rep, 2022; 12(1):16036; doi: 10.1038/s41598-022-20279-4

18.

Lee

, Seo

, Park

, et al. Neural network-based method for diagnosis and severity assessment of Graves’ orbitopathy using orbital computed tomography. Sci Rep, 2022; 12(1):12071; doi: 10.1038/s41598-022-16217-z

19.

, Li

, Liu

, et al. DMs-MAFM+EfficientNet: A hybrid model for predicting dysthyroid optic neuropathy. Med Biol Eng Comput, 2022; 60(11):3217–3230; doi: 10.1007/s11517-022-02663-4

20.

, Chen

, Zhang

, et al. T₂-weighted MR imaging-derived radiomics for pretreatment determination of therapeutic response to glucocorticoid in patients with thyroid-associated ophthalmopathy: Comparison with semiquantitative evaluation. J Magn Reson Imaging, 2022; 56(3):862–872; doi: 10.1002/jmri.28088

21.

Jiang

, Luo

, Wang

, et al. Evaluation on auto-segmentation of the clinical target volume (CTV) for Graves’ ophthalmopathy (GO) with a fully convolutional network (FCN) on CT images. Curr Med Imaging, 2021; 17(3):404–409; doi: 10.2174/1573405616666200910141323

22.

Jiang

, Zhou

, He

, et al. [Using stacked neural network to improve the auto-segmentation accuracy of Graves’ ophthalmopathy target volumes for radiotherapy]. Sheng Wu Yi Xue Gong Cheng Xue Za Zhi, 2020; 37(4):670–675; doi: 10.7507/1001-5515.202002025

23.

Karlin

, Gai

, LaPierre

, et al. Ensemble neural network model for detecting thyroid eye disease using external photographs. Br J Ophthalmol, 2022; 107(11):1722–1729; doi: 10.1136/bjo-2022-321833

24.

Huang

, Ju

, Li

, et al. An intelligent diagnostic system for thyroid-associated ophthalmopathy based on facial images. Front Med (Lausanne), 2022; 9:920716; doi: 10.3389/fmed.2022.920716