Automated Retinal Imaging System (ARIS) Compared with ETDRS Protocol Color Stereoscopic Retinal Photography to Assess Level of Diabetic Retinopathy

Abstract

Background:

Early Treatment Diabetic Retinopathy Study (ETDRS) seven-standard-field color stereoscopic retinal photography (ETDRS photos) has been a gold standard for determining diabetic retinopathy (DR) severity. The Automated Retinal Imaging System (ARIS™, model 110, Visual Pathways, Inc., Prescott, AZ) acquires seven-sequential color stereoscopic digital images (ARIS images) by a semiautomated technician-run process generally corresponding to ETDRS photos. We assessed the correlation between a single semiautomated ARIS imaging session without any re-imaging and ETDRS photos performed by a certified photographer for the determination of DR severity.

Methods:

Two independent masked readers graded mydriatic ARIS images and ETDRS photos. A third masked retinal specialist adjudicated discrepancies. Correlation between the two modalities was compared using weighted-κ statistics.

Results:

We evaluated 211 eyes of 106 patients with varying levels of DR. Partially ungradable images were present in 3.4% of ETDRS photos versus 31.8% of ARIS images. Exact agreement and agreement within one level between ETDRS photos and ARIS images using only completely gradable image sets occurred in 69% (κ=0.81) and 90% of cases, respectively. Exact agreement for clinically significant macular edema was 92.1% (κ=0.59). There was 100% agreement for eyes with high-risk proliferative DR. Within one level of DR severity, 100% agreement occurred for the following: questionable nonproliferative DR (NPDR), moderate NPDR, and severe NPDR.

Conclusions:

Results suggest that semiautomated ARIS images compare favorably with ETDRS photos when full image sets can be obtained; however, partially ungradable image sets occurred almost 10 times more frequently with ARIS images than with ETDRS photos. In the two-thirds of cases where ARIS images can be utilized, ARIS can obtain retinal images comparable to ETDRS photos while requiring less highly trained personnel than generally needed for standard ETDRS photos.

Introduction

A longstanding gold standard for determining severity of diabetic retinopathy (DR) is grading of Early Treatment Diabetic Retinopathy Study (ETDRS) seven-standard-field 35-mm stereoscopic color retinal images (ETDRS photos).^1,2 The ETDRS grading scale, an extension of the modified Airlie House classification, provides the basis for classifying and characterizing diabetic retinopathy.¹ Although other imaging modalities exist, the 35-mm color retinal slides provide high-resolution color images and low initial equipment cost and have been the established standard for more than 40 years.¹ Slides, however, have drawbacks, including cost, time delays with film handling and processing, physical storage space requirements, difficulty in sending or sharing acquired images, and, with the advent of digital photography, unavailability of 35-mm slide film and processing. These disadvantages can inhibit the flow of healthcare information and make it difficult for patients to provide future doctors with past images for comparison and documentation, necessitating repetition of retinal imaging.³ In addition, if the images are obtained for clinical trials, there are often additional technical requirements, including photographer certification, stereoscopic image acquisition, precise image field definitions, and image quality guidelines.

Because of these limitations of ETDRS photos, alternative methods of fundus photography have been investigated.^4,5 Digital imaging modalities, including digital photography, compare favorably with 35-mm slides and are used in clinical practice and in numerous clinical trials, including the Diabetic Retinopathy Clinical Research network and the Epidemiology of Diabetes Interventions and Complications studies.^4

–8 To date, these modalities include mydriatic and nonmydriatic retinal cameras, digital-video imaging, and other modifications to ETDRS standard photos, such as nonstereoscopic (monoscopic) images and the acquisition of fewer fields than the seven standard retinal fields.^{3,5,7,9
–11}

The Automated Retinal Imaging System (ARIS™, model 110,Visual Pathways, Inc., Prescott, AZ) is a digital imaging system that acquires seven sequential color stereoscopic digital images (ARIS images), approximating the ETDRS photo fields, by a semiautomated, technician-run process. Because of the automation of the process with ARIS images, only minimally trained technicians are required for acquisition as opposed to highly trained, certified retinal photographers. If accurate and effective, this semiautomated process may represent significant operational cost savings for clinical, clinical trial, and telemedicine programs.

ARIS acquires seven standard stereoscopic fields approximately as defined by the ETDRS protocol. However, the feasibility, effectiveness, and utility of the ARIS images have yet to be compared directly with those of ETDRS photos. The present study compares the agreement of these two imaging modalities with clinical DR severity and DR lesion-level findings.

Subjects and Methods

This single-center, masked, multireader study evaluated the agreement in assessing severity of DR at the retinal lesion level between ARIS images and ETDRS photos. Subject eligibility was determined from medical record review of the most recently diagnosed clinical level of DR. Based on the most recent clinical exam for DR, subjects were selected sequentially to ensure adequate distribution of various levels of DR, ranging from no DR (ETDRS level 10) to high-risk proliferative DR (PDR) (ETDRS level 75). At least 80% of subjects enrolled had documented DR on clinical examination, at least 50% had varying levels of nonproliferative DR (NPDR), and at least 30% had PDR. All patients were recruited during regularly scheduled appointments from the clinic population of the Beetham Eye Institute at the Joslin Diabetes Center, a tertiary eye care center specializing in eye disease due to diabetes, in Boston, MA. Inclusion criteria were age 18 years or older, type 1 or type 2 diabetes mellitus as defined by the American Diabetes Association, and willingness to sit through photography and ARIS imaging sessions. Patient exclusion criteria were a history of conditions in either eye that may preclude pupil dilation, use of eye drops (mydriatic or miotic) that would alter pupil size or reactivity, and prior laser treatment or media opacities precluding adequate imaging of the retina. The research was conducted according to the tenets of the Declaration of Helsinki and received Joslin Committee of Human Studies review, and patients consented to sit through photography and imaging sessions after explanation of the nature of the study.

Two hundred eleven eyes of 106 patients were evaluated in the study. Patients were recruited to ensure an adequate representation of the various levels of DR, ranging from no DR to high-risk PDR. The recruitment of patients was weighted towards those with mild to moderate levels of DR because more severe changes of DR would most likely be easier to detect. Therefore, detection of the more subtle changes of mild to moderate NPDR would provide a more rigorous test of lesion detection capability by the ARIS imaging system.

Each patient underwent a comprehensive eye exam including pupil dilation followed by a dilated retinal exam by a retina specialist. During the same visit, patients underwent imaging by an ophthalmic photographer certified for capturing ETDRS photos to obtain seven-standard-field stereo digital images of each eye. Mydriatic ARIS imaging was conducted by an ETDRS-certified photographer using the ARIS model 110 (2003), a semiautomated imaging system that has 26 internal fixation targets, seven of which are automatically and sequentially turned on during the preprogrammed ETDRS7 imaging procedure. For this study, ARIS images were taken in seven predetermined fields using the preprogrammed ETDRS7 imaging procedure. Multiple wavelengths that yield differential light penetration into the retina are used. Multilingual voice prompts in the system directed the patient to fixate for each of the seven fields. Of important note is that only one imaging attempt was made with the ARIS system, and no attempt was made to retake ARIS images of poor quality. Re-imaging of poor quality images with the ARIS system would disrupt the automated image capture and require a trained photographer to identify ungradable images, which is contrary to the purpose for which the system was designed. For each field, high-resolution digital stereo pairs of images were collected using single exposures of approximately 100 ms in duration. All ARIS digital images were stored electronically in a proprietary format that could be automatically exported using any of the common image formats. Color images were formed by combining the red and green wavelength images utilizing retinal feature registration and a proprietary method of generation and addition of a blue wavelength image.

After adequate pupil dilation, ARIS imaging was performed followed by seven-standard-field stereo ETDRS photography taken by a study-certified ETDRS photographer on 35-mm slide film. ETDRS photos were processed through the Beetham Eye Institute Photography Department according to standard clinical practice and ETDRS Reading Center protocol.

ARIS images were displayed and graded on a 21-inch Sony (Tokyo, Japan) Trinitron^® cathode ray tube color monitor at a resolution of 1,280×1,024×24 bits. Images were viewed monoscopically and stereoscopically and as “red free” images (digital filtering of red and blue channels). Stereoscopic viewing of ARIS images was achieved using liquid crystal display shuttered goggles (NuVision 60GX^®, MacNaughton, Inc., Beaverton, OR) where the left and right images of the stereo pair were sequentially displayed on the display monitor at 140 Hz, allowing a flicker-free three-dimensional image to be perceived. ARIS images were graded using two methods: Color Only (ColorOnly) and Color Plus (ColorPlus). Grading by the ColorOnly method involved grading the captured ARIS color images without any image adjustments made by the reader. The ColorPlus images were graded after the reader was free to adjust contrast, alignment, and color to assist in grading. Grading of 35-mm slide ETDRS images was performed on a standard slide light box through Donaldson stereo viewers according to ETDRS protocol.

Each eye was evaluated for the presence and severity of specific index retinal lesions and degree of diabetic macular edema according to the ETDRS extension of the Modified Airlie House Classification of Diabetic Retinopathy.¹ These index lesions were hemorrhages and/or microaneurysms (HMa), venous beading (VB), intraretinal microvascular abnormalities (IRMA), neovascularization elsewhere (NVE), neovascularization of the disc (NVD), lesions of diabetic macular edema and clinically significant macular edema (CSME), and hard exudates (HE). Each eye was assigned a clinical level of DR severity based on these findings.¹

One grader (J.D.C.) graded all ARIS images, and a second grader (T.J.M.) graded all ETDRS images. Grader disagreements were adjudicated by a third, independent retinal specialist (L.P.A.). All graders were masked to patient demographics, medical information, results of the other imaging modalities, and results from the clinical exam by a retina specialist. The two initial graders entered findings directly into a patient-specific, de-identified, electronic findings template modified from the Wisconsin Reading Center ETDRS retinal evaluation form.¹ The results were recorded on the electronic medical record templates by ETDRS photographic field, and these templates were electronically verified to ensure that there were no omissions or internal inconsistencies. A separate section was included on the template for identifying nondiabetic retinal lesions and for grader comments.

Statistical analysis

The adjudicated grades derived from the ETDRS images were considered the reference standards. The clinical ETDRS level of DR severity and the presence/absence of ETDRS index retinal lesions were compared. Agreement of clinical ETDRS level of DR severity between ARIS and ETDRS images was cross-tabulated, and both simple kappa (κ) and weighted κ statistics (κ _w) were calculated. A weight of 1 was used for exact agreement, disagreements between absence and definite presence of a characteristic received 0 weight, and those between absence and questionable presence (wherein the evaluator was between 50% and 90% certain of the presence of the characteristic) received a weight of 0.75. Eyes with photographs classified as ungradable were excluded from the analysis. Guidelines for interpretation were based on those of Landis and Koch¹² as used in ETDRS Report Number 10: 0.0–0.2=slight agreement, 0.21–0.40=fair agreement, 0.41–0.60=moderate agreement, 0.61–0.80=substantial agreement, and 0.81–1.00=almost perfect agreement. Severity of DR and ETDRS retinal lesion grading comparisons were tested using κ statistics for agreement, sensitivity/specificity, and positive/negative predictive values. All statistical analyses were performed using SAS (version 9.2) (SAS, Inc., Cary, NC) and the VassarStats Website for statistical computing (http://faculty.vassar.edu/lowry/VassarStats.html).

Results

Two hundred eleven eyes of 106 patients with diabetes mellitus were evaluated. One patient had a prosthetic eye. Baseline patient characteristics are shown in Table 1. DR severity and lesion prevalence derived from ETDRS photos are shown in Table 1 and Figure 1, respectively. The prevalence of DR and DR lesions did not reflect the prevalence of lesions or severity distribution of DR in the clinic, but rather was intentionally enriched with subjects having DR so as to more rigorously test the camera system. In this study only 11% of subjects enrolled did not have DR, in contrast to general clinic populations, which vary from 60% to 80% with no DR.

FIG. 1.

Diabetic retinopathy retinal lesions: 35-mm Early Treatment Diabetic Retinopathy Study photos compared with ARIS ColorPlus (n=211 eyes). CSME, clinical significant macular edema; HMA, hemorrhage/microaneurysm; HE, hard exudates; IRMA, intraretinal microvascular abnormalities; NVD, new vessels at the disc; NVE, new vessels elsewhere.

Table 1.

Baseline Demographic and Diabetic Retinopathy Severity Levels

Demographic or level	Value
Age (years)	49±15 (range, 19–78)
Gender (male:female)	52%/48%
Diabetes type (1/2)	57%/43%
Diabetes duration (years)	23.7±13.7 (range, 1–66)
Hemoglobin A1c	7.7±1.3% (range, 5.1–12%)
Diabetic retinopathy severity level on 35-mm ETDRS slides (n=211 eyes) [n (%)]
No diabetic retinopathy	23 (10.9)
Questionable NPDR	2 (2.4)
Mild NPDR	86 (40.8)
Moderate NPDR	18 (8.5)
Severe NPDR	2 (1.0)
Early PDR	7 (3.3)
PDR high risk	7 (3.3)
Quiescent PDR	51 (24.2)
Ungradable	12 (5.7)
Diabetic macular edema severity level on 35-mm ETDRS slides (n=211 eyes) ) [n (%)]
No diabetic macular edema	170 (80.6)
Macular edema questionable	12 (5.7)
Macular edema, not clinically significant	3 (1.4)
CSME, threatening center	1 (0.5)
CSME, involving center	6 (2.8)
Ungradable	19 (9.0)

CSME, clinical significant macular edema; ETDRS, Early Treatment Diabetic Retinopathy Study; NPDR, nonproliferative diabetic retinopathy; PDR, proliferative diabetic retinopathy.

Some images were ungradable for specific lesions in a particular field because of either poor overall image quality or inability to grade at least three disc areas of retina within a single field. These ungradable images occurred in at least one retinal quadrant in 3.4% of ETDRS as compared with 31.8% of ARIS ColorPlus images. Excluding the eye image sets containing ungradable images in one or more retinal quadrants, exact agreement for DR severity occurred in 68% of cases (95% confidence interval, 62–75%). As shown in Table 2, agreement within one level of clinical DR severity was observed in 88% of cases (95% confidence interval, 84–93%). Exact agreement occurred for all eyes with high-risk criteria. In addition, agreement within one level of DR severity occurred for all eyes with questionable NPDR, moderate NPDR, or severe NPDR on ETDRS photos.

Table 2.

Cross-Tabulation of Level of Diabetic Retiinopathy in ARIS ColorPlus and 35-mm Early Treatment Diabetic Retinopathy Study Photos

	Grading by 35-mm ETDRS photos
Grading by ARIS clinical photos	DR absent	Questionable NPDR	Mild NPDR	Moderate NPDR	Severe NPDR	PDR<high risk	High-risk PDR	Quiescent PDR	Ungradable	Total (%)
DR absent	18^a	3^b	1	0	0	0	0	0	0	22 (10)
Questionable NPDR	1^b	1^a	2^b	0	0	0	0	0	0	4 (2)
Mild NPDR	4	1^b	54^a	3^b	0	0	0	1	0	63 (30)
Moderate NPDR	0	0	23^b	8^a	1^b	2	0	2	0	36 (17)
Severe NPDR	0	0	3	4^b	1^a	0^b	0	0	1	9 (4)
PDR<high risk	0	0	0	0	0^b	4^a	0^b	2	0	6 (2)
High-risk PDR	7	0	0	0	0	1^b	6^a	1^b	0	15 (7)
Quiescent PDR	0	0	1	0	0	0	0^b	43^a	0^b	44 (21)
Ungradable	0	0	0	1	0	0	0	0^b	11^a	12 (6)
Total (%)	30 (15)	5 (2)	84 (4)	16 (8)	2 (1)	7 (3)	6 (3)	49 (23)	12 (6)	211 (100)

The measures graded here did not include photos that cannot be graded by ARIS and 35-mm ETDRS film. n (included)=198.

Simple κ statistic, 0.59 (95% confidence interval 0.50–0.67); weighted κ statistic (linear scale), 0.77 (95% confidence interval 0.70–0.83).

Perfect agreement: 68% (95% confidence interval 62–75%).

Within one-step agreement: 88% (95% confidence interval 84–93%).

DR, diabetic retinopathy; NPDR, nonproliferative diabetic retinopathy; PDR, proliferative diabetic retinopathy.

Table 3 and Figure 2 show the agreement of ARIS ColorPlus with ETDRS photos at different DR levels. Comparing eyes with no to mild NPDR with moderate NPDR or worse, ARIS photos showed agreement with ETDRS photos 92% and 81% of the time, respectively (substantial agreement, κ=0.65±0.08 and 0.62±0.05, respectively). Comparing eyes with very severe NPDR or better to eyes with PDR (including PDR with less than high-risk criteria, PDR with high-risk criteria, and quiescent PDR), ARIS photos showed near perfect agreement, matching ETDRS photos 93% of the time (κ=0.85±0.04). ARIS photos had a positive predictive value of 0.93 for ruling-in the presence of any DR. The negative predictive value of ARIS photos in ruling-out moderate NPDR or worse was 0.96.

FIG. 2.

Diabetic retinopathy threshold: 35-mm Early Treatment Diabetic Retinopathy Study photos compared with ARIS ColorPlus (n=198 eyes). NPDR, nonproliferative diabetic retinopathy; PDR, proliferative diabetic retinopathy.

Table 3.

Comparison by Different Diabetic Retinopathy Level for ARIS ColorPlus Compared with 35-mm Early Treatment Diabetic Retinopathy Study Photos (n=198)

DR level	Perfect agreement	Simple κ (mean±SE)	Sensitivity	Sensitivity 95% CI	Specificity	Specificity 95% CI	PPV	NPV
DR absent (n=30) vs. DR present (n=168)	92%	0.65 (0.08)	0.98	0.96–1.00	0.60	0.42–0.78	0.93	0.82
DR absent or mild NPDR (n=119) vs. moderate NPDR or worse (n=79)	81%	0.62 (0.05)	0.95	0.90–1.00	0.71	0.63–0.79	0.69	0.96
Very severe NPDR or better (n=136) vs. PDR (n=62)	93%	0.85 (0.04)	0.92	0.85–0.99	0.94	0.90–0.98	0.88	0.96

Eyes graded as questionable nonproliferative diabetic retinopathy (NPDR) were considered to have NPDR.

CI, confidence interval; DR, diabetic retinopathy; NPV, negative predictive value; PDR, proliferative diabetic retinopathy; PPV, positive predictive value.

Lesion-by-lesion comparisons of ARIS photos to ETDRS gold standard photos are shown in Figure 1 and Table 4. Agreement with ETDRS photos was 60–90% for both ColorPlus and ColorOnly. Correlations had substantial κ agreement for NVD (κ=0.69±0.1), moderate agreement for CSME (κ=0.59±0.12), HMa (κ=0.54±0.02), NVE (κ=0.56±0.05), and HE (κ=0.47±0.04), fair agreement for IRMA (κ=0.29±0.03), and slight agreement for VB (κ=0.18±0.04). ARIS ColorPlus detected key DR lesions as accurately as ETDRS film photos in 94% of cases of NVD, 92% of NVE, 92% of CSME, 87% of HE, 76% of IRMA, 77% of VB, and 62% of HMa (Fig. 1 and Table 4). ARIS photos demonstrated very high negative predictive values for nearly all lesions (mean, 0.93; range, 0.84–0.99 [see Table 4]). ColorPlus and ColorOnly grading had near perfect agreement for all lesions and thus did not add substantially to ColorOnly's ability to detect lesions or match ETDRS grading.

Table 4.

Comparison by Lesion over Multiple Fields for ARIS ColorPlus Compared with 35-mm Early Treatment Diabetic Retinopathy Study Photos (n=211 Eyes)

									κ_w
	% gradable fields with lesion	Perfect agreement	Sensitivity	Sensitivity 95% CI	Specificity	Specificity 95% CI	PPV	NPV	Mean±SE	95% CI
HMA	51%	62%	0.9	0.87–0.93	0.64	0.60–0.68	0.72	0.86	0.54 (0.02)	0.50–0.58
VB	10%	77%	0.45	0.35–0.55	0.82	0.80–0.85	0.23	0.93	0.18 (0.04)	0.12–0.26
IRMA	23%	76%	0.43	0.36–0.48	0.92	0.90–0.94	0.61	0.84	0.29 (0.03)	0.22–0.35
HE	10%	87%	0.73	0.65–0.82	0.9	0.88–0.92	0.43	0.97	0.47 (0.04)	0.40–0.54
NVE	8%	92%	0.52	0.41–0.63	0.98	0.97–0.99	0.66	0.96	0.56 (0.05)	0.47–0.65
NVD	5%	94%	0.78	0.51–1.05	0.97	0.96–1.00	0.58	0.99	0.69 (0.1)	0.51–0.88
CSME	9%	92%	0.47	0.21–0.72	0.98	0.96–1.00	0.7	0.95	0.59 (0.12)	0.35–0.82

CI, confidence interval; CSME, clinically significant macular edema; HE, hard exudate; HMA, hemorrhage or microaneurym; IRMA, intraretinal microvascular abnormality; κ _w, weighted kappa statistic; NPV, negative predictive value; NVD, neovascularization of the disc; NVE, neovascularization elsewhere; PPV, positive predictive value; VB, venous beading.

Discussion

This was a single-center, masked instrument, category 3 validation study that adhered to the recommendations of the American Telemedicine Association guidelines for telehealth programs for diabetic retinopathy.¹³ A category 3 American Telemedicine Association validation study demonstrates the capability to match dilated ETDRS photos for the evaluation of DR. All persons with diabetes may potentially benefit from such an evaluation, and this study design and recruitment were tailored to potentially assess performance in this diverse population.

In this study, when gradable images are obtained, the ARIS imaging system produced results similar to ETDRS photos for the evaluation of clinical DR severity. ARIS images also showed high negative predictive values in ruling out DR. The agreement obtained with this automated imaging system was comparable to those from previous reports using nonautomated technician-driven digital imaging systems.¹⁴ However, as discussed below, the substantial percentage of ungradable images may affect efficiency and performance of the current ARIS systems in larger teleophthalmology implementations.

Compared with ETDRS photos, the ARIS system had nearly 10 times more ungradable retinal fields, and ungradable fields approached one-third of all ARIS images (3.4% ETDRS, 31.8% ARIS). This high percentage of ungradable images could clearly impact efficiency and usefulness of the ARIS system in clinical or research settings. However, when evaluating gradable fields present for each eye, DR severity could be determined by the ARIS system in 93.4% of eyes, which is similar to that reported in other teleophthalmology systems.^15,16

When comparing ARIS ColorPlus with ETDRS photos in lesion-by-lesion analyses, ARIS ColorPlus detected ETDRS retinopathy lesions to a similar extent as ETDRS photos in the majority of the cases. More than 90% of NVD and NVE was detected by ARIS. ARIS ColorPlus and ETDRS photos showed a high level of agreement with respect to clinically significant macular edema at 92% (κ=0.59±0.12). The sensitivity and specificity for detecting sight-threatening PDR were 92% and 94%, respectively. The ungradable/technical failure rate with ARIS images was 6.6%. The ARIS system compares favorably with the Australian National Health and Medical Research Council guidelines (approximately 60% and 90–95%) and the United Kingdom's Exeter Standards (approximately 80% and approximately 95%) in terms of sensitivity and specificity, respectively.

The lesions having the lowest κ values were VB (0.18±0.04) and IRMA (0.29±0.03), which also were among the least prevalent lesions at 10% and 23%, respectively. Other lesions had a high percentage of exact agreement (VB=77%, IRMA=77%) and high negative predictive values (VB=0.93, IRMA=0.84). If these lesions were more prevalent, it is possible that a higher κ value may have been observed as has been previously described.^17,18

Seven-standard-field ETDRS 35-mm stereoscopic fundus photography has been the gold standard for evaluating DR severity; however, obtaining ETDRS photos has certain drawbacks, including cost and the requirement for certified photographers. Previous studies have investigated alternatives to using standard ETDRS photography, including using monoscopic digital imaging and photographing fewer than seven fields. Although these methods have been shown to be time saving, they may be associated with decreased specificity for assessing certain lesions such as retinal thickening and elevated neovascularization.^19,20 The key advantage of the ARIS system is that it does not need a skilled photographer to image a retinal area equivalent to ETDRS photos. Based on the results of this study, the ARIS system compares favorably with ETDRS photos for evaluating DR when gradable images are obtained. The ARIS system replaces the variability that may arise with multiple operators with a standardized automated protocol that can achieve results consistent with the recommendations of established standards and guidelines.

Results using ARIS, when gradable, compared favorably with mydriatic ETDRS film photography obtained by certified personnel for determining clinical DR severity and individual DR lesions. These findings suggest that ARIS could provide a less labor-intensive and lower-cost alternative to ophthalmic photographer–obtained ETDRS protocol imaging, especially if a higher percentage of gradable images could be obtained across a broader population. Individual lesion identification in ARIS images had both a high specificity and high negative predictive value, indicating lesions graded as “not present” by ARIS were unlikely to have been missed. These results suggest that when the extent of ungradable images is not a limiting factor, ARIS images may be useful for determining retinopathy level as well as the presence and severity of individual ETDRS retinal lesions.

The ARIS is a semiautomated system that does not require significant operator input. Although only one operator used the ARIS in this study and the impact of multiple operators was not explicitly evaluated, substantial intra-operator variability is not expected based on the automated operation of this machine. The semiautomation of this retinal imaging process may improve access to retinal imaging because of less training requirements and may optimize workflow, thus enhancing efficiency. However, the ARIS imaging system could be optimized by substantially reducing the number of ungradable fields. Additional future innovations such as applying automated software algorithms and computer lesion detection might be applied to this technology once this hurdle is overcome. Such features combined with the limited expertise needed to run the system would be a significant benefit to many teleophthalmology programs. If future generation imaging systems incorporate a similar semiautomated process with enhanced image quality and gradability, they would likely have a significant positive impact on clinical care.

With ungradable image rates of over 30%, there may be a significant impact on efficiency and usefulness of the ARIS system. However, despite nearly 10 times more ungradable fields in ARIS images as compared with ETDRS photos (31.8% vs. 3.4%), the overall ungradable rate for an eye imaged with ARIS was 6.6%. The better correlation at the eye level is likely due to the acquisition of multiple retinal fields as is done with ETDRS photos. Unlike most teleophthalmology programs, which take one or two images of the disc and macula, the ARIS acquires multiple retinal images, including the optic disc, macula, temporal macula, and four peripheral retinal fields. Such coverage may lead to substantial agreement with a low overall ungradable rate despite having a substantial number of ungradable individual fields. Imaging of the retinal periphery is more technically demanding, and to achieve consistent results often requires a skilled retinal photographer. The ARIS acquires multiple retinal images that include the optic disc, macula, temporal macula, and four peripheral retinal fields without the need for a skilled photographer. The addition of the peripheral retinal fields compared with the usual one- or two-field retinal imaging results in a sevenfold increase in the number (1–2 vs. 14) and two- to threefold increase of retinal area (30–45° vs. 90°) that may be evaluated for diabetes pathology. Despite the number of ungradable individual fields with ARIS, there is substantial agreement with ETDRS photos and low ungradable rate for eyes overall.

In summary, when gradable images can be acquired using ARIS, they are comparable to ETDRS photos for determining clinical severity of DR based on individual ETDRS retinal lesions. Accuracy increases with more severe DR. Although this tool has the potential to reduce the need for a highly trained ophthalmic photographer to obtain the images and may be more time efficient, these benefits must be considered in light of nearly 30% of images being ungradable and a loss of sensitivity for VB and IRMA. Retinal imaging systems that automate the acquisition of multiple retinal fields may eventually prove to be a useful alternative to the labor- and resource-intensive ETDRS photography in large telemedicine programs where automation of workflow is becoming essential.

Footnotes

Author Disclosure Statement

No competing financial interests exist.

References

Grading diabetic retinopathy from stereoscopic color fundus photographs—an extension of the modified Airlie House classification. ETDRS report number 10. Early Treatment Diabetic Retinopathy Study Research Group. Ophthalmology, 1991; 98:786–806.

Fundus photographic risk factors for progression of diabetic retinopathy. ETDRS report number 12. Early Treatment Diabetic Retinopathy Study Research Group. Ophthalmology, 1991; 98:823–833.

Tennant

, Greve

, Rudnisky

, Hillson

, Hinz

. Identification of diabetic retinopathy by stereoscopic digital imaging via teleophthalmology: a comparison to slide film. Can J Ophthalmol, 2001; 36:187–196.

Ahmed

, Ward

, Bursell

, Aiello

, Cavallerano

, Vigersky

. The sensitivity and specificity of nonmydriatic digital stereoscopic retinal imaging in detecting diabetic retinopathy. Diabetes Care, 2006; 29:2205–2209.

Vujosevic

, Benetti

, Massignan

, Pilotto

, Varano

, Cavarzeran

, Avogaro

, Midena

. Screening for diabetic retinopathy: 1 and 3 nonmydriatic 45-degree digital fundus photographs vs 7 standard early treatment diabetic retinopathy study fields. Am J Ophthalmol, 2009; 148:111–118.

Chow

, Aiello

, Cavallerano

, Katalinic

, Hock

, Tolson

, Kirby

, Bursell

, Aiello

. Comparison of nonmydriatic digital retinal imaging versus dilated ophthalmic examination for nondiabetic eye disease in persons with diabetes. Ophthalmology, 2006; 113:833–840.

Cavallerano

, Aiello

, Cavallerano

, Katalinic

, Hock

, Kirby

, Aiello

. Joslin Vision Network Clinical Team: Nonmydriatic digital imaging alternative for annual retinal examination in persons with previously documented no or mild diabetic retinopathy. Am J Ophthalmol, 2005; 140:667–673.

, Hubbard

, Danis

, Esquivel

, Florez-Arango

, Krupinski

. Monoscopic versus stereoscopic retinal photography for grading diabetic retinopathy severity. Invest Ophthalmol Vis Sci, 2010; 51:3184–3192.

Nayak

, Bhat

, Acharya

, Lim

, Kagathi

. Automated identification of diabetic retinopathy stages using digital fundus images. J Med Syst, 2008; 32:107–115.

10.

Rudnisky

, Tennant

, Weis

, Ting

, Hinz

, Greve

. Web-based grading of compressed stereoscopic digital photography versus standard slide film photography for the diagnosis of diabetic retinopathy. Ophthalmology, 2007; 114:1748–1754.

11.

Massin

, Erginay

, Ben Mehidi

, Vicaut

, Quentel

, Victor

, Marre

, Guillausseau

, Gaudric

. Evaluation of a new non-mydriatic digital camera for detection of diabetic retinopathy. Diabet Med, 2003; 20:635–641.

12.

Landis

, Koch

. The measurement of observer agreement for categorical data. Biometrics, 1977; 33:159–174.

13.

, Horton

, Bursell

, Cavallerano

, Zimmer-Galler

, Tennant

, Abramoff

, Chaum

, Debuc

, Leonard-Martin

, Winchester

. American Telemedicine Association Diabetic Retinopathy Telehealth Practice Recommendations Working Group: Telehealth practice recommendations for diabetic retinopathy, second edition. Telemed J E Health, 2011; 17:814–837.

14.

Cavallerano

, Cavallerano

, Katalinic

, Blake

, Rynne

, Conlin

, Hock

, Tolson

, Aiello

. Joslin Vision Network Research Team: A telemedicine program for diabetic retinopathy in a Veterans Affairs Medical Center—the Joslin Vision Network Eye Health Care Model. Am J Ophthalmol, 2005; 139:597–604.

15.

Choremis

, Chow

. Use of telemedicine in screening for diabetic retinopathy. Can J Ophthalmol, 2003; 38:575–579.

16.

Higgs

, Harney

, Kelleher

, Reckless

. Detection of diabetic retinopathy in the community using a non-mydriatic camera. Diabet Med, 1991; 8:551–555.

17.

Cicchetti

, Feinstein

. High agreement but low kappa: II. Resolving the paradoxes. J Clin Epidemiol, 1990; 43:551–558.

18.

Bursell

, Cavallerano

, Clermont

, Birkmire-Peters

, Aiello

. Joslin Vision Network Research Team: Stereo nonmydriatic digital-video color retinal imaging compared with Early Treatment Diabetic Retinopathy Study seven standard field 35-mm stereo color photos for determining level of diabetic retinopathy. Ophthalmology, 2001; 108:572–585.

19.

Rudnisky

, Tennant

, de Leon

, Hinz

, Greve

. Benefits of stereopsis when identifying clinically significant macular edema via teleophthalmology. Can J Ophthalmol, 2006; 41:727–732.

20.

Bresnick

, Mukamel

, Dickinson

, Cole

. A screening approach to the surveillance of patients with diabetes for the presence of vision-threatening retinopathy. Ophthalmology, 2000; 107:19–24.