Abstract
Background:
Diabetic retinopathy (DR) is a leading cause of low vision and blindness. We evaluated the feasibility of using a handheld, noncontact digital retinal camera, Pictor, to obtain retinal images in dilated and undilated eyes for DR screening. We also evaluated the accuracy of ophthalmologists with different levels of training/experience in grading these images to identify eyes with vision-threatening DR.
Methods:
A prospective study of diabetic adults scheduled to have dilated eye exams at Duke Eye Center from January to May 2014 was conducted. An imager acquired retinal images pre- and postdilation with Pictor and selected 1 pre- and 1 postdilation image per eye. Five masked ophthalmologists graded images for gradability (based on image focus and centration) and the presence of no, mild, moderate, or severe nonproliferative DR (NPDR) or proliferative DR (PDR). Referable disease was defined as moderate or severe NPDR or PDR on image grading. We evaluated feasibility based on the graders’ evaluation of image gradability. We evaluated accuracy of identifying vision-threatening disease (severe NPDR or PDR documented on dilated clinical examination) based on the graders’ sensitivity and specificity of grading referable disease.
Results:
Images were gradable in 86-94% of predilation and 94-97% of postdilation photos. Compared to the dilated clinical exam, overall sensitivity for identifying vision-threatening DR was 64-88% and specificity was 71-90%.
Conclusions:
Pictor can capture retinal images of sufficient quality to screen for DR with and without dilation. Single retinal images obtained using Pictor can identify eyes with vision-threatening DR with high sensitivity and acceptable specificity compared to clinical exam.
Diabetic retinopathy (DR) is the leading cause of new blindness in adults 20-74 years of age. 1 It has been estimated that 95% of subjects with DR can avoid vision loss if referred in time for treatment. 2 While it has been recommended that all patients with diabetes should have at least annual dilated retinal exams, 3 it is estimated that fewer than 50% of these patients have an annual eye exam and only about 60% receive vision-saving treatments.4,5 Missing these patients is largely due to the lack of effective screening strategies that enable early detection, especially in underserved locations and populations that lack access to care. 4
The ideal DR screening strategy remains unclear but with advancements in ophthalmic photography and communication technologies, there has been growing interest in telemedicine.4,6,7 While in-person clinical exam by an eye care provider is the gold standard for diagnosing DR, the standard for photographic detection and classification of DR is mydriatic (dilated) 7-standard field stereoscopic 35-mm color 30-degree retinal photographs that arose from the Early Treatment Diabetic Retinopathy Study (ETDRS).8,9 The ETDRS used retinal tabletop camera systems, but these machines are large and not easily portable. With large populations needing to be screened, both ease of use and camera portability are important considerations.
One study evaluated photographs for image quality and retinal findings using the handheld NM-100 camera (Nidek Inc, Gamagori, Japan) that attached to a large tabletop power unit weighing nearly 40 pounds. 10 They found that the quality of digital images from the Nidek camera was not suitable for the diagnosis of DR. Since then, there has been an advent of smaller cameras, including the use of mobile phones, which can obtain high quality retinal photographs.11-13 Current smartphone-based devices range from the D-eye (20° field of view) to the Ocular CellScope (55° field of view).11,14 Recently, 1 study compared single-photo smartphone ophthalmoscopy (D-eye device) with slit-lamp biomicroscopy following dilation; the authors reported agreement between the 2 methods in 85% of eyes. 14 However, no published study has investigated the ability of a portable handheld retinal camera to screen for DR in undilated eyes compared to the dilated clinical examination.
A new FDA-approved portable camera that has shown promise in obtaining high-quality retinal images is Pictor (aka SmartScope, Volk Optical Inc, Mentor, OH) (Figures 1a-1b). It weighs 450 mg and is handheld, portable, noncontact, and nonmydriatic. It captures JPEG digital images with a 45° field of view and 1920 × 1440 image resolution. It can acquire both color and red-free retinal photos simultaneously (Figures 1c-1d) and has shown promise as a portable handheld retinopathy of prematurity screening tool. 15
Images of the Pictor camera in the photographer’s hand (side view) (a) and from the photographer’s perspective (live-view monitor) (b) and example of a slide set composed of the color (c) and corresponding red-free (d) retinal images of a left eye taken with the Pictor camera showing good focus, centration of the macula, and the optic nerve and temporal vascular arcades in view.
The purpose of this prospective study was to evaluate the feasibility of using Pictor as a screening tool to obtain retinal images in both dilated and undilated eyes and the accuracy of ophthalmologists at different levels of training/experience in grading these images for referable disease to identify the presence of vision-threatening DR.
Patients and Methods
This study received institutional review board approval from the Duke University Health System and complied with the Health Insurance Portability and Accountability Act of 1996. Written informed consent was obtained from all participants.
Eligibility
We serially enrolled all eligible subjects (diabetic patients ≥18 years old) undergoing a dilated eye examination by a board-certified ophthalmologist at Duke Eye Center from January to May 2014.
Sample Size Calculation
Prior to the commencement of this study, a sample size calculation indicated that to appropriately power our study to detect a sensitivity of 0.85 (95% CI, 0.75-0.95), a sample size of 51 subjects was required.
Procedures
The imager was a first-year ophthalmology resident (WZ) who was trained to use the Pictor by reading the user’s manual and practicing on un-dilated adult volunteers. The imager was masked to the clinical examination findings. Before dilation, the imager used Pictor to acquire retinal images of both eyes. Following dilation, the subject underwent their routine clinical eye examination, followed by postdilation imaging with Pictor. During imaging, the imager attempted to obtain a focused retinal image centered on the macula with the optic nerve and superior and inferior vascular arcades in view (Figure 1c). The imager recorded the number of photos acquired per eye and the average image acquisition time needed to satisfy the imaging goal. The camera was set to save the color and corresponding red-free images.
Image Grading
After images were obtained, the imager selected the best pre- and postdilation image pairs (an image pair comprised of a color image and its corresponding red-free retinal image) based on macula-centration and focus for each imaged eye. An electronic slideshow was compiled of these image pairs where each slide showed a single image. The image pairs were shown on consecutive slides, showing the color image first (Figure 1c), followed by its corresponding red-free image for grading (Figure 1d). Thus, up to 4 image pairs (ie, 8 slides) were included for each patient—1 of the right eye predilation and postdilation, and similarly for the left eye. Right and left eyes were separated in the slideshow so that each eye was graded independently.
Pictor image grading was performed by 5 ophthalmologists with different levels of training/ experience (1 general ophthalmologist [PN], 3 medical retina fellows [MJA, AF, TS], and 1 medical retina attending [SGS]) masked to demographic and clinical information. Grading was performed independently. Graders were introduced to the grading system prior to grading the electronic slideshow. Graders rated image pairs as gradable or not based on their ability to assess the level of DR. If an image was considered gradable, the grader then graded the level of DR as none, mild, moderate, or severe nonproliferative diabetic retinopathy (NPDR), or proliferative diabetic retinopathy (PDR), according to ETDRS classification. 9 Lastly, graders were asked to comment on the helpfulness of the red-free photo in grading the image pairs.
Statistical Analysis
SAS 9.3 (SAS Institute Inc, Cary, NC) was used for all statistical analysis. For assessment of accuracy of Pictor image grading, the reference standard was the dilated clinical examination findings from the same day as Pictor imaging. In a DR screening program, it is important to identify cases that have vision-threatening DR as these eyes have a high risk for morbidity. Eyes with either PDR or severe NPDR on clinical exam were considered to have “vision-threatening DR” because PDR can cause vision loss from sequelae of ischemia-induced neovascularization, and patients with severe NPDR have a high likelihood of converting to PDR within a short time frame. 3 Those with no, mild, or moderate NPDR on clinical exam were considered not to have vision-threatening DR. To assess our grading accuracy, Pictor images were graded for “referable disease” (moderate or severe NPDR, or PDR) versus “nonreferable disease” (no or mild NPDR) by our graders. The sensitivity and specificity of each grader’s evaluation of referable disease present in Pictor images was calculated against the reference standard of the dilated clinical exam findings to identify vision-threatening DR eyes.
Analysis was performed for all images. Any ungradable images were grouped with “referable disease,” as these eyes would have failed image screening. Analysis was carried out separately for images acquired predilation and postdilation. As most patients have 2 eyes, we also calculated each grader’s sensitivity and specificity of grading for referable disease by patient, by linking the grading for both eyes together so that if either eye was graded to have “referable disease,” the patient would fail the screening test and require a standard dilated clinic examination. For this part of the analysis, the single monocular patient was excluded as it was felt that any monocular patient should be automatically referred for a dilated clinical eye exam.
Results
Fifty-eight diabetic subjects were enrolled, with 56 subjects (n = 111 eyes) included in our study (Figure 2). One subject only had 1 eye, due to a history of enucleation of the other eye following trauma. On average, 7 ± 2 images were taken of each eye predilation and 7 ± 3 images postdilation by the imager to satisfy imaging goals (P = .6). The average time for image acquisition was 5 ± 3 minutes for undilated eyes (range: 1-16 minutes) and 3 ± 2 minutes for dilated eyes (range: 1-7 minutes) (P < .01). Of the 214 (109 predilation and 105 postdilation) image pairs, graders rated images as gradable in 86-94% of predilation and 94-97% of postdilation photos. Of ungradable images, the majority (67-100%) belonged to eyes with vision-threatening DR (Table 1).
Demographics of study population and information of acquired Pictor images.
Number of Ungradable Pictor Images (n) According to Each Grader and the Corresponding Clinical Examination Diagnoses (Dx) for All Predilation and Postdilation Images.
NPDR, nonproliferative diabetic retinopathy; PDR, proliferative diabetic retinopathy. Bolded values highlight the ungradable images that have vision-threatening DR (severe NPDR and PDR) on clinical examination.
General ophthalmologist grader.
Medical retina fellow with 11 months of fellowship training.
Medical retina attending.
On dilated clinical examination (our reference standard) of the 111 eyes screened on the same date as Pictor imaging, 22 eyes (20%) had no DR, 14 eyes (13%) had mild NPDR, 29 eyes (26%) had moderate NPDR, 7 eyes (6%) had severe NPDR, and 39 eyes (35%) had PDR. Thus, there were 46 eyes with vision-threatening DR, and 65 eyes without.
Considering each eye independently, graders had a sensitivity of 64-88% in predilation and 65-87% in postdilation images, and a specificity of 72-84% in predilation and 71-90% in postdilation images for identifying vision-threatening DR eyes by grading images for referable disease (Table 2). Considering both eyes together, graders had a sensitivity of 91-100% and specificity of 38-81% in predilation images, and sensitivity of 85-100% and specificity of 53-84% in postdilation images for identifying patients with vision-threatening DR by grading for referable disease (Table 2).
Sensitivity (Sn) and Specificity (Sp) of Each Grader for Correctly Grading for Referable Disease (Moderate or Severe NPDR, PDR, or an Ungradable Image) Compared to the Clinical Exam Diagnosis of Vision-Threatening Diabetic Retinopathy (Defined as Severe NPDR and PDR on Dilated Clinical Exam) by Eye and by Patient.
NPDR, nonproliferative diabetic retinopathy; PDR, proliferative diabetic retinopathy.
Images of each eye were evaluated separately.
Patients were grouped according to the eye with worse disease.
General ophthalmologist grader.
Medical retina fellow with 11 months of fellowship training.
Medical retina attending.
All 5 graders found that displaying the corresponding red-free images with the color-images was useful in over 80% of postdilation images, while 4 of 5 graders found the red-free image useful in evaluating predilation images.
One grader (SGS, grader 5) also performed the reference dilated clinic examination on 14 study subjects. She was asked to recuse herself from grading any images she recognized. She did not recognize any of the subjects while grading images.
Discussion
We found that Pictor could take retinal images of sufficient quality to screen for DR through dilated and nondilated pupils. While a single Pictor image has a narrow field-of-view (45°), it was sufficient to screen for DR with high sensitivity.
Logistically, Pictor offers qualities that make it a versatile and effective screening tool. It is lightweight (~1 pound) and transportable in a bag or briefcase. The number of images it took to satisfy our imaging goal was similar between undilated and dilated eyes. However, image acquisition time was faster for dilated eyes. Images obtained from both undilated and dilated eyes could be graded with high sensitivities and specificities (Table 2) suggesting that if Pictor is used to screen for DR, imaging could occur primarily through undilated eyes. If images are unable to be captured through any undilated eyes, then dilation could be used in just this subset of eyes. This paradigm would reduce the time, expense, and patient discomfort associated with having their eyes dilated. Also, Pictor can simultaneously acquire color and red-free images (Figures 1c-1d), which our graders felt was helpful in grading for DR as red-free images emphasize the presence of heme and blood vessel pathology—important features of DR. Notably, of those retinal images that the graders found ungradable, the majority of those belonged to eyes with vision-threatening DR (Table 1). In our study, subjects with poor quality images were likely to have vision-threatening disease, fail the screening test, and require a standard diagnostic clinical examination.
Studies have shown that nonstereoscopic DR screening protocols using fewer than 7 fields-of-view may be a feasible way of detecting referable disease.4,16-22 Single-field retinal photography has shown sensitivities of 38-100% and specificities of 75-100% for detecting vision-threatening DR compared with dilated ophthalmoscopy, and sensitivities of 61-90% and specificities of 85-97% for detecting vision-threatening DR when compared with the 7-standard fields of view from the ETDRS (Table 3).17-19 Our results suggest that single retinal images obtained using Pictor can identify eyes with vision-threatening DR with high sensitivity and specificity comparable to results published from other 1-field DR retinal imaging studies using tabletop cameras (Table 3).
Comparison of Diabetic Retinopathy Screening Studies Using 1-Field Imaging.
7-sf, 7 standard fields; ETDRS, Early Treatment of Diabetic Retinopathy Study; NPDR, nonproliferative diabetic retinopathy; PDR, proliferative diabetic retinopathy; Sn, sensitivity; Sp, specificity.
Sensitivity and specificity of identifying vision-threatening DR for each eye in this study.
Sensitivity and specificity of identifying vision-threatening DR by patient (paired eyes) in this study.
A DR screening program should identify disease with high risk of leading to vision loss so that appropriate referral can be made for examination and possible treatment, but not include a high number of false positives leading to unnecessary referrals. The British Diabetic Association (now Diabetes UK) suggested a required screening standard for DR of at least 80% sensitivity and 95% specificity. 23 In our study, looking at patients as a whole (taking both eyes into account), all graders had sensitivities >80% (85-100%) (Table 2). While specificity values for our graders did not meet the 95% goal, we believe that with formal training, both the sensitivities and specificities of screening for vision-threatening DR using our system can be increased.
The sensitivity and specificity of identifying vision-threatening DR for each eye varied depending on levels of training/experience of our 5 graders. Ophthalmologists with medical retinal training had higher sensitivities compared to the ophthalmologist without medical retinal training in identifying eyes with vision-threatening DR. With regard to specificity, the comprehensive ophthalmologist had the highest specificity values, followed by medical retina fellows-in-training, and by the medical retina attending. In our study, our graders did not undergo formal training for image grading for screening purposes given their ophthalmology background. We postulate our findings are a reflection of the bias that each grader may have based on clinical experience where those who see more significant disease may have a tendency to grade disease as more severe, resulting in lower specificity. While it has been suggested that even nonphysician graders are able to provide good detection of DR from retinal photographs with appropriate training, 24 our study highlights the need for formal and validated training protocols for graders in screening programs.
Our study has limitations. Our study sample may not be representative of all patients with diabetes as our population was older, skewed toward patients with more significant disease, and included patients who were not treatment-naïve. While our sample size was small, it was appropriately powered to detect a sensitivity of 85%. We did not record the racial and ethnic makeup of our population because we included all eligible subjects regardless of their race or ethnicity. Furthermore, we did not formally train our graders on how to grade images for referable DR disease. While this study evaluated grading and identifying DR, we did not evaluate screening for clinically significant macular edema (CSME), which can result in vision impairment in the diabetic population. Further studies are merited to investigate the ability to screen for CSME and strategies to increase both sensitivity and specificity by evaluating the ability and utility of imaging more than a single field of view and developing a way to systematically train graders to accurately identify referable disease or apply computer-automated grading assessments. The cost effectiveness and patient satisfaction of using a nonmydriatic screening tool (like Pictor) compared to an annual dilated clinic examination by an eye care provider should also be explored.
Conclusions
In conclusion, Pictor shows promise as a truly portable screening tool for DR. Using single-field retinal photos centered on the macula, the images obtained using Pictor were of sufficient quality and could be graded to identify patients at risk for vision-threatening DR with high sensitivity. A DR screening paradigm involving the use of a truly portable, noncontact, nonmydriatic retinal camera that does not require pupillary dilation on every patient could not only reduce the discomfort, time, and expense involved in screening, but also increase patient satisfaction. This may encourage more patients to get their eyes screened for DR annually and in turn decrease the incidence of vision loss due to undiagnosed and untreated DR.
Footnotes
Abbreviations
CSME, clinically significant macular edema; DR, Diabetic retinopathy; ETDRS, Early Treatment Diabetic Retinopathy Study; NPDR, nonproliferative diabetic retinopathy; PDR, proliferative diabetic retinopathy.
Declaration of Conflicting Interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: The authors have completed the ICMJE Form for Disclosure of Potential Conflicts of Interest. For disclosures reported by SGP, PN, and SWC, the funding organization had no role in the design or conduct of the study; collection, management, analysis, or interpretation of the data; preparation, review, or approval of the manuscript; or the decision to submit the manuscript for publication. There are no other disclosures to report.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: SGP is supported by NIH K23EY024268. SWC is supported by grants from the Duke Endowment. PN is supported by grants from the Duke Endowment and has a patent Segmentation and identification of layered structures in images US 20110182517 A1 issued. These funding organizations had no role in the design or conduct of this study.