Abstract
We evaluated the accuracy of diagnoses made from pictures taken with the built-in cameras of mobile phones in a ‘real-life’ clinical setting. A total of 263 patients took part, who photographed their own lesions where possible, and provided clinical information via a questionnaire. After the teledermatology procedure, each patient was examined face-to-face and a gold standard diagnosis was made. The telemedicine data and pictures were diagnosed by 15 dermatologists. The 299 cases contained 1–22 clinical images each (median 3). Nine dermatologists finished all the cases and the remaining six completed some of them, thus providing 2893 decisions. Overall, 61% of all cases were rated as possible to diagnose and of those, 80% were correct in comparison with the face-to-face diagnosis. Image quality was evaluated and the median was 5 on a 10-point scale. There was a significant correlation between the correct diagnosis and the quality of the photographs taken (P & 0.001). In nearly two-thirds of all cases, a teledermatology diagnosis was possible; however, there was insufficient information to make a telemedicine diagnosis in about one-third of the cases. If applied carefully, mobile phones could be a powerful tool for people to optimize their health care status.
Introduction
Studies concerning mobile teledermatology1–5 (i.e. teledermatology applications that do not depend on stationary equipment) have shown that mobile phones may be a useful and reliable tool for dermatological intervention. However, most studies have been conducted in idealised study settings, so there are still questions concerning the use of mobile teledermatology in real clinical settings. We have conducted a study that simulates teledermatology procedures in a real-life clinical setting using mobile phones in the hands of patients to transmit clinical history and images.
Methods
Patients of the dermatology outpatient service of the Medical University of Vienna were recruited for the study. Prior to participation, they were given information about the study and were asked to provide consent. A numerical identification code was used to ensure participants’ privacy. The study was approved by the appropriate ethics committee.
A mobile phone with an integrated camera was given to each participant. Patients were instructed on how to take pictures of their skin diseases without assistance from technical staff. Assistance was provided for inaccessible body regions (e.g. lesions on the back). An adhesive ruler 2 cm long was placed next to the regions of interest before being photographed. The patients then sent their images via multimedia messaging service mode (MMS) for subsequent analysis without further assistance.
Participants were asked to complete an interactive, computer-based questionnaire. This questionnaire provided additional data on: (1) duration of skin disease; (2) estimated size of the dermatological disorder; (2) affected body parts; (3) presence and intensity of itching and pain; (4) surface and character of the damaged skin; (5) whether the lesion felt warmer than the surrounding skin; (6) allergies; (7) previous contact with animals; (8) subjective wellbeing. General data on age, bodyweight, height, level of education and profession was also acquired.
Mobile phones
Three different mobile phones were used to capture the images of the skin conditions:
mobile phone 1: Nokia N95, (Nokia, Finland) with a 5 Mpixel camera (2592 × 1944 pixels), 1.6 s shutter delay, auto focus and 20x digital zoom;
mobile phone 2: Sony Ericson K810i, (Sony, UK) with a 3.2 Mpixel camera (2048 × 1536 pixels), 2 s shutter delay, auto focus and 4x digital zoom;
mobile phone 3: Samsung SGH-U600, (Samsung, South Korea) with a 3.2 Mpixel camera (2048 × 1536 pixels), 0.8 s shutter delay, auto focus and 16x digital zoom.
The different mobile phones were sequentially assigned to the cases. All mobile phones saved the images in JPEG (Joint Photographic Experts Group) format. According to the MMS protocol, all images were reformatted to an image size of 640 × 480 pixels prior to transmission.
Gold standard
After the teledermatology procedure, each patient was examined by the outpatient unit staff, i.e. residents in training, under close supervision of the board-certified consultant on duty. The resulting face-to-face diagnoses were used as the reference or gold standard.
Classification of cases
All cases were classified into five categories using a 5-point scale (1 = easy to 5 = very difficult) to rate the difficulty of making the correct diagnosis. This scoring of case difficulty was performed by an expert panel of three experienced dermatologists and differences were resolved by consensus.
Evaluation
Using web-based survey software (Unipark 6.5) the images of the skin lesions together with the completed questionnaires were presented to 15 board-certified dermatologists, who were blinded to the clinical diagnoses. Each teledermatologist worked independently, without knowledge of their fellow rater's diagnoses.
Teledermatologists were first asked to assess the digital information for each case, deciding whether or not a diagnosis would be possible on the supplied information and images (Figure 1). When answered affirmatively, they were prompted by the computer to provide a diagnosis and optional differential diagnoses. Then they were asked if they had enough diagnostic confidence to provide treatment based on the information given. In this case they were only allowed to give one telemedicine diagnosis (i.e. in the evaluation it had to agree exactly with the face to face diagnosis to be considered correct). If they did not have enough diagnostic confidence, diagnoses or possible differential diagnoses were denned as correct if they matched one of the differential diagnoses listed in the gold standard.
Decision tree for telediagnosis. There were three different diagnostic scenarios: (1) not enough confidence to give a telediagnosis; (2) enough confidence to give a telediagnosis, but not enough confidence to give teleadvice for treatment, i.e. the teledermatologists considered at least one possible differential diagnosis; (3) the teledermatologist had enough confidence to make a telediagnosis and give teleadvice for treatment, i.e. there was only one diagnosis considered. If a teledermatologist did not make a diagnosis, they had to state the reasons for this.
In addition, the teledermatologists’ subjective rating of difficulty of the cases and diagnostic confidence, as well as picture quality and information given by the patients, was scored on a 10-point visual analogue scale (VAS) (1 = best to 10 = worst). The teledermatologists were also asked to decide on the need for additional diagnostic procedures. The teledermatology diagnoses and feedback were automatically saved by the computer software. The accuracy of the teledermatology diagnoses was compared with the face-to-face diagnoses. A standard textbook was used as a reference for dermatological terminology. 6 The study's primary evaluation criterion was the degree of agreement between teledermatology diagnosis and face-to-face diagnosis.
Results
In total, 277 individuals were asked to participate in the study and 14 of them (5%) declined. Thus, 263 volunteers were included into the study. Some patients had more than one skin lesion, so there were 299 cases evaluated in total. The sample consisted of 62% women and 38% men. The median age was 39 years (interquartile range, IQR 28–49). Half of the volunteers had higher education (beyond high school). There was a broad range of skin diseases, similar to the usual spectrum seen at our outpatient unit, including tumours, naevi, inflammatory/infectious skin diseases, ulcers and wounds, see Table 1. Most participants (81%) needed assistance with handling the mobile phones, partly because of the difficulty in reaching certain parts of the body where their skin disease was located, partly because of problems with handling the phone despite detailed instruction (i.e. images were out of focus, distance from phone to lesion too close or too far). Most people aged over 60 years needed assistance not only with the handling of mobile phones but also with the computer-based questionnaire.
Locations of the skin lesions. Some patients had lesions in more than one region
Evaluation output
The 299 cases contained 1–22 clinical images each (median 3). The cases were sent to 15 teledermatologists who were asked to evaluate the cases within 4 months. Evaluation of cases was only possible in sequential order, without the possibility of omitting cases. After the deadline, nine dermatologists had finished all the cases and the remaining six had completed some of them. In total, 2893 decisions were made. Overall, 61% (1775) of all cases (2893) were assigned a diagnosis and optional differential diagnoses. Out of the 1775 cases where diagnosis was possible, the teledermatologist gave a remote treatment recommendation in 56% (988).
Comparison with face-to-face diagnosis
Considering all cases, including the ones that were rated not possible to diagnose, 49% (1426) of the telemedicine diagnoses were correct when compared to face-to-face diagnosis. Considering only the cases for which a diagnosis was made by the teledermatologists, 80% (1426) of diagnoses were correct. For those cases where the dermatologists felt confident enough to give telemedicine advice for treatment, only one diagnosis (and no differential diagnosis) was considered for evaluation. Out of these cases for which diagnosis was possible and treatment was possible, the diagnosis was correct in 78% (775). The correlation between the confidence of the teledermatologist and the decision to treat the patient was significant (P & 0.001). This shows that only dermatologists certain about their diagnosis suggested treatment.
Diagnostic difficulty of cases
There was a significant correlation between a correct diagnosis and the rating of difficulty by the expert panel (P & 0.001). Of all the cases that were assessed as very easy, 74% of telemedicine diagnoses were correct; taking only the number of diagnoses considered as possible, there were 90% which were correct. The percentage of correct diagnoses decreased steadily with the increase of complexity, regardless of the calculation base (Table 2).
Correct diagnosis with reference to complexity
Based on all diagnoses given, the median subjective rating of difficulty by the teledermatologists was 3 (IQR 2–4) on the VAS. There was a significant correlation between the rating of difficulty by the teledermatologists and the rating of difficulty by the expert panel (P & 0.001).
The median diagnostic confidence was 3 (IQR 2–4). The correlation between correct diagnosis and diagnostic confidence was also significant. Nevertheless, in over three-quarters of all diagnoses considered possible, the teledermatologists found that a more thorough diagnostic process was necessary and only 24% of cases were diagnosed without a request for further testing. The correlation between the raters’ confidence and the need for further examination was not significant (P = 0.09).
Picture quality
Most of the pictures were rated between 4 and 5 (median 5, IQR 4–6) for picture quality (see Table 3). There was a significant correlation between the correct diagnosis and the quality of the photographs taken (P & 0.001). The average picture quality per mobile phone was very similar, with the median for each phone being identical and all other values almost the same, which indicates no significant difference in the quality of the pictures taken by the individual phones. The majority of correct diagnoses were made from pictures taken by mobile phone 2 (51%). Pictures taken with the other two phones had slightly fewer correct diagnoses (phone 1: 48%; phone 3: 48%). Chi-squared analysis revealed that there was no significant difference between the percentage of correct diagnoses among the three phones.
Picture quality of the cases in which diagnosis was possible, as determined by the teledermatologists
Information quality
The informative value of additional text obtained a median rating of 5 (IQR 5–6). In 39% of cases (1118) the teledermatologists did not have enough diagnostic confidence based on the information given, to make a telemedicine diagnosis. This was either due to insufficient image quality (46%), lack of text based information (38%) or incomplete depiction of the skin lesion (15%). In 2% of cases the text based information provided by the patients themselves was found to be misleading or contradictory.
Discussion
Our study evaluated mobile teledermatology in a real-life clinical setting. Patients were able to take the pictures themselves when possible and they were given a simple questionnaire instead of having a medical history taken. In nearly two-thirds of the cases, a teledermatology diagnosis was possible. In eight out of ten cases the telemedicine diagnosis was concordant with the face-to-face diagnosis. However, there was insufficient information to make a telemedicine diagnosis in about one-third of the cases.
Our results confirm the findings of previous studies that showed the feasibility of teledermatology via digital camera or mobile phone. Three studies showed slightly better results (correct answers of 80–90%)2,4,7 and two showed slightly worse results (correct answers of 68 and 77%).3,8 However, with 299 cases, the sample size of our study was larger: the previous studies had 51–228 participants. Furthermore, 15 dermatologists participated in our study, compared with 2 or 3 in other studies.
The high number of correct answers indicates that the teledermatologists only diagnosed cases where they were certain. In 71% of cases, difficulty was rated 2 or 3 by the teledermatologists, showing that many cases seemed easy for them. The subjective difficulty rating was significantly concordant with the rating of difficulty by the expert panel. This indicates that the teledermatologists were able to judge the difficulty of many cases correctly. Their levels of confidence were very high in most of the cases. Ultimately, the correctness of the diagnosis depended on the difficulty and complexity of the case and on the confidence of the rater.
Both the quality of the pictures and the quality of the additional information were regarded as suboptimum by the teledermatologists: 60% of the images were rated 4 or 5 on the 10-point VAS scale, and 51% of additional information was rated 5 or 6. Nevertheless, most cases were correctly diagnosed (Figure 2), but it should be noted that the diagnosis of cases with poor quality pictures and/or information (e.g. Figure 3) resulted in incorrect answers more often, e.g. those cases in which additional information was rated 9 were wrong in 56%. The correlation between the quality of the information and correct diagnoses was especially strong. This indicates that the questionnaire has potential for improvement. There was no substantial difference concerning picture quality between the individual phones. Previous studies found no significant difference between teledermatology studies using a digital camera8–12 and studies using a mobile phone for image acquisition.2,4,5,13,14
The diagnoses of xanthelasma (a) and of the typical target lesions in erythema exsudativum multiforme (b) were made easily by the teledermatologists. (a) incomplete depiction of the skin lesion; (b) low image quality (the skin lesion is out of focus)
The teledermatologists gave a treatment recommendation in only about half of the cases where a diagnosis was possible, showing that they were not always comfortable with remote diagnosis. In addition, teledermatologists recommended further examination in 78% of the cases, albeit it did not affect the accuracy of their diagnoses. The level of diagnostic confidence and the need for further examination were not correlated. This indicates unfamiliarity with this diagnostic method. Ebner et al. 4 came to similar conclusions, as the two teledermatologists recommended face-to-face consultation in 47% and 41% of cases, respectively.
There were some limitations to our study. Our gold standard was face-to-face diagnosis, and it is possible that the in-person examinations were diagnostically inaccurate (in the absence of a more rigorous accuracy assessment). However, face to face diagnosis is the gold standard for diagnoses in many dermatological diseases. Furthermore, we only simulated the telemedical aspect, as patients were in the clinic and partially assisted.
The most common application of teledermatology is for communication between medical specialists. Direct communication between patient and expert is more complex. It requires the active participation of the patients, and extensive testing, in order to avoid mismanagement. At present there is hesitation to use such an approach. 15 A persisting core problem is that mobile phones are not certified medicinal products. Therefore, there are no standards for mobile phones. This hinders their legalization in telemedicine.
In the present study one-fifth of the diagnoses that were made with high certainty (intention to treat remotely) were not correct when compared with the gold standard. We included a broad variety of dermatological diseases although there are some that are more suitable for teledermatology diagnosis than others. Skin diseases that were deemed easy to diagnose by the teledermatologists comprised: burns, palmar/plantar eczema, hypertrophic scars/keloids, psoriasis, actinic keratosis, seborrhoic dermatitis, atopic dermatitis, acne and viral/drug exanthema. These diseases showed great congruence between teledermatology and the gold standard. By matching teledermatology and gold standard diagnoses, we hope to define disease groups of great teledermatology potential for future work. 16
In conclusion, mobile phone-based teledermatology has the potential to be used as a triage tool for outpatients, where it might improve time management and reduce costs in the health care system. If applied carefully, mobile phones could be a powerful tool for people to optimize their health care status.
Footnotes
Acknowledgements
The research was funded by the Jubiläumsfonds zur Förderung der Forschungs- und Lehraufgaben der Wissenschaft of the Austrian National Bank (Oesterreichische Nationalbank).