Abstract
Tertiary teledermatology (TTD), where a general dermatologist consults a specialized dermatologist on difficult cases, is a relatively new telemedicine service. We evaluated TTD in a Dutch university hospital, where 13 general dermatologists used TTD to consult 11 specialized dermatologists and two residents at the university medical centre. We measured the avoided referrals to the university centre, the usability of the system and the user acceptance of it. During a three-month study, general dermatologists consulted via TTD 28 times. In 17 of the consultations (61%), the general dermatologists would have referred their patients to the university centre if teledermatology had not been available. Referral was not necessary after teledermatology for 12 of these 17 consultations (71%). The mean usability score (0–100) of all the users was 80. All dermatologists were satisfied with TTD (mean satisfaction of 7.6 on a 10-point scale) and acceptance was high. The baseline measurements showed that half of tertiary referrals were suitable for TTD. These results suggest that TTD reduces unnecessary physical referrals and that users are satisfied with it. A large-scale evaluation is now required.
Introduction
The use of teledermatology has become widely accepted, especially for communication between general practitioners and dermatologists. Besides making dermatology services accessible in remote areas, 1,2 it also makes the referral process more efficient, safer and more convenient for patients. 3–5 Although most evaluation studies on teledermatology are small-scale pilots, 6 reviews of teledermatology studies have shown teledermatology to be diagnostically accurate, reliable and in some cases cost-efficient. 4,5,7–9 Most studies report high physician satisfaction with teledermatology. 7–9 Patient satisfaction and acceptance of teledermatology is also high. 1,5,7–13
Besides the above uses, dermatologists also use teledermatology to communicate with each other. This is called tertiary teledermatology (TTD, see Figure 1). TTD offers a number of advantages over conventional practice. 14 It can be used to seek advice on difficult cases that require specialized treatment or knowledge. This can help solve problems in the conventional referral process, where patients are sometimes referred to the wrong subspecialty, or where patient information is sometimes missing or incomplete. These problems lead to longer consultation times and longer waiting lists. Teledermatology can reduce these problems by formalizing patient data entry and guiding better triage. Furthermore, in current practice without TTD, communication via unsecure email messages is increasing, including clinical photographs and patient data for consultation. 15 A secure TTD web application, which features adequate identification, authentication and authorization, could replace email traffic and make this process safer and more standardized. Few studies have been published on TTD, 14 and its value in practice has not previously been investigated.

Teledermatology
The aim of the present study was to evaluate the feasibility and acceptance of TTD in a pilot study in the Dermatology Department of a Dutch university hospital. We addressed the following research questions:
Does TTD reduce physical referrals between general and academic dermatologists? Can we create a usable TTD system? Do dermatologists accept this new technique?
Methods
The Department of Dermatology of the Academic Medical Centre (AMC), a Dutch university hospital, diagnoses and treats patients referred by general practitioners (secondary care) and by dermatologists or other specialists (tertiary care). We categorized tertiary referrals as: Inter-departmental Referrals, Second Opinion Referrals and Specialized Referrals. Inter-departmental Referrals are referrals by other specialists from within the hospital. Second Opinion Referrals are patient-initiated referrals after consultation with a dermatologist outside the hospital. Specialized Referrals are also from outside the hospital. We divided them into two subgroups: referrals from general dermatologists outside the hospital (Specialized Referrals A) and referrals from all other specialists outside the hospital (Specialized Referrals B). In 2007, the dermatology department received 1238 tertiary referrals, which amounts to 12% of all referrals they received. The baseline measurement encompassed all tertiary referrals. The pilot study focused on Specialized Referrals A.
Baseline measurement
For two months (December 2007–January 2008), the number and type of all tertiary referrals to the Dermatology Department of the AMC, resulting in a face-to-face consultation, were collected by means of a checklist. The checklist included the following items:
The reason for the referral; The time it took to perform the consultation; Whether or not important patient information, which should be present at the time of the consultation, was missing; Whether the referral was correct (e.g. right subspecialist, right urgency); Whether the referral would have been suitable for teledermatology. The latter was based solely on the academic dermatologist's prediction.
The checklist was completed anonymously for each Inter-departmental Referral, Second Opinion Referral and Specialized Referrals A and B. All the residents and dermatologists of the AMC Dermatology Department participated in the baseline measurements. To check if all referrals were reported during the baseline measurement, data in the hospital administration system from January 2007 to January 2008 were compared with the referrals reported in the checklists. In this comparison, Specialized Referrals A and B were merged, as the hospital administration system did not differentiate between Specialized Referral A and B.
Tertiary teledermatology system
We used the KSYOS Teledermatology Consultation System (KSYOS TeleMedical Centre, the Netherlands) for the TTD communication. This system is widely used for secondary care teledermatology in Holland and was adapted for the needs of TTD. We adapted the system based on interviews with four of the participating academic dermatologists. They specified what information they would require from the general dermatologists during a TTD consultation. The web-based TTD consultation system stored the patient data in a central database, making it accessible to the users via an Internet connection and a web browser. Users accessed the system by logging in using either a username/password combination or a health-care provider identification card. 16 The HTTPS secure Internet protocol was used to protect data transfer.
A maximum of four digital photographs, accompanied by structured textual data containing patient history and other information, could be sent. A teleconsultation consisted of two parts: one for the general dermatologist and one for the academic dermatologist. The general dermatologist's part contained fields for:
General patient data (e.g. name, address, birth date); Photographs; Reason for referral; Reports of any previous examinations; Medical history; Medication usage; The patient's history, including sexually transmitted diseases, tropical dermatology and allergies; A selection box to assign a subspecialty, such as psoriasis, pigmentation and paediatric dermatology.
The part for the academic dermatologist contained fields to describe findings, diagnosis, advice and remarks. For each subspecialty, at least one academic dermatologist with extensive knowledge in that area participated in the study.
Consultation process
General dermatologists initiated the teleconsultations and remained responsible for the treatment throughout the TTD process. Patient data and clinical photographs were first assessed and answered by a pre-assigned academic dermatology resident. The resident selected the most suitable academic dermatologist based on the indicated subspecialty or the description of the problem. The academic dermatologist checked the answers given by the resident and corrected them if needed. He or she then sent the response back to the general dermatologist. For educational purposes, the resident checked the answers given by the academic dermatologists. If the academic dermatologist asked for additional information or the general dermatologists had a follow-up question, the above process could be repeated in a second round. After this, no more rounds were possible, and the general dermatologist closed the teleconsultation.
When a new teleconsultation or a response to a teleconsultation was available, automatic notification messages were sent by email to alert either the academic dermatology resident or the general dermatologist. To alert an academic dermatologist about a new or follow-up teleconsultation, the dermatology resident personally created a notification email.
Pilot study
Ten general dermatologists from seven hospitals and three dermatologists from private practice were invited to send teleconsultations to the AMC for a period of three months in 2008. A total of 11 academic dermatologists and 2 academic dermatology residents of the AMC were involved in answering the teleconsultations. A teleconsultation had to be answered by the academic dermatologists within five working days after the teleconsultation had been sent. Patients were asked for their informed consent for sending their information over the Internet. Ethics permission was not required for the study. Dermatologists received no reimbursement.
All general dermatologists were given a digital camera (Easy Share C713, Kodak) so they could take the required photographs. All participants received a user's manual for the teleconsultation system. On request, they were also given demonstrations by one of the researchers to show them how to take clinical photographs. The photographs were taken at a resolution of 1.2 megapixels.
Statistics
User statistics were recorded. Response time was calculated based on working days from 08:30–17:30, not counting weekends. One researcher analysed the teleconsultations to determine if the advice given in a teleconsultation was of a therapeutic or diagnostic nature. Two academic dermatologists classified all the teleconsultations into a subspecialty for comparison with the subspecialties indicated by the general dermatologists.
Avoided physical referrals
During every teleconsultation, the system asked evaluation questions. At the start of each teleconsultation the general dermatologist was asked: would he/she have referred the patient to the academic centre if teledermatology was not available? After each teleconsultation, the general dermatologist was asked: would he/she still refer the patient to the academic centre? A referral was considered to be avoided when the answer to the first question was ‘Yes’ and the answer to the second questions was ‘No’. The general dermatologist was also asked: (1) was the answer of the academic dermatologists useful? and (2) did they learn from the teleconsultation? Academic dermatologists were asked about their confidence level in their diagnosis and if they wanted to see the patient for a physical consultation.
User acceptance and satisfaction
After the study was finished, a questionnaire was sent to all participating general and academic dermatologists and residents. The questionnaire contained a generic part, a user-specific part and a part to determine system usability.
The first part contained the same questions for all participants: (1) did you experience any problems while using the system? (2) which data entry fields were useful? (3) do you believe TTD could have a role in continuing medical education? (4) were four photographs sufficient for a teleconsultation? They were also asked to rate their satisfaction (on a 10-point scale, where 10 was the highest level of satisfaction) regarding the TTD system, the conduct of the study and TTD as a concept.
The second part was user-specific and was aimed at general, academic or resident dermatologists. For each user group, a set of propositions was created, with which the respondent either agreed or disagreed. General dermatologists were asked about the time spent on performing a teleconsultation and again whether they felt they had learned from using TTD. Academic dermatologists were asked about time spent and how they felt about the interaction with the residents. The residents were asked if they had learned from TTD and about the interaction with the academic dermatologists.
The third part of the questionnaire contained the System Usability Scale (SUS), a validated scale to determine the usability of the system. 17 The SUS results in a score between 0 and 100, where acceptable systems score above 70, better products score in the range from 77 to 89 and superior systems score 90 or higher. 18
Results
Baseline measurement
During the two-month baseline measurement period, 181 tertiary referrals were recorded by checklist (Table 1). Of the 181 referrals measured, 78 (43%) were indicated as Specialized Referrals (A and B). The hospital administration system recorded 149 Specialized Referrals (A and B) to the Dermatology Department during the baseline measurement period. This indicates that no checklist results were received for 71 (48%) of the consultations that were conducted during the baseline measurement period.
Baseline measurements. Results of tertiary referrals to the academic hospital
*Not available: no answer was provided
†Referrals from other departments within a hospital
‡Patient initiated referrals after they had seen a different dermatologist
§Referrals by dermatologists from other hospitals or private practice
**Referrals from all specialists other than dermatologist from other hospitals or private practice
Of the 181 tertiary referrals, 24 (13%) were referrals from general dermatologists (Specialized Referrals A). About half (n = 11) of these were considered suitable for teledermatology. The main reason given for the referral was diagnostic advice (n = 18); the reason for the remaining 6 referrals was starting new treatment. In 13 (54%), the consultation lasted 10–20 min and in four (17%) it took 5–10 min. In total, six of the consultations (25%) missed some patient data. The reasons given for the 4 incorrect referrals were: ‘not the right specialist’ (n = 2), ‘not the right subspecialist’ (n = 1); and not specified (n = 1).
Pilot study
During the pilot study, a total of 28 teleconsultations were performed; 9 general dermatologists and 11 academic dermatologists sent and answered one or more teleconsultations. There were 8 male and 20 female patients. The mean age of the patients was 39 years.
Statistics
The mean response time — the mean time between the teleconsultation sent by the general dermatologist and the response sent back by the academic dermatologist — was 2.2 days for the first round. In ten of the cases (36%), the general dermatologists initiated a second round. The mean response time for the second round was also 2.2 days. The mean response time from the start of the teleconsultation by the general dermatologist to the final response by the academic dermatologist was 3.0 days. The usage of the data entry fields in a teleconsultation is summarised in Table 2.
The text fields available in the teledermatology system, their usage during 28 teleconsultations and their usefulness as judged by 18 users
*Diagnostic test I and II were text fields in which results of diagnostic test could be entered (e.g. dermatoscopy, pathology, lab report)
The reason for the teleconsultations was diagnostic advice in 17 of the cases (61%), therapeutic advice in 5 cases (18%) and both diagnostic and therapeutic in 6 cases (21%). The general dermatologists indicated a specialism in 11 of the teleconsultations (39%). Paediatric dermatology and the subfield of dermatology concerning pigmented disorders were the most frequently selected specialisms.
Avoided physical referrals
In 17 of the 28 teleconsultations (61%), the general dermatologist indicated that the patient would have been physically referred to the academic centre if TTD were not available. In this group, 5 patients (29%) were physically referred after teleconsultation, while 12 of the physical referrals (71%) to the academic centre were avoided. In addition, one of the teleconsultations resulted in an additional physical referral. This means that without TTD the general dermatologist would not have referred the patient to an academic centre, but after the teleconsultation the patient was referred at the academic dermatologist's request.
The general dermatologists indicated that the answer of the academic dermatologist was useful in 23 of the teleconsultations (82%), and that they had learned from the teleconsultation in 24 of the teleconsultations (86%).
The overall confidence in the diagnosis was moderate: very sure (n = 1), sure (n = 2), fairly sure (n = 4), somewhat unsure (n = 6) and unsure (n = 2). For the remaining 13 teleconsultations the academic dermatologist indicated that no diagnosis was made through teleconsultation.
User acceptance and satisfaction
The response rate of the usability and satisfaction questionnaire was 82%. This amounted to 8 out of 11 for the academic dermatologists, 2 out of 2 for the residents, and 8 out of 9 for the general dermatologists.
In the generic part, 22% reported that they experienced some problems in using the system. These included not being able to log in from home, small text fields leading to extensive scrolling, not having the log-in codes at hand, not being able to use the system on mobile devices and not getting a notification when a new consultation was received. A comparison between the actual usage of the information entry fields and the percentage of respondents who felt the field was useful is shown in Table 2. Regarding the other questions, 15 respondents (83%) reported that 4 photographs were sufficient for a teleconsultation; and 13 respondents (72%) felt that the TTD could be used for continuing medical education. On average, the respondents were satisfied about all three matters. The TTD system, the conduct of the pilot study and TTD as a concept all scored 7.6 out of 10.
An overview of the answers provided in the second part of the questionnaire is shown in Tables 3–5. General dermatologists indicated that they spent 5–10 min entering the information for a teleconsultation. Six of them (75%) indicated that it took less than 5 min to take pictures of the patient and seven of them (88%) believed that they had learned from participating in TTD. All the academic dermatologists said they could complete a teleconsultation in 5–10 min and thought the process, where the resident acted as the first station for the teleconsultations, was very effective. The two residents were also positive about the set up; they both indicated that they had learned from the interaction with the academic dermatologist and that a teleconsultation often resulted in additional communication with an academic dermatologist to discuss or clarify the case.
Evaluation questionnaire (the original questions were in Dutch). Results from eight general dermatologists
Evaluation questionnaire (the original questions were in Dutch). Results from eight academic dermatologists
Evaluation questionnaire (the original questions were in Dutch). Results from two residents
System usability
The mean SUS score of the general dermatologists was 78. The mean SUS score of the academic dermatologists was 81. The mean SUS score of the resident dermatologists was 86. The overall mean SUS score was 80.
Discussion
The results of our pilot study demonstrate that a tertiary teledermatology system has the potential to reduce physical referrals between general and academic dermatologists. They also show that the TTD system studied was feasible and usable, and that general as well as specialist dermatologists accepted the system.
At baseline, almost half of the referrals from general dermatologists to specialized dermatologists (Specialized Referrals A) were suitable for TTD. Using TTD resulted in a 71% reduction of physical referrals to the academic hospital in the patient group selected by the general dermatologists. The acceptance of TTD was high and all users reported a high level of satisfaction with it.
TTD can be used for different purposes, such as prevention of referrals, second opinions, continuing medical education and international expert groups; it can be a valuable addition to health-care systems. 14 The system and the way it is used might vary according to the purpose, but in the present pilot trial we demonstrated that it is relatively simple to use and effective for several of these purposes. TTD was used to improve access to the expert opinions of specialized dermatologists. This resulted in a more efficient tertiary referral process, with 71% avoided referrals. Secondary uses of TTD in the pilot study were education and acquiring advice from a colleague, without the intention to refer. The general dermatologists sent 11 teleconsultations without any intention to refer the patient physically. Nevertheless, at the academic dermatologist's request, one of these teleconsultations did result in a physical referral of the patient. This represents an improvement in the quality of care: the patient was treated by a physician better suited for the ailment, which would not have happened without TTD, or at least not so quickly.
The general dermatologists and the residents both indicated that they learned from using TTD. The general dermatologists indicated a learning effect in 86% of the cases, both during system evaluation and afterwards in the questionnaire. This educational benefit of TTD is a form of continuing medical education. Both residents indicated that they experienced a learning effect, along with the fact they learned to work with teledermatology at an early stage, which will benefit the use of teledermatology in the future.
We found similar results for TTD in this study compared to results found for secondary teledermatology on preventable referrals, 19 acceptance and satisfaction. 5,7–9 Only a few studies on TTD concerning preventable referrals have been published, and none of them have reported quantitative data. 20–22 One publication reported on physician acceptance, which was high. 23 Several publications reported using TTD for continuing medical education, which seems to be one of the main purposes for which these systems are used. 14
The main reason for a tertiary referral was diagnostic advice, both for the baseline measurement (72% of the cases) and the pilot study (82% of the teleconsultations). In 82% of the teleconsultations, the general dermatologists indicated that the teleconsultation was useful, but no information was available about the reason for the remaining 18%. In 13 teleconsultations the academic dermatologists indicated that they made no diagnosis; 11 of these teleconsultations were therapeutic, and hence there was no request for a diagnosis since it had already been made. The confidence of the academic dermatologists in their diagnosis through TTD was moderate. However, the advice given in these teleconsultations was enough for the general dermatologists to decide whether or not to refer a patient. The academic dermatologists often made a differential diagnosis, which reduces confidence in a single diagnosis.
The comparison of the number of checklists received in the baseline measurement with the hospital administration information indicated that we only obtained the data for half of the consultations. This could have been caused by time restrictions or forgetfulness of the dermatologists, or even faulty recording in the hospital information system.
Based on the data we collected with the checklist, it is clear that TTD has potential value. No major technical problems were reported in the evaluation. The reported problems concerned human error and were resolved quickly. It was not possible to implement automatic notification email messages to the academic dermatologists, but having the residents send notification messages by email solved this problem. Although the small scale of this pilot study prevents us from making firm conclusions, the results certainly indicate that TTD can improve communication between dermatologists.
One possible defect in this pilot study was selection bias. The participating dermatologists were highly motivated and monitored. Consequently it was unlikely that they would not have accepted TTD. Second, no follow-up data was available on whether or not patients were seen live by an academic dermatologist after TTD. Finally, patient acceptance and satisfaction was not measured in this pilot. More research on a larger scale is therefore needed to evaluate if and how tertiary teledermatology improves the tertiary referral process. Because confidence in diagnosis was moderate, future research could focus on the diagnostic reliability of TTD. Finally, TTD should be validated in different settings to generalize any conclusions.
Footnotes
Acknowledgements
A potential conflict of interest exists since JP van der Heijden and L Witkamp are employees of the KSYOS TeleMedical Centre.
