Development and Implementation of a Methodology for Quality Assessment of Asynchronous Teleconsultations

Abstract

Background:

There is a lack of evidence regarding audits or quality analysis of telehealth strategies in clinical practice. Our aim is to develop and implement a methodology for quality assessment of asynchronous teleconsultations.

Materials and Methods:

A random sample of asynchronous teleconsultations performed by the specialists from the Telehealth Network of Minas Gerais (TNMG), a public telehealth service in Brazil, was selected. The responses were evaluated regarding size, objectivity, quality, ethics, courtesy, and grammar, and received a score for each category: 1 = fair, 2 = moderate, and 3 = good. As each domain has a different importance in rating the overall quality of teleconsultation, each one was assigned a different weight, and a final score was calculated.

Results:

A total of 576 teleconsultations were assessed. Overall, the scores were good or moderate for all items. Only a few cases were classified as fair. Among medical specialties, pediatrics was the one that proportionally received the highest number of fair classifications, and the item “quality of the answers” was the one with highest number of worse classifications for this specialty. Corrective actions were implemented. With regard to the nonmedical specialties, the majority of the items were classified as good or moderate, and in rare cases some items received the fair rating.

Conclusion:

The methodology showed to be useful to evaluate the teleconsultation service. We established six domains that we considered important components to be assessed. This assessment was essential to identify the priority areas to receive correct actions. It may be easily replicated in other services worldwide.

Introduction

Telehealth is recognized as one of the most rapidly growing areas in health today. It has received large amounts of investment from public and private sectors in the last two decades¹ and a large number of telehealth initiatives have been implemented in various countries around the world.

There is evidence that telehealth can drive improvements in timeliness of care and enhance quality of care, by better supporting chronic disease management, application of best practices, and improvement of knowledge and skill development in local care providers.^{2

–12} However, most of the initiatives are still pilot studies and small-scale implementations.¹³ Therefore, there is concern about the analysis of factors that help scaling up and sustainability of such projects.¹⁴ A lot of attention has been drawn on technical, behavioral, economical, and organizational factors,^1,15

–18 but there is lack of evidence regarding audits or quality analysis of telehealth strategies in clinical practice.

For developing countries, the most important telehealth application is tele-assistance, which includes teleconsultation and telediagnostic tools to support health professionals and to extend specialized services to remote and isolated areas.¹⁹ Teleconsultation can be defined as a second opinion system that allows an information exchange between distant health care professionals and local health care professionals to discuss a clinical case when a specialist is not locally available.²⁰ It can be real-time/synchronous (with simultaneous interaction) or store-and-forward/asynchronous (store-and-forward, in a time-independent basis). Low-cost technology and low bandwidth make store-and-forward teleconsultation the most frequent form of telehealth used in developing countries.²⁰ A well-established teleconsultation service among health providers may have good outcomes, which include timely access to correct medical information, quality improvement of the diagnosis and treatment process, increased physician trust, and significant improvement in the total quality of health care.³

The Telehealth Network of Minas Gerais (TNMG), formed by a partnership of seven public universities,¹⁹ is the largest telehealth service in Latin America. The TNMG started performing asynchronous teleconsultations for primary care in 2007 and currently attends 902 cities in 6 states in different Brazilian regions. Teleconsultations are provided for 813 cities in Minas Gerais State, southeastern Brazil. Professionals in remote cities access the TNMG system via internet and present the case, attaching an image or other files as needed. One of the specialists on duty in the TNMG receives the request, analyzes it, and responds within 24 h, or forwards it to a subspecialist, who responds to it in 48 h.²¹

The service is stable, mature, and integrated to health care, sponsored by the Ministry of Health and Minas Gerais State Government. Until December 2018, more than 125,000 teleconsultations have been performed. TNMG monitors monthly performance indicators, which include the number of cases dealt, percentage of active cities (requested at least one teleconsultation in the current month), time to first response, cost and user feedback on perceived utility, satisfaction, and percentage of patient referral avoided. The active monitoring has shown that more than 95% health care professionals of the attended cities are satisfied with the service provided. However, this indicator is not enough to guarantee the overall quality of teleconsultations.

Only objective tools enable the assessment of the quality of the teleconsultations,¹⁹ and we are not aware of another study that described in detail the quality assessment of asynchronous teleconsultations, in a way it can be replicated. Based on the available literature, both patients and medical professionals appear to be generally satisfied with teleconsultation services.^22

–27 However, this assessment has limitations. The positive evaluation by the patients could be a result of telemedicine removing problems found to cause dissatisfaction in nontelemedicine care, including easier access to specialists, appointment scheduling, travel time, improved effectiveness, and financial savings. Overcoming common issues, as well as geographic barriers, can make certain patients more optimistic about experiences with telemedicine, mainly in a primary care context.^28
–30 Patients seem to find telemedicine to be a good solution to overcome many of the barriers they regularly face while seeking medical care. Medical professionals also appreciate the educational possibilities of teleconsultations, as well as their ability to reduce unnecessary specialist referrals.²⁸

Therefore, the objective of this study is to develop and implement a methodology for quality assessment of asynchronous teleconsultations.

Materials and Methods

Study Design and Setting

This is an observational, clinical record review study conducted with retrospective information extracted from the database of the TNMG. This kind of investigation may help us to decipher what went wrong, what decision could be taken to minimize problems running to improve the quality of care provided, and what corrective actions can be taken subsequently.³¹

The TNMG teleconsultation service uses a self-developed asynchronous web-based system. All teleconsultations take place among health care professionals, since up to 2019, Brazil's Board of Physicians (Conselho Federal de Medicina) does not authorize teleconsultations between physicians and patients. Process flow is the same for medical and nonmedical specialties. Registered professionals, previously trained, access the TNMG teleconsultation system from remote municipalities via internet to submit a clinical case or a general educational matter, when the specialty is chosen. There is the possibility to attach images and/or other types of files. The most demanded specialties are dermatology, internal medicine, family medicine, pediatric, gynecology/obstetric, nursing, and dentistry. For these specialties, there is an on-duty specialist from Monday to Friday with different working hours according to the demand of each specialty. So, depending on the chosen specialty, a specific on-duty specialist receives the teleconsultation, answering on average 80% of demand and forwarding 20% to off-duty focal subspecialists (Fig. 1). On-duty specialists receive by working hours and have an answer time limit of 24 h. Off-duty focal subspecialists, with a 48-h time limit, receive by teleconsultation.⁹ There are focal specialists in more than 40 subspecialties,¹⁹ to support primary care professionals from low-resource settings. Teleconsultants, usually university hospital employees, are selected based on curriculum and interview, and are previously trained by the clinical coordinator (C.G.P.).

Fig. 1.

Telehealth Network of Minas Gerais teleconsultation workflow.

Developing the Methodology for Teleconsultation Quality Assessment

The investigators proposed the assessment in six domains, described below. As each of the domains has a different importance in rating overall quality of the teleconsultation, each one is assigned a different weight, according to its impact in the quality of teleconsultation. The domains assessed and the weight of each of them were based on the opinions of teleconsultation users in service audits in previous years or by direct contact with them in previous studies.^32
–34 We have identified, for example, that the quality of information was the most valuable item according to their opinion, but they were not satisfied with answers that were too broad, missing the main focus of the doubt, and/or too long. Answers receive a score for each category: 1 = fair, 2 = moderate, 3 = good, based on an expert panel composed of clinicians, experts in telehealth, and one epidemiologist:

Quality of information: assessed whether the answer contained the best available evidence on the topic (weight 0.5).

Objectivity: assessed whether the response was focused on the question or the answer was too broad, missing the main point (weight 0.15).

Size: evaluated whether the amount of text in the response was compatible with the proposed question, that is, if information was missing (too short) or if there was too much text with unnecessary information (weight 0.15).

Ethics: evaluated whether the answer contained criticism that denigrated another health professional, use of improper words, violation of codes of professional conduct, suggestions that contradicted the assistance flow of the guidelines of public health agencies (weight 0.1).

Courtesy: evaluated whether the response was written in a respectful and polite way (weight 0.05).

Grammar: evaluated whether the spelling of the response was correct, with no typing errors and with understandable content (weight 0.05).

Score 1 is given to the answer that exhibits at least one of the features exposed in Table 1. The answer is classified as moderate (score 2) if there were just a mild problem, which would not allow rating it as good, but would also not meet criteria to classify it as fair. In the score 3 group are the answers that are evidence based, with an adequate size for the question asked, replying the inquiry in a clear and objective way, written in a correct, comprehensible, and polite manner, and also respecting the telehealth's ethical and legal principles.

Table 1.

Criteria for Classifying the Teleconsultation as “Fair” in Each of the Six Domains

1. Quality of information

Absence of evidence-based response (best available evidence)

Response based only on common sense (lay knowledge)

2. Objectivity

Absence of focus on the question of the applicant

Wordy response (long response without objectivity)

3. Size

Incomplete response (containing insufficient data)

Wordy response (long response without objectivity)

Very long response to the teleconsultation's objectives

4. Ethics

Criticizing other professionals' conduct

Improper name quotes

Response to a personal/private teleconsultation^a

Incompatible content with the professional duties of the applicant (e.g., pharmacotherapy for nonprescriber professionals)

Response with procedures at odds with the assistance flow established by the SES-MG

5. Courtesy

Rude or discourteous response

6. Grammar

Spelling or grammatical mistakes

Incomprehensible content

It was classified as “fair” when it was sent to a physician, as this practice is not allowed in Brazil.

SES-MG, State Health Department of Minas Gerais.

A final score is then calculated by the multiplication of each category's score by its corresponding weight and sum of all the obtained results, and rated.

Testing the Methodology for Teleconsultation Quality Assessment

A sample of all consecutive teleconsultations performed by the TNMG's specialists, from January to February 2014, was analyzed. This sample was obtained from a computerized database that archives all the TNMG teleconsultations. They were split in two categories: (1) medical teleconsultations and (2) nonmedical teleconsultations (nursing, dentistry, physiotherapy, pharmacy, psychology, nutrition, and speech therapy teleconsultations).

Two senior investigators (J.X.M. and C.G.P.), who were more familiar with the teleconsultation system, assessed the teleconsultations. Data were extracted from the records using a structured instrument and included the professional category of the practitioner who requested the teleconsultation, the health care professional who answered the teleconsultation and his/her professional category, teleconsultations, and teleconsultation responses.

To ensure an adequate concordance level between them, a pilot study was performed using a sample of teleconsultations from December 2013. They analyzed samples of 30 teleconsultations and started the formal analysis when kappa interrater reliability index was higher than 0.8. Cohen's kappa is a robust statistic, calculated by the formula (observed agreement − chance agreement)/(1 − chance agreement). It can range from −1 to +1, where 0 represents the extent of agreement that can be expected from random chance, and 1 represents perfect agreement between the raters. When it is higher than 0.8, the level of agreement is considered almost perfect.³⁵

Subsequently, the medical doctor investigator (J.X.M.) analyzed the medical teleconsultations and the physiotherapist investigator (C.G.P.) analyzed the nonmedical ones. Teleconsultations of all medical and nonmedical specialties and subspecialties were assessed. However, for the purpose of this study, only the results of the professionals on duty are shown.

Results

A total of 576 teleconsultations were analyzed. Of those, 73% were medical teleconsultations: family medicine (40%), dermatology (15%), pediatrics (11%), and gynecology (7%). The nonmedical ones (27%) comprehend nursing (17%), pharmacy (2%), dentistry (3%), physiotherapy (2%), psychology (1%), nutrition (1%), and speech therapy (1%).

The description of the results of teleconsultations' quality assessment is shown in Figures 2 and 3 . Overall, the scores were good or moderate for all items, and only a few cases were classified as bad or fair. In medical specialties, pediatrics was the one that proportionally received the highest number of fair classifications, and the item “quality of the answers” was the one with the highest number of worse classifications for this specialty. With regard to the nonmedical specialties, the majority of the items were classified as good or moderate, and in rare cases, some items received the fair rating.

Fig. 2.

Score of medical teleconsultations.

Fig. 3.

Score of nonmedical teleconsultations.

From the initial stratification, the data shown in the graphs were multiplied by the respective weights and are presented in Table 2. All the medical specialties had a median score of 3 (good), except pediatrics, with a median score of 2.9 (interquartile range 2.3–3.0). With regard to the nonmedical specialties, nursing, physiotherapy, and nutrition had a median final score classified as moderate or good (2.5–3.0).

Table 2.

Final Scores (Interquartile Range) for Each of the Six Domains According to Weights Applied for Each Domain

	FINAL SCORE
Medical	3.0 (2.5–3.0)
Dermatology	3.0 (2.9–3.0)
Family medicine	3.0 (2.5–3.0)
Gynecology	3.0 (2.9–3.0)
Pediatrics	2.9 (2.3–3.0)
Nonmedical	3.0 (2.5–3.0)
Nursing	2.5 (2.5–3.0)
Dentistry	3.0 (3.0–3.0)
Physiotherapy	2.5 (2.5–3.0)
Pharmacy	3.0 (3.0–3.0)
Psychology	3.0 (3.0–3.0)
Nutrition	2.6 (2.6–3.0)
Speech therapy	3.0 (3.0–3.0)

Discussion

This study describes an objective methodology for quality assessment of asynchronous teleconsultations, which was developed in a large-scale Brazilian Telehealth service, but can be easily replicated in other services worldwide. We established six domains, which we considered important components to be assessed.

Quality assessment has been little used in telehealth. Most studies describe numbers and user feedback.^{3,22

–27} The telehealth experiences have been expanding and the services are growing. To achieve the benefits of teleconsultations in high quality, by rapid decision-making, shortening diagnosis time, faster and better patient management, educational value, increasing the confidence, and reducing unnecessary referrals and procedures in patient care, the quality should be assured. Therefore, it is essential to assess whether the quality of the services is being maintained.

The TNMG has always been monitoring health professionals' satisfaction with the teleconsultations, by applying a survey at the end of each teleconsultation. Satisfaction rate has always been more than 90%.²² However, relying the quality assessment of a teleconsultation service on a satisfaction survey only may not be appropriate. Response rates tend to decline over time, affecting representativeness. Health care professionals from resource-constrained areas have limited access to continued education. This fact may lead to two consequences: (1) they may not be aware that the response could be improved, so they may rate up the quality of response or (2) they fear losing the service if they rate the quality down, although this is not true. There is also the possibility of response bias: professionals who respond to the survey may be the ones who are more satisfied with it, or the ones who are more unsatisfied.³⁶ Therefore, an unbiased and evidence-based assessment by an auditor not involved in those teleconsultations seems more appropriate. The audit methodology reported in this article extended the findings from health professionals' feedback, and pointed out flaws, which were very important for improvement.

For our service, the results of this analysis were very important to get to know the performance of each health professional who answered the teleconsultations, to recognize where there was room for improvement, for personalized feedback, development of new protocols, and to indicate the professionals who needed further training and the ones who needed to be dismissed. For example, professionals who did not score 3.0 for each item assessed in all teleconsultations were contacted for personalized feedback and each mistake was pointed out. One professional was dismissed, as the TNMG clinical coordinator observed in the results of this assessment that the majority of the teleconsultations she answered had problems with quality.

Wootton et al. describe a methodology for quality assurance, based on their experience in two store-and-forward telehealth networks. The authors proposed a performance measurement framework and indicators to be regularly monitored, including performance indicators, outcomes data, user feedback, and overall teleconsultation quality.³⁷ We agree with the authors, but their methodology for overall teleconsultation quality assessment may not be appropriate for large-scale services. In our study, we aimed to assess the performance of each health professional who answered the teleconsultation, in our multiprofessional service. The methodology described by Wootton et al. comprehends 17 questions, to be assessed by multiple observers (at least 4), and the authors suggested assessing 1 teleconsultation per month,³⁸ what in most cases will not be representative. That is why we proposed a method with the main aspects to be assessed, in a way it can be performed at regular intervals, and can make possible immediate feedback for the health professional who answered the teleconsultation. It appears suitable for routine adoption, in a way that services can adapt for their reality. The weights were arbitrary, and for other contexts may be changed accordingly.

Thijssing et al. developed a questionnaire, based on the CQ-Index framework, for measuring the quality of care delivered through teleconsultations as perceived by patients. The quality aspects were categorized as follows: (1) access to care, (2) communication and information, (3) interpersonal conduct, (4) patient management role, (5) competence, (6) organization of care, (7) continuity of care, (8) effective and safe care, and (9) costs and compensation.²³ The authors also proposed to investigate in future research comprehensiveness, relevance, unambiguousness, reliability, and internal cohesion of the questionnaire. Focus groups were conducted with patients,²³ to investigate what was important. In our case, teleconsultation is among health professionals, and most of the time, patients do not have access to teleconsultation content.

It is important to highlight that this assessment does not substitute the monthly performance indicators, as well as the regular assessment with the users. They provide complementary information, to perform a thorough assessment of the telehealth service. A health professional who is not satisfied or who thinks the quality of the teleconsultation response is not good enough may not use the service again, so regular monitoring of teleconsultation quality by periodic audit is essential.

The present work has certain limitations. For example, before it could be used routinely in other services, the quality assessment methodology would require validation in other contexts. Further research may test this methodology in other services and geographic contexts. It is important to consider that different types of teleconsultations may need different methods of assessment. Therefore, it is important to investigate whether the methodology would be appropriate to assess synchronous teleconsultations and teleconsultations between health care professionals and patients. In Brazil, a change in regulation by the Brazilian Board of Physicians has been discussed and, from May 2019, teleconsultations with patients might be allowed. If this is the case, both models will coexist (teleconsultations with health professionals and with patients), and it is essential to define whether this methodology is useful for both models, or whether a different methodology for quality assessment applies for each type. In conclusion, the use of the proposed quality-assessment method is feasible in practice. It extends the findings from health professionals' feedback and may point out flaws to be improved. It could be easily replicated in other telehealth services worldwide, to improve the quality of their teleconsultations.

Footnotes

Disclosure Statement

No competing financial interests exist.

References

Whitten

, Holtz

. Provider utilization of telemedicine: The elephant in the room. Telemed J E Health, 2008; 14:995–997.

Chen

, Sun

, Yang

, Liu

, Zhang

, Peng

, et al. Clinical and economic outcomes of telemedicine programs in the intensive care unit: A systematic review and meta-analysis. J Intensive Care Med, 2018; 33:383–393.

Deldar

, Bahaadinbeigy

, Tara

. Teleconsultation and clinical decision making: A systematic review. Acta Inform Med, 2016; 24:286.

Hamine

, Gerth-Guyette

, Faulx

, Green

, Ginsburg

. Impact of mHealth chronic disease management on treatment adherence and patient outcomes: A systematic review. J Med Internet Res, 2015; 17:e52.

Hjelm

. Benefits and drawbacks of telemedicine. J Telemed Telecare, 2005; 11:60–70.

Lin

M-H

, Yuan

W-L

, Huang

T-C

, Zhang

H-F

, Mai

J-T

, Wang

J-F

. Clinical effectiveness of telemedicine for chronic heart failure: A systematic review and meta-analysis. J Investig Med, 2017; 65:899–911.

Marcin

, Rimsza

, Moskowitz

. The use of telemedicine to address access and physician workforce shortages. Pediatrics, 2015; 136:202–209.

Marcolino

, de Queiroz Oliveira

, Ribeiro

. Text messaging for prevention and treatment of cardiovascular disease: Time to move on. Heart, 2017; 103:893–894.

Marcolino

, Maia

, Alkmim

MBM

, Boersma

, Ribeiro

. Telemedicine application in the care of diabetes patients: Systematic review and meta-analysis. PLoS One, 2013; 8:e79246.

10.

Marcolino

, Oliveira

JAQ

, D'Agostino

, Ribeiro

, Alkmim

MBM

, Novillo-Ortiz

. The impact of mHealth interventions: Systematic review of systematic reviews. JMIR Mhealth Uhealth, 2018; 6:e23.

11.

Verhoeven

, Tanja-Dijkstra

, Nijland

, Eysenbach

, van Gemert-Pijnen

. Asynchronous and synchronous teleconsultation for diabetes care: A systematic literature review. J Diabetes Sci Technol, 2010; 4:666–684.

12.

Zhang

, Wang

, Zhu

, Thapa

, Switzer

, Hess

, et al. Expansion of telestroke services improves quality of care provided in super rural areas. Health Aff (Millwood), 2018; 37:2005–2013.

13.

Hall

, Fottrell

, Wilkinson

, Byass

. Assessing the impact of mHealth interventions in low- and middle-income countries—What has been shown to work?. Glob Health Action, 2014; 7:25606.

14.

Tracy

, Rheuban

, Waters

, DeVany

, Whitten

. Critical steps to scaling telehealth for national reform. Telemed J E Health, 2008; 14:990–994.

15.

van Dyk

. A review of telehealth service implementation frameworks. Int J Environ Res Public Health, 2014; 11:1279–1298.

16.

Miller

. Solving the disjuncture between research and practice: Telehealth trends in the 21st century. Health Policy, 2007; 82:133–141.

17.

Stroetmann

, Kubitschke

, Robinson

, Stroetmann

, Cullen

, McDaid

, et al. How can telehealth help in the provision of integrated care? Copenhagen: WHO Regional Office for Europe, 2010:28.

18.

Nelson

. Telemedicine and telehealth: The potential to improve rural access to care. AJN The Am J Nurs, 2017; 117:17–18.

19.

Marcolino

, Figueira

, Dos Santos

, Cardoso

, Ribeiro

, Alkmim

. The experience of a sustainable large scale Brazilian Telehealth Network. Telemed J E Health, 2016; 22:899–908.

20.

Nerlich

, Balas

, Schall

, Stieglitz

, Filzmaier

, Asbach

, et al. Teleconsultation practice guidelines: Report from G8 Global Health Applications Subproject 4. Telemed J E Health, 2002; 8:411–418.

21.

Alkmim

, Figueira

, Marcolino

, Cardoso

, Pena de Abreu

, Cunha

, et al. Improving patient access to specialized health care: The Telehealth Network of Minas Gerais, Brazil. Bull World Health Organ, 2012; 90:373–378.

22.

Soriano Marcolino

, Minelli Figueira

, Pereira Afonso dos Santos

, Silva Cardoso

, Luiz Ribeiro

, Alkmim

. The experience of a sustainable large scale Brazilian telehealth network. Telemed J E Health, 2016; 22:899–908.

23.

Thijssing

, Tensen

, Jaspers

. Patient's perspective on quality of teleconsultation services. Stud Health Technol Inform, 2016; 228:132–136.

24.

Fatehi

, Martin-Khan

, Smith

, Russell

, Gray

. Patient satisfaction with video teleconsultation in a virtual diabetes outreach clinic. Diabetes Technol Ther, 2015; 17:43–48.

25.

Petcu

, Kimble

, Ologeanu-Taddei

, Bourdon

, Giraudeau

. Assessing patient's perception of oral teleconsultation. Int J Technol Assess Health Care, 2017; 33:147–154.

26.

Viers

, Lightner

, Rivera

, Tollefson

, Boorjian

, Karnes

, et al. Efficiency, satisfaction, and costs for remote video visits following radical prostatectomy: A randomized controlled trial. Eur Urol, 2015; 68:729–735.

27.

Banks

, Farr

, Salisbury

, Bernard

, Northstone

, Edwards

, et al. Use of an electronic consultation system in primary care: A qualitative interview study. Br J Gen Pract, 2018; 68:e1–e8.

28.

Whitten

, Love

. Patient and provider satisfaction with the use of telemedicine: Overview and rationale for cautious enthusiasm. J Postgrad Med, 2005; 51:294.

29.

Gustke

, Balch

, West

, Rogers

. Patient satisfaction with telemedicine. Telemed J, 2000; 6:5–13.

30.

Nesbitt

, Hilty

, Kuenneth

, Siefkin

. Development of a telemedicine program: A review of 1,000 videoconferencing consultations. West J Med, 2000; 173:169.

31.

Sarkar

, Seshadri

. Conducting record review studies in clinical practice. J Clin Diagn Res, 2014; 8:JG01.

32.

Pessoa

, Sousa

, Ribeiro

, Oliveira

, Silva

, Alkmim

, et al. Description of factors related to the use of the teleconsultation system of a large telehealth service in Brazil—The Telehealth Network of Minas Gerais. J Int Soc Telemed Ehealth, 2016; 4:e4.

33.

Alkmim

, Marcolino

, Figueira

, Sousa

, Nunes

, Cardoso

, et al. Factors associated with the use of a teleconsultation system in Brazilian primary care. Telemed J E Health, 2015; 21:473–483.

34.

Marcolino

, Passos

PFO

, França

ALN

, Marino

BCA

, Sousa

, Alkmim

, et al. Clinical quality control of a large Brazilian teleconsultation service: The Telehealth Network of Minas Gerais Global Telemedicine and eHealth Updates: Knowledge Resources. Basel, Switzerland: International Society for Telemedicine & eHealth (ISfTeH), 2014:91–95.

35.

McHugh

. Interrater reliability: The kappa statistic. Biochem Med, 2012; 22:276–282.

36.

Cull

, O'connor

, Sharp

, Tang

SfS

. Response rates and response bias for 50 surveys of pediatricians. Health Serv Res, 2005; 40:213–226.

37.

Wootton

, Liu

, Bonnardot

, Venugopal

, Oakley

. Experience with quality assurance in two store-and-forward telemedicine networks. Front Public Health, 2015; 3:261.

38.

Wootton

, Liu

, Bonnardot

. Assessing the quality of teleconsultations in a store-and-forward telemedicine network. Front Public Health, 2014; 2:82.