Quality of Telehealth Services in Primary Care in the State of Minas Gerais

Abstract

Objective:

Telemedicine is seen as an important tool to face contemporary health challenges. The factors that help improve quality in these services must be studied. The objective of this work was to assess the quality of telehealth primary care services offered in the State of Minas Gerais —Brazil, based on the ISO 13131 standard.

Methods:

This cross-sectional analytical study with a quantitative approach was conducted in the cities in the state of Minas Gerais that use telehealth services. A stratified sample composed of 385 cities was used. A questionnaire, based on ISO 13131 on the quality of telehealth services, was prepared, used, and verified for its consistency. Quality levels in telehealth were built from the data. To analyze the quality of care, data from the Ministry of Health's quality improvement program were used, involving 366 surveyed cities. Logistic regression was performed to verify the association between quality of telehealth and quality of care.

Results:

The research identified that 64% of the cities had records of telehealth activities, and 51% of cities had high quality telehealth services. There was no association between quality of telehealth and quality of care; only the dimensions of Quality and Risk Management were associated with quality of care.

Conclusion:

The developed instrument enabled the quality of telehealth actions to be verified. The State of Minas Gerais has high-quality telehealth services.

Introduction

Telemedicine has been a tool in the Brazilian Unified Health System (Sistema Único de Saúde [SUS]) for a long time, due to the geographic, demographic, and organizational characteristics of the country,¹ and it is used mainly to provide distance services and professional improvement. Of the government policies, we highlight the Brazilian Telehealth Network Program (Programa Telessaúde Brasil Redes).² Minas Gerais is one of the States that has participated in this program since its initiation in 2003, with wide coverage to provide telehealth services.

The importance and the positive contribution of telehealth resources to access health services has already been recognized.³ Although telehealth has many positive attributes for health care, the additional risks that it may pose to the quality of services offered at a distance must be recognized. This resource is sensitive to technical issues (equipment, platforms, software, etc.), due to the absence of face-to-face contact,⁴ infrastructure issues along with legal limitations and ethical discrepancies.^5
–7

Quality management practices and total quality are widespread in the field of care, clinical and hospital protocols,⁸ but the evaluation of quality telehealth in Brazil and in the world is still developing. Specific guidelines for quality of telemedicine are published for a few countries, generally those with a history of use or strong involvement with telemedicine, especially Australia,^9

–12 Canada,^13,14 and the International Organization for Standardization's Technical Committee on health informatics—ISO/TC 215 belonging to the ISO4 study group. The assessment of quality in telehealth starts with important initiatives related to the guidelines of telehealth projects. Examples include the Canadian Telestroke Implementation Kit¹⁵ and the European Code of Good Practice for Telehealth Services,¹⁶ which focuses on supporting surveillance and alarm services for people at home. The American Telemedicine Association¹⁷ prioritized the development of standards and guidelines, involving standards to assist risk management, create new services, and improve practice.¹⁸

To standardize the quality of services in 2014, the international ISO (ISO/TS 13131-2014) published recommendations and quality guidelines for telehealth services to deliver medical assistance through information and communications technologies, using risk management. The ISO/TS 13131 proposes guidelines for quality and financial management; planning of services, workforce, and health care; responsibilities of health care organization; facilities; technology; and information management.⁴ The quality characteristics considered by the ISO/TS 13131 specification include accessibility, accountability, adequacy, competence, confidentiality, continuity, reliability, efficiency, effectiveness, inclusion, security, transparency, and usability. The guidelines for ISO 13131 specification were conceived to encompass the design of telehealth services, health professionals who use telehealth, and telehealth security.⁴

A study by Taylor⁹ in relation to the application of ISO 13131 in Australia highlights several important aspects, including required changes in the planning service for clinical practice to consider settings for telehealth; new aspects of workforce planning to ensure that managers and clinical workers have additional skills related to telehealth; adaptation of health planning considering improved systems for sharing patient care and changes in face-to-face care processes for remote care; and new and comprehensive responsibilities for health care professionals to determine whether the use of telehealth is appropriate to deliver a particular service.^9,10,19

The structuring of ISO in Canada telehealth (called Virtual Health Standard) is designed to ensure that all patients who could benefit from Virtual Health services have access to them and that the services they receive will be of high quality, efficient, and protect their safety.¹⁹ The standard offers organizations that provide Virtual Health services with guidance on how to ensure quality and safety in their institutional and virtual contexts; policy makers a plan for requirements to establish organizations that receive and/or provide virtual health services in their jurisdictions; and external assessment bodies with measurable requirements to include in assessment programs.

In Brazil, to build a telehealth network with integrated and cohesive information, with parameters that allow good quality service to achieve measurable results at the administrative, economic, and assistance context for the SUS, the quantitative results on access to consultations and diagnostics via telemedicine was well as questions that favor the use of this resource must be studied. These indicators of structure, processes, and results can effectively assist in the assessment and construction of the overall quality of these services. Brazil already has a participating ISO/TC215 committee dedicated to the review and preparation for the application of ISO13131.

The objective of this work was to evaluate the quality of telehealth services offered in Minas Gerais (MG), based on the ISO 13131 standard, which provides for quality parameters in telehealth services and to verify their relationship with health care.

Materials and Methods

This cross-sectional analytical study employed a quantitative approach. Secondary data about the employment of telehealth services were obtained from the database of the Brazilian National Program for Improving Access and Quality of Primary Care (Programa Nacional de Melhoria do Acesso e da Qualidade da Atenção Básica [PMAQ]),²⁰ second cycle carried out in 2014, and from the database of the National Telehealth Project Brazil Networks (Projeto Nacional de Telessaúde Brasil Redes), which identifies the use of teleconsulting and telediagnosis in the State of Minas Gerais. This process was used to define the sample of the municipalities in Minas Gerais ranked according to their use of telehealth resources.

The data related to the quality of telehealth services were collected from a questionnaire prepared by the first author of this article based on ISO 13131, in 2019. Through it, the quality level of telehealth in the participating municipalities was identified. The questionnaire was answered by the municipal Health Care Manager, the Coordinator of Basic Health Care, or the Technical Representative for Telehealth in the municipality.

The questionnaire consisted of 74 multiple-choice questions separated by an initial sections with the participant's data—9 questions, and 5 more sections about quality research parameters in telehealth according to ISO 13131, which are (A) Quality and risk management—8 questions; (B) Financial and quality management of telehealth services—20 questions; (C) Service, workforce, and health care planning—11 questions; (D) Accountability—9 questions; and (E) Facility, technology, and information management—17 questions. Cronbach's alpha consistency analysis²¹ was performed to assess the reliability of the questionnaire, which assesses the internal consistency and reliability of a test or a questionnaire based on the items in it.

The municipalities were selected from a ranked sample considering the degree of telehealth resources used, totaling 385 municipalities in Minas Gerais. The questionnaire was applied exclusively online, and 366 valid responses were obtained.

To analyze the quality of care, data were used from the PMAQ 2 cycle database, conducted in 2014, by the Ministry of Health. A quality-of-care typology in the municipalities was created, based on the care variables.

Then, we analyzed whether the quality in telehealth is associated with the quality of care. The outcome variable was the quality of telehealth service offered, according to the dimensions present in ISO 13131.

The Chi-Square test was employed. To assess the relationship between the quality of telehealth services and the quality of care, a logistic regression model adjusted by the PMAQ stratum was used, which groups the municipalities according to socioeconomic and demographic variables. The analyses were performed by using the STATA software version 12.0 considering a significance of 5%.

The study was submitted to the National Ethics Committee and approved on May 22, 2019 as CAAE: 12735019.1.0000.5149.

Results

According to the questionnaire structured for the study, Table 1 shows that most municipalities have high-quality telehealth (50.8%); 37.2% of the municipalities have average quality telehealth; and 12.0% of the municipalities have low-quality telehealth. According to Cronbach's alpha value (0.69), the questionnaire has substantial internal consistency.

Table 1.

Distribution of Telehealth Services Quality According to Level

LEVEL	N	%	CRONBACH'S ALPHA
Low	44	12.0	0.6879
Medium	136	37.2
High	186	50.8

Minas Gerais, 2019.

Source: Prepared by the author.

Table 2 presents the distribution of quality telehealth services according to the adequacy of the parameters based on ISO 13131, by size and consistency.

Table 2.

Distribution of Telehealth Services Quality According to Adequacy of Parameters Defined Based on ISO 13131, by Dimension and According to Cronbach's Alpha

DIMENSIONS	MEET OR DO NOT MEET	n	%	CRONBACH'S ALPHA
A. Quality and risk management	No	87	23.8	0.5175
	Yes	279	76.2
B. Financial and quality management of telehealth services	No	127	34.7	0.8044
B. Financial and quality management of telehealth services	Yes	239	65.3
C. Service, workforce, and health care planning	No	212	57.9	0.7274
	Yes	154	42.1
D. Accountability	No	74	20.2	0.7434
	Yes	292	79.8
E. Facility, technology, and information management	No	31	8.5	0.6485
	Yes	335	91.5

Minas Gerais, 2019.

Source: Prepared by the author.

The Cronbach's alpha of each dimension presents a substantial level of consistency, with the dimension (B) (financial and quality management of telehealth services) being considered to exhibit strong internal consistency.

The vast majority of municipalities met the dimension of quality (E) (facilities, technology, and information management), with 91.5%, and the least attended dimension is (C) (service, workforce, and health care planning), with only 42.1%, which is the only dimension that most municipalities (57.9%) were unable to achieve.

This indicates that physical infrastructure issues are not a problem for telehealth activities in municipalities of Minas Gerais and explains that the current obstacle is specialized labor to implement care and manage telehealth services.

As presented in Table 3, referring to the quality of care, most municipalities (66.6%) provide average quality, with 20.0% classified as low, and 13.5% as high quality of health care/assistance.

Table 3.

Distribution of the Surveyed Cities According to Programa Nacional de Melhoria do Acesso e da Qualidade da Atenção Básica Stratification and Quality of Care Levels in Primary Care

VARIABLES	n	%
PMAQ stratum
1	220	60.27
2	66	18.08
3	21	5.75
4	44	12.05
5	9	2.47
6	5	1.37
Care quality
Low	73	20.00
Medium	243	66.58
High	49	13.42

Minas Gerais, 2014.

Source: Prepared by the author.

PMAQ, Programa Nacional de Melhoria do Acesso e da Qualidade da Atenção Básica.

Of the municipalities that make up the sample, 60% are small with up to 10,000 inhabitants (belong to stratum 1 of the PMAQ); 18% of the municipalities have up to 20,000 inhabitants (stratum 2); 12% of the municipalities have a population of up to 100,000 inhabitants (stratum 4); 5.8% of the cities have a population between 50,000 and 100,000 inhabitants (stratum 3); 2.5% of the municipalities have a population of between 100,000 and 500,000 inhabitants (stratum 5); and 1.4% of the municipalities have a population above 500,000 inhabitants (stratum 6).

Comparing the quality of care with the levels of telehealth (Table 4), it has been indicated that only the dimension of Quality and Risk Management exhibited a significant association. Of the municipalities with a high ranking in the PMAQ, 87.8% meet the dimension of Quality and Risk Management (p = 0.016).

Table 4.

Result of the Association by Dimension of Quality in Telehealth and Levels of Quality of Care

		QUALITY OF CARE LEVELS
TELEHEALTH	MEET OR DO NOT MEET	LOW		MEDIUM		HIGH		p
TELEHEALTH	MEET OR DO NOT MEET	n	%	n	%	n	%	p
A. Quality and risk management	No	25	34.2	55	22.6	6	12.2	0.016^a
	Yes	48	65.8	188	77.4	43	87.8
B. Financial and quality management of telehealth services	No	30	41.1	82	33.7	14	28.6	0.328
	Yes	43	58.9	161	66.3	35	71.4
C. Service, workforce, and health care planning	No	43	58.9	135	55.6	33	67.3	0.306
	Yes	30	41.1	108	44.4	16	32.7
D. Accountability	No	17	23.3	49	20.2	7	14.3	0.473
	Yes	56	76.7	194	79.8	42	85.7
E. Facility, technology, and information management	No	8	11.0	19	7.8	3	6.1	0.588
	Yes	65	89.0	224	92.2	46	93.9

Minas Gerais, 2019.

Chi-square test significant at 5%.

Regarding the association between levels of quality in telehealth and care/assistance quality, in Table 5, no association was identified.

Table 5.

Association Between Quality in Telehealth and Quality of Care Levels

	QUALITY OF CARE
TELEHEALTH	LOW		MEDIUM		HIGH		p
TELEHEALTH	n	%	n	%	n	%	p
Final level
Low	14	19.2	27	11.1	2	4.1	0.128^a
Medium	26	35.6	92	37.9	18	36.7
High	33	45.2	124	51.0	29	59.2

Minas Gerais, 2019.

Chi-square test significant at 5%.

The logistic regression model adjusted by stratum (Table 6) found that the dimension of Quality and Risk Management correlated with the quality of care. Municipalities with medium quality were 2.12 times more likely to meet the Quality and Risk Management dimension than municipalities with low quality (odds ratio [OR] = 2.12 95% confidence interval [CI] = 1.16–3.89). Municipalities with high quality were 3.64 times more likely to meet the Quality and Risk Management dimension than municipalities with low quality (OR = 3.64 95% CI = 1.32–10.00).

Table 6.

Logistic Regression of Quality Level of Telehealth and Quality of Care

TELEHEALTH	QUALITY OF CARE	OR	p	95% CI
A. Quality and risk management	Low	1.00	—	—
	Medium	2.12	0.015^a	1.16–3.89
	High	3.64	0.012^a	1.32–10.00
B. Financial and quality management of telehealth services	Low	1.00	—	—
	Medium	1.30	0.346	0.75–2.26
	High	1.78	0.153	0.81–3.91
C. Service, workforce, and health care planning	Low	1.00	—	—
	Medium	1.18	0.548	0.69–2.02
	High	0.70	0.367	0.33–1.51
D. Accountability	Low	1.00	—	—
	Medium	1.27	0.474	0.66–2.42
	High	2.04	0.158	0.76–5.46
E. Facility, technology, and information management	Low	1.00	—	—
	Medium	1.38	0.472	0.57–3.31
	High	1.94	0.348	0.49–7.76
Final level	Low	1.00	—	—
	Medium	1.28	0.363	0.75–2.20
	High	1.69	0.166	0.80–3.56

Minas Gerais, 2019.

Regression adjusted by PMAQ stratification.

OR significant at 5%.

OR, odds ratio; CI, confidence interval.

Discussion

In view of the quality of services provided via telehealth, the questionnaire structured, based on ISO 13131, provided useful guidelines to assist the development of telehealth services. This study demonstrated that telehealth services offered to the population in Minas Gerais are of medium (50.8%) or high quality (37.2%), with more municipalities participating since the structuring of the national program in 2003. A study conducted in Australia⁹ also found similar results, with most telehealth services achieving a certain level of quality.

This study assessed the quality of telehealth service in a Brazilian state within the scope of SUS. However, as advocated by ISO itself, the design and implementation of risk management, safety, and quality management require specific frameworks for telehealth that must take into account the different needs of a specific organization, its particular objectives, context, structure, operations, processes, functions, projects, products, services or assets, and specific practices employed.^4,22 This study was able to identify the strengths and weaknesses for structuring quality telehealth services in the country, based on a questionnaire designed for this purpose, based on ISSO 13131.

In the dimension of Quality and Risk Management, 64.4% of the municipalities have teams that use telehealth. Most are small municipalities and do not have centers with medical specialties.^6,9,23,24

The financial and quality management of telehealth services analyzes the specificities of how telehealth resources are employed. Telediagnosis, asynchronous teleconsulting, and tele-education are the resources most used by municipalities. This dimension also refers to the ease of access to telehealth activities and the provision of technological improvement in health units, aspects that highlight the importance of planning and managerial provision in the implementation of quality telehealth, which corroborates other studies.²⁵ Several studies^5,24
–26 emphasized the importance of planning the technological infrastructure of health services to conduct quality telemedicine activities.

The dimensions of Service, Workforce, and Health Care Planning constitute the most fragile dimension for quality telehealth services in Minas Gerais. This was the only dimension in which most municipalities did not meet. As mentioned by Wen,²⁵ telemedicine does not have an exclusive focus on technology, communication, and IT, but it is a multiprofessional activity, which involves management with planning. The dimension has important aspects such as continuous planning and training of human resources for telehealth and integration of primary care with other levels of care facilitated via telehealth. Other studies^25,26 also point out the difficulties for primary health care to integrate properly with other levels of system complexity as well as identify problems in the planning of human resources for the health area.

This dimension indicates that the primary care professional has not been trained to provide telehealth. Further, telehealth in many municipalities is not used to reduce waiting time in specialized care. Most cities do not have criteria for adapting patients to services and care via telehealth, which was considered essential by Canadian Guildlines¹⁰ for telehealth practices and by Australian studies.^9,11

The Accountability dimension identifies whether patients are informed when the case will be referred for evaluation via telehealth. Most patients and health professionals approve the use of the telehealth resource. A negative highlight in this dimension is that 55.6% of the participating managers said they were unaware of the laws that govern telehealth activities. According to the general norms for quality processes,²⁷ the people involved must have knowledge of the organizational procedures and guidelines to provide quality service. Studies in Canada^14,19 and Australia^9,10 also indicate the need to sensitize those involved in the telehealth process, such as patients and health professionals, to minimize the risks inherent to the use of this resource.

In the dimension about Facility, technology, and information management, the municipalities have physical infrastructure of information technology in primary care, but problems of access to technology remain, since 24.7% of the interviewees considered that the Internet capacity and maintenance of the devices are inadequate. Technical support for telehealth does not refer only to the identification and repair of breakdowns,⁸ as technology^24,28 is constantly evolving and requires constant investigation and evaluation, for innovative health and data management.

This study did not find an association between quality telehealth and quality of care. Although telehealth has been used for many years in Brazil, its use for primary care assistance is still small.²³ The teleconsultations conducted are still not able to have a relevant impact on the care process, as they are still negligible in relation to the consultations carried out within the sphere of primary care in these surveyed municipalities and in the country as a whole.^1,29 This situation may be interfered with by the lack of correlation between assistance and the quality of telehealth services.

This outcome could also be influenced by the fact that ISO 13131 includes more parameters⁴ for financial management, facility management, and data security,³⁰ which most municipalities provide; however, the most recurrent parameters for the quality of care are related to the implementation, use, and infrastructure of resources to provide health care.³¹ This result emphasizes the need to further develop this theme in Brazil.

Despite the lack of association between quality telehealth and quality of care, the dimension of Quality and Risk Management in telehealth was associated with the level of quality of care. This dimension considered effective use and the level of use of telehealth; the number of primary care teams in the municipality; if the teams are from rural or urban areas; the percentage of teams using telehealth; year the municipality joined the program; and if the municipality has any center with medical specialties. This association reveals that these points influence the final quality of care provided to the patient.

This work has demonstrated that the ISO/TS 13131 specification provides useful guidance to assist development of telehealth services, but each organization is responsible for developing appropriate guidelines for each health service. As Taylor⁹ stated, the transition from a face-to-face service to a telehealth service will require a change in the way health services design and manage safety and quality.

Among the analyses conducted in this study, a reflection of the results presented was sought, specifically in a Brazilian context.²⁰ Taylor⁹ also pointed to guidelines for quality in the provision of telehealth services. Innovation is necessary in aspects of workforce planning to ensure that managers and health workers have the additional skills related to telehealth, including qualifications and skills to safely provide high-quality health care. Expanding health professionals' knowledge and responsibility is important to determine whether the use of telehealth in the delivery of a health service is appropriate.

The results point to the need for advances in the telehealth planning process, as specified in the Canadian ISO15, which recommends that a telehealth service should have a document describing the service and the health care process via Virtual Health in the context of the health program. These involve: Objectives; to whom and when the service is available; Service Level Agreements; Adequate clinical objectives and model of care or shared care; Ability of patients to choose the health service they consider appropriate to access; Expected duration of the service, when they occur; and criteria for inclusion and exclusion of patients.

Despite all these results, this study has some limitations. One of them is the adjustment variable of the study. The PMAQ²¹ stratum was used for the adjustment, but other epidemiological and sociodemographic parameters could assist in understanding the association identified between the size of the municipality and the quality in telehealth. The study was carried out by an interview, and personal perceptions may interfere with some responses. Despite the difficulty in reaching the municipalities, the sample obtained was highly representative in relation to the range of the study.

Conclusion

The telehealth resources have been used in the municipalities of Minas Gerais, which mostly provide high or medium degrees of quality telehealth. Smaller municipalities have higher levels of use and quality in telehealth, demonstrating the importance of telehealth for these locations.

The result of the questionnaire based on ISO 13131 found that the main difficulties or obstacles present in telehealth, highlighting issues related to risk management, specialized human resources, and management of telehealth assistance. Telemedicine does not have an exclusive focus on technology, communication, and IT, but it is a multiprofessional activity, which involves management and planning.

Considering the findings of this study and based on reflections from the literature, we can systematize the following aspects that can help to improve the quality of telehealth services in Brazil: structurally reinforce the specific responsibilities of municipal managers and greater involvement of technical references of municipalities in telehealth management and quality, with the sharing of responsibilities with the centers and health professionals; adaptation of health planning for better data sharing and patient care in a safe way; and changes in planning beyond the face-to-face clinical practice, which consider and adapt to the differences in care via telehealth.

The lack of association between the general levels of telehealth quality and quality of health care expands perspectives on the factors related to the dimensions of quality that should or should not be prioritized in the structuring of telehealth activities. Telemedicine represents an advance in health care; however, its development and use are still far below its potential.

Footnotes

Acknowledgments

The authors thank the team at the Health Technology Center of Federal University of Minas Gerais, and they also thank all who, in some way, have contributed to the completion of this study.

Author Disclosure Statement

No competing financial interests exist.

Funding Information

This study did not receive additional or specific funds for its performance.

References

Maldonado

JMSV

, Marques

, Cruz

. Telemedicine: Challenges to dissemination in Brazil. Cad Saude Publica, 2016; 32:e00155615.

Brazil. Ministério da

Saúde

. Gabinete do

Ministro

. “Portaria n° 2.546, de 27 de outubro de 2011. Redefines and expands the Telehealth Brazil

Program

, which is now called the Telehealth Brazil Networks Program (Telessaúde Brasil Networks). Brasília

Brazil

: Diário Oficial da União. 2011.

Khoja

, Durrani

, Nayani

, Fahim

. Scope of policy issues in eHealth: Results from a structured literature review. J Med Internet Res, 2012; 14:34–42.

ISO Technical Specification ISO/TS 13131:2014 —Health informatics—Telehealth services—Quality planning guidelines

International Organization for Standardization

. 2014. Available at www.iso.org/iso/home/store/catalogue_tc/catalogue_detail.htm?csnumber=5305 (last accessed June 19, 2018 ).

World Health Organization. Telemedicine: opportunities and developments in Member States: report on the second global survey on eHealth. World Health Organization Global Observatory for eHealth. 2010. Available at https://apps.who.int/iris/handle/10665/44497 (last accessed May 12, 2020 ).

Harzheim

, Katz

, Ferri

, Fernandes

, Barbosa

. Guia de avaliação, implantação e monitoramento de programas e serviços em telemedicina e telessaúde. Porto Alegre, Brazil: Universidade Federal do Rio Grande do Sul, Hospital Alemão Oswaldo Cruz, 2017.

Messina

, Dos Santos

, Fernández

, Melo

MCB

, Alves

. Desarrollo de la telesalud en América Latina: Aspectos conceptuales y estado actual. Comissão Economica para America Latina e Caribe das Nações Unidas. Chile, 2013.

Quinto

, Gastal

. Acreditaçäo hospitalar: proteçäo dos usuários, dos profissionais e das instituiçöes de saúde. Acreditaçäo hospitalar: Proteçäo dos usuários, dos profissionais e das instituiçöes de saúde. Porto Alegre, Brazil: Editora Dacasa 1997:133–133.

Taylor

. Applying International Guidelines for Telehealth Services: A Case Study. Proceedings of the 8th Australasian Workshop on Health Informatics and Knowledge Management. Sydney, Australia: HIKM, 2015; 87–95.

10.

Herrington

, Zardins

, Hamilton

. A pilot trial of emergency telemedicine in regional Western Australia. J Telemed Telecare, 2013; 19:430–433.

11.

ACSQHC

Standard

, Quality Improvement

Guide

. Australian commission on safety and quality in health care. Safety and quality improvement guide standard 1. Sydney, Australia: Governance for Safety and Quality in Health Service Organisations. 2012:31.

12.

ACRRM—Australian College of Rural and Remote Medicine. ATHAC telehealth standards framework ACRRM telehealth guidelines. Australia. 2016:12. Available at www.ehealth.acrrm.org.au/sites/default/files/ACRRM%20Telehealth%20Guidelines_2016.pdf (last accessed April 02, 2020 ).

13.

Canada's Health Informatics Association. Canandian telehealth report. Canada. 2015. Available at https://livecare.ca/sites/default/files/2015%20TeleHealth-Public-eBook-Final-10-9-15-secured.pdf (last accessed June 16, 2020 ).

14.

College of Physicians and Surgeons of

ALBERTA

. Telemedicine—standard of practice. CPSA, Canada. 2014. Available at www.cpsa.ca/standardspractice/telemedicine/ (last accessed September 26, 2020 ).

15.

Telestroke Implementation Toolkit. Canadian heart and stroke foundation. 4th Edition. Canada. 2013. Available at www.strokebestpractices.ca/wp-content/uploads/2013/09/Telestroke-Best-Practice-Recs-and-Toolkit-201309_30_2013.pdf (last accessed September 20, 2020 ).

16.

TeleSCoPE. Telehealth services code of practice for Europe. 2013. Available at https://www.joinup.ec.europa.eu/collection/ehealth/document/telescope-telehealth-services-code-practice-europe-telescope (last accessed October 18, 2020 ).

17.

ATA—American Telemedicine Association. Practice guidelines. Available at https://www.americantelemed.org/resource_categories/practice-guidelines/ (last accessed September 30, 2020 ).

18.

Santos

. An updated view of the process of incorporating technologies into Brazil's Primary Health Care Network. Latin Am J Telehealth, 2019; 6:112–125.

19.

National Standard of Canada. CAN/HSO 83001:2018. Virtual Heath. ICS CODE: 11.020.01. Canada. 2018:31. Available at https://www.store.healthstandards.org/products/virtual-health-can-hso-83001-2018-e (last accessed October 18, 2020 ).

20.

Brazil Ministério da Saúde. Secretaria de Atenção à Saúde. Primary Care Department. Improving access and quality in primary care national program (PMAQ): Instrumento de avaliação externa para as equipes de atenção básica. Brasília, Brazil: Ministério da Saúde, 2013.

21.

Vieira

Alfa de Cronbach. In: VIEIRA, Sonia. Blog da Profa Sonia Vieira Estatística. São Paulo. Available at http://soniavieira.blogspot.com/2015/10/alfa-de-cronbach.html (last accessed August 1, 2019 ).

22.

IEC 80001-1:2010: Application of risk management for IT-networks incorporating medical devices. Part 1: Roles, responsibilities and activities. Geneva, Switzerland: International Organization for Standardization, 2010.

23.

Brazil. Ministério da Saúde, ed. Plano de ações estratégicas para o enfrentamento das doenças crônicas não transmissíveis (DCNT) no Brasil: 2011–2022. Brasília, Brazil: Ministério da Saúde, 2011.

24.

Brazil Ministério da Saúde. Estratégia E-saúde para o Brasil. 2014. Available at www.cspace.eportuguese.org/tiki-download_file.php?fileId=1173 (last accessed October 28, 2018 ).

25.

Wen

. Telemedicine and telehealth: an overview in Brazil. Public Informatics, 2008; 10:7–15.

26.

Tončinić

, de Wildt-Liesveld

, Vrijhoef

. Evaluation of a digital platform that engages stakeholders in the co-creation of healthcare innovations: A mixed-methods study. Int J Care Coord, 2020; 23:33–42.

27.

Brazilian Association of Technical Standards. Quality management systems: Fundamentals and vocabulary: NBR ISO 9001. Rio de Janeiro, Brazil. Brazilian Association of Technical Standards, 2000.

28.

Atlas of eHealth country profiles:. The use of eHealth in support of universal health coverage: Based on the findings of the third global survey on eHealth 2015. Geneva, Switzerland: World Health Organization, 2016.

29.

Sabbatini

RME.

A telemedicina no Brasil: Evolução e perspectivas. Informatica em Saúde: Uma Perspectiva Multiprofissional Dos Usos E Possibilidades. São Caetano Do Sul, Brazil: Yendis, 2012:1–16.

30.

Hee

, et al. A study on establishing a technical safety assessment system for the Korean telemedicine system. J Korean Med Assoc, 2015; 58:1159–1170.

31.

Brazil Ministério da Saúde. Secretaria de Atenção à Saúde, Primary Care Department. National program for improving access and quality in primary care. Health closer to you: Access and quality national program to improve access and quality of primary care (PMAQ-AB). Brasília, Brazil: Ministério da Saúde, 2012.