A comprehensive evaluation tool to assess community capacity and readiness for virtual care implementation

Introduction

Despite virtual care's increasing prevalence in healthcare delivery, the rapid evolution of technology often outpaces the development of standardized implementation protocols necessary to direct safe and effective use.^1,2 As virtual care solutions become increasingly integral to efficient healthcare delivery, this gap between technologies and their implementation protocols can impede equitable and sustainable health system adoption and integration.^3,4

The complexity of implementing virtual care programs in existing healthcare systems is difficult to overestimate. In some settings, virtual care might be deployed efficiently with seamless integration into existing healthcare structures. ⁴ In contrast, in rural and remote communities with unique challenges and characteristics including geographic remoteness, limited workforce and variable expertise and experience in digital health, the lack of guidelines can lead to inefficiencies, underutilization or even mismanagement of these technologies.^2,5 As a result, there is a growing recognition of the need for tools and frameworks that can standardize the implementation of virtual care, ensuring that all communities, regardless of their geographic location or socio-economic status, can benefit from these technological advancements.^6,7

The purpose of this paper is to describe a new comprehensive evaluation tool designed to assess community capacity and readiness for virtual care implementation, including its development process and functionality. The tool evaluates several key areas including community clinical needs, technological infrastructure, human resources and clinical workflows, to identify and mitigate potential barriers to deploying virtual care technologies. We assess the utility of the tool in informing the implementation of a telerobotic ultrasound system in an isolated northern Canadian community restricted to fly-in only accessibility. Informed by experiences from the Virtual Care and Robotics Program at the University of Saskatchewan as well as the literature surrounding eHealth and digital health readiness, this tool can play an important role in qualifying remote sites as appropriate candidates for virtual care services. We make this tool available to virtual care and information technology (IT) practitioners, leaders and researchers to use or adapt for their local context.^8,9,10

Methods

Process for development

The comprehensive evaluation tool was developed using an iterative and multi-stakeholder collaborative approach, engaging stakeholders from diverse backgrounds in a cyclical process of feedback and refinement. ¹⁵ The development process acknowledged that a co-creation engagement approach with diverse partners is needed to fully understand key contextual characteristics. ¹⁵ Highlighting the importance of individual and group agency in implementation design, ¹⁶ an environment was fostered where stakeholders were encouraged to make meaningful and purposeful contributions, influencing specific aspects of the tool while responding to the diverse needs and insights of all parties involved. The development of the tool drew upon insights and recommendations from representatives from the Virtual Care and Robotics Program, the Saskatchewan Ministry of Health, eHealth Saskatchewan, Northern Medical Services, and clinicians with experience working in Saskatchewan's remote communities. Collectively, this multidisciplinary group brought together experts and leaders in the fields of IT, healthcare and project management, while facilitating collaboration across multiple levels of government. This co-creation process allowed for the incorporation of local insights and needs into the tool, ensuring it was adaptable and responsive to the unique challenges of each community.

The first interactions between stakeholders to inform the development of the framework occurred organically through in-person and virtual meetings, emails and telephone calls from March to July 2018. This initial phase of development involved defining the primary categories for evaluation, followed by a delineation of specific areas of assessment within each category. Multiple cycles of stakeholder feedback and revisions aimed to capture all critical aspects necessary to comprehensively assess the compatibility of virtual care systems within the distinctive fabric of any community. Subsequently, the evaluative domains were visually mapped using user-friendly flowcharts, completed by January 2019 before later simplification into a checklist manual by May 2023 (Figure 1), with the goal of enabling communities to drive their own evaluations in collaboration with the Virtual Care and Robotics Program, enhancing local engagement and implementation.

OPEN IN VIEWER

Figure 1.

Developmental process and timeline.

Results

Components of the tool

Three domains were identified as critical to assess community capacity and readiness for virtual care implementation and were included in the evaluation tool: (1) clinical needs assessment, (2) technology readiness assessment and (3) human resources and workflow assessment (Figure 2). The complete tool is available in Supplemental Information for virtual care and IT practitioners, leaders and researchers to use or adapt for their local context.

OPEN IN VIEWER

Figure 2.

Three domains of the tool – clinical needs assessment, technology readiness assessment, and human resources and workflow assessment.

(1) Clinical needs assessment

A foundational component of the tool is the facilitation of a clinical needs assessment within remote communities. This assessment is designed not to impose a predetermined set of criteria but to assist communities in identifying and articulating their specific healthcare needs. By doing so, the tool places the power of decision-making into the hands of the community members themselves, recognizing that they are best positioned to understand their unique healthcare challenges and priorities.

The tool guides communities through a comprehensive evaluation of their healthcare landscape, including identifying prevalent health issues, gaps in current healthcare services and potential areas where virtual care could be most beneficial. It offers guidance on how to conduct this analysis, suggesting critical focus areas and recommendations for collaborating with healthcare authorities and service providers to gather necessary health information to support the assessment's conclusions. Ultimately, the community leads the discovery and decision-making process, ensuring that the implementation of virtual care is tailored to address their most pressing clinical needs, and fostering a sense of ownership and commitment among its members.

(2) Technology readiness assessment

Virtual care technologies have baseline system requirements that must be met to ensure the consistency, quality and reliability of their functions. Before communities can be considered for implementation, their healthcare facilities must have the requisite IT infrastructure and support systems in place to meet or exceed these standards. The tool assesses facilities’ IT environments with a focus on essential elements including communication bandwidth, Wi-Fi access point distribution and Quality of Service (QoS). QoS is the use of mechanisms to manage and prioritize network traffic, ensuring performance levels for specific technologies in environments where bandwidth is limited. In instances where a community's IT framework exhibits deficiencies, our team offers guidance and mentorship to help bridge these gaps, thereby facilitating the pathway towards technological readiness.

Our experience in dealing with technical challenges has highlighted the critical role of designating a specialized technical liaison within each community. This role is critical for streamlining the management of technical issues related to virtual care systems. The liaison acts as a primary point of contact and first responder to any technical challenges, liaising with our technical team for efficient testing and troubleshooting. This approach provides a reliable and responsive first-line of defence to maintain smooth operations by addressing technical issues as they arise.

(3) Human resources and workflow assessment

The integration of a human resources and workflow analysis component into the tool reflects an understanding of the practical side of virtual care implementation. Recognizing that technology is just one facet of successful virtual care delivery, this component focuses on a team evaluation of the available human resources and existing healthcare workflows within the healthcare facility and ensuring alignment with new operational protocols. This includes analyzing the skill sets of local healthcare workers, their capacity to adapt to virtual care technologies, and the need for further training or support.

Additionally, the tool evaluates the potential impact on the workload of healthcare providers, the adaptability of current practices to incorporate virtual care, and the overall readiness of the healthcare system to embrace and sustain this change. The tool maps out a step-by-step workflow standard for the newly operating technology, ensuring that duties and responsibilities are assigned to and accepted by specific individuals. By including this human resources and workflow evaluation, the tool ensures that the implementation of virtual care is not only technologically feasible but also practically viable and compatible with the community's existing healthcare structure.

Case study: results from the Fond du Lac pilot

The evaluation tool was successful in qualifying the community of Fond du Lac for the implementation of telerobotic ultrasound services. In addition to confirming this compatibility, the tool facilitated a nuanced understanding of the community's unique needs and capacities. Highlights of the evaluation and implementation process are summarized in Figure 3.

OPEN IN VIEWER

Figure 3.

Highlights of Fond du Lac’s evaluation and implementation of telerobotic ultrasound.

Discussion

The comprehensive evaluation tool presented in this paper serves as a practical guide to assessing the capacity and readiness for virtual care implementation in rural and remote communities with variable resources, expertise and experience. The tool addresses the critical gap between the rapid evolution of virtual care technologies and standardized protocols necessary for their effective, equitable and sustainable deployment in remote communities. By assessing community capacity and readiness across multiple domains, including clinical needs, technological infrastructure, human resources and workflows, the tool provides a systematic approach to identifying and mitigating potential barriers to virtual care implementation.

Key readiness types commonly identified in the eHealth and digital health readiness literature include organisational readiness, technological/infrastructural readiness, healthcare provider readiness, engagement readiness, societal readiness, core readiness, government readiness and public/patient readiness. ²³ However, Mauco et al. found that none of these readiness types are comprehensively addressed by current e-health readiness assessment frameworks. ²³ The authors emphasize the need for context-specific tools that cater to the unique challenges faced in specific regions, including infrastructure limitations, socio-cultural dynamics and varying levels of ICT literacy. ²³ There is a notable absence of standardized and reliable tools for assessment of e-health readiness and consistent application across diverse healthcare settings, contributing to variable results and high failure rates. ²⁴ In contrast to prior research on eHealth and digital health readiness, one of the key distinctions of the tool which we present in this paper is its algorithmic approach. Unlike most prior studies which focus on identifying readiness types and constructs for evaluation, this tool provides a finite, structured set of instructions to solve specific problems surrounding various stages of readiness. As a result, it offers a more actionable, reproducible and transparent process, making the steps required for implementation clearer and more defined. The tool supports the domains included by Katz et al. in their modified Delphi study evaluating constructs of a telemedicine framework for designing and implementing telemedicine interventions. ²⁴

Although telehealth adoption surged during the COVID-19 pandemic, many vulnerable populations, such as those in lower socioeconomic areas, people with disabilities, and individuals with low digital literacy, face significant barriers to accessing these services. ²⁵ Previous research into identifying and assessing these barriers has highlighted that co-designing telehealth-enabled models of care with consumer involvement to ensure culturally appropriate solutions is essential to bridging gaps to equitable access. ²⁵ Furthermore, a multi-stakeholder approach that includes a focus on workforce training and improving digital health literacy is deemed crucial for success. ²⁵ These identified themes are congruent with not only the multi-stakeholder approach for designing the tool, but the collaborative processes within, emphasizing the practical side of workflow planning and workforce training. The tool expands on the existing literature by offering organizations an actionable, step-by-step approach, helping to parameterize and navigate the complexities of virtual care adoption. Prior work has highlighted the challenges faced by remote and rural communities in adopting virtual care solutions due to a lack of standardized implementation protocols. ²⁶ Additionally, with the underrepresentation of well-defined frameworks for interacting with Indigenous communities to deliver virtual healthcare, previous studies have indicated a heightened potential for causing harm.^25–28 The tool builds upon this literature by offering a practical framework that draws upon firsthand experiences and insights gained from implementing virtual care solutions in remote settings. By engaging stakeholders from various sectors including healthcare, IT and project management, the tool is both comprehensive and user-friendly, guided by principles of equity, practicality, sustainability and flexibility. ¹⁶

By engaging a diverse group of representatives, the tool was not only customized to address the unique needs and challenges of communities, but also responded to the priorities of multiple stakeholders. For example, the Ministry of Health was motivated to produce a tool that was transparent and politically defendable in the context of resource investment into virtual care, whereas members of the Virtual Care and Robotics Program needed an actionable framework to scale up virtual care operations in a practical and standardized fashion.

The pilot study in Fond du Lac demonstrated the utility and effectiveness of the evaluation tool in qualifying a remote community for the implementation of telerobotic ultrasound services. By systematically assessing clinical needs, technological infrastructure and human resources, the tool identified and addressed key considerations for successful implementation. The incorporation of virtual care technologies such as telerobotic ultrasound through the application of the evaluation tool has the potential to increase service capacity, reduce costs, promote interprofessional collaboration, enhance patient/provider experience and ultimately improve the health of more underserved populations such as Fond du Lac.

Engaging closely with Fond du Lac's team members to complete all sections of the comprehensive evaluation tool highlighted the critical importance of collaborative processes in securing the buy-in necessary for successful virtual care implementation. By mentoring the clinical needs assessment, walking through workflows, helping to assign tasks and responsibilities, working with IT personnel and enlisting a local technical liaison, the research team effectively built relationships, garnered commitment and gained the necessary influence for adoption and uptake. Tools that promote interdisciplinary collaboration and co-development of implementation processes between service administrators and sites, like the method taken in the comprehensive evaluation tool, may be an effective approach towards achieving influence and fostering a sense of ownership from impactful stakeholders.

Despite these promising findings, this study has several limitations that warrant discussion. First, the pilot study was conducted in a single remote community, limiting the generalizability of the findings. Future research should aim to validate the tool in diverse settings to ensure its applicability across different contexts. Second, while the tool assesses community capacity and readiness for virtual care implementation, it assumes the broader system is ready and does not address systemic issues such as funding models or policy frameworks, which may impact the sustainability of virtual care initiatives. Third, to validate the selection of telerobotic ultrasound for Fond du Lac as the most appropriate clinical solution to address the community's healthcare needs, an outcomes assessment will need to be conducted. Finally, ongoing evaluation and refinement of the tool will be necessary to keep up with the evolving landscape of virtual care technologies and best practices.

In conclusion, this comprehensive evaluation tool may contribute to the building of a battery of standardized evaluation and implementation guides for the deployment of virtual care technologies in remote and underserved communities. By systematically assessing community capacity and readiness, the tool provides a practical framework for identifying and addressing barriers to virtual care adoption. Moving forward, further research and collaboration will be essential to refine and validate the tool, ensuring its effectiveness in promoting equitable access to virtual care services across diverse communities.