Abstract
Introduction:
In the wake of COVID-19, widespread adoption of telehealth has led to variations in implementation that have implications for patient safety and access to health care. We developed a 60-item Telehealth Access Checklist, which aims to address this challenge by supporting health care organizations in assessing their telehealth platforms for issues that may pose access challenges.
Objectives:
This feasibility study evaluates the Telehealth Access Checklist among three different large health care organizations. Eleven different reviewers ranging from primary care physicians to telemedicine directors evaluated the checklist’s content for efficacy and utility. Suggestions were made for checklist improvement.
Results:
The preliminary testing revealed overwhelming support for not only the concept of the 60-item checklist but also the checklist itself. Reviewers were able to identify aspects of their telehealth technologies that could be optimized to improve accessibility. They found the checklist to be an effective guide to delve deeper into their respective facility’s capabilities and identify areas needed for improvement to meet the needs of all populations. Recommendations for improvements to the checklist ranged from changes in formatting and phrasing to enhancements such as broadening the scope to include other modes of telehealth, such as remote patient monitoring.
Conclusion:
A self-assessment checklist may serve to improve telehealth technology in meeting the needs of all populations by enabling the identification of technological features that should be optimized.
Introduction
In response to COVID-19, immense transformations of health care have occurred. Efforts ranging from hospitals creating new intensive care units to the rapid development and implementation of new technologies have taken place out of both necessity and due to regulatory changes intended to broaden care access. 1 One of the more apparent developments has been the expansion and refinement of telehealth.2–4 The use of telehealth has swelled, and widescale adoption of the telehealth care modality has resulted in extensive variations in practice with direct implications for patient safety and health care access. 5 These variations in practice are due to numerous factors, including some organizations racing to establish telehealth access points as quickly as possible with limited time for full evaluation for safety and health care access implications, a lack of usability testing and standardization of telehealth platforms, and the need to customize telehealth platforms to meet existing health care environment constraints such as interoperability with other information technology systems.6–7
A telehealth standard of care, promoted at federal and state levels, outlines a provider’s responsibility for ensuring that virtual telehealth visits match the requirements of in-person visits, such as performing an adequate patient evaluation.8–9 However, to perform an adequate patient evaluation, properly document the visit, and establish a suitable patient-provider relationship, the provider must have infrastructure and technology that can enable the standard of care. Currently, there are few, if any, tools that support health care facilities in assessing their infrastructure and technology for meeting standard of care needs that serve all populations. 10 There have been efforts to support health care facility assessment of other technologies, such as an electronic health record (EHR) usability and safety assessment tool that elicits common usability issues. Using this tool, health care facilities have identified several usability and safety issues in their EHRs, with recommendations provided for how to address these issues. 11 The EHR assessment tool serves as an example of a low-cost, scalable method for a health care facility to assess technology capabilities. Use of these types of tools provides a rapid assessment method that can benefit health care facilities and patients, especially those facilities with limited resources to conduct safety.
Evaluations of technology should also consider health care access and the opportunity for everyone to receive rightfully needed or deserved health care regardless of social or economic standing. 12 In turn, researchers have developed an evidence- and consensus-based Digital Health care Framework to guide users when generating solutions involving digital technologies. 13 The framework, which aims to ensure solutions improve health care inequities rather than exacerbate them, is built upon three domains. Through the consideration of key characteristics involving patient and community, system, and health information technology, health systems can address disparities during every phase of digital health care. Ultimately, the framework views equity and access as an essential domain of quality digital health care, and in considering frameworks such as this, health care systems can improve their overall digital care practices for underserved communities and move towards a more complete standard of care.
To further address this challenge, we have developed a telehealth checklist to support health care organizations in assessing their current telehealth platform capabilities to identify opportunities for telehealth access improvements. This checklist supports identification and discussion of potential issues that may contribute to telehealth technologies not serving all populations to their fullest ability. The checklist is not intended to provide solutions to these challenges. Rather, health care facilities should work with their telehealth developers and other partners to address the challenges that are identified through use of the tool.
To evaluate the value and usefulness of this checklist, we conducted pilot testing across three health care organizations. Our evaluation focused on whether the checklist was perceived as valuable and useful, whether it supported health care facilities in assessing their own telehealth capabilities, and how the tool could be designed to better support the cyclical review of telehealth capabilities.
Methods
CHECKLIST DEVELOPMENT AND OVERVIEW
The checklist development process was guided by a multidisciplinary approach to promote a comprehensive review of barriers to telehealth access. A critical component of this process was the development of the personas (Fig. 1) that represented key user groups that may face telehealth access challenges. These personas were used to inform the checklist’s design by highlighting real-world barriers and user needs.
Example of a persona used to develop checklist.
The persona development process began with an effort to understand who might use telehealth and their access barriers, focusing on perceptions of care quality, trust (e.g., data security and privacy), and ease of use. Personas were then created to represent individuals and groups facing access challenges, considering factors such as race, sex, age, education, socioeconomic status, geography, and insurance. These personas accounted for overlapping and interacting factors that collectively shaped access barriers. Ultimately, a total of six personas were developed that focused on key groups that may encounter access challenges, such as low digital literacy, visually impaired, intellectual disabilities, and the deaf/hard of hearing.
The research and development of personas was led by two research scientists and iteratively reviewed by a multidisciplinary human factors team that included a registered nurse, a palliative care physician, and a chief medical officer specializing in telehealth. Feedback from this team was integrated into the personas, promoting their relevance and applicability. In addition to the persona development, feedback from telehealth reviews, satisfaction surveys, and social media complaints was also considered to inform the checklist. These sources provided valuable insights into user experiences and further helped identify key barriers to access. While the personas were not included in the final checklist, they played a pivotal role in shaping its content by identifying key user needs and barriers, ultimately ensuring that the checklist addressed critical factors affecting telehealth access.
CHECKLIST FUNCTIONALITY
The checklist is organized to consider the entirety of the patient journey. In turn, it encourages evaluators to consider the health care facility’s capabilities over the course of a typical visit beginning with preappointment considerations, considerations during the appointment, and postappointment considerations. The checklist also includes scenario-specific considerations that may be applicable to some but not all organizations, such as patients with low digital literacy or those who have visual impairment. The checklist features instructions for use that describe the purpose of the checklist, its intended audience, how to use the checklist, and the telehealth considerations organized in a temporal format. The checklist is user-agnostic in that a review of telehealth capabilities by the health care facility can be conducted either by a single person familiar with the telehealth technology or in a small team with members from direct care, management, and director-level positions. It is recommended the checklist be completed among a team of two to four members who know the organization’s telehealth resources well.
To use the checklist and assess telehealth capabilities, the user first reads Column 1 (Considerations). The checklist can be found in Supplementary Data. This column proposes patient/provider needs that should be considered in the improvement of accessible telehealth. The user then rates the facility by assigning a maturity level (ranging from developing to defined to managed) by checking the appropriate level in Columns 2–4 with the specific consideration in mind. The ratings are defined as follows:
While the ratings are not associated with any numerical values, they serve as a visual representation and reminder of the multitude of detail that is required of optimized and accessible telehealth. Therefore, there is no final score or rating when the participants complete the checklist. Should the user not be able to appropriately rate the facility’s capabilities and would like to refer the consideration to a team member, they are able to refer that consideration to their team member in Column 5. If the consideration is not applicable or not possible for the facility, they are instructed to mark Column 6. An example of the structure of the checklist can be found below in Table 1.
Instructional Example
PARTICIPATION
The research team used a convenience sampling of three health care facilities to apply and evaluate the checklist. Each participating organization had a point of contact that acted as a liaison and assisted in facilitating the use of the checklist. Additional participants were encouraged. Ultimately, each organization had two to three additional reviewers (excluding the liaison) participate in the review. Organization A was a multihospital system of the United States Western region, organization B was a multihospital system of the United States Mountain region, and organization C was a multihospital system of the United States Eastern region. A breakdown of the participants can be found in Table 2. All participants received informed consent prior to being involved in this study. This study was approved by the MedStar Institutional Review Board.
Summary of Participants
PROCEDURES
A human factors researcher led recruitment and initiated a 30-min virtual meeting with the liaison from each organization to (1) reiterate the purpose of the feasibility testing, (2) to explain the procedures of the testing, and (3) to answer any questions. To ensure the research team did not influence the participants’ reactions or experience, the meeting did not include instructions on how to use the checklist but instead only explained the purpose of the checklist and the procedures of the review.
The research team instructed the participating organizations to gather a team of two to four members with a strong working knowledge of their telehealth capabilities (preferably of a primary care facility, although not required). Once the organizations had a team assembled, they were instructed to review the checklist in its entirety (which took approximately 45 min) and shortly thereafter, complete a five-question, open-ended evaluation form. This survey acted as a high-level evaluation of the checklist and included queries on first impressions, whether the checklist fulfills its intended purpose, obstacles or challenges during use, additional considerations that should be added to the checklist, and how likely they are to use it in the future. The participating organizations were given the freedom to complete the evaluation at their leisure and to reach out with any questions. The results of each evaluation were aggregated into a spreadsheet and reviewed by the research team for themes and areas for improvement. An institutional review board approval was obtained prior to the beginning of the study.
Results
There was variability in maturity ratings within and across the three organizations, Table 3. Organization A rated nearly all the 60 considerations of the checklist as managed (n = 50 of 60, 83%), with some considerations (n = 9 of 60, 15%) assessed to be developing. The considerations rated as developing were mainly related to aspects such as software that provides transcripts after a telehealth appointment for the deaf/hard of hearing or ensuring compatibility with screen-reading software for the visually impaired. Organization B also rated most considerations to be managed (n = 50 of 60, 83%), with the rest of the considerations being rated as defined (n = 10 of 60, 17%). The considerations marked as defined were related to technology access and appointment assistance, such as establishing methods to assess a patient’s technological needs or establishing mechanisms to send and track referrals. Organization C measured the checklist considerations on a more granular level by location and stated that the capabilities depended on the location 55% of the time. Of the capabilities that were implemented organization-wide, 2 of 60 (3%) were still developing, 7 of 60 (12%) were defined, and 18 of 60 (30%) were successfully managed. Organization C areas of improvement are related to patients that are hard of hearing, have low digital literacy, and limited English proficiency.
Ratings of Each Organization’s Assessment
Organization C stated that each facility is separately responsible for several telehealth considerations. Therefore, a new column was added ad hoc to denote this dependency.
The evaluation feedback from each organization was synthesized and is described below.
IMPRESSIONS OF THE CHECKLIST
First impressions of the checklist were positive, and all organizations found the checklist to be thorough and insightful. Importantly, participants found that it covered multiple domains of telehealth access well. However, some participants would prefer a more systematic table of contents and definitions of terms that orients the reader to the contents prior to starting. For example, participants requested a comprehensive list of specific access considerations (e.g., deaf/hard of hearing, visual impairment, and physical disability) along with their associated definitions to better understand the scope of unique scenarios and their associated considerations.
CHECKLIST PURPOSE
Although all three organizations found that the checklist does serve the purpose of assessing telehealth platform capabilities to identify opportunities for access improvements, one participant noted that it is limited in scope. The checklist is mainly applicable to audio/video visits. It does not include other elements of telehealth like remote patient monitoring (RPM) or asynchronous care. To establish a more complete assessment of technology, it was noted that those other modalities of telehealth should be considered for inclusion in a future integration of the checklist.
POTENTIAL USE OF THE CHECKLIST
All three organizations unanimously agreed that this checklist will be used in the future once a final version becomes available. They noted likely use cases such as when developing new telehealth programs from inception, when migrating from one video platform to another, and a supplement to financial/staffing projections and assessing compliance issues. Others urged the checklist to be published and made available not only to current telehealth providers but also to learners such as medical students and residents.
ADDITIONAL CONSIDERATIONS
One organization noted no additional considerations were needed, while the other two offered additional insight. First, the checklist should be reassessed periodically to stay current and relevant. The participant added as artificial intelligence (AI) becomes more ubiquitous, there may be increased risk of hidden biases within tools used during telehealth visits leading to inequitable outcomes. A challenge remains around how organizations should screen for this possibility and how this screening should be implemented into the checklist. A second improvement revisits a shortcoming from earlier in the evaluation—the checklist should include categories for nonvideo types of telehealth such as RPM and asynchronous care.
Discussion
Designing telehealth applications to improve accessibility is critical to offering care to all patients. Our checklist and feasibility study focused on whether the checklist was useful and if it fulfilled its purpose of assessing telehealth through a more accessible lens. The results show that sites using the checklist were able to identify areas for improvement, and looking across sites, there was variability in the maturity level ratings, suggesting the checklist is responsive to differences by consideration. While these results are promising, additional testing of the checklist is needed to assess potential variations in acceptance and usability for organizations of different sizes and technological capabilities. For instance, smaller facilities may be unable or unwilling to undertake an assessment of this size, and further testing of the checklist may promote more scalable and practical versions of the assessment that consider a spectrum of required capabilities rather than only catering to the largest health systems.
Organizations that participated in the feasibility study collectively agreed that the checklist has tremendous utility and value. Its breadth of scope prompted participants to delve deeper into their respective facilities’ capabilities and effectively indicate areas needed for improvement. It also raises the important question of how health care organizations should implement telehealth solutions at the organizational level. For example, some organizations are prescriptive about telehealth capabilities and features, with each specific health care facility following certain guidelines, while others enable some amount of facility level customization. For example, organization C allows customization at the facility level, which might offer benefits to aligning the technology with subpopulation needs but may also pose challenges with standardization of key technological capabilities. The motivation behind providing customization at the facility-level should be researched further, as it may explain whether governance of technologies or access to resources (whether they be financial, technical, or personnel-related) are the overriding factors in how capabilities are allocated across a system. Additional data are needed to understand how these different models of governance might impact telehealth capabilities and patient access.
While reception of the checklist was positive, it is not without opportunities for improvement. For instance, the checklist only considers traditional video/audio-based telehealth visits while overlooking less common but equally important modes of telehealth such as RPM, asynchronous visits, mobile health (m-health) services, E-consultations (provider-to-provider communication), and text-based visits. Additionally, the checklist offers no plan or scope of maintenance, which is necessary to remain relevant and useful in the age of rapidly developing technologies and AI. Future work to further evaluate and validate the checklist can include reliability testing in which multiple evaluators of the same organization independently use the checklist and compare results. The checklist may also benefit from validation testing in which criterion are created to test the checklist’s objective and purpose. Of course, an established method of keeping the checklist current, useful, and effective is also needed to maintain relevance.
Conclusions
The overall feasibility study demonstrated that the accessibility-focused telehealth checklist is a valuable tool for identifying appropriate telehealth considerations and prompting deeper organizational assessment. While it was well-received, the checklist would benefit from an expanded scope to include other methods of telemedicine and validation to ensure long-term utility. There are limitations to our study. The checklist aims to focus on numerous considerations of digital telehealth technology and appointments, but it is in no way exhaustive. Additionally, the checklist only serves as a method of assessment and does not provide specific solutions if areas for improvement are identified.
Footnotes
Author Disclosure Statement
No competing financial interests exist.
Funding Information
This work was supported by grant number R18HS029117 from the Agency for Health care Research and Quality.
Supplemental Material
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
