Using the hierarchy of intervention effectiveness to improve the quality of recommendations developed during critical patient safety incident reviews

Abstract

Our Canadian multi-site academic health sciences centre uses a standardized process to review critical patient safety incidents and develop recommendations to prevent incident reoccurrence. We recognized an opportunity to enhance recommendation development by integrating the Hierarchy of Intervention Effectiveness (HIE), a human factors framework, into the incident review process. This project aimed to increase the proportion of system-focused recommendations from critical incident reviews from 16 to 30% over 16 months. A multi-intervention strategy included (1) standardizing the incident analysis review template; (2) earmarking time for recommendation development during reviews; (3) providing participants with just-in-time education and tools; and (4) initiating HIE-based recommendation classification during incident reviews. Statistical process control p-Chart analysis showed an increase in system-focused recommendations from 16 to 30% over 16 months. The HIE promotes system-level change to prevent critical incidents, which other organizations may benefit from incorporating in their patient safety reviews.

Introduction

Improving the safety of care delivery is a global health priority.¹ In Canada, it has been estimated that every 1 in 17 hospital stays results in at least one harmful event.² In the seminal report, “To Err Is Human,” the Institute of Medicine called for the integration of human factors into healthcare to reduce patient safety incidents,³ defined as “an event or circumstance that could have resulted, or did result, in unnecessary harm to a patient.”^4-6 Further, in the World Health Organization’s Global Patient Safety Action Plan, the authors acknowledge the role that human factors can play in the development of high reliability and resilient systems.¹ Human factors is the discipline concerned with understanding the interactions between humans and their environments to optimize system performance.⁷ Human factors can be applied to understand what led to a patient safety incident and guide the development of recommendations to prevent incident reoccurrence.¹

The Hierarchy of Intervention Effectiveness (HIE), a Human Factors Framework (HFF), has the potential to improve patient safety by designing systems with increased resiliency. The HIE asserts that system-focused changes (e.g., forcing functions, automation and computerization, and simplification and standardization) are more effective at addressing contributory factors identified from patient safety incident reviews than people-focused changes (e.g., education and training, rules and policies, and reminders, checklists, and double checks).⁸ While each level of the HIE can play an important role in error prevention, people cannot be expected to compensate for weak systems.⁹ Selecting changes designed to fix the system, rather than individual behaviour, should be prioritized whenever possible.⁹

At the meso-level of the Canadian healthcare system, hospitals have a responsibility to identify and learn from patient safety incidents.⁵ Our Canadian multi-site academic health sciences centre uses a standardized process to review potential critical patient safety incidents and develop recommendations to prevent incident reoccurrence. Improving the quality of recommendations for patient safety incidents was identified as an organization-wide priority following a retrospective review of recommendations for critical incidents. From January 2020 to December 2022, 16% of recommendations developed for critical incidents were system-focused recommendations. This is not surprising as in healthcare, incident review teams gravitate to people-focused strategies to improve human performance and minimize the potential for adverse events.⁸ Humans are fallible, so there is a need to support people through system resilience improvement.

We aimed to enhance the recommendations developed through our incident review process by integrating the HIE to drive system-focused change from January 2023 to April 2024. By integrating the HIE into our incident review process, we acknowledge what the Institute For Safe Medication Practice has long noted—improvement strategies with the greatest impact on patient safety and the ability to sustain improvement are those that make it hard for practitioners to do their job wrong, and easy for them to do it right.¹⁰ In this article, we share our approach to the integration of the HIE into our incident review process, report on the results of implementation, and share our lessons learned to support other organizations interested in integrating HFF into their organizational review processes.

This project was approved by the organization’s Quality Improvement Review Committee (QIRC) (ID#22-0338).

Methods

Orbitational context

Our multi-site academic health sciences centre has nine hospital sites and also provides post-acute and community care. Incident reviews move through a five-phase process: incident reporting, incident screening, incident analysis and recommendation development, recommendation implementation, and sharing review findings. Patient Safety & Quality Improvement (PSQI) Specialists from the corporate Quality & Safety (Q&S) team facilitate this process for the review of critical patient safety incidents. The incident analysis and recommendation development phase are usually completed during a one-hour virtual meeting where representatives from clinical operations, professional practice, Q&S, and patient experience come together to determine what happened, the contributing factors to the incident, incidental findings, and recommendations to prevent incident reoccurrence. This project targeted the incident analysis and recommendation development phase of this process.

Change idea development

Using the Model for Improvement, this Quality Improvement (QI) project aimed to increase the proportion of system-focused recommendations developed from critical patient incident reviews from 16 to 30% over 16 months (January 2023 to April 2024).

We implemented change ideas to enhance the recommendations stemming from our patient safety incident review process. To identify change ideas, we first held a virtual one-hour working meeting in January 2022 with clinical, operational, and executive leaders across the organization’s multiple sites spanning acute care to rehabilitation. These individuals were selected as they have a key role in the review process for critical incidents (e.g., facilitating the analysis meetings where recommendations are developed and supporting team members through the review process). The working meeting was facilitated by two PSQI Specialists, and themes were identified related to strengths and areas for improvement in the incident analysis and recommendation development phase of the organization’s review process. During the meeting, attendees developed strategies for the standardization of this phase across the organization to improve the quality of recommendations. The organization’s Healthcare Human Factors (HHF) team also introduced attendees to the HIE and solicited their feedback on its applicability to the incident review process. Ultimately, this meeting resulted in the identification of change ideas, which were implemented from January to September 2023 (Table 1).

The project was paused from April to September 2022 for the organization-wide implementation of a new health information system. The project resumed with the validation of four change ideas with leaders who facilitate incident reviews and PSQI Specialists. The project was paused again from February to June 2023, due to healthcare human resource challenges.

Table 1.

Change idea summary.

Change idea	Details
1. Standardize the incident analysis template	• Template used across all sites and programs to support incident review discussion and documentation
1. Standardize the incident analysis template	• Standardized template appearance and formatting to improve incident review within our governance committees
2. Allocate 30 minutes in incident reviews for focused recommendation development	• Anecdotally, at baseline, most incident reviews allocated 5-10 minutes for recommendation development
	• HHF expert opinion suggested a minimum of 30 minutes is needed to apply the HIE for the development of higher-quality recommendations
3. Provide just-in-time HIE education and tools for incident review facilitators	• Just-in-time education and tools provided to leaders who facilitate critical incident reviews from August to September 2023
	• Education provided within existing structures (e.g., leadership-focused email communication and governance committee meetings)
	• Developed an HHF-approved HIE reference for leaders that includes prompting questions to steer the discussion toward the development of system-focused recommendations and organization-specific examples of each level of the HIE
	• PSQI Specialists also received education on the HIE to support leaders with its application
4. Initiate HIE-based recommendation classification during incident reviews	• Classification of recommendations according to the HIE and documentation on the incident analysis template
	• Visualization of the number of people: system-focused recommendations developed

Analysis methods

The outcome measure aligned with the project aim and was the proportion of system-focused recommendations developed from critical patient incident reviews obtained from the organization’s incident reporting and learning system. During the implementation of the change ideas, a Statistical Process Control (SPC) p-Chart was used to analyze the outcome measure. QI Macros^® was used to analyze data according to established healthcare rules from the Institute for Healthcare Improvement (IHI) for differentiating between common and special cause variation.¹¹

Process measures regarding change idea fidelity were obtained by a voluntary survey administered via Google Forms^©12 with weekly calendar reminders. It was completed by PSQI Specialists post-critical incident review that required the development of recommendations for the prevention of incident reoccurrence. A change was made to two survey questions in February 2024, which were approved by the QIRC. The first change added the incident identification number and removed the PSQI Specialist name for follow-up on survey completion compliance. The second change rewrote the last survey question monitoring the classification of recommendations during critical incident reviews to improve understanding.

Process measures captured through the survey included (1) receipt of a sequence of events at least 24 hours before the incident review meeting; (2) time allocated to the discussion and development of recommendations during the incident review meeting and whether it was sufficient from the perspective of the PSQI Specialist; (3) use of the incident analysis template to facilitate recommendation discussion and documentation; and (4) HIE-based recommendation classification during incident reviews.

Results

The project measures and results are summarized in Table 2.

Table 2.

Project measures and results.

	Result
Outcome measure
The proportion of system-focused recommendations developed from critical patient incident reviews.	30% (87% increase)
Process measures
Receipt of a sequence of events at least 24 hours before the incident review.	67% (16/24)
Time allocated to the discussion and development of recommendations during the incident review and whether it was sufficient.	71% (17/24) of critical incident reviews had sufficient time allocated for recommendation development
	Time allocated for recommendation development:
	• Less than 30 minutes: 29% (5/17)
	• 30-44 minutes: 18% (3/17)
	• 45-59 minutes: 41% (7/17)
	• 60+ minutes: 12% (2/17)
Use of the incident analysis template to facilitate recommendation discussion and documentation.	50% (12/24)
HIE-based recommendation classification during incident reviews.	38% (9/24)

For the outcome measure, the baseline average of system-focused recommendations developed from January 2020 to December 2022 (16%) was obtained from the SPC p-Chart’s centre line (Figure 1). The Centre Line (CL) represents the process average of the dataset. To determine the post-intervention average, a process change was shown from January 2023 (implementation of the first change idea). The average of system-focused recommendations post-intervention implementation (30%) was obtained from this recalculated CL. It is noteworthy that, for this project, the process change implemented from January 2023 does not indicate the presence of sustained special cause variation as a result of the project’s change ideas. The process change was done solely to determine the change in the average number of system-focused recommendations developed.

Figure 1.

Outcome measure: Proportion of System-Focused Recommendations (p-Chart). The baseline average of system-focused recommendations developed between January 2020 and December 2022 (16%) was obtained from the SPC p-Chart’s centre line. Special cause variation (astronomical points at January to February 2023 and September to October 2023) may indicate a temporary effect of change idea #1 and change idea #2 implementation.

The percent change formula ((V₂ − V₁)/V₁*100) was used to calculate the change in the baseline average of system-focused recommendations compared to the average post-intervention implementation.

Process measure data were obtained by completion of a voluntary survey by PSQI Specialists. This survey had a 77% (24/31) completion rate.

Discussion

In this QI project, we applied an HFF to move the organization toward developing more system-focused recommendations with the potential to improve the resilience of our healthcare system. We successfully met the project’s aim to increase the proportion of system-focused recommendations developed from critical patient incident reviews from 16 to 30% over 16 months (January 2023 to April 2024). Our approach was designed to follow the HIE to increase our opportunity for change idea sustainability as three of the four change ideas were a standardization for implementation across our multi-site organization. Education and training, while necessary at the start of the project, were used to support this work rather than as the main tool for improvement.

Process measure data were obtained by completion of a voluntary survey by PSQI Specialists. The following information was also identified: (1) a new standard was established where if a sequence of events was not completed before the critical incident review, the review was rescheduled. The 24-hour pre-incident review timeframe was less important than the completion of the task; (2) 45 minutes was identified as the optimal time to allocate for recommendation development (including documentation of the recommendation, the recommendation owner(s), and an estimated due date for completion) in 41% (7/17) of critical incident reviews; and (3) the survey did not consistently capture critical incident reviews requiring meetings on two or more separate occasions where the incident analysis template was used during only the latter meeting(s).

While the literature supports the integration of human factors to improve patient safety,^1,8,10,13 we were unable to identify other published quality projects where the application of the HIE is directly embedded into the patient incident review process. A systematic review to identify a user-friendly and validated method that prioritizes recommendations to implement following analyses of adverse events failed to identify a method.¹⁴ Work by Bos K et al. indicated the cause of recurrent sentinel events seems to lie in the quality of recommendations.¹⁵ They also identified human factors as a field with the potential to significantly increase the quality and safety of healthcare.¹⁵ Their follow-up work presents a recommendation improvement matrix, but its purpose is also to select interventions most likely to have a positive impact on healthcare safety.¹⁶ In our organization’s incident review process, the expectation is for all recommendations developed during the review to be implemented for the prevention of incident reoccurrence. Utilizing the HIE is meant to support the initial development of recommendations to include system-focused solutions.

This project contributes to the body of work that emphasizes the need to incorporate human factors into patient safety. Future work will explore the potential for artificial intelligence to develop system-focused recommendations for critical incident reviews. This innovation could enable recommendation development that is less reliant on the experiences of the individuals participating in incident reviews, and incorporate learning from incident review data and literature while mitigating the manual effort that currently limits its feasibility. Additionally, future research could evaluate the efficacy of these system-focused recommendations, measuring their impact on improving preventable patient harm rates.

Considerations for health leaders

This project answers the global call to integrate human factors into patient safety.¹ It also highlights that effective leadership in the healthcare system is essential to ensure the delivery of high-quality care and patient safety.¹⁷ The successful integration of the HIE in the patient incident review process was the outcome of supportive transformational leadership from key collaborators throughout various steps of this work.

Guiding the project’s change management work was LEADS (a leadership capability framework) which outlines activities to employ to effectively lead through change.¹⁸ Our project also aligns with IHI’s Whole System Quality (WSQ) framework, where its leadership principles foster a culture that promotes inquiry, reflection, systems thinking, and purpose.¹⁹ Revising the patient incident review process to incorporate the HIE required navigating the organization’s political environment to influence leaders and gain project support. We referenced the WSQ framework when introducing this up-and-coming work to organization leaders. The WSQ framework also aligns with our organization’s Q&S Strategic Action Plan, strategically aligning our work with our organization’s vision, values, and valid evidence for decision-making.¹⁸

This QI work demonstrated systems and critical thinking.¹⁸ We encouraged innovation to create systemic change¹⁸ when we introduced this novel use of the HIE in the patient incident review process to leaders. We mobilized knowledge and orchestrated change¹⁸ when we provided just-in-time education using the existing governance committee meeting structure rather than introducing extra educational sessions. Communicating effectively¹⁸ with our organization’s quality governance committees was essential to reach those who lead and participate in incident reviews. The terms of reference for the governance committees were also updated with content formalizing their accountability for recommendation review, appropriateness, and completion. Aligning the rules and policies of the committees responsible for governing the organization’s quality and safety of healthcare framed the purpose of the HIE-related QI work as it was cascaded to incident review participants.

Limitations

Initiated in January 2022, this QI project was limited by its non-continuous nature. The planning and implementation phases were separated by 12 months and the implementation phase was staggered over 9 months. These interruptions were unavoidable, stemming from the organization’s transition to a new health information system and subsequent healthcare human resource challenges. These also likely contributed to change fatigue within the Q&S team and the organization while they were still navigating the COVID-19 pandemic. While a reflection of the reality of healthcare today, this adversely affected the ability to closely follow up on process measures and change idea fidelity, as well as the decreased voluntary process measure survey response rate (77%). Despite delays, at no point did Q&S consider stopping this work. Strategic change management and long-term perseverance were required to incorporate the HIE into the incident review process. Another limitation was the variability in the interpretation of the HIE. Recommendation classification is unlikely to have been consistently accurate as it depended on PSQI Specialists who are fallible in their perfect understanding and application of the HIE—despite the organization having a specialized HHF department onsite that supports the organization’s patient safety and QI work. We acknowledge this level of specialized knowledge and support, and the presence of a corporate Q&S team that can focus on patient safety-related QI work may make this project less broadly applicable and could be a practical concern when considering the feasibility of this work. Creating organization-specific examples of recommendations for each level of the HIE and incorporating HIE education in new leader onboarding has spread awareness of the HIE beyond PSQI Specialists and enhanced discussion during incident reviews. Our organization has seen a decrease in its overall preventable patient harm rate; however, we are unable to directly associate this with the incorporation of the HIE, as our organization had multiple interventions occurring simultaneously to reduce preventable patient harm.

Conclusion

Developing strategies to prevent patient harm is of utmost importance to healthcare organizations. A multi-prong approach to incorporate the HIE into the incident review process was implemented, successfully meeting the project’s aim of increasing the proportion of system-focused recommendations developed from critical incident reviews. The change ideas and change management methods described could apply to other organizations on a similar journey of incorporating HFF into their incident review process. Healthcare cannot continue to rely on interventions that improve individual performance to reduce and minimize the impact of safety events.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Ethical approval

Institutional review board approval was not required. The project was reviewed and approved by the organization's Quality Improvement Review Committee.

ORCID iDs

Melissa F. Lan

Laura D. Pozzobon

References

World Health Organization . Global Patient Safety Action Plan 2021-2030: Towards Eliminating Avoidable Harm in Health Care. 1st ed.; 2021.

Patient Harm in Canadian Hospitals? It Does Happen. CIHI. Accessed October 4, 2024. https://www.cihi.ca/en/patient-harm-in-canadian-hospitals-it-does-happen

Institute of Medicine . Building a Better Delivery System: A New Engineering/Health Care Partnership. National Academies Press (US); 2005.

Excellent Care for All Act , 2010. Accessed October 4, 2024. https://www.ontariocanada.com/registry/view.do?postingId=4544

Healthcare Excellence Canada . Canadian Incident Analysis Framework; 2012.

World Health Organization . World Alliance for Patient Safety Report on the Web-Based Modified Delphi Survey of the International Classification for Patient Safety; 2007.

What Is Ergonomics (HFE)? International Ergonomics Association. https://iea.cc/about/what-is-ergonomics/. Accessed October 4, 2024.

Cafazzo

St-Cyr

. From discovery to design: the evolution of human factors in healthcare. Healthc Q. 2012;15(sp):24-29. doi:10.12927/hcq.2012.22845

Institute For Safe Medication Practice . Medication Error Prevention “Toolbox.”; 1999. https://www.ismp.org/resources/medication-error-prevention-toolbox. Accessed February 17, 2022.

10.

Institute For Safe Medication Practice . Education is “predictably disappointing” and should never be relied upon alone to improve safety. ISMP Medication Safety Alert! Acute Care. 2020;25(11):1-4.

11.

Healthcare Rules | IHI Rules | Control Chart Rules . https://www.qimacros.com/control-chart/healthcare-rules/. Accessed October 4, 2024.

12.

Lan

. Recommendations Project: POST-DEBRIEF SURVEY. Google Forms. 2022. https://forms.gle/R6mo3Gfu37cmqrECA

13.

Soong

Shojania

. Education as a low-value improvement intervention: often necessary but rarely sufficient. BMJ Qual Saf. 2020;29(5):353-357. doi:10.1136/bmjqs-2019-010411

14.

Bos

Van Der Laan

Dongelmans

. Prioritising recommendations following analyses of adverse events in healthcare: a systematic review. BMJ Open Qual. 2020;9(4):e000843. doi:10.1136/bmjoq-2019-000843

15.

Bos

Dongelmans

Greuters

Kamps

Van Der Laan

. The next step in learning from sentinel events in healthcare. BMJ Open Qual. 2020;9(1):e000739. doi:10.1136/bmjoq-2019-000739

16.

Bos

Van Der Laan

Groeneweg

, et al. Grading recommendations for enhanced patient safety in sentinel event analysis: the recommendation improvement matrix. BMJ Open Qual. 2024;13(2):e002592. doi:10.1136/bmjoq-2023-002592

17.

Ahmed

Ellahham

Soomro

Shams

Latif

. Exploring the impact of compassion and leadership on patient safety and quality in healthcare systems: a narrative review. BMJ Open Qual. 2024;13(Suppl 2):e002651. doi:10.1136/bmjoq-2023-002651

18.

LEADS Global . LEADS - In A Caring Environment Capabilities Framework. https://www.leadsglobal.ca/leads-framework/. Accessed September 20, 2024.

19.

Sampath

Rakover

Baldoza

Mate

Lenoci-Edwards

Barker

. Whole System Quality: A Unified Approach to Building Responsive, Resilient Health Care Systems. IHI White Paper. Institute for Healthcare Improvement; 2021. https://www.ihi.org/