A Likert-Type scale for evaluating the “bottom line” of patient safety

Abstract

The formally reported number of adverse events may be open to ambivalent interpretation – actual higher prevalence of adverse events versus a patient safety culture supporting reporting and learning. Many methods appearing in the literature that are not based on reporting systems struggle for adequately assess the precise level of prevalence of adverse events. Confronting this challenge in patient safety research, we suggest evaluating the perceived state of “almost no adverse events” in the ward, by using a short Likert- type scale we developed for this purpose. Some evidence for its reliability and validity are presented using two samples (99 head nurses, and 383 nurses). As was expected, leadership had a significant direct effect on the measured state of “almost no adverse events” as well as an indirect effect mediated successively by psychological safety, and safety behavior.

Keywords

Patient safety Adverse events Likert-type scale Formal reporting Safety behavior Psychological safety Leadership

Background

Improving patient safety, meaning the prevention of errors and adverse effects to patients associated with the provision of healthcare, is imperative in healthcare organizations (Kohn, Corrigan, & Donaldson, 2000; Makary & Daniel, 2016). Patient safety is ultimately reflected by the actual prevalence of adverse events, defined as injuries related to medical management in contrast to disease complications (Brennan et al., 1991). Much research has focused on the factors influencing the prevalence of adverse events. Nevertheless, we assert that the straightforward approach of using formally reported adverse events might be open to ambivalent interpretation.

Reporting adverse events is a major mean of learning about and improving patient safety (Vogus, Sutcliffe, & Weick, 2010). To continuously improve the system, adverse events must be analyzed, their root causes determined (to understand how ‘latent errors’ play a role in a chain of events that sets up the occurrence of an ‘active error’), and actions taken to implement the needed changes (Leape, 2002). Cumulative empirical findings, however, reveal that the reporting levels are still low (Mitchell et al., 2016; Wachter, 2010).

Several causes may explain the persistent gap between the actual and the reported prevalence of adverse events (Leape, 2002). These may be grouped into two main categories that rationalize why the prospective benefits of reporting an adverse event outweigh its price:

“It isn’t beneficial” – First, low levels of trust in management’s commitment to leading a change, and timely engagement in rectifying the underlying system shortcomings indicated by the reported incidents, make the time, and effort involved in reporting seem unjustified. Considering the heavy workloads typical of many healthcare settings, investing time in reporting that does not seem to lead to actual improvement in patient safety is a serious disincentive (e.g., Frankel, Leonard, & Denham, 2006; Mahajan, 2010; Ramírez et al., 2018).

“It isn’t safe” – The second obstacle is low psychological safety, meaning the belief that one is able to express oneself without fear of negative consequences (Liang, Farh, & Farh, 2012). Psychological safety might be especially low in healthcare cultures that emphasize professional accountability and lean on the assumption that “one must not be wrong” (e.g., Conchie, Taylor, & Donald, 2012; Mariner & Miller, 2001).

Thus, reporting an adverse event may be perceived not only as futile but also as risky and harmful.

Consequently, as has been previously recognized, the number of formally reported adverse events may have two contrary interpretations. On the one hand, lower reporting levels may indicate higher patient safety; yet, on the other hand, they may indicate low levels of psychological safety, learning, and patient safety (Vogus, Sutcliffe, & Weick, 2010).

This ambiguity calls for assessing the actual prevalence of adverse events by looking beyond the formally reported numbers. Prior research used various means to achieve this. Some studies retrospectively analyzed patient administrative records, referring to specific types of adverse events that can be extracted from these records (e.g., Budnitz, Shehab, Kegler et al., 2007; Otchi, Esena, Srofenyoh et al., 2019), or to adverse events resulting in prolonged hospital stay, disability at the time of discharge, or death, since these are extractable from patient records (Forster et al., 2003). An ambitious prospective approach employs an observational design (e.g., Andrews et al., 1997), or collection of data from the medical teams along the hospital stay (e.g., Michel, Quenon, de Sarasqueta et al., 2004). These prospective methods require extensive effort and resources so as not to miss any relevant information (Thomas, & Petersen, 2003; Rafter et al., 2015). Further, it was found that staff members’ ability to identify adverse events that occurred within a specified period (the median was four days) was inferior to the retrospective methods in identifying adverse events (Michel et al., 2004).

Other studies used the staff’s perception of the prevalence of adverse events. Team members used a Likert-type scale ranging from “never” to “frequently” to report their assessments regarding specific types of adverse events such as falls (Boamah, Laschinger, Wong et al., 2018). This less labor-intensive method, however, still suffers the disadvantage of excluding less common types of adverse events.

To meet, at least partially, the challenge of gauging the state of patient safety, meaning the degree to which the unit is approaching the goal of “almost no adverse events”, we developed a short Likert-type scale assessing the staff perception of their ward in terms of this ideal state.

Method

As a first step, we built, as recommended (Norman, 2010), a symmetric, seven-point scale, with a clear middle point: 1–very strongly disagree; 2–strongly disagree; 3–disagree; 4–neutral, 5–agree; 6–strongly agree, 7–very strongly agree.

As seen in Table 1, we wanted the scale to be comprehensive, to include all types of events, not only specific predetermined ones. It was also intended, by asking respondents to compare their ward to similar ones, to take into account features of the ward that are relevant to the prevalence of adverse events such as type, size and intensity of activity.

Table 1.

Frequencies, means, and standard deviations for nurses’ demographic characteristics for the two sample.

	Sample 1 (Head nurses) N = 99				Sample 2 (Nurses) N = 383
					Academic	Non- Academic
Education	Academic	Non- Academic		Missing	273 (71.3%)	108 (28.3%)		Missing
Gender	Women	Men		Missing	Women	Men		Missing
Gender	74 (74.7%)	15 (15.2%)		10 (7.3%)	260 (69.3%)	115 (30%)
	Mean		St. D.		Mean		St. D.
Age	39.771		16.029		38.459		9.031
Tenure	19.232		9.679		14.429		8.813

To test the scale’s reliability and validity, we, first, presented the items to a group comprising five head nurses and twenty nurses from different healthcare organizations. They confirmed the scale’s face validity.

The scale was thereafter administered to a sample of 99 head nurses (see Table 1), and an exploratory factor analysis (extraction method: principal component analysis) was conducted.

The results indicated that, as was expected, all the items were loaded on a single factor (see Table 2).

Table 2.

Principal component analysis of the scale items with two samples.

	Component 1
The items:	Head nurses sample (N = 99)	Nurses sample (N = 383)
In this ward the number of adverse events is extremely low.	.833	.896
In this ward, there are fewer adverse events than in equivalent wards with similar workloads.	.894	.904
This ward actually succeeded in reaching a state of “almost no adverse events”.	.924	.893
Remarkably, this ward succeeded in reducing the number of adverse events occurring within it.	.881	.799

We then tested for construct validity with a sample of 383 nurses from a variety of healthcare organizations and wards (see Table 1).

Based on previous research, we expected that the construct measured by the scale, meaning the level to which the ward reaches the optimal state of “almost no adverse events”, would positively correlate with three variables. The first variable was nurse safety behavior, which includes safety compliance, as well as proactive safety behaviors such as engaging in quality improvement, and speaking up. We assessed this variable with Neal and Griffin’s six-item safety behavior scale (2006), and the ten-item voice scale by Liang, Farh, and Farh (2012). This scale’s Cronbach’s alpha was 0.96 (see Table 3).

Table 3.

Research variable means, standard deviations, inter-correlations, and scale reliabilities- Nurses sample (N = 383).

	Mean	St. D.	1	2	3	4	5	6	7	8	9
Measured construct (“Almost no adverse events”)	3.86	1.66	.89
Safety behavior	4.34	1.61	.62**	.96
Leadership	4.14	1.67	.62**	.79**	.95
Psychological safety	4.05	1.60	.54**	.68**	.71**	.82
Perceived number of adverse events during last six months	4.65	3.32	–.18**	–.09	–0.13*	–.05	1
Gender	–	–	–.01	–.10	–.03	–.08	.02	1
Age	38.46	9.03	–.01	–.02	.01	.02	.08	–.02	1
Tenure	14.43	8.81	–.01	–.08	–.04	–.02	.07	.003	.93**	1
Academic education	–	–	0.004	.05	.01	.05	.13*	–.05	–.04	–.05	1

*Correlation is significant at the 0.05 level (2-tailed); **Correlation is significant at the 0.01 level (2-tailed).

The second variable was the head nurse’s leadership. We assessed this variable with Paglis and Green’s eight-item scale (2002). This scale’s Cronbach’s alpha was 0.95 (see Table 3).

The third variable was the nurse’s psychological safety. We assessed this variable with Liang, Farh, and Farh’s three-item scale (2012). Cronbach’s alpha for this scale was 0.82 (see Table 3).

We also expected that there would be a negative correlation between the construct measured by the scale and the estimated number of adverse events in the last six months as estimated by the nurse. Due to memory gaps and variance in the wards’ type, size, and workloads affecting the ability of the respondents to adequately note the exact number of adverse events, we expected that this correlation would be moderate.

Further, based on previous research, we also expected nurses’ safety behavior will mediate the effects of psychological safety (e.g. Liang, Farh, and Farh’s, 2012), and leadership style (e.g. Boamah, Laschinger, Wong et al., 2018; McFadden, Stock, & Gowen, 2015) on patient safety. Further, since leadership influences psychological safety (e.g. Walumbwa, & Schaubroeck, 2009‏), we anticipated finding that psychological safety mediates some of the effect of leadership on patient safety behavior and the resulting construct measured by the scale.

Results

As seen in Table 3, Cronbach’s alpha reliability of the scale was 0.89.

First, as was expected, and seen in Table 3, there were rather high positive significant correlations between the construct measured by the scale and the three predictors, patients safety behavior (r = 0.62, p < 0.01), psychological safety (r = 0.54, p < 0.01), and leadership (r = 0.62, p < 0.01). Further, as was expected, there was a significant moderate negative correlation between the construct measured by the scale, and the number of adverse events in the last six months as estimated by the nurse (r = –.219, p < 0.01).

Second, we conducted two mediation analyses using SPSS’s PROCESDS software package, following Hayes’ recommended three steps (2017). As expected, the results (see Table 4) of the first step showed that both predictors, nurse’s psychological safety (b = 0.21, [0.13 0.30]), and head nurse’s leadership (b = 0.63, [0.55 0.72]), made a significant positive contribution to the construct measured by the scale.

Table 4:

Mediation analyses – Indirect effects of psychological safety and leadership on the measured construct (“almost no adverse events”) mediated by safety behavior.

STEP 1 Safety Behavior (R² = 0.67***)	Direct effect (SE)		P value	Boot 95% CI

Psychological safety	0.21 (0.04)		0.000	[0.13, 0.30]
Leadership	0.63 (0.04)		0.000	[0.55, 0.72]
Gender	–0.21 (0.12)		0.050	[–0.42, –0.00]
Age	0.012 (0.01)		0.396	[–0.02, 0.04]
Tenure	–0.02 (0.02)		0.150	[–0.05, 0.02]
Academic education	0.10 (0.11)		0.381	[–0.12, 0.31]
STEP 2 Measured construct (”almost no adverse events”) (R² = 0.46***)
Safety Behavior	0.31 (0.07)		0.000	[0.17, 0.44]
Psychological safety	0.11 (0.06)		0.070	[–0.01, 0.23]
Leadership	0.34 (0.07)		0.000	[0.20, 0.49]
Gender	0.07 (0.14)		0.622	[–0.21, 0.35]
Age	–0.04 (0.02)		0.036	[–0.08, –0.003]
Tenure	0.04 (0.02)		0.028	[0.01, 0.08]
Academic education	–0.08 (0.15)		0.564	[–0.37, 0.20]
STEP 3: Indirect effects of nurse psychological safety and leadership on the measured construct ("almost no adverse events") mediated by safety behavior
		Indirect Effect (SE)			Boot 95% CI
Psychological safety -> Safety behavior -> Measured construct		0.07 (0.03)			[0.02, 0.12]
Leadership -> Safety behavior -> Measured construct		0.19 (0.05)			[0.106, 0.29]

The second step showed that safety behavior significantly (b = 0.31, [0.17 0.44]) predicted this construct beyond the contribution of psychological safety and change initiating leadership.

The third step revealed that there were significant indirect effects of leadership (ab = 0.07, [0.02 0.12]), and psychological safety (ab = 0.07, [0.02 0.12]) on the construct via safety behavior. Last, as expected, there was also a significant direct effect (B = 0.34, [0.20, 0.49]) of leadership but not of psychological safety on the construct beyond their contribution via safety behavior.

Further, we conducted an analysis using SPSS PROCESS (Model 6) looking for the indirect effect of leadership on the construct mediated successively by psychological safety and safety behavior. As expected, this indirect effect was significant (ab = 0.05, [0.02, 0.09]).

Conclusions

It is important to be aware of the possibly ambiguous implication for the level of patient safety of formally reported adverse event occurrences. The results of our analyses imply that our proposed scale is adequate for use in future research seeking to establish the “bottom line” of patient safety, meaning the degree to which the ward is achieving the goal of “almost no adverse events”. The results indicate its reliability and provide initial support for its construct validity. Beyond the expected correlations of the construct measured by the scale with psychological safety, leadership, and safety behavior, the pattern of the direct and indirect effects are in congruence with previous research (Boamah, Laschinger, Wong et al., 2018; Liang, Farh, and Farh’s, 2012; Walumbwa, & Schaubroeck, 2009). These results show that leadership is an important determinant of psychological safety, which in its turn contributes to safety behaviors as voice. Further, leadership also contributed directly to safety behavior, and to the measured construct, while psychological safety had no direct effect on this construct. Yet, future research may add support to the scale construct validity by using data from different sources.

Previous measurement of safety culture relies on staff perception i.e. the Safety Attitudes Questionnaire (Sexston et al., 2006), and the AHRQ HSOPS (Sorra, & Dyer, 2010). These large- scale multi- dimensional measurements incorporate items about perceived safety. Relying on the front line providers' perception regarding the state of reaching “no adverse event” by their work unit cansimilarly contribute to patient safety research by providing a relatively efficient method for assessing the perceived pervasiveness of adverse events in the unit .

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Sarit Rashkovits

References

Kohn

Corrigan

Donaldson

MS.

(Eds.). To err is human: Building a safer health system. ( Vol. 6). Washington, DC: National Academies Press, 2000. ‏

Makary

Daniel

Medical error—the third leading cause of death in the US. Bmj 2016; 353: i2139.

Brennan

Leape

Laird

, et al. Incidence of adverse events and negligence in hospitalized patients: Results of the harvard medical practice study I. N Engl J Med 1991; 324: 370–376.

Vogus

Sutcliffe

Weick

KE.

Doing no harm: Enabling, enacting, and elaborating a culture of safety in health care. AMP 2010; 24: 60–77.

Leape

LL.

Reporting of adverse events. N Engl J Med 2002; 347: 1633–1638.

Wachter

RM.

Patient safety at ten: Unmistakable progress, troubling gaps. Health Aff (Millwood) 2010; 29: 165–173.

Mitchell

Schuster

Smith

, et al. Patient safety incident reporting: a qualitative study of thoughts and perceptions of experts 15 years after ‘to err is human. BMJ Qual Saf 2016; 25: 92–99.

Frankel

Leonard

Denham

CR.

Fair and just culture, team behavior, and leadership engagement: the tools to achieve high reliability. Health Serv Res 2006; 41: 1690–1709.

Mahajan

RP.

Critical incident reporting and learning. Br J Anaesth 2010; 105: 69–75.

10.

Ramírez

Martín

Villán

, SINOIRES Working Groupet al. Effectiveness and limitations of an incident-reporting system analyzed by local clinical safety leaders in a tertiary hospital: Prospective evaluation through real-time observations of patient safety incidents. Medicine (Baltimore) 2018; 97: e12509.

11.

Liang

Farh

JL.

Psychological antecedents of promotive and prohibitive voice: a two-wave examination. AMJ 2012; 55: 71–92.

12.

Conchie

Taylor

Donald

IJ.

Promoting safety voice with safety-specific transformational leadership: the mediating role of two dimensions of trust. J Occup Health Psychol 2012; 17: 105–115.

13.

Mariner

Miller

FH.

Medical error reporting: Professional tensions between confidentiality & liability. Boston: Massachusetts Health Policy Forum, www.sihp. brandeis.edu/mhpf/prof_liability_Issue_Brief.pdf (2001 accessed 24 October 2002).

14.

Budnitz

Shehab

Kegler

, et al. Medication use leading to emergency department visits for adverse drug events in older adults. Ann Intern Med 2007; 147: 755–765.

15.

Otchi

Esena

Srofenyoh

, et al. Types and prevalence of adverse events among obstetric clients hospitalized in a secondary healthcare facility in Ghana. Journal of Patient Safety and Risk Management 2019; 24: 238–244. ‏

16.

Cho

Ketefian

Barkauskas

, et al. The effects of nurse staffing on adverse events, morbidity, mortality, and medical costs. Nurs Res 2003; 52: 71–79.

17.

Forster

Murff

Peterson

, et al. The incidence and severity of adverse events affecting patients after discharge from the hospital. Ann Intern Med 2003; 138: 161–167.

18.

Andrews

Stocking

Krizek

, et al. An alternative strategy for studying adverse events in medical care. Lancet 1997; 349: 309–313.

19.

Michel

Quenon

de Sarasqueta

, et al. Comparison of three methods for estimating rates of adverse events and rates of preventable adverse events in acute care hospitals. Bmj 2004; 328: 199.

20.

Thomas

Petersen

LA.

Measuring errors and adverse events in health care. J Gen Intern Med 2003; 18: 61–67.

21.

Rafter

Hickey

Condell

, et al. Adverse events in healthcare: learning from mistakes. QJM: An International Journal of Medicine 2015; 108: 273–277.

22.

Boamah

Laschinger

HKS

Wong

, et al. Effect of transformational leadership on job satisfaction and patient safety outcomes. Nurs Outlook 2018; 66: 180–189.

23.

Norman

Likert scales, levels of measurement and the “laws” of statistics. Adv Health Sci Educ Theory Pract 2010; 15: 625–632.

24.

Neal

Griffin

MA.

A study of the lagged relationships among safety climate, safety motivation, safety behavior, and accidents at the individual and group levels. J Appl Psychol 2006; 91: 946–953.

25.

Liang

Farh

JL.

Psychological antecedents of promotive and prohibitive voice: a two-wave examination. Academy of Management Journal 2012; 55: 71–92. ‏

26.

Paglis

Green

SG.

Leadership self‐efficacy and managers' motivation for leading change. J. Organiz. Behav 2002; 23: 215–235. ‏

27.

McFadden

Stock

Gowen

CII.

Leadership, safety climate, and continuous quality improvement: impact on process quality and patient safety. Health Care Manage Rev 2015; 40: 24–34.

28.

Walumbwa

Schaubroeck

Leader personality traits and employee voice behavior: mediating roles of ethical leadership and work group psychological safety. J Appl Psychol 2009; 94: 1275–1286. ‏

29.

Hayes

AF.

Introduction to mediation, moderation, and conditional process analysis: a regression-based approach. 2nd ed. New York City: Guilford Press, 2017.

30.

Sexton

Helmreich

Neilands

, et al. The safety attitudes questionnaire: psychometric properties, benchmarking data, and emerging research. BMC Health Serv Res 2006; 6: 44.

31.

Sorra

Dyer

Multilevel psychometric properties of the AHRQ hospital survey on patient safety culture. BMC Health Serv Res 2010; 10: 199.