Measuring Quality Improvement in Public Health

Abstract

There is growing interest and investment in improving the quality of public health services and outcomes. Following the lead of other sectors, efforts are underway to introduce systematic quality improvement (QI) tools and approaches to state and local public health agencies. Little is known, however, about how to describe and reliably measure the level of QI maturity within a public health agency. The authors describe the development of a QI Maturity Tool using research from the fields of organizational design, psychology, health care, and complexity theory. The 37-item assessment tool is based on four quality domains derived from the literature: (a) organizational culture, (b) capacity and competency, (c) practice, and (d) alignment and spread. The tool was designed to identify features of an organization that may be enhancing or impeding QI; monitor the impact of efforts to create a more favorable environment for QI; and define potential cohorts of public health agencies for evaluation purposes. The article presents initial steps in testing and validating the QI Maturity Tool including: (a) developing a theoretical framework, (b) assuring face and content validity, (c) determining the tool’s reliability based on estimates of internal consistency, (d) assessing the dimensionality, and (f) determining the construct validity of the instrument. The authors conclude that there is preliminary evidence that the QI Maturity Tool is a promising instrument. Further work is underway to explore whether self-reported survey results align with an agency’s actions and the products of their QI efforts.

Keywords

quality improvement health department validity reliability measurement tool

The adoption of quality improvement (QI) techniques and methods from private industries and the health care arena has spread to public health. The growing interest and momentum have led to several national initiatives promoting the application of QI in health departments (Beitsch, Thielen, Mays et al., 2006; Corso, Lenaway, Beitsch, Landrun, & Deutsch, 2010; Davis, Solomon, & Gorenflo, 2010; Gillen, McKeever, Edwards, & Thielen, 2010) and the advent of formal definitions (Department of Health and Human Services [DHHS], 2008; Riley et al., 2010). Recently, the U.S. Department of Health and Human Services (DHHS) developed six priority areas for the improvement of quality in public health (Honore & Scott, 2010). This DHHS report serves as a call to action necessitating the need to monitor progress in order to inform and mobilize action toward improved quality. In addition to the recently identified priority areas, the federal government has begun to heavily invest in building QI capacity as evidenced by recent Affordable Care Act funding for the National Public Health Improvement Initiative. The Centers for Disease Control and Prevention has provided $42.5 million in support to 75 state, tribal, local, and territorial public health agencies in an effort to “support innovative changes in key areas that improve the quality, effectiveness and efficiency of the public health infrastructure” (Office of State, Tribal, Local and Territorial Support, CDC, 2010). These new developments require methods for measuring QI capacity, efforts, and impact.

The establishment of a national voluntary accreditation program by the Public Health Accreditation Board (PHAB) also has raised the visibility and commitment to QI. According to PHAB (www.phaboard.org), the program has recently begun accepting applications with the goal of improving and protecting the health of every community by “advancing the quality and performance of public health departments.” Achieving accreditation is expected to convey a commitment to quality and high performance that meets or exceeds the national standards (Lenaway, Corso, & Bailey, 2007). Accreditation has been described as a “platform” for QI (Russo, 2007), and with the launch of this program and the continued efforts to promote the application of QI in health departments across the country, there is an increasing need to reliably assess an organization’s QI efforts or level of QI maturity and to assure that we are in fact capturing the underlying constructs of interest. Measuring QI maturity is not a new concept and it has proven useful in the health care sector (Lombarts, Rupp, Vallejo, Klaxinga, & Sunol, 2009), yet there are no similar tools that have been developed and tested with a public health agency.

QI Maturity: Underlying Constructs

The literature suggests that QI should be part of a public health department’s culture in order for efforts to be most successful (Bialek, Carden, & Duffy, 2010) and this may require transformational change in an organization’s structure and management mindset (Riley, Parsons, Duffy, Moran, & Henry, 2010) in order to intentionally diffuse QI throughout an agency or public health system (Corso et al., 2010). Bialek, Carden, and Duffy (2010) claim that programs, services, and resources should be aligned with an organization’s guiding principles and strategic efforts in order to create an environment in which QI can flourish. Yet, this type of environment requires staff members to be equipped with the knowledge and skills needed to scrutinize and adapt existing processes, activities and services to achieve improvements (Gorenflo, 2010). This often requires that agencies provide opportunities and resources for staff and teams to get trained and practice QI techniques and methods (Davis, 2010).

Given the importance of an organization’s QI culture, capacity, practice, and ability to align and diffuse efforts, the authors sought to develop a reliable and valid tool that would measure all of these overarching constructs. Although efforts to assess the status of QI within local and state health departments have been underway for several years (Beitsch, Leep, Shah, Brooks, & Pestronk, 2010; Madamala, Sellers, Pearsol, Dickey, & Jarris, 2010), to our knowledge, there are no validated instruments developed for use in a health department setting that formally address all of these core areas. Recently, Riley and colleagues (2010) proposed a technique through the use of a radar chart with a 5-point rating scale to assess the current culture of an organization. While this technique assesses six dimensions of quality culture, to date, there are no published studies documenting the psychometric properties of this approach.

Developing an assessment tool that can be applied with confidence is a critical first step in adequately measuring QI efforts and subsequently conducting research to investigate the relationship between QI and specific outcomes of interest (e.g., accreditation status, customer satisfaction, community health indicators). The focus of this article and the analysis is to provide an overview of the development of a QI Maturity Tool and its accompanying psychometric properties. The tool was designed to:

Identify features of an organization that may be enhancing or impeding QI approaches.

Monitor the impact of efforts to create a more favorable environment for QI to flourish.

Define potential cohorts of public health agencies for evaluation purposes.

The tool was developed by the authors, in consultation with experts on QI in public health, as part of their evaluation of a 16-state demonstration to enhance quality capacity within state and local health departments (Gillen et al., 2010; Joly, Shaler, Booth, Conway, & Mittal, 2010). Although we had initially hoped to perform factor analysis on a list of nearly 100 items we identified in the literature as measuring specific aspects of QI, this more traditional approach for scale development was not possible as part of our evaluation. We were limited in terms of the number of questions we could ask, the time frame available for designing this evaluation instrument, and our approach for simultaneously validating the tool when no known comparison tools existed in public health. Below, we have described our process and our attempt to assess the attributes of the QI Maturity Tool given the constraints we faced as part of our evaluation.

Method

Instrument Development

The QI Maturity Tool was developed by the evaluation team based on a review of the research on how organizations successfully adopt changes and innovative practices to improve quality. These studies borrow from the fields of organizational design, psychology, health care, and complexity theory in examining how agencies make, sustain, and spread change. Studies on the adoption and use of QI within an organization also were incorporated and the evaluation team reviewed existing survey questions developed by public health researchers and national public health associations. Finally, the team examined published questionnaires from the health care sector designed to measure aspects of culture, maturity, and the implementation of QI collaboratives.

As mentioned above, the 37-item survey is based on four quality domains derived from the literature: (a) organizational culture, (b) capacity and competency, (c) practice, and (d) alignment and spread. These domains include a corresponding set of items that measure distinct subdomains herein referred to as dimensions. The majority of items included a closed Likert-type scale response option where 1 corresponds to strongly disagree and 5 corresponds to strongly agree. A response of “I don’t know” also was included as an alternative given that the concept of QI remained relatively new for some of the participating health departments (see Table 1 ).

Table 1.

Quality Improvement Maturity Tool

	Strongly Disagree				Strongly Agree	I Don’t Know
Domain and Survey Items	1	2	3	4	5	9
Domain #1: QI Organizational Culture
1. Leaders (e.g., board, senior management team) of my public health agency are receptive to new ideas for improving agency programs, services, and outcomes.	□	□	□	□	□	□
2. The impetus for improving quality in my public health agency is largely driven by external forces (e.g., Board of Health, funders, accreditation, regulation, peer pressure).	□	□	□	□	□	□
3. The impetus for improving quality in my public health agency is largely driven by an internal desire to make our services and outcomes better.	□	□	□	□	□	□
4. The board and/or the management team of my public health agency work together for common goals.	□	□	□	□	□	□
5. Agency data are shared with staff for performance improvement purposes.	□	□	□	□	□	□
6. When things go wrong, my public health agency looks at matters in a respectful way without blaming others.	□	□	□	□	□	□
7. Staff consult with, and help, one another to solve problems.	□	□	□	□	□	□
8. Staff members are routinely asked to contribute to decisions at my public health agency.	□	□	□	□	□	□
Domain #2: QI Capacity and Competency
9. The leaders of my public health agency are trained in basic methods for evaluating and improving quality, such as Plan-Do-Study-Act.	□	□	□	□	□	□
10. Staff at my public health agency who provide public health services are trained in basic methods for evaluating and improving quality, such as Plan-Do-Study-Act.	□	□	□	□	□	□
11. Many individuals responsible for programs and services in my public health agency have the skills needed to assess the quality of their program and services.	□	□	□	□	□	□
12. My public health agency has objective measures for determining the quality of many programs and services.	□	□	□	□	□	□
13. Many individuals responsible for programs and services at my public health agency routinely use systematic methods (e.g., root cause analysis) to understand the root causes of problems.	□	□	□	□	□	□
14. Many individuals responsible for programs and services at my public health agency routinely use best or promising practices when selecting interventions for improving quality.	□	□	□	□	□	□
15. Programs and services are continuously evaluated to see if they are working as intended and are effective.	□	□	□	□	□	□
16. My public health agency has designated a Quality Improvement Officer.	□	□	□	□	□	□
17. The quality of many programs and services in my agency is routinely monitored.	□	□	□	□	□	□
18. There is an established process for identifying quality improvement priorities within many programs and services in my public health agency	□	□	□	□	□	□
Domain #3: QI Practice
19. Has your public health agency ever implemented a formal process to improve the performance of a specific service or program, process or outcome?
□ Yes
□ No (Skip Q34–Q37.)
□ I don’t know. (Skip Q34–Q37.)
20. In the past 12 months, how many formal projects has your public health agency implemented to improve the quality of a service, process or outcome? For the purposes of this question, a “project” is defined as a systematic quality improvement initiative that includes an aim statement; a workplan with tasks, responsibilities and timelines; intervention strategy(ies); and measures for tracking change.
____ # of projects (If the number is “0”, skip Q25, Q26; Q34–37)
□ Don’t know
a. Specific project topics: ______________________________________
21. How long has your public health agency been engaged in established and consistent efforts to improve the quality of services?
□ No systematic efforts currently in place to improve quality
□ < 1 year
□ 1–2 years
□ 3–4 years
□ 5+ years
22. What types of quality improvement methods and techniques has your agency used in the past year (check all that apply)?
□ Brainstorming □ Fishbone or equivalent design to identify root causes
□ Priority-setting □ Trend chart, run chart or control chart
□ Root cause analysis □ Business process re-engineering
□ Flow chart □ Plan-do-study-act cycle
□ Other: ___________________________________
□ None
□ I don’t know
Domain #4: Alignment and Spread
23. Job descriptions for many individuals responsible for programs and services at my public health agency include specific responsibilities related to measuring and improving quality.	□	□	□	□	□	□
24. Agency staff is aware of external quality improvement expertise to help measure and improve quality.	□	□	□	□	□	□
25. Staff members at all levels participate in quality improvement efforts.	□	□	□	□	□	□
26. Customer satisfaction information is routinely used by many individuals responsible for programs and services in my public health agency.	□	□	□	□	□	□
27. Good ideas for measuring and improving quality in one program or service USUALLY are adopted by other programs or services in my public health agency.	□	□	□	□	□	□
28. My public health agency allocates sufficient time for staff to participate in quality improvement efforts.	□	□	□	□	□	□
29. Accurate and timely data are available for program managers to evaluate the quality of their services on an ongoing basis.	□	□	□	□	□	□
30. Many individuals responsible for programs and services in my agency have the authority to change practices or influence policy to improve services within their areas of responsibility.	□	□	□	□	□	□
31. Improving quality is well integrated into the way many individuals responsible for programs and services work in my public health agency.	□	□	□	□	□	□
32. Many individuals responsible for programs and services in my public health agency find implementing methods for assessing and improving the quality of services to be challenging.	□	□	□	□	□	□
33. Spending time and resources on quality improvement is worth the effort.	□	□	□	□	□	□
34. Quality improvement efforts mostly happen in only one program area of my public health agency.	□	□	□	□	□	□
35. When a quality improvement effort succeeds, there are sufficient resources to sustain improvements on an ongoing basis.	□	□	□	□	□	□
36. Staff integrates lessons from successful quality improvement efforts into daily practice and operations.	□	□	□	□	□	□
37. Before initiating a quality improvement effort, short-term, intermediate and long term desired targets are established.	□	□	□	□	□	□

Organizational culture

This domain is defined as the values and norms that pervade how the agency interacts with its staff and stakeholders. A total of 8 items were used to assess several underlying constructs revealed through our literature review as elements essential to a QI organizational culture: leaders/staff support improvement and change, (Berlowitz et al., 2003; Lukas et al., 2007; Nolan & Shall, 2007; Seid, Lotstein, & Williams, et al., 2006; Riley, Parsons, et al., 2010), motivation for change (Nolan & Shall, 2007), shared vision and goals (Leonard, 2007), transparency (Baldridge National Quality Program, 2008; Berlowitz et al., 2003), collaborative versus punitive orientation (Berlowitz et al., 2003; Rondeau & Wagner, 2002), decision making (Rondeau & Wagner, 2002), and the use of reliable data (Berlowitz et al., 2003; Davis, 2010; Rondeau & Wagner, 2002).

Capacity and competency

This domain includes 10 items that measure the skills, functions, and approach used within an organization to assess and improve quality. The items were based on existing assessment efforts (Davis, 2010) and foundational elements identified in the literature related to building QI capabilities (Berlowitz et al., 2003; Riley, Parsons, et al., 2010; Turning Point Performance Management Collaborative, 2007).

QI practice

Unlike the other domains, this portion of the instrument includes a series of four objective items to assess the number, type, and length of formal QI efforts. Previous surveys were reviewed to assure that the QI methods and techniques identified were relevant and feedback from national experts was sought to help identify appropriate wording for these items.

Alignment and spread

This domain focuses on the extent to which QI supports (and is supported by) the organization as well as the diffusion of QI within the agency. The 15 items are based on existing literature about the importance of: accurate and timely data (Berlowitz et al., 2003; Davis, 2010; Rondeau & Wagner, 2002) customer service, authority to make changes, integration of QI into daily practice (Berlowitz et al., 2003; Riley, Parsons, et al., 2010), and characteristics of change that spread easily (Rogers, 2006).

Study Participants and Data Collection

Data used in this study were based on a larger evaluation survey distributed to all local health departments (n = 1,161) among the 16 states participating in the Multistate Learning Collaborative. A description of this national collaborative and its evaluation is reported elsewhere (Gillen et al., 2010; Joly et al., 2010). The QI Maturity Tool was embedded in a web-based evaluation survey that totaled 61 items and included skip patterns. The survey was administered in February–March, 2009, to a health department’s lead public health official or a member of the senior management team. As an incentive to participate, respondents were provided an agency-specific report upon completion of the survey in PDF format and available for downloading. This report summarized the agency’s responses and included areas of strength and opportunity related to QI.

Data from this larger survey were merged with the 2008 Profile Survey administered by the National Association of County and City Health Officials (NACCHO). Agency characteristics (e.g., jurisdiction, budget, staff size) were pulled from the NACCHO database in order to minimize the length of the survey.

Statistical Analyses and Psychometric Testing

Descriptive statistics for each item were calculated including the distribution of valid responses. Bivariate analyses were used to determine potential differences between respondents and nonrespondents. All analyses were performed using statistical analysis software version 9.2 and statistical package for the social sciences version 17.0.

Reliability

The reliability of the QI Maturity Tool was calculated by measuring the internal consistency of the various domains and dimension based on Cronbach’s α, the most common estimate of internal consistency (Cronbach, 1951). In this study, internal consistency was defined as the extent to which items in a given domain or dimension are correlated with each other. Generally, a minimum score of .70 is ideal (Nunnally & Berstein, 1994).

Face and content validity

Face and content validity were explored during the instrument development phase to help provide assurance that the tool measures the elements it was designed to capture. Generally, face and content validity are established by reviewing the literature and engaging experts to comment on the tool based on their expert judgment (Grant & Davis, 1997). In order to assess face validity, a small group of experts commented on whether the items measured what they were intended to measure. A four-step process was used to establish content validity. First, a review of the literature was conducted to identify a set of items that measure all aspects of the phenomenon of interest. Second, a National Advisory Group was convened to provide an initial review of this instrument. Members included public health professionals and QI experts. Third, cognitive interviewing was conducted with two local health departments to assess the comprehension of the questions and the appropriateness of the response options, techniques described by Willis (1999). Finally, the QI Maturity Tool was pilot tested with eight local health departments and one state public health agency from across the country.

Dimensionality

Based on our review of the literature, we anticipated that each domain would measure distinct underlying dimensions and we sought to test this assumption. Two rounds of factor analysis using principal component analysis were used to test the dimensionality of our instrument. First, we performed factor analysis assigning each item to one of our hypothesized domains in order to identify corresponding dimensions. Second, we conducted factor analysis based on all items in the model with no restrictions to compare whether or not the factor structure was consistent with our previous results and to determine if new or different dimensions emerged, also known as exploratory factor analysis. Prior to computing factor scores we ran diagnostic tests to determine sampling adequacy and sufficient correlations (e.g., Bartlett’s test of sphercity, Kaiser’s Measure of Sampling Adequacy). We used the promax with Kaiser normalization data rotation method and reviewed factor loadings that were greater than or equal to 0.40 for any given dimension.

Construct validity

The evaluation team assessed construct validity by testing the a prior hypothesis that the practice of QI would be positively correlated with the dimensions in each of the three domains. We anticipated that an agency’s QI culture, capacity, and competency, and QI alignment would be positively associated with the number of QI projects undertaken by that agency. In order to test our hypothesis, we identified the number of QI projects in the past 12 months reported by an agency (Q20), including those with no projects and correlated this variable with the dimensions that were generated based on our factor analysis.

Results

Respondents

Descriptive information about the survey respondents is provided in Table 2 . Overall, 60% of the local health departments participated in the survey and 92% of these agencies had descriptive data available in the 2008 NACCHO database. Approximately 82% of the surveys were completed by the lead health official in the department. No differences were found between participating and nonparticipating local health departments (LHDs) based on the following continuous variables: population size of jurisdiction (t value = −.80, df = 647, p > .43), full-time equivalents (t value = −.53, df = 611, p > .60), or expenditures (t value = −.56, df = 615, p > .58). Additional analyses also revealed no differences based on type of agency when comparing county-based versus all other (e.g., city, regional) types (X ₂ = 1.6465, df = 5, p > .8956). Psychometric testing was completed on a total of 468 (72%) health departments that responded to all relevant QI items. Statistical analyses revealed no significant differences in health department characteristics between respondents who completed the entire survey and those who missed QI items.

Table 2.

Descriptive Information on Respondents (n = 650)

Characteristic	n	(%)
Completed surveys	690	60
Matched with NACCHO 2008 Profile	650	92
Survey completed by administrator
Agency expenditures
Less than $500,000	139	21
$500,000–$3,000,000	290	45
More than $3,000,000	188	29
Missing	33	5
Agency size (full-time equivalents)
Less than 10	194	30
10–49	268	41
More than 50	151	23
Missing	37	6
Agency jurisdiction/population size
Less than 50,000	403	62
50,000–500,000	217	33
More than 500,000	29	5
Missing	1	<1
Geographic jurisdiction served
City	29	5
County	493	76
City/county	48	7
Town/township	21	3
Multicounty/district/region	42	6
Other	16	3
Missing	1	<1

Note. NACCHO = National Association of County and City Health Officials.

Descriptive Findings

A summary of the descriptive findings can be found in Table 3 . The valid responses are high (above 90%) for all items with the exception of Questions 34–37 which were part of a skip pattern based on a previous response that indicated a participant’s agency had never implemented a formal QI process. Programming features of the online survey precluded respondents from providing multiple answers to the same item, therefore eliminating the need to recode multiple responses as “missing.” In general, there were no items on the Likert-type scale questions with more than 50% of the answers falling in one response category. Furthermore, despite initial concerns about the lack of awareness regarding QI efforts, there were few “don’t know” responses ranging from less than 1% to 3% on any given item.

Table 3.

Descriptive Information on QI Items (n = 650)

		Responses (%)
		Strongly Disagree				Strongly Agree
Item Description^a	M (SD)	1	2	3	4	5	DK^b	9^c
Domain #1: QI Organizational Culture
1. Leaders receptive to ideas for improving quality	4.3 (0.8)	1	2	9	33	47	<1	8
2. Impetus for improving quality is external	2.8 (1.1)	11	29	28	18	6	<1	8
3. Impetus for improving quality is internal	4.4 (0.8)	1	2	7	33	50	<1	8
4. Leaders work together for common goals	4.2 (0.8)	1	2	12	36	40	<1	8
5. Agency data shared for improvement purposes	4.2 (0.9)	1	2	11	39	38	0	8
6. Matters reviewed in respectful way	4.2 (0.7)	1	1	10	43	37	<1	8
7. Staff help solve problems	4.5 (0.7)	<1	1	6	33	52	0	8
8. Staff routinely contribute to decisions	4.4 (0.7)	1	1	8	34	49	0	8
Domain #2: QI Capacity and Competency
9. Leaders are trained in basic QI methods	3.4 (1.2)	6	14	23	30	16	3	11
10. Staff members are trained in basic QI methods	3.0 (1.1)	9	20	29	23	9	3	11
11. Staff has skills to monitor quality of programs	3.7 (1.0)	3	10	23	36	20	0	8
12. Agency has objective quality measures	3.5 (1.0)	3	11	28	35	15	<1	9
13. Staff uses methods to identify root causes	2.7 (1.1)	12	27	29	17	5	3	11
14. Staff uses best or promising practices	3.4 (1.2)	2	6	22	44	17	3	9
15. Programs are continuously evaluated	3.6 (0.9)	2	10	28	38	13	1	8
16. Agency has a Quality Improvement Officer	2.4 (1.5)	38	17	10	10	14	3	10
17. The quality of programs is routinely monitored	3.6 (1.0)	3	9	27	33	20	<1	8
18. Process exists for establishing QI priorities	3.0 (1.2)	9	23	25	23	11	1	8
Domain #4: QI Alignment and Spread
23. Job descriptions include QI responsibilities	3.0 (1.2)	11	24	23	21	11	2	11
24. Staff is aware of external QI expertise	3.1 (1.1)	6	21	28	27	7	3	11
25. Staff at all levels participate in QI	3.5 (1.1)	4	14	23	33	19	<1	11
26. Customer satisfaction information is routinely used	3.6 (1.1)	4	14	19	32	22	<1	11
27. QI efforts usually adopted by other programs	3.3 (1.0)	4	13	29	33	10	3	11
28. Agency allocates sufficient time for QI	3.0 (1.1)	9	23	28	22	10	<1	8
29. Accurate and timely data available for QI	3.1 (1.1)	7	21	33	24	9	<1	8
30. Staff have authority to make change	3.7 (1.0)	4	8	21	40	19	2	8
31. Improving quality is integrated into agency practice	3.4 (1.0)	3	12	32	32	13	<1	8
32. Implementing QI is challenging among staff	3.7 (0.9)	2	8	26	38	17	<1	10
33. Spending time and resources on QI is worth it	4.3 (0.9)	1	2	13	30	45	<1	9
34. QI mostly happens in one area of agency	2.2 (1.1)	14	16	11	5	1	<1	53
35. Sufficient resources exist to sustain improvements	3.3 (1.1)	2	6	18	15	4	<1	53
36. Lessons from QI are integrated into practice	3.8 (0.8)	1	4	11	23	8	<1	53
37. Targets are determined before implementing QI	3.5 (1.0)	1	6	14	19	6	<1	53
Domain #3: QI Practice Responses (%)
Ever implemented QI 51 = Yes; 32 = No; 7 = Don’t Know; 10 = Missing
Number of QI projects, last 12 months 3.2 (4.4) = M (SD); 1–45 = Range; 60 = Missing
Length of time engaged in QI 5 = < 1 Yr; 9 =1–2 Yrs; 28 > 2 Yrs; 4 = No QI; 54 = Missing
Tools used 1 = 1 Tool; 30 = 2–4 Tools; 16 = 5+ Tools; 54 = Missing

QI = quality improvement.

^aFull item wording in Table 1.

^bDK = Don’t know.

^c9 = Missing data.

Specific findings revealed that approximately 51% of agencies reported ever implementing a QI project. This finding was consistent with results of the 2008 NACCHO profile and provided some level of assurance that sites were reporting their efforts in a similar manner across two separate surveys (Beitsch et al., 2010).

Reliability Findings

Internal consistency reliability estimates were calculated on all dimensions and all domains except for QI practice, given the type of response options for these items. Results corresponding to each domain revealed Cronbach’s α estimates of .76 or higher indicating satisfactory reliability. Alpha scores were calculated for QI organizational culture in two ways: (a) the complete 8-item scale was assessed and (b) a 7-item scale was used resulting in the elimination of Item 2 pertaining to external drivers of QI culture. This question was subsequently eliminated to better reflect the desired measure of an organization’s internal culture. Additionally, as mentioned above, the number of valid responses (see Table 3) available for four of the items in the alignment and spread domain were limited due to a skip pattern and consequently dropped from additional testing. Therefore, the internal consistency of this entire domain and a reduced version that removed Items 34–37 were explored.

Table 4 summarizes the reliability results by domain and dimension. Overall, the QI capacity and competency domain had the highest Cronbach’s α score at .89. The reliability estimates for the reduced and nonreduced culture domain were .81 and .76, respectively. The α estimates for the full and reduced alignment and spread domain were both .87. The α scores for the dimensions ranged from .67 to .89. All but two dimensions (commitment, investment) were within the ideal range.

Table 4.

Internal Consistency and Item Correlations (n = 468)

Domains & Dimensions	# Items	Chronbach’s α
QI Organizational Culture^a	7	.81
Commitment	3	.67
Collaboration	4	.80
QI Capacity & Competency	10	.89
Skills	2	.84
Methods	6	.87
Investment	2	.69
QI Alignment & Spread^b	11	.87
Integration	8	.89
Authority	1	NA
Implementation	1	NA
Value	1	NA

^aQ2 excluded from domain.

^bQ34–37 excluded from domain.

NA = not available; QI = quality improvement.

Validity Findings

Content and face validity

As mentioned previously, a National Advisory Group was convened to provide input during the development and testing stage of the QI Maturity Tool. Additionally, cognitive interviewing and pilot testing were conducted and subsequent changes were incorporated to streamline and strengthen the instrument thus providing initial confidence in the tool’s face and content validity. During this stage of the validation process, item changes were based on expert suggestions from the National Advisory Group and feedback received from practitioners who completed the tool.

The first draft of the tool included a series of 95 questions assessing various facets of a public health department’s level of QI maturity based on the literature. Questions that were redundant, vague, problematic to interpret, or difficult to answer were eliminated or revised, resulting in a 37-item survey.

Dimensionality

The suitability of the data was assessed and the diagnostic test results revealed satisfactory findings indicating the appropriateness of conducting factor analysis. The factor loadings for each item and the corresponding hypothesized dimensions based on our initial factor analysis are provided in Table 5 . The findings revealed a two-factor structure for the organizational culture domain that we termed: (a) commitment and (b) collaboration. Commitment was defined as a willingness to adopt new ideas among leaders and a desire to improve services and outcomes. This factor or dimension included 3 items assessing receptivity to QI, the impetus for improving quality, and the experiences of leadership based on common goals. The collaboration dimension or factor was defined as a mutually supportive work environment that engages staff and creates an environment where QI can thrive. The 4 items in this factor focused on problem solving, staff involvement, data sharing, and shared accountability. Both of these factors accounted for 63% of the variance explained.

Table 5.

QI Maturity Tool Factor Loadings and Alignment With Hypothesized Domains (n = 468)

	Factors and Factor Loadings
Domains and Items	1	2	3	4	5	6	7	8	9
QI Organizational Culture^a
Leaders receptive to ideas for improving quality^*	−0.123	0.865
Impetus for improving quality is internal	0.007	0.711
Leaders work together for common goals^*	0.079	0.738
Agency data shared for improvement purposes	0.476	0.372
Matters reviewed in respectful way^*	0.737	0.060
Staff help solve problems^*	0.915	−0.118
Staff routinely contribute to decisions^*	0.868	−0.030
QI Capacity & Competency
Leaders are trained in basic QI methods^*			−0.073	0.932	0.040
Staff members are trained in basic QI methods^*			0.073	0.855	0.049
Staff has skills to monitor quality of programs^*			0.760	0.190	−0.183
Agency has objective quality measures^*			0.660	0.172	0.080
Staff uses methods to identify root causes^*			0.471	0.381	0.042
Staff uses best or promising practices^*			0.832	0.005	−0.182
Programs are continuously evaluated^*			0.914	−0.094	−0.019
The quality of programs is routinely monitored^*			0.737	−0.177	0.306
Agency has a Quality Improvement Officer^*			−0.230	0.107	0.997
Process exists for establishing QI priorities^*			0.441	−0.058	0.574
QI Alignment & Spread^b
Job descriptions include QI responsibilities^*					0.871	−0.029	−0.23	0.204
Staff is aware of external QI expertise					0.531	0.448	−0.320	0.030
Staff at all levels participate in QI^*					0.876	−0.153	0.026	−0.035
Customer satisfaction information routinely used^*					0.678	−0.074	0.177	0.051
QI efforts usually adopted by other programs^*					0.847	−0.121	0.033	−0.038
Agency allocates sufficient time for QI^*					0.541	0.293	0.131	−0.196
Accurate and timely data available for QI^*					0.501	0.284	0.154	−0.136
Improving quality is integrated into agency practice^*					0.668	−0.055	0.339	0.057
Staff has authority to make change^*					0.031	0.069	0.870	0.090
Implementing QI is challenging among staff^*					0.048	0.083	0.105	0.949
Spending time and resources on QI is worth it					−0.226	0.956	0.124	0.094

Note: Bolded items reflect corresponding factor. ^aQ2 excluded from domain.

^bQ34–37 excluded from domain.

*Consistent factor loading findings were found in model two when all items were included together versus separated by hypothesized domain.

The capacity and competency domain revealed three factors or dimensions that we labeled: (a) skills, (b) methods, and (c) investment. When combined, these three factors accounted for 72% of the variance explained. The skills dimension was defined as leadership and staff training in basic methods for evaluating and improving quality and included 2 items assessing training experience. The methods dimension was defined as the application of QI tools, approaches and data to systematically assess and improve quality. This factor included 6 items measuring the ability of staff to monitor quality, identify root causes, and implement best practices as well as the availability of quality measures within the agency and current evaluation practices. Finally, the investment dimension was defined as dedicated staff and priorities for QI. This dimension included 2 items measuring existing processes for establishing QI priorities and core staff responsible for QI. The item measuring the development of QI priorities had a moderate cross-loading with the methods dimension.

The alignment and spread domain indicated a four-factor solution. The factors or dimensions were labeled: (a) integration, (b) authority, (c) value, and (d) implementation. When combined, the four-factor solution accounted for 70% of the variance explained. Integration was defined as agency policies and practices that support QI. The integration dimension focused on the alignment of QI responsibilities with job descriptions, the use of customer satisfaction data, the availability of accurate and timely data, the incorporation of QI into daily practice, the amount of awareness about QI expertise, and the level of participation, adoption, and time available for QI efforts. The authority dimension was defined as the ability of staff to make and implement decisions that affect quality and included 1 item assessing staff authority to make needed change. The value dimension was defined as the perception that QI is worthwhile. It included 1 item about the importance of devoting time and resources to QI efforts. While Item 24 was retained in the integration dimension, there was a moderate cross-loading with the value dimension as well. Finally, the implementation factor was defined as the perception that QI is challenging to implement. This factor included 1 item assessing the ease of QI implementation.

Our second factor analysis model was intended to determine if the dimensions would remain the same when all factors were combined in the model. The results indicated that 3 of the 28 items loaded with a different factor, Item 24 became a separate dimension, Items 3 and 33 also resulted in a new dimension. While Item 5 loaded with the anticipated dimension (collaboration), the results revealed that it had a moderate cross-loading with the integration factor as well.

Construct validity. Correlation results based on the number of QI projects by each dimension revealed positive moderate correlations with the three capacity and competency dimensions: skills (.35), methods (.36) and investment (.45), and with the integration dimension (.40). Weak correlations were identified with both of the culture dimensions, collaboration (.18), and commitment (.26).

Discussion

The work presented here represents the initial steps in testing and validating the QI Maturity Tool including: (a) developing a theoretical framework, (b) assuring face and content validity, (c) determining the tool’s reliability based on estimates of internal consistency, (d) assessing the dimensionality, and (e) determining the construct validity of the instrument. We believe the combined steps and techniques used, coupled with the results outlined above, provide preliminary evidence that the QI Maturity Tool is generally a reliable measure with distinct dimensions. Yet, further refinement and testing is necessary, particularly with the single item dimensions and 6 items (Q3, Q5, Q18, Q24, Q32, and Q33) that will likely need to be removed, reworded, or reassigned.

Our analyses demonstrated face and content validity. This suggests that the items in the tool appropriately reflect the components and characteristics generally known, or believed to be associated with QI Maturity based on the four quality domains and corresponding dimensions. However, while the construct validity testing provided early evidence about expected correlations, additional testing and investigation is needed. We anticipated that QI practice would be associated with all dimensions in the instrument. Yet, our findings revealed weak correlations with the culture domain suggesting a limited association between the number of QI projects implemented by an organization and an agency’s level of QI commitment and collaboration. It is possible that this weak correlation is due to a lack of understanding regarding QI and the use of QI rhetoric which can differ fairly dramatically from the practice of QI (Zbaracki, 1998).

Implications for Policy and Practice

The application of this instrument is relevant to public health practice and policy. National organizations and associations, as well as state and local public health agencies have expressed interest in using this tool to assess efforts, determine priorities for action, and build support for QI. The refinement of this tool also has implications for research, particularly if a comparative database is created that will allow agencies to track progress over time, compare their findings with other health departments, and explore the impact of QI maturity on health outcomes. The Hospital Survey on Patient Safety Culture developed in 2004 by the Agency for Healthcare Research and Quality (AHRQ) may provide a good model for this work. This survey was designed to help hospitals assess the culture of safety in their organizations (Agency for Healthcare Research and Quality [AHRQ], 2011). Since its inception, hundreds of United States and international hospitals have administered the survey and used the findings to guide policy and programmatic decisions. This tool has undergone a fair amount of psychometric testing (Sorra & Dyer, 2010) and a comparative database has been developed. Although the AHRQ tool is designed for a health care setting, we believe the model is worth exploring in public health. It is likely that there will be many relevant lessons learned from this effort and others that are applicable as we continue to strengthen the QI Maturity Tool and the administration process and as we develop a data repository that is available for practitioners and researcher.

Limitations and Next Steps

Although the early psychometric testing revealed several positive results and the preliminary findings suggest that this is a promising tool for measuring QI maturity in a health department setting, we recognize the need to continue refining and validating this instrument. The concept of QI in public health remains relatively new and, to date, little work has been done to test and validate existing measurement instruments in this area. Further validation studies with the QI Maturity Tool are needed to expand our construct validation work and to explore concurrent and criterion validity. Additional efforts also are needed to further substantiate and expand our initial psychometric findings with a broader group beyond agency administrators.

This study represents early efforts to develop a tool that can be used to assess an agency’s level of QI organizational culture, capacity and competency, practice, and alignment, and spread. Modifications have been made to this instrument since its initial administration in 2009 including the expansion of those dimensions that are based on single items. Additionally, ongoing efforts are underway to test additional concepts and features of the tool. For example, we know from research conducted by Zbaracki (1998) that it is important to distinguish the rhetoric of QI from the practice of QI when defining and evaluating adoption and spread. Zbaracki (1998) stresses the need to probe beyond “ceremonial” or “rhetorical” components that embrace the language of QI but fail to become “reality” by putting QI into practice. Building off this research, we conducted a series of case studies to explore the extent to which reported survey results align with an agency’s QI understanding, actions, and products. We also explored the sophistication of the QI techniques being used and whether or not they are sustained within the organization. Analysis of this data is currently underway.

In addition to exploring the issue of rhetoric, we also recognize the need to administer this tool to all staff in a health department and to adequately assess the test–retest reliability of the instrument. While the survey administration efforts were consistent with others in the field (Beitsch et al., 2010; Madamala et al., 2010), the reliance on self-reported data by agency administrators alone may not be sufficient or consistent with results reported by all staff. While it was not feasible as part of the overall evaluation efforts to administer the survey twice in a repeated design or to administer it to all staff, we believe these are important next steps to pursue in the near future.

Finally, our team hopes to continue improving the tool by further refining the domains, dimensions, items, and response options based on analyses beyond the Multi-State Learning Collaborative (MLC) evaluation. For example, the “I don’t know” category may be eliminated as awareness of QI continues to increase in public health settings. Additionally, we intend to develop a shorter version of this instrument based on ongoing analysis on the best predictors within a given domain and the instrument’s dimensionality.

Conclusions

Measuring QI in public health is in its formative years. Yet, while much work remains to be done, the analyses suggest that the QI Maturity Tool is one of the few, if not the only, existing reliable and valid measurement instruments designed for a public health department setting. The tool measures four domains of quality based on the literature including: organizational culture, capacity and competency, practice, and alignment and spread. This study holds promise for measuring QI maturity in a health department setting and using this instrument to better understand the relationship between quality domains and specific outcomes of interest.

Footnotes

Acknowledgments

The authors are grateful to Leslie Beitsch, MD, JD, Mary Davis, PhD, MSPH, Brenda Henry, PHD, MPH, and William Riley, PhD for their guidance. The authors also wish to thank all of the health departments that participated in our survey and Bruce Clary PhD for his assistance with validity testing.

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was part of a larger national evaluation effort of the Multi-State Learning Collaborative funded by the Robert Wood Johnson Foundation, grant # 64232.

References

Agency for Healthcare Research and Quality. (2011). Surveys on patient safety culture. Retrieved from www.ahrq.gov/qual/patientsafetyculture/

Baldridge National Quality Program. (2008). Criteria for performance excellence Gaithersburg, MD: National Institute of Standards and Technology, Department of Commerce.

Beitsch

L. M.

Leep

Shah

Brooks

R. G.

Pestronk

R. M.

(2010). Quality improvement in local health departments: Results of the NACCHO 2008 survey. Journal of Public Health Management and Practice, 16, 49–54.

Beitsch

L. M.

Thielen

Mays

Brewer

R. A.

Kimbrell

Chang

C. …

Landrum

L. B.

(2006). The multistate learning collaborative, states as laboratories: Informing the national public health accreditation dialogue. Journal of Public Health Management and Practice, 12, 217–231.

Berlowitz

D. R.

Young

G. J.

Hickey

E. C.

Saliba

Mittman

B. S.

Czarnowski

… Moskowitz

M. A.

(2003). Quality improvement implementation in the nursing home. Health Services Research, 38, 65–83.

Bialek

Carden

Duffy

G. L.

(2010). Supporting public health departments' quality improvement initiatives: Lessons learned from the public health foundation. Journal of Public Health Management and Practice, 16, 14–18.

Corso

L. C.

Lenaway

Beitsch

L. M.

Landrum

L. B.

Deutsch

(2010). The national public health performance standards: Driving quality improvement in public health systems. Journal of Public Health Management and Practice, 16, 19–23.

Cronbach

L. J.

(1951). Coefficient alpha and the internal structure of tests. Psykometrika, 16, 297–333.

Davis

(2010). Opportunities to advance quality improvement in public health. Journal of Public Health Management and Practice, 16, 8–10.

10.

Davis

P. B.

Solomon

Gorenflo

(2010). Driving quality improvement in local public health practice. Journal of Public Health Management and Practice, 16, 67–71.

11.

Department of Health and Human Service. (2008). Consensus statement on quality in the public health system. Retrieved from www.hhs.gov/ash/initiatives/quality/quality/phqf-consensus-statement.pdf

12.

Gillen

S. M.

McKeever

Edwards

K. F. K.

Thielen

(2010). Promoting quality improvement and achieving measurable change: The lead states initiative. Journal of Public Health Management and Practice, 16, 55–60.

13.

Gorenflo

(2010). Achieving a culture of quality improvement. Journal of Public Health Management and Practice, 16, 83–84.

14.

Grant

J. S.

Davis

L. L.

(1997). Selection and use of content experts for instrument development. Research in Nursing & Health, 20, 269–274.

15.

Honore

P. A.

Scott

(2010). Priority areas for improvement of quality in public health Washington, DC: U.S. Department of Health and Human Services.

16.

Joly

B. M.

Shaler

Booth

Conway

Mittal

(2010). Evaluating the multi-state learning collaborative. Journal of Public Health Management and Practice, 16, 61–66.

17.

Lenaway

Corso

Bailey

(2007). Accreditation as an opportunity to strengthen public health: CDC's perspective. Journal of Public Health Management and Practice, 13, 332–333.

18.

Leonard

B. A.

(2007). Adapting quality improvement to public health: Highlights and conclusions. Conference Summary. Robert Wood Johnson Foundation. Retrieved from www.nnphi.org/CMSuploads/AdaptingQI_Final-051807edit-80435.pdf

19.

Lombarts

M. J.

Rupp

Vallejo

Klazinga

N. S.

Sunol

(2009). Differentiating between hospitals according to the “maturity” of quality improvement systems: A new classification scheme in a sample of European hospitals. Quality and Safety in Health Care, 18, i38–i43.

20.

Lukas

C. V.

Holmes

S. K.

Cohen

A. B.

Restuccia

Cramer

I. E. Shwartz, M.

Charns

M. P.

(2007). Transformational change in health care systems: An organizational model. Health Care Management Review, 32, 309–320.

21.

Madamala

Sellers

Pearsol

Dickey

Jarris

P. E.

(2010). State landscape in public health planning and quality improvement: Results of the ASTHO Survey. Journal of Public Health Management and Practice, 16, 32–38.

22.

Nolan

K. M.

Schall

M. W.

(Ed.). (2007). Spreading improvement across your health care organization. Oakbrook Terrace, IL: Joint Cogmmission Resources

23.

Nunnally

J. C.

Berstein

I. H.

(1994). Psychometric theory (3rd ed.). New York, NY: McGraw-Hill.

24.

Office for State, Tribal, Local and Territorial Support. National Public Health Improvement Program. CDC. (September 2010). Retrieved from www.cdc.gov/ostlts/NPHII/

25.

Riley

W. J.

Moran

J. W.

Corso

L. C.

Beitsch

L. M.

Bialek

Cofsky

(2010). Defining quality improvement in public health. Journal of Public Health Management and Practice, 16, 5–7.

26.

Riley

W. J.

Parsons

H. M.

Duffy

G. L.

Moran

J. W.

Henry

(2010). Realizing transformational change through quality improvement in public health. Journal of Public Health Management and Practice, 16, 72–78.

27.

Rogers

E. M.

(2006). Diffusion of innovation (4th ed.). New York, NY: The Free Press.

28.

Rondeau

K. V.

Wagar

T. H.

(2002). Organizational learning and continuous quality improvement: Examining the impact on nursing home performance. Healthcare Management Forum, 15, 17–23.

29.

Russo

(2007). Accreditation of public health agencies: A means, not an end. Journal of Public Health Management and Practice, 13, 329–331.

30.

Seid

Lotstein

Williams

V. L.

Nelson

Lurie

Ricci

K. A. …

Stern

(2006). Quality improvement: implications for public health preparedness Pittsburgh, PA: RAND.

31.

Sorra

J. S.

Dyer

(2010). Multilevel psychometric properties of the AHRQ hospital survey on patient safety culture. BMC Health Services Research, 10, 1–13.199

32.

Turning Point. Performance Management Collaborative. (2007). Turning point. Retrieved from www.turningpointprogram.org/Pages/perfmgt.html

33.

Willis

G. B.

(1999). Cognitive interviewing: A “how to” guide. American statistical association. Retrieved from appliedresearch.cancer.gov/areas/cognitive/interview.pdf

34.

Zbaracki

M. J.

(1998). The rhetoric and reality of total quality management. Administrative Science Quarterly, 43, 602–636.