Implementation Research as Applied Science: Bridging the Research to Practice Gap

Abstract

Implementation research is intended to address challenges posed by the slow adoption of evidence-based science by the medical and health promotion practice community. A case study approach is used to illustrate and discuss the use of Quality improvement and Evaluation as an applied approach to implementation science in contrast of more classic purposes of research. Quality improvement was the implementation model used to facilitate organizational change needed to adopt the use of texting to report sexually transmitted infection test results in over a fifth of Florida’s larger county health departments. Both quantitative and qualitative methods were used to evaluate implementation. All seven participating county health departments were successful in enrolling clients in texting with extensive variation (24% to 72%) in texting enrollment at the end of the 10-month study. Statistically significant outcomes for those enrolled in texting were recorded through Florida’s online sexually transmitted infection reporting system in the form of increased number of people receiving early (1-4 days) treatment and reductions in delayed (≥8 days) or no treatment. This study illustrates an applied approach to implementation research which may be critical to adapt emerging evidence and technologies to the multiple and complex characteristics of the diverse populations served by health promotion institutions.

Keywords

implementation science quality improvement organization change translational research mixed methods evaluation sexually transmitted infection test results digital communication mobile technology

Introduction

One of the greatest challenges for medicine and population health has been the slow pace of the practice communities in adopting the latest evidence-based science (Meslin, Blasimme, & Cambon-Thomsen, 2013; Riley, 2017; Westfall, Mold, & Fagnan, 2007; Woolf, 2008), which exacerbated by inequities in health care delivery and allocation of resources (Institute of Medicine [IOM], 2003; Krieger, Rowley, Herman, Avery, & Phillips, 1993), contributes to the quality chasm (IOM, 2001). The challenges for public health and health promotion agencies to adopt newly emerging 21st-century technologies and evidence-based medicine can be even more profound. In addition to the slow pace of adopting emerging science and the chronic health disparities found with private sector health system institutions, public agencies, in general, can have the conventional trappings of government bureaucracy, as well as facing the challenges of organizational culture’s resistance to change (Davis et al., 2014; Ravasi & Schultz, 2006; Schein & Schein, 2017). With these extensive impediments to change, agencies responsible for health promotion face daunting barriers to adopting and adapting new technologies to their service delivery, creating a rich environment for implementation research.

Background

Rather than creating new evidence associated with intervention research, a major purpose of implementation research is to bridge the gap between the research and practice communities (Brown et al., 2017; Nilsen, Neher, Ellström, & Gardner, 2017; Riley, 2017; Tabak, Khoong, Chambers, & Brownson, 2012). Within that context, one purpose of implementation research is to reveal the underlying principles of change within health care practice organizations, consistent with developing and testing theory associated with many forms of research (Damschroder et al., 2009; Kirk et al., 2016; Nilsen, 2015; Sales, Smith, Curran, & Kochevar, 2006). Alternative to developing and testing theory and underlying principles, a more pragmatic view of implementation research would be adapting science to achieve outcomes in specific settings, which is more associated with applied research such as engineering (Livingood et al., 2011). For example, the main focus of engineering is not the development and understanding of underlying theories or principles, but rather the adaptation of existing array of scientific principles and technologies to each unique setting.

Viewing implementation research as a more applied research endeavor, consistent with engineering’s emphasis on applying an array of scientific principles or technologies to unique settings, this study used quality improvement (QI), complemented with evaluation methods, as an implementation mechanism to adopt and adapt emerging digital communication technologies, more specifically text messaging, to public health service delivery. In this case, QI is an implementation model borrowed from business management that was developed over decades starting with W. E Deming’s work in operational/industrial engineering and management science (Deming, 2000). These pragmatic origins do not emphasize theory but are more focused on application of principles and technologies in practical settings to improve quality and productivity.

Implementation research can involve a variety of approaches derived from the social and behavioral science such as Diffusion of Innovation, Organization Change, and Re-AIM Evaluation (Rabin, Brownson, Haire-Joshu, Kreuter, & Weaver, 2008). QI derived from management science and operational engineering is not typically cited in the social and behavioral science literature (Woodhouse et al., 2013), but has been widely adopted as an essential component of public health practice as reflected in the public health agency accreditation standards. Key components of QI have been identified as (1) QI Team Functioning/Group Dynamics, including direct involvement of members providing the service, collaboration across job responsibilities, motivation to implement change, commitment to continuous improvement, and active exchange of Information, ideas, and suggestions; (2) Organization Support, including: both affected unit and county health department (CHD) management; and (3) Tracking and Use of Data including: Goals are readily measurable, use of measurements to plan changes, and affected unit continuously tracks progress (Livingood, Bilello, & Lukens-Bull, 2019).

The purpose of this case study is to illustrate this applied research approach using a merged model of QI and evaluation (Woodhouse et al., 2013) as a framework for adapting recently emerged digital technologies to public health agency settings. The emphasis for this approach is on the output and outcome measures associated with successful implementation, not the development or testing of theory or proving the effectiveness of an intervention that is associated with intervention research.

Method

Design

A case study approach is used to illustrate and discuss the issues and merits of an applied science approach to implementation research versus approaches emphasizing development and testing of theory or approaches focused on proving cause-and-effect relationships. This study was granted exempt status by the institutional review board of the home academic institution.

The purpose of this case study was to illustrate the use of QI and evaluation (Woodhouse et al., 2013) as a model for implementation research within the context of applied versus pure research or intervention research approaches. Consistent with the general purpose of the case study approach, this report is intended to enable “in-depth, multi-faceted explorations of complex issues in their real-life settings” (Crowe et al., 2011). In particular, it is presented to demonstrate and stimulate discussion of an applied approach to implementation research that is focused on adapting principles and technology to unique settings rather than to develop and test theoretical principles. This opportunity to use QI to adopt and adapt mobile health technology to increase efficiency and improve outcomes was identified through a participatory research process involving state and county public health agency officials (Livingood, Bilello, Choe, & Lukens-Bull, 2018). Data were collected in 2016 and 2017 and analyzed in 2017.

Setting and Participants

All counties in Florida have CHDs, but the counties they serve and the resources supporting the CHDs vary extensively, at least partially due to the differences in the populations and wealth of the counties they serve. Counties vary in size from just over 8,000 to over 2.7 million people (33 of Florida’s 67 counties have less than 100,000 and 12 have less than 20,000 people). The 2015 county poverty rate ranged from 9.5% to 30.6%. The 2015 Full-Time Equivalent (FTE) employees for CHDs in Florida ranged from 8 to 759. Taking into account the very different population sizes, the CHD 2015 FTE expenditure rate (annual dollars per person in county) ranged from $27.9 to $229.2 with the average expenditure rate for Florida CHDs of $36.5. Despite Florida’s reputation as a centralized system of CHDs, the adage “When you’ve seen one CHD, you’ve seen one CHD” applies. Smaller counties did not participate in this study, primarily because they tend to rely on larger CHDs for many of the sexually transmitted infection (STI) services. Over 20% of the larger CHDs (7/33 counties with over 100,000 populations) committed to the study with others indicating an interest in adopting the new technology, but not committing to the time frame or requirements. Table 1 provides more CHD details, showing extensive variations in disease burden and resources available to each participating CHD.

Table 1

County Health Department (CHD) Demographics, Key Health Statistics, and CHD Characteristics

County	2016 County Size^a	2015 % Black^b	2015 % Hispanic^b	% Below Poverty^c	2017 County Health Rank/67 Florida Counties^b	2016 CHD Employee FTE^d	2016 CHD FTE Rate/Person^d	2015 CHD Expend in Millions^e	2015 CHD expend per Person^e	2016 Gonorrhea Rate/100,000^f	2016 Chlamydia Rate/100,000^g
CHD A	178,465	6.4	6.6	12.2	22	66	38.9	4.5	27.1	29.9	186.0
CHD B	263,496	20.7	9.5	24.3	26	227	87.7	16.6	65.3	229.8	862.7
CHD C	375,888	9.5	16.8	15.2	21	105	29.1	7.7	22.2	127.8	389.9
CHD D	512,368	5.6	13.4	14.0	35	172	34.5	11.3	22.9	75.6	278.8
CHD E	960,730	11.5	9.8	14.4	28	629	65.9	49.7	52.7	163.5	427.7
CHD F	1,376,238	17.8	26.9	17.0	29	365	26.9	39.4	29.5	113.4	594.0
CHD G	2,712,945	19.0	66.6	20.4	23	770	28.5	71.1	26.8	174.2	469.2

U.S. Census Population Estimate as of July 1, 2016 (https://factfinder.census.gov/). ^bFlorida Department of Health County health profile (http://www.flhealthcharts.com/ChartsReports/rdPage.aspx?rdReport=ChartsProfiles.CountyHealthProfile). ^cFlorida Department of Health Percentage of individuals below poverty level (http://www.flhealthcharts.com/Charts/OtherIndicators/NonVitalIndDataViewer.aspx?cid=294); County Health Rankings and Roadmaps (http://www.countyhealthrankings.org/app/florida/2017/overview). ^dFlorida Department of Health County Public Health Department Full-Time Employees (http://www.flhealthcharts.com/charts/OtherIndicators/NonVitalIndNoGrpDataViewer.aspx?cid=0332). ^eFlorida Department of Health County Public Health Department Expenditures in Dollars (http://www.flhealthcharts.com/charts/OtherIndicators/NonVitalIndNoGrpDataViewer.aspx?cid=0333). ^fFlorida Department of Health Gonorrhea Cases (http://www.flhealthcharts.com/charts/OtherIndicators/NonVitalSTDDataViewer.aspx?cid=0145). ^gFlorida Department of Health Chlamydia Cases (http://www.flhealthcharts.com/charts/OtherIndicators/NonVitalSTDDataViewer.aspx?cid=0145).

Measures

Use of QI methods to implement a change in practice requires a distinction between (1) the methods used by each CHD to apply QI methods versus (2) the impact of using QI, the process and outcome measures associated with change in organization practice, in this case, texting STI results to clients/patients.

STI Texting Process (Impact) Measures

The primary process measure used to gauge organization uptake of the texting notification process for STI test results was the number of clients enrolled in the text messaging program. Clients had to proactively consent to be notified through use of texting. Each CHD selected their own targets for texting enrollment, ranging from 24% to 90%. Test and enrollment data were recorded by CHD staff into the PRISM (Patient Reporting Investigation Surveillance Manager), an electronic information and data management system, developed by the Florida Department of Health (FDOH) in collaboration with the Centers for Disease Control and Prevention (CDC; Shiver, Schmitt, & Cooksey, 2009). Data included the client cell phone number and mobile phone carrier to enable PRISM to initiate test result notification electronically. Test results were coded to call CHD prearranged number without CHD identification, test, or results information. The PRISM system had to be modified by FDOH to accommodate this additional data and use of PRISM to trigger the texted notification process. Consequently, the mobile carrier also had to be compatible with PRISM.

Texting STI Results Outcome Measures (Impact)

The primary outcome measure for change in organization practice related to texting STI results was the time to treatment following the lab results for gonorrhea and chlamydia tests being entered into PRISM. Dates of treatment were entered into PRISM with other treatment data by CHD staff. Time to treatment was calculated by FDOH staff using PRISM data, primarily measured by number of days to treatment after test results were entered into PRISM. Time to treatment was grouped as 0 (same day treatment), 1 to 4 days, 5 to 7 days, and 8 days and over or no treatment. Because same day treatment primarily included presumptive treatments, which are typically not affected by different notification procedures, same day treatment comparisons were excluded from the analysis.

Measures of Organization Use of QI Processes

Data on organization use of QI processes were collected and reported electronically to the study team using previously distributed reporting forms included QI team membership, time and date of QI team meetings, attendance at the meeting, subject matter of the meeting particularly related to progress with texting enrollments and problems encountered in achieving targets, and QI processes such as root cause analysis to resolve problems. The QI team chairperson also attended the QI Collaborative monthly phone conference meetings where progress and challenges for texting enrollment were discussed.

Survey Organization Use of QI

A QI postintervention survey was also conducted with participating CHD staff to assess staff perceptions of using QI to support organization change (adoption of an option for client enrollment for texting STI results). The survey included 10 Likert-type ordinally scaled QI items developed from previous QI research (Livingood et al., 2012; Livingood et al., 2013; Livingood et al., 2015) and open-ended items to obtain staff perspectives of the QI efforts and texting project. The scaled 10 items were aggregated under 3 major constructs; Management/Unit support (3 items), Measurement and Tracking (3 items), and QI Team Engagement (4 items).

Attempts were also made to collect cost-effective data as gauged by staff time dedicated to STI services. These data were collected through time on task instruments and retrospective survey instruments designed by the study team and reported electronically to the CHD staff performing the tasks.

Processes (Recruitment and Analysis)

CHDs were recruited through an ongoing Public Health Practice Based Research Network that involved both the Florida county health agencies and the FDOH (Livingood et al., 2018). The invitation included the stated “purpose of this project is to monitor, support and assess CHD QI efforts to implement a STI texting initiative.” Invitations came from a CHD director who was chairing the FDOH supported Infectious Disease Council, and who also served as a co–prinicipal investigator for the project. The invitations described the expectations for participation that reflected key qualities of QI, including: commitments of the CHD to provide both the option for clients to receive STI test results through mobile technology and the use of QI processes to facilitate the organization change in providing services (see Table 2).

Table 2

Participation Requirements for CHDs Related to Key QI Characteristics

QI Characteristic	Participation Requirement
QI Team Functioning	1. QI team to be established to review the results of the change to the STI program. 2. QI team uses QI methods to improve CHD STI texting implementation efforts.
Tracking and use of data	1. Collect and report data on process and outcome measures used to continually improve the implementation of the texting initiative.2. Collect baseline time on task data to provide STI test results to clients including phone calls, field visits, and follow-up contact efforts. 3. QI methods to monitor CHD STI texting implementation efforts.
Organization support	1. CHD director support for project. 2. CHD work with FDOH to adapt the texting initiative protocol for their use.

NOTE: CHD = county health department; QI = quality improvement; STI = sexually transmitted infection; FDOH = Florida Department of Health.

Analysis primarily included descriptive statistics including measures of central tendency, proportions, and graphs consistent with QI techniques for most quantitative data and content analysis for qualitative text. For impact evaluation purposes, the chi-square test was used to assess differences in the outcome of proportion of participants who were treated early (1-4 days) or who had delayed (≥8 days) or no treatment, for clients using texting to receive results compared with clients who did not enroll for texted results. The chi-square was also used to analyze statistical difference in enrollment proportions for STI texted results among participating CHDs.

Results

Public Health Impact Process and Outcomes

The major public health outcome of concern for this project was time to treatment following a positive STI test. A statistically significant difference in the proportion of treated participants based on the time to treatment was observed between people using the texting versus those who did not enroll in texting, based on analysis of the PRISM data (Bilello, Livingood, Lukens-Bull, Smotherman, & Choe, 2019). Texting enrolled clients significantly sought early treatment (within 1-4 days) at a higher rate than those who did not enroll in treatment. Conversely, clients who did not enroll in texting did not seek treatment or sought it more than 8 days after testing at a significantly (p < .001) higher rate than those who were enrolled for texted results (Bilello et al., 2019).

The primary process measure and the most direct indicator of organization behavior in implementing the texting option was the percentage of clients who obtained a STI test who also enrolled in the texting program. Consistent with the very different CHD characteristics and targets for enrollment, the CHDs performed very differently (p = .0001) in enabling client enrollment in texting. Some CHDs took more time to implement the texting program, with some starting as early as May, but all of the participating CHDs fully implementing the texting project in the designated clinics by August 2016 (see Table 3). After implementing the texting project, one CHD reached over 90% enrollment very early and another CHD struggled to achieve 25% enrollment over the course of the project (See Table 3). CHD B implemented the texting project in a one day per week (Monday) clinic which never went above 17 STI tests per week. This CHD was also substantially effected by holiday closures and weather (hurricane) closures. This CHD decreased from a high of 17 tests in one week to only 1 to 5 tests each week in November and December, followed by discontinuing their involvement in the project. Others in the larger CHDs implemented in multiple clinics, with CHD E reaching over 200 STI tests in one week.

Table 3

Percent Enrollment for Texting by CHD Clients Obtaining STI Test

CHD	Target	May	June	July	August	September	October	November	December	January	February
CHD A	90		88	75	84	89	75	100	100^a
CHD B	40				15	17	21	20	22	25	29
CHD C	70	33	47	28	24	18	39	29	12	21	24
CHD D	30	60^b	24	25	23	29	34	17	32	29	21
CHD E	24	61	26	32	31	71	73	70	45	26	37
CHD F	50	98	95	100	97	95	86	69	64	76	75
CHD G	65		37	58	57	58	68	60	71	60	51

NOTE: CHD = County Health Department; STI = sexually transmitted infection.

Dropped to 1-5 tests per week. ^bOne week of services.

QI Processes

The QI information and data provided by the CHDs provided important insights into the extensive variations in texting enrollments. All CHDs initially reported their QI team membership, QI team meetings, numbers of STI tests, and STI texting enrollments. Each CHD QI team leader participated in the monthly QI consortium conference call. In addition to the extensive variations in the previously described CHD populations served, accessible resources and disease burden, the qualitative data collection and analysis revealed extensive variations in texting enrollment including use of same day (presumptive) treatment, client access to mobile technology, incompatible mobile carriers used by clients, apparent differences in experience and commitment to QI by CHDs, different starting points for each CHD (readiness), staffing changes (loss of staff), number and types of clinics where texting was implemented, and increased availability and use of CHD electronic health record (EHR) patient portal. The level of experience and commitment to QI appeared to vary across the agencies, and this also appeared to impact the effectiveness of using QI to implement the use of this emerging technology.

The six participating CHDs at the conclusion of the project yielded 43 survey participants, ranging from 2 to 9 participants from the different CHDs, a small sample size that challenged statistical analysis. After reducing the responses to a dichotomous variables and aggregating the responses under the three major constructs, a significant difference (p = .027) was observed from pre- to postperceptions showing movement to agree/strongly agree for the construct QI team engagement (team functioning and group dynamics).

Discussion

The results of this QI/evaluation case study clearly documented substantial progress in adopting emerging technologies by public health agencies while documenting the extensive variations among those agencies. Much of the discussion of the purpose of implementation research is more aligned with definitions of research related to the development of theory (Damschroder et al., 2009; Kirk et al., 2016; Nilsen, 2015; Sales et al., 2006) or as intervention research, neither of which reflects the purpose of the approach to research described in this case study. For example, the IOM (2007) report on QI and implementation research assessed the relationship of the two terms (QI and implementation research) within the context of definitions of research associated with intervention research or more pure research, without discussion of its value as an applied science for bridging the gap between research and practice.

Greenhalgh, Robert, Macfarlane, Bate, and Kyriakidou’s (2004) first recommendation for the next generation of diffusion of health service innovations is that it be “theory driven.” Kirk et al. (2016) also insists on the importance of theoretical frameworks for implementation research. If focused only on developing and testing theories, implementation research might miss many of the opportunities for achieving the critical purposes for which it is intended, bridging the gap between research and practice. Alternatively, a more applied research emphasis (Livingood et al., 2011), focused on adapting an existing array of scientific theories, principles and technologies to local problems, might have more potential for achieving broad adoption, diffusion, and adaptation of proven evidence practices. This study illustrates an approach to and the value of such an applied research focus.

Holding implementation research to the tenants of intervention research (Wilson et al., 2017), with research designs intended to prove a clear cause-and-effect relationship, also undermines the goals of implementation research. Why have a separate nomenclature for implementation research if it is essentially the same as intervention research? If evidence already exists (the cause and effect relationship established), would the purpose of implementation research primarily be to conduct research in other situations with the intent to have more generalizable results (Brown et al., 2017) or external validity? It is not at all clear if this approach would overcome the challenges that implementation research is intended to overcome or simply create more challenges (seeking universal application and acceptance). The IOM report on QI and implementation research (IOM, 2007) also tended to assess the relationship of the two terms within the context of definitions of research associated with intervention research or more pure research, without discussion of its value as an applied science, similar to engineering.

This case study illustrates the use of QI/evaluation model of implementation research, employing both QI and evaluation methods (Woodhouse et al., 2013). Merging evaluation with QI captures the applied engineering methods of QI and the research methods associated with evaluation to provide a model for implementation research. In particular, mixed-method evaluation research includes statistically assessed quantitative data and the more in-depth insights associated with qualitative methods (Palinkas et al., 2011; Plano Clark, 2010). Additionally, evaluation methods that focus on monitoring process and outcomes and on lessons learned adds research dimensions to QI models, complementing Rabin et al.’s (2008) implementation research emphasis on developing “understanding.”

This case study application of QI and evaluation illustrated the very different challenges that each organization (CHD) faced in implementing a texting STI notification protocol. In addition to the different characteristics of the CHDs and the populations they serve, extensive variations in texting enrollment were influenced by same day treatments, different mobile carriers, client access to mobile technology, apparent differences in experience with and commitment to QI, different starting points for each CHD (readiness), staffing changes (loss of staff), number and types of clinics where texting was implemented, increased availability and use of CHD EHR patient portal.

Although not the direct focus of the organization and staff behavior, the public health outcome of time to treatment was an important evaluation measure. This study was not intended to be a conclusive intervention study that would typically require an experimental (randomized control) design. However, a primary concern in deciding the focus of this intervention was the potential impact on public health outcomes from practitioners’ perspective (Livingood et al., 2018). Therefore, monitoring the outcome was critical to clarify that the intervention did not result in negative consequences. The fact that the improvement in outcomes was statistically significant was also important for continued expansion of these implementation efforts by the practice communities and further development of policy by CDC and FDOH. Notwithstanding the value of these outcome results, monitoring outcomes for QI implementation should not be interpreted as a mandate for intervention research design, which would substantially impede adoption by practice communities.

This applied approach (Livingood et al., 2011) to implementation research may be particularly useful from a practical point of view. Health promotion and public health agencies continue to face a growing STI epidemic (CDC, 2017). This applied emphasis is particularly well suited for this kind of real-life research interest of practice communities, including practice-based research networks. It is also very compatible with recently emerging public health accreditation standards which heavily emphasize QI (Honoré et al., 2011; Ingram, Bender, Wilcox, & Kronstadt, 2014). In addition to this applied approach having value to implementation research, it may also be very applicable to overcoming many of the challenges of translational research, particularly the later phases of T3 and T4 where implementation is a critical issue (Trochim, Kane, Graham, & Pincus, 2011).

Limitations

A case study approach, by its nature has limited generalizability, and that is true for this study. The more “in-depth, multi-faceted explorations of complex issues in their real-life settings” (Crowe et al., 2011), associated with case studies, was the purpose of this study. Other study designs that were previously discussed have advantages in testing theory or causality, and a more intervention study design could particularly be applied to texting STI results to determine if the texting results has a more universal impact on early treatment of STIs. While this would provide stronger evidence for broad dissemination of texting results, it would not address the unique application issues encountered by each organization, which a QI/evaluation approach does provide, as illustrated here. Practitioners in particular may be much more interested in an approach emphasizing unique application rather than generalizability to other settings.

Conclusions

This study discusses and illustrates the need for a very adaptive approach to implementing emerging evidence and technologies in very different and diverse health promotion organizations, particularly health promotion organizations and public health agencies. Implementation research may be a critical strategy for bridging the gap between research and practice including T3 and T4 translational research, as well as a mechanism to overcome health disparities and the quality chasm. However, the role of implementation research may need to be clarified vis-a-vis an applied science that is intended to adapt established principles, evidence and technologies to unique settings, similar to engineering, versus simply replicating intervention research in other settings or limiting implementation research to a role of developing and testing theory that is more associated with classically defined purposes of research.

Footnotes

Authors’ Note:

This implementation research would not have been possible without the collaboration and support of both the state and county agencies. In particular, the authors wish to acknowledge the Florida Department of Health for its instrumental leadership and support for sexually transmitted infection services innovations and its commitment to quality improvement processes at both the state and county level. The authors also wish to acknowledge the participating County Health Departments for their support for implementing texting and quality improvement processes to improve texting adoption and enrollment. This project was substantially supported by the Robert Wood Johnson Foundation. It was conducted in collaboration with the Florida Public Health Practice Based Research Network, the Florida Department of Health and Florida county health departments.

ORCID iD

William C. Livingood

References

Bilello

Livingood

Lukens-Bull

Smotherman

Choe

(2019). Texting test results reduces the time to treatment for sexually transmitted infections. Journal of Public Health Management and Practice, 25, 165-170.

Brown

C. H.

Curran

Palinkas

L. A.

Aarons

G. A.

Wells

K. B.

Jones

. . . Cruden

(2017). An overview of research and evaluation designs for dissemination and implementation. Annual Review of Public Health, 38, 1-22.

Centers for Disease Control and Prevention. (2017). STDs at record high, indicating urgent need for prevention. Retrieved from https://www.cdc.gov/media/releases/2017/p0926-std-prevention.html

Crowe

Cresswell

Robertson

Huby

Avery

Sheikh

(2011). The case study approach. BMC Medical Research Methodology, 11, 100. doi:10.1186/1471-2288-11-100

Damschroder

L. J.

Aron

D. C.

Keith

R. E.

Kirsh

S. R.

Alexander

J. A.

Lowery

J. C.

(2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4, 50. doi:10.1186/1748-5908-4-50

Davis

M. V.

Mahanna

Joly

Zelek

Riley

Verma

Fisher

J. S.

(2014). Creating quality improvement culture in public health agencies. American Journal of Public Health, 104, e98-e104.

Deming

W. E.

(2000). Out of the crisis. Cambridge: MIT Press.

Greenhalgh

Robert

Macfarlane

Bate

Kyriakidou

(2004). Diffusion of innovations in service organizations: Systematic review and recommendations. Milbank Quarterly, 82, 581-629.

Honoré

P. A.

Wright

Berwick

D. M.

Clancy

C. M.

Lee

Nowinski

(2011). Creating a framework for getting quality into the public health system. Health Affairs, 30, 737-745.

10.

Ingram

R. C.

Bender

Wilcox

Kronstadt

J. A.

(2014). Consensus-based approach to National Public Health Accreditation. Journal of Public Health Management and Practice, 20, 9-13.

11.

Institute of Medicine. (2001). Crossing the quality chasm: A new health system for the twenty-first century. Washington, DC: National Academies Press.

12.

Institute of Medicine. (2003). Unequal treatment: Confronting racial and ethnic disparities in health care. Washington, DC: National Academies Press.

13.

Institute of Medicine. (2007). The state of quality improvement and implementation research: Expert views: Workshop summary. Washington, DC: National Academies Press.

14.

Kirk

M. A.

Kelley

Yankey

Birken

S. A.

Abadie

Damschroder

(2016). A systematic review of the use of the Consolidated Framework for Implementation Research. Implement Science, 11, 72. doi:10.1186/s13012-016-0437-z

15.

Krieger

Rowley

D. L.

Herman

A. A.

Avery

Phillips

M. T.

(1993). Racism, sexism, and social class: Implications for studies of health, disease, and well-being. American Journal of Preventive Medicine, 9(6 Suppl.), 82-122.

16.

Livingood

W. C.

Allegrante

J. P.

Airhihenbuwa

C. O.

Clark

N. M.

Windsor

R. C.

Zimmerman

M. A.

Green

L. W.

(2011). Applied social and behavioral science to address complex health problems. American Journal of Preventive Medicine, 41, 525-531.

17.

Livingood

W. C.

Bilello

L. A.

Choe

Lukens-Bull

(2018). Enhancing the science of discovery in public health systems and services research through participatory research methods. Population Health Management, 21, 155-162.

18.

Livingood

W. C.

Bilello

L. A.

Lukens-Bull

(2019). Seeing the quality improvement forest through the quality improvement trees: A meta-synthesis of case studies in Florida and Georgia. Sage Open Medicine, 7. doi:10.1177/2050312119845703

19.

Livingood

W. C.

Marshall

Peden

Gonzalez

Shah

G. H.

Alexander

. . . Woodhouse

(2012). Health districts as quality improvement collaboratives and multijurisdictional entities. Journal of Public Health Management and Practice, 18, 561-570.

20.

Livingood

W. C.

Sabbagh

Spitzfaden

Hicks

Wells

Puigdomenech

. . . Wood

D. L.

(2013). A quality improvement evaluation case study: Impact on public health outcomes and agency culture. American Journal of Preventive Medicine, 44, 445-452.

21.

Livingood

W. C.

Peden

A. H.

Shah

G. H.

Marshall

N. A.

Gonzalez

K. M.

Toal

R. B.

. . . Woodhouse

L. D.

(2015). Comparison of practice based research network based quality improvement technical assistance and evaluation to other ongoing quality improvement efforts for changes in agency culture. BMC Health Services Research, 15, 300.

22.

Meslin

E. B.

Blasimme

Cambon-Thomsen

(2013). Mapping the translational policy “valley of death.” Clinical and Translational Medicine, 2(1), 1-8.

23.

Nilsen

(2015). Making sense of implementation theories, models and frameworks. Implement Science, 10, 53.

24.

Nilsen

Neher

Ellström

P. E.

Gardner

(2017). Implementation of evidence-based practice from a learning perspective. Worldviews Evidenced Based Nursing, 14, 192-199.

25.

Palinkas

L. A.

Aarons

G. A.

Horwitz

Chamberlain

Hurlburt

Landsverk

(2011). Mixed method designs in implementation research. Administration and Policy in Mental Health, 38, 44-53.

26.

Plano Clark

V. L

. (2010). The adoption and practice of mixed methods: U.S. trends in federally funded health-related research. Qualitative Inquiry, 16, 428-440.

27.

Rabin

B. A.

Brownson

R. C.

Haire-Joshu

Kreuter

M. W.

Weaver

N. L.

(2008). A glossary for dissemination and implementation research in health. Journal of Public Health Management and Practice, 14, 117-123.

28.

Ravasi

Schultz

(2006). Responding to organizational identity threats: Exploring the role of organizational culture. Academy of Management Journal, 49, 433-458.

29.

Riley

W. T.

(2017). Behavioral and social sciences at the National Institutes of Health: Adoption of research findings in health research and practice as a scientific priority. Translational Behavioral Medicine, 7, 380-384.

30.

Sales

Smith

Curran

Kochevar

(2006). Models, strategies, and tools: Theory in implementing evidence-based findings into health care practice. Journal of General Internal Medicine, 21(Suppl. 2), S43-S49.

31.

Schein

E. H.

Schein

(2017). Organizational culture and leadership (5th ed.). Hoboken, NJ: John Wiley.

32.

Shiver

S. A.

Schmitt

Cooksey

(2009). Use of a business approach to improve disease surveillance data management systems and information technology process in Florida’s Bureau of STD Prevention and Control. Public Health Reports, 124(Suppl. 2), 98-102.

33.

Tabak

R. G.

Khoong

E. C.

Chambers

D. A.

Brownson

R. C.

(2012). Bridging research and practice: Models for dissemination and implementation research. American Journal of Preventive Medicine, 43, 337-350.

34.

Trochim

Kane

Graham

M. J.

Pincus

H. A.

(2011). Evaluating translational research: A process marker model. Clinical and Translational Science, 4, 153-162.

35.

Westfall

J. M.

Mold

Fagnan

(2007). Practice-based research: “Blue Highways” on the NIH roadmap. Journal of the American Medical Association, 297, 403-406.

36.

Wilson

P. M.

Sales

Wensing

Aarons

G. A.

Flottorp

Glidewell

. . . Straus

(2017). Enhancing the reporting of implementation research. Implement Science, 12, 13. doi:10.1186/s13012-017-0546-3

37.

Woolf

S. H.

(2008). The meaning of translational research and why it matters. Journal of the American Medical Association, 299, 274-286.

38.

Woodhouse

L. D.

Toal

Nguyen

Keene

Gunn

Kellum

. . . Livingood

W. C.

(2013). A merged model of quality improvement and evaluation: Maximizing return on investment. Health Promotion Practice, 14, 885-892.