Examining Complex Program Adoption Decision-Making in a Health Pilot for U.S. Women Military Veterans: A Single Revelatory Case Study

Abstract

A single retrospective, qualitative case study was conducted to examine the decision-making factors that impacted the adoption of a women-specific health transition pilot as a permanent program. The pilot was adopted as a permanent program prior to completing all expected pilot sessions and before the final pilot evaluation report was produced. This study examined factors affecting the program adoption decision at the societal, organizational, interpersonal, and individual levels. To examine this research question, the socioecological model was used as a theoretical framework to identify and understand the decision-making factors and their interactions at the various socio-ecological levels. Semi-structured interviews were conducted with 15 Veterans Health Administration (VHA) Transition Training pilot stakeholders. The study identified 12 predominant factors that facilitated the pilot’s adoption as a permanent program. The paper discusses implications and considerations for practitioners for making evidence-based decisions in highly complex contexts.

Plain language summary

A case study was conducted to examine the decision-making factors that impacted the adoption of a women’s-specific health program for transitioning service members as a permanent program. The pilot was adopted as a permanent program prior to completion of all expected pilot sessions and before the final pilot evaluation report was produced. This study examined the decision-making factors that affected the program adoption decision at levels of the socioecological model, including societal, organizational, interpersonal, and individual levels. Interviews were conducted with pilot stakeholders to understand the decision-making factors. The study identified 12 predominant factors (and 24 overall factors) that facilitated the pilot being adopted as a permanent program. The paper discusses implications and considerations for practitioners for making evidence-based decisions in highly complex contexts.

Keywords

change management leadership evidence-based decision-making women veterans case study

Introduction

As evidenced by recent Government Accountability Office (GAO) efforts, U.S. government agencies face mounting pressure to eliminate duplication, fragmentation, and overlap and to identify cost saving and revenue generating opportunities (GAO, 2018). As such, government agencies and programs face significant pressure to provide evidence justifying their programs’ existence. However, questions remain: for instance, which factors and individuals determine if a program is effective? How are decisions made to initiate, adopt, scale, or eliminate programs?

Significant decisions are never straightforward and, unlike a simple math equation, evidence-based decision making must balance science and art (Brownson, Chriqui, & Stamatakis, 2009). Evidence-based decision making is achieved by combining scientific evidence and consideration of “values, resources, and context” (Brownson et al., 2018, p. 30). The definition of enough evidence is subjective and determined by the parties involved in designing, implementing, and making decisions about the intervention (Brownson et al., 2018).

The Veterans Health Administration (VHA) Women’s Health Transition Training pilot provided a means to explore the complexity of evidence-based decision making. Research has shown that the transition from military to civilian life is complex and challenging. The reintegration process involves many changes as an individual moves through a transition that involves changing structures, geographic location, career, family roles, support systems, social networks, community, culture, and life roles (Castro et al., 2014; Stein-McCormick, 2013).

When service members leave the military, they also leave an institution of distinct rules that is culturally different from the civilian sector (Cooper et al., 2018). The success of the transition to civilian life is impacted by biological, psychological, and social factors at home and in the community (Strong et al., 2018). Studies have shown that women and men have different experiences during and after the transition from military to civilian life. A 2011 study found that male Veterans were more likely to report an easy adjustment to civilian life compared to female Veterans (Thompson et al., 2011).

Factors that can impact a former servicewoman’s reintegration into the civilian world include the availability of women-specific VA policies and services; access to employment resources and higher education; access to mental health resources; and social stigmas related to being a female Veteran (Strong et al., 2018). Although VA has made a concerted effort in the past decade to increase and improve women-specific health services, only 34% of eligible women use VHA for their health needs, compared to 43% of eligible men (VA/DoD, 2017).

To ease the transition from military to civilian life for women Veterans, VHA Women’s Health Services and the Department of the Air Force Women’s Initiative Team designed the VHA Women’s Health Transition Training pilot to increase knowledge of VHA benefits for servicewomen as they transition from the military to civilian life. The half-day curriculum includes an overview of VHA, a discussion of available women’s health services, a visit to a Department of Veterans Affairs (VA) medical center, and a briefing on transition assistance benefits and enrollment information (Department of Veterans Affairs, 2015; Department of Veterans Affairs/Department of Defense, 2017; Department of Veterans Affairs National Center for Veterans Analysis and Statistics, 2016; Department of Veterans Affairs National Center for Veterans Analysis and Statistics, 2017; Department of Veterans Affairs Office of Mental Health and Suicide Prevention, 2018; Department of Veterans Affairs Office of Mental Health and Suicide Prevention, 2020).

In October 2017, VHA began piloting the training at 11 U.S. Air Force bases. In June 2019, the training program became permanent and was implemented worldwide across all military branches, before the completion of all expected pilot sessions and before the final pilot evaluation report was produced. This study aimed to understand the decision-making factors that resulted in adoption of the VHA women’s health transition program as a permanent program.

Theoretical Framework

Most existing models to understand important factors in decision making are reductive and focus on single levels (e.g., individual and cognitive factors). Turner et al. (2017) systematic scoping review on evidence use in innovation adoption decision making, noted a dearth of literature on using multi-level models to understand the use of evidence in decision making. Turner et al. (2017, p. 8) stated that “much of the existing literature on evidence use in decision making on innovation has focused on processes at a particular level or not been explicit about the need to study processes at different levels.”

In response to this finding, Turner et al. (2017) created a conceptual framework to demonstrate the interactions between evidence use and processes at different contextual levels, including professional, organizational, and local system levels. Although the Turner et al. (2017) model improved on existing decision-making factor models, the socioecological model provided a more comprehensive theoretical framework to understand decision making from four levels in the context of this single case study, not just the three included in the Turner et al. (2017) conceptual framework.

Our hypothesis, given our extensive review of the decision-making literature, is that many factors at several levels influenced the decision to adopt the pilot as a permanent program. The socioecological model was used as a theoretical framework to articulate and recognize the likely existence and interaction of many factors that led to the Women’s Health Transition Training becoming a permanent program (Agency for Toxic Substances and Disease Registry [ATSDR], 2015). The socio-ecological model considers the relationships and interplay of factors at the individual, interpersonal, organizational, and societal levels. For decades, this model has been used to describe the complex interplay of factors involved in public health concerns and associated interventions. The model has also been used in decision-making research in a health context (Qiao et al., 2015) but is a relatively novel framework for understanding factors impacting decision making.

An extensive literature review was completed to identify a priori factors at all levels of the socio-ecological model. Program adoption, implementation science, evidence-based decision making, and change readiness emerged as the most relevant search terms; the associated concepts from these search results most widely informed the conceptual framework that guided the study design and execution (Figure 1).

Figure 1.

Factors identified in the literature to support program adoption decision making.

In this single case study, the decision to adopt the VHA Women’s Health Transition Training pilot was made several months before the expected completion of the pilot and final outcome evaluation. This quality improvement study, designed in coordination with Women’s Health Transition Training stakeholders from VA and the Department of the Air Force, explored what decision-making factors impacted the program adoption decision at all levels of the socioecological model—individual, interpersonal, organizational, and societal.

Methods

The VHA Women’s Health Transition Training pilot offered a unique opportunity to examine program adoption decision making retrospectively. Although program adoption decision making is routine in the public sector, this was the first study of its kind to explore and understand the decision-making factors that contributed to the adoption of the Women’s Health Transition Training as a permanent program. These study findings are illustrative of public sector challenges and the findings are transferable to understanding the larger challenge of public sector program decision-making.

Per Yin (2009), the pilot was a candidate for a single case study because “how” or “why” questions were posed. This single case study was considered revelatory because the pilot provided an opportunity to observe a phenomenon that has not been previously observed and studied (Yin, 2009, pp. 47–48), particularly in the Air Force and VA. This quality improvement study used three qualitative methods to examine three research questions. This article focuses only on the decision-making factors research question: What decision-making factors affected the program adoption decision at the societal, organizational, interpersonal, and individual levels?

Participants and Data Collection

From October through December 2019, we conducted semi-structured interviews with 15 VHA Women’s Health Transition Training pilot stakeholders. Interviews were used to understand process and context that could not be gleaned from documents, as advised in the qualitative research and methods literature (Patton, 2015). Interviewees were classified into three groups:

•Implementer: responsible for developing the pilot curriculum and program evaluation;

•Influencer: stakeholders external to VA and the Department of Defense (DoD) who influenced the pilot implementation decision-making process, including federal advisory committees; veteran service organizations (VSOs), or congressional entities; and

•Decision maker: an individual from one of the primary interagency committees who made decisions related to pilot progress.

We recruited interviewees with the assistance of VHA’s Office of Women’s Health Services practice partners. All pilot stakeholder groups involved in the decision-making process were invited to interviews to ensure representativeness of study participants. Sixty-five percent of eligible interviewees participated in telephone interviews, with interviews averaging 56 min. We conducted semi-structured interviews to explore the:

•Participant’s role in the decision-making process;

•Participant’s perception of how the decision was made; and

•Factors that impacted the decision-making process.

We digitally recorded and transcribed the interviews using the free service provided by Otter.ai, Inc., and the principal investigator validated transcript content accuracy.

Data Analysis

We used inductive and deductive approaches (hybrid analysis approach) to identify a priori decision-making factors and allow for emergence of novel factors (Padgett, 2012). Both hand coding and MAXQDA coding processes were employed to analyze interview data (VERBI Software, 2019). We initially coded six interviews in MAXQDA using the a priori codebook, developed from decision-making factors identified during an extensive literature review.

To further refine the codebook and properly account for emerging codes, all 15 interviews were hand coded in Microsoft Word. The new codes identified in the hand coding process were compiled into a revised Excel codebook. While defining and identifying definitions and examples for the codes, the codebook was further refined and some a priori and emergent codes were collapsed into similar codes. A second coder independently coded 20% of the interview transcripts to ensure interrater reliability and construct validity. We resolved any code discrepancies using the consensus method to determine the appropriate code assignment.

We used several MAXQDA visualization tools to analyze the data within and between the interviews, including code frequencies (by both number of coded segments and interviewees cited); code prevalence across interviewees; and code co-occurrences (e.g., relationships between multiple decision-making factor codes). We also examined codes by stakeholder group (e.g., decision maker); pilot phase (e.g., Pilot Exploration & Approval, Pilot Implementation, Pilot Adoption Decision); and levels of the socio-ecological model (e.g., organizational) to understand the potential variances in factors at different times with different stakeholders and at different levels of the socioecological model.

The process to stratify and understand the strength of evidence of each decision-making factor involved many components. The aforementioned data views and dimensions used in MAXQDA, combined with the principal investigator’s detailed knowledge of the interview data, served as the basis for understanding the strength of evidence for each factor. Whether factors were predominant was primarily determined by code frequencies, secondarily by the number of phases in which a factor occurred and, finally, by the context in which the factor was mentioned.

Results

Sample Characteristics

The findings below are based on interviews with 15 pilot stakeholders. Four interviews were with decision makers, seven were with influencers, and four were with implementers.

Qualitative Analysis Findings

A thematic analysis of the interview transcripts yielded 12 predominant factors facilitating the adoption of the pilot as a permanent program. Appendix A displays these factors and their associated definitions.

Figure 2 illustrates that various factors at different levels of the socioecological model impacted the decision to adopt the pilot as a permanent program. Often, the same factors were present at multiple levels of the socioecological model (e.g., understanding of the need). The analysis also revealed that different factors mattered during different pilot phases (i.e., Pilot Exploration & Approval, Pilot Implementation, and Pilot Adoption Decision). In addition to the factors identified at the socioecological model levels, interviewees asserted that a combination of several factors of varying importance impacted the decision to adopt the pilot as a permanent program.

Figure 2.

Most predominant observed decision-making factors by socio-ecological model level.

Most Predominant Decision-Making Factors

Twelve factors were consistently cited by interviewees and deemed to be the most predominant factors in the decision (see predominant factor determination method in data analysis section). The 12 most predominant factors are detailed and supported with data in the following section; predominant factor definitions are included in Appendix A.

Understanding of the Need

Understanding of the need is the extent to which an individual/group/organization is aware an issue exists and values it as a problem. Understanding of the need, cited by all interviewees (n = 15), was the primary antecedent to pilot development in that stakeholders recognized a problem existed and, more importantly, believed and valued this as a problem that required action. And it is not a single issue; rather, it is a complex web of issues including the transition process, women veterans, suicide, homelessness, employment, and VHA utilization that served as the basis for pilot development. Interviewees shared that, because the pilot was framed as a solution to numerous complex issues, many different stakeholders were supportive based on their stake.

For example, the interagency transition stakeholders were supportive because of their oversight in the military to veteran transition space. The Health Executive Committee Women’s Health Working Group was interested from a women’s health perspective, while senior VA leaders were focused on the suicide prevention aspect of the pilot. One leader shared about the appeal of the pilot on the grounds of suicide prevention:

… there is data and research to suggest that if you can get women into VA mental health care there is a protective effect against suicide. So anything that gets women into the VA at higher rates … and if this [program] can get them into treatment if they need it, I think … why wouldn’t you do it?

Pilot Data

The pilot data construct refers to qualitative and quantitative data as indicators of pilot success. Convincing pilot data was cited at a high frequency (n = 14) and was also noted as one of the most important factors in the decision to adopt the pilot as a permanent program. Pilot data was cited as being important to individuals, groups, DoD and VA, and to outside influencer organizations.

Pilot data came in many forms, including personal stories; quantitative and qualitative data presented through graphical displays; personal testimonial videos; and toward the end of the pilot period, actual increases in VHA enrollment of transitioning servicewomen because of pilot attendance. VHA Women’s Health Transition Training pilot decision makers and influencers viewed the pilot data as compelling and promising enough to endorse the pilot on these grounds. One interviewee spoke about the evolution of personal stories into pilot data that helped to influence senior leader support across the three pilot phases. Personal stories were used in the Pilot Exploration and Approval phase to demonstrate the need for the pilot. As the pilot was implemented and data collected, the emphasis shifted from personal stories to pilot data. One interviewee stated: “… when we gave the preliminary data, I think that helped our senior leaders go okay, not only have we heard the stories, but now we see the data to support them.”

In addition to qualitative and quantitative pilot evaluation data, the implementation team leveraged participant testimonial videos to highlight the pilot’s positive impact. One influencer shared the impact of these testimonials, in coordination with preliminary pilot outcome data, on the decision-making process:

I think a big part of that was how it was briefed initially, where … they did a video with interviews of servicewomen who had gone through … this program and said basically, if it hadn’t been for this program, I would never have known that the VA was even an option for me…. It’s really showing how much of a need there is for women-specific education about the VA that really set the expectation for leadership to say, yeah, this is something we really, truly need. And then when they were able to document their outcomes and show these women talking about how this affected them and how now they’re enrolled in care and that they have all these wonderful glowing things to say about it, I think that really struck a chord, with leadership saying, okay, well, this is something we can do that obviously makes a difference.

Senior Leader Support

Senior leader support of the pilot was cited by most interviewees (n = 14) as a key factor in the decision to make the pilot permanent. Interviewees expressed that the pilot would not have progressed beyond exploration without senior leader support. Senior leaders were often the decision makers, making their support a necessary step in pilot progression.

Senior leader support was secured early in the process and the positions and roles of senior leaders facilitated the pilot being moved from a concept to wide-scale implementation faster than most usual implementations. Additionally, senior leader support helped overcome early resistance from other stakeholders opposed to the pilot. Interviewees spoke of the imperativeness of gaining and maintaining senior leader support in progressing the pilot:

And so we were just getting some resistance … but we knew we had support from leadership at the top level. So … we needed to get this in front of senior leaders to formally say yes and in front of everybody, so that we could continue to move on…. And you know, VA and DoD leadership were beyond supportive. … we couldn’t have asked for anything better.

Interviewees relayed that senior leaders and decision makers are subject to both internal and external pressures when prioritizing decisions. Often the “loudest” issues—those on the media’s and Congress’s radar—become those priorities. The highly complex array of issues addressed by the pilot fit those criteria, as noted by an interviewee:

When you have a bazillion competing challenges—when you’re worried about electronic health records and you’re worried about suicide and you’re worried about the backlog of claims and you’re worried about, you know, 16,000 things all at the same time and running an organization with 360,000 employees, balancing priorities is really difficult. And certainly the priorities that are … making the news clips every day and that you’re having to answer congressional letters about—it’s a little easier to stay focused on those.

Visibility

The visibility construct, cited by 13 interviewees, is comprised of the general visibility of women veteran and military transition issues and of the pilot in the media. Visibility of women veterans’ issues contributed to the understanding of the need and sense of urgency to act. In addition, interviewees stated that visibility of women veterans’ issues was aided by a number of factors, including VSOs, the media, and women veterans being elected to Congress and advocating for other women veterans:

… the fact that women veterans started getting a lot of attention in the public eye had to play into it. I’ve been working on women veterans’ issues for quite some time and in the past few years they’ve gotten a lot more attention than they used to. There have been a lot of headlines about women veterans in terms of the suicide rate, women veterans and homelessness, women veterans and mental health challenges, sexual assault of women while they’re in the military, the number of women veterans running for Congress, number of veterans actually getting elected to Congress. Just awareness of women, military women, and women veterans has gone up significantly. And the level of interest being paid to these issues by members of Congress, by the press, by veteran service organizations…. I think all of that comes into play.

Additionally, the importance of pilot publicity and visibility was noted by multiple interviewees. An article released on August 7, 2018 (Taylor, 2018), raised the pilot’s visibility and led to additional interviews and articles with other media outlets later in the implementation stage. The articles, in addition to advertising to prospective pilot participants, were reviewed by Congress, federal advisory committees, and VSOs. As shared by an interviewee, the media articles helped gain public endorsements from a federal advisory committee and VSO:

I think one key thing that happened was that we got an article in an Air Force publication about the training that generated a lot of interest, not just from servicewomen, but … [from] stakeholders, like Congress and advocacy groups.

Collaboration

The interagency collaboration around women’s health demonstrated by VHA Women’s Health Services and the Department of the Air Force Women’s Initiative Team was cited by 12 interviewees as facilitating the pilot’s adoption as a permanent program. The collaboration was noted by interviewees as unique, as interviewees shared that interagency efforts are rare, particularly efforts formed organically, as in the case of this pilot. According to one interviewee:

I think it was the first of its kind for that kind of VA, DoD interaction around women’s health issues…. I don’t know, maybe it’s happened in other areas, but for women’s health, I think we hadn’t had those cross-agency discussions to that same extent.

Organizational Characteristics

Organizational characteristics include many embedded components, including organizational structure and culture, values, size, and resources. Many interviewees (n = 12) mentioned at least one facet of organizational characteristics as a facilitator for the pilot’s adoption as a permanent program, such as how both departments established permanent offices and structures to promote women’s issues in the respective organizations. This organizational structure, as described by an interviewee, exemplifies the organizations’ commitment to women and serves as recognition of the need for women-specific services:

… we finally have women’s offices in these organizations … as more and more women have joined the military and our numbers have increased exponentially. I think that they’re finally … actually doing things on behalf of servicewomen and women veterans that are needed.

Interviewees noted changes in military culture that have resulted from an influx of women. One interviewee discussed changes in military culture that have occurred over the past few decades:

… I think that our senior leaders recognize the importance of having women in the service…. It’s not just about … having women for the sake of having women. It’s the readiness and security of the nation issue…. I think they take very seriously the policies and the processes and how they impact the women that serve in the culture that women are serving in. … it’s a very male dominant organization, the military at large. … but I think that the culture is changing.

Readiness for Change

Readiness for change is a multi-faceted and complex construct that refers to organizational members’ resolve to implement a change and a shared belief in their collective capability to do so (Bouckenooghe et al., 2009; Castaneda et al., 2012; Holt & Vardaman, 2013; Satterfield et al., 2009). Readiness for change, cited by 12 interviewees, is differentiated from organizational characteristics in that it is a fluid posture and mindset, whereas organizational characteristics are largely related to the organizational infrastructure. However, both constructs are inextricably linked, as organizational readiness for change impacts organizational characteristics like values, organizational structure, and resource allocation.

The implementation team and original champions and supporters of the pilot exhibited certain dimensions of readiness for change (e.g., collective commitment) but were operating in a broader political environment that had different concerns and considerations (e.g., climate of change). Thus, readiness for change was not uniform across stakeholder groups and motivations for change varied by individuals and stakeholder groups.

For example, some stakeholder groups, particularly senior leaders, were supportive of the change due to their understanding of the need and proximity to political pressure. Other stakeholder groups, particularly groups whose programs were directly adjacent to the pilot, had different concerns about how the pilot would impact the implementation of their programs. These factors and vantage points affected willingness and readiness to embrace change. Ultimately, any resistance to change within the organization was usurped by leadership and decision-maker support by virtue of the pilot being adopted as a permanent program.

Readiness for change is impacted by both internal (organizational) and external (societal) forces. One interviewee spoke about the progressive military culture, driven primarily by internal forces, which has led the way societally for larger social movements, including racial integration, equal pay, and inclusion of women. The interviewee indicated that this innovative military culture, including the rotation of individuals into and through leadership positions, was a facilitator for the adoption of an innovative program:

The military historically has … led the way [when] it comes to inclusivity…. When you look at the integration of blacks and whites … it kind of happened here before it happened in society…. And so … I don’t think it was driven by Congress, the shifts that are happening. I think it was internal. I think Congress may have helped in some areas, but I think it’s more a desire to always be innovative and to recognize the changes necessary and that you need perspectives from everyone in order to be successful in the things that we have to do to keep the country safe.

Congressional Influence

Interviewees (n = 11) referenced Congressional influence, including specific House and Senate committees, the National Defense Authorization Act (NDAA), and House resolutions, as influencers in the pilot decision. Congressional representatives became involved in the pilot after attending a webinar on pilot progress and participating in subsequent meetings and briefings. In turn, these Congressional representatives demonstrated their support by including provisions for the pilot to be adopted by all military branches in base text in House Resolution (H.R.) 2500, the National Defense Authorization Act for Fiscal Year 2020, and two separate bills: H.R. 2941, Servicewomen’s Health Transition Training Act of 2019, and H.R. 2942, HEALTH Act.

An interviewee spoke about women veterans as an ever-increasing priority for Congress. The congressional infrastructure (e.g., task forces and caucuses) and current congressional representation are reflecting these priority shifts and the sense of urgency to act:

Two new veteran and servicewomen groups on the Hill have been formed. So there’s the Women Veterans Task Force and then there’s the Servicewomen and Women Veterans Caucus. … there are seven women veterans now elected members of Congress. So I think that … new lawmakers … along with … concerns about high suicide rates … coupled with sexual assault has, you know, created a degree of knowledge that people want to do something about.

Coordinated Implementation Plan

Interviewees (n = 11) cited a coordinated and organized implementation plan as a reason the pilot was supported. The coordinated implementation plan included how the pilot was delivered (e.g., the pilot curriculum was delivered by women veterans, not by men or women who had not served), the communication methods employed, and the stakeholders (internal and external) briefed on the pilot at different junctures.

Interviewees stated that an important component of the coordinated implementation plan was that it allowed the program to be steps ahead to seize opportunities when they arose. This included moving forward with planning the training before having any seed money to implement the pilot. Interviewees shared the importance of this preparation and how being in “the right place, at the right time, with the right plan” and prepared with a “shovel ready” program were facilitating factors. According to one interviewee:

I would say that sometimes within the Department of Defense and within the VA, things can sometimes move slowly…. This one happened fairly quickly, which I’m somewhat amazed at, but I think it all comes down, once again, to when you have a good program and a good team that’s working on it, and you use the process correctly to your benefit, then you’re able to get things through.

Priority Alignment

Interviewees (n = 10) cited how the pilot aligned to organizational priorities and strategies. The pilot was described as an initiative that touched on many of the hot button topics and priorities that concerned the Joint Executive Committee (JEC), the highest decision-making authority between VA and DoD. This priority alignment generated interest when the pilot was initially proposed as a JEC agenda item. One interviewee stated:

This … really affects a lot of different things that the JEC cares about. And it also really scratched an itch as far as suicide prevention efforts and transition, which are all really key priorities and have been for years for both departments.

Priority alignment was also noted as a factor in the influencer interviews. The pilot was aligned with the priorities of many entities, including VSOs, Congress, and federal advisory committees.

Passion and Persistence

Passion and persistence refer to the resolve and dedication of the pilot implementation team in seeing the pilot through from start to finish. Interviewees (n = 9) shared that the pilot was led by a group of individuals who believed in the program and did not give up on their vision, even amid adversity. One interviewee shared how this resolve was the single most important factor in getting the pilot in front of the highest-level decision-making entities:

I think top, top, first and foremost, it’s the passion of the people who have developed this program and seen it through to this point. It wouldn’t even have hit our radar at the JEC if it hadn’t been for … just sort of getting it in front of people.

Additionally, this follow through, driven by passion and persistence, was noted by an interviewee as one of the top two factors for why this pilot was successfully adopted:

Some pilot programs kind of die on the vine. But this one … started out strong and finished strong…. Everyone involved with it had, you know, passion and desire … and leadership making it work…. Those factors are why we’re here.

Sense of Urgency

Personal connections to the issue and an understanding of the need created a sense of urgency among individual implementers, influencers, and decision makers that something needed to be done—and soon. Interviewees (n = 9) noted that initially, the sense of urgency was largely at the individual, interpersonal, and organizational levels, but as outside entities were engaged, this sense of urgency was shared by societal influences:

When you look at the stats for women Vets on suicide or homelessness or health issues and concerns, I think everyone takes very seriously that the suicide rate for female veterans is alarming…. And so I think that if there’s anything that could help to reduce that or help to make sure that veterans at least know that resources are available if they find themselves in that dark place, then it’s something worth looking into and trying.

Combination of Factors

In addition to the individually named factors, most interviewees (n = 9) spoke about multiple factors occurring synergistically as the reason the pilot was adopted. Interviewees noted that this pilot occurred during an important time in history for women, both societally and within VA and DoD. In March 2017, the Marines United Scandal—which was centered around the distribution of nude pictures of servicewomen without their consent to 30,000 male service members in a closed Facebook group—thrust DoD’s treatment of servicewomen into the spotlight.

Outside the DoD, the #MeToo movement was in full swing when the pilot was initiated, bringing attention and justice to women survivors of sexual assault and harassment. Know Your Value, a program designed to stymy pay inequality and help women grow professionally, was cited as another manifestation of society reacting to injustices against women. Additionally, there were many books and documentaries cited that shined a light on women in the military.

“The Invisible War” was a harrowing 2012 documentary about sexual assault and rape in the military. Three books that share true stories of women in service were noted as impactful for a broader audience: Ashley’s War: The Untold Story of a Team of Women Soldiers on the Special Ops Battlefield; Fight Like a Girl: The Truth Behind How Female Marines Are Trained; and Shoot Like a Girl: One Woman’s Dramatic Fight in Afghanistan. Additionally, it was noted that there are record numbers of women veterans in Congress and in positions of leadership within DoD and VA and in VSOs, bringing a voice to a previously underrepresented demographic.

Passages were coded as a combination of factors when there were references to a perfect storm of multiple factors, like those noted above, that impacted the decision (e.g., pilot data, congressional attention, VSO involvement, broader societal forces). The fact that this construct emerged supports the assertion that there was no single influencing factor, but rather a confluence of complex factors at various levels of the socio-ecological model at different times in the implementation process that contributed to the context in which the decision was made. One interviewee stated:

… it’s when you have all these things going at the same time—when you have lawsuits being filed and films, documentary films, coming out, and women going and meeting with members of Congress—and it’s like all of these things happening together that can make change happen synergistically in a way that, if you took one or two of those prongs away, it would just fall over. It wouldn’t have the stability to keep going.

Response Trends by Stakeholder Type

Predominant factors were largely universally cited across stakeholder groups, but there were other factors cited more frequently in some groups than others. Additionally, we found the level of involvement with the pilot had an inverse relationship with the decision-making authority. Many high-level entities had limited interaction with the pilot (e.g., JEC), but had the ultimate decision-making authority to adopt the program. Conversely, individuals implementing the pilot were responsible for delivering the curriculum and evaluating the program but had no decision-making authority.

All or most decision makers cited the pilot data, senior leader support, uniqueness of the grassroots movement, priority alignment, and risk perception as reasons supporting the pilot’s adoption. Their vantage point was more from an organizational perspective of how effective the pilot is and where it fits into their organizational portfolio based on existing programs and the proposed pilot characteristics. And the decision-maker vantage point only represented the Pilot Implementation and Pilot Adoption Decision phases of the process, which also accounts for a different emphasis on facilitating factors.

Influencers, on the other hand, based on their position outside the departments and their involvement in the Pilot Implementation phase, were more likely to cite broader influences, such as the visibility of the issue and the role of influencer organizations (e.g., Congressional, VSOs or VA/DoD advisory bodies), in the decision-making process. The influencer groups, albeit important, were not involved with the daily implementation of the pilot, nor were they involved in the formal decision-making process. Consequently, they imparted a broader perspective of the context in which this decision was made.

The implementation team, with the closest proximity to the process, was more likely to note process-oriented factors that impacted the decision, such as garnering senior leader support (often achieved through personal connections); navigating bureaucracy; the role of the coordinated implementation plan in the initiation and implementation process; and the importance of interagency collaboration. Additionally, the implementation team was the only group of stakeholders that provided a full perspective of the pilot from pilot exploration to the adoption decision.

In a pilot like this with many stakeholders, it is important to understand the predominant factors that were more universally recognized across all stakeholder groups and those factors that played a more nuanced role in the decision-making process. This study found that the stakeholder interviewee type influenced the factors discussed, largely because of the level of involvement and vantage point of the interviewee. Whether a factor was predominant, moderate, or of limited evidence was not as important as understanding that all the factors worked synergistically, at different levels of the socio-ecological model, at different times, to influence the pilot’s adoption.

Discussion

This research question explored the factors that impacted the decision to adopt the pilot as a permanent program. The findings, as revealed in the semi-structured interviews, indicated that there were 24 total factors of varying importance that impacted the decision-making process. Factors were identified at all levels of the socio-ecological model (i.e., individual, interpersonal, organizational, societal) and were noted as being important during different phases of the pilot (i.e., pilot exploration and approval, pilot implementation, and pilot adoption decision). Although there were 12 factors deemed predominant based on a number of measurements (e.g., code frequency, number of phases impacted, and the nature of the discussion in which the factor was mentioned), it was clear that a combination of factors acting in coordination with each other ultimately impacted the decision to adopt the pilot as a permanent program. Of these twelve predominant factors, findings in this study confirmed what is known in the literature, yet this study highlights ways in which they manifest that have important implications for other decision-making contexts. These factors are discussed below.

Interpersonal Factors

Navigating bureaucracy, political savvy, personal connections, trust, and passion and persistence all connected into a conglomeration of interpersonal level facilitating factors. The emergence of these factors, in coordination with factors such as understanding of the need and empathy, highlights the importance of networks and assembling a strong implementation team that understands the bureaucratic process. March (1994), in his discussion of multiple-actor factors, noted that decisions can often be made based on who is paying attention, who is participating, and who is alert and persistent. Rhodes (1997) referred to similar concepts as a network. Networks have characteristics and rules predicated on an exchange between interacting organizations. These networks, as was the case with this pilot, occur and work in parallel with hierarchical processes (Rhodes, 1997). And as Greenhalgh et al. (2004) noted, these network forces can generate pressure for organizations to adopt an innovation.

Timing and Context

The perception, visibility, and broader societal influence factors that emerged all signal that timing and context mattered in this decision. The issues facing women veterans are very public, as evidenced by the number of articles, books, documentaries, and public scandals on the topic. Some interviewees noted that it would have been a very poor decision if VHA had not supported the pilot based on the ongoing women’s movements (both inside and outside DoD and VA). Yanovitzky (2002) demonstrated that public attention to an issue can impact legislative and policy actions. The pressure and immediacy of some issues can lead to the generation of immediate, short-term solutions to a problem; but Yanovitzky (2002) found that once media attention subsides, more long-term solutions are typically implemented.

Political Pressure

The study findings very clearly indicated that leaders’ decisions are impacted by external forces, including timing and context. Anderson and Jakobsen (2018, p. 13) generated a political pressure hypothesis that “the more pressure from political principals for adoption of an organizational innovation, the more likely an organization is to adopt the innovation.” But amidst the inevitable (and persistent) pressure, how can leaders focus on making evidence-based decisions that will be sustainable over time? And how can organizations recognize and acknowledge the role of political pressure and still make sound organizational decisions?

Especially in the case of this pilot, Congress got involved and there is a clear tension between the legislative and executive branches because of the resource dependency. Braun and Gilardi (2006, p. 310) spoke of the political pressure from a resource-dependency perspective, “…this can give rise to coercive pressure because the wishes of the political principals are directly or indirectly linked to the organization’s dependency on financial, legislative, or hierarchical resources from the political principals. Based on such dependencies, political principals can make it very costly (or very beneficial) to not adopt (or adopt) a specific organizational innovation.”

Evidence-Based Implementation Science Protocols

Study participants discussed that an explicit, evidence-based implementation science framework was not used for this implementation, although the program aligned with many of the Expert Recommendations for Implementing Change (ERIC) criteria. Future VA implementations may consider the use of evidence-based implementation science practices during all project phases.

Explicitly using an implementation science framework can help ensure program fidelity (Moir, 2018). Although an implementation science framework is not a panacea, it can provide a systematic way to think through complex systems before embarking on a new program intervention. Adhering to an implementation science framework also means building evaluation into the implementation design, which can further assist in evidence-based decision making.

As evidenced in this pilot, having a program evaluation and associated pilot data is necessary, but not sufficient, for decision making. Even with an evaluation in place, there was still a question about what was considered enough evidence. For future implementations, VA could benefit from engaging decision makers in the program evaluation design process to ensure all stakeholders understand what is being measured and that what is being measured is meaningful to decision making.

Public Value

Government, as a public enterprise, is concerned with public value—that is, using taxpayer money wisely to generate and implement effective interventions (Moore, 1997). There are many considerations for public value, including issues of risk and the ethics of prioritizing and potentially marginalizing some groups.

A key facet to ensuring public value is determining how to allocate scarce resources. Similar to what happened in this pilot, the literature identified that public sector resource allocation must respond to risks. And the greater the risk of not doing something, the more likely an intervention is to be allocated resources (Daniels, 2016). Tools for guiding resource allocation are limited, particularly because of their limits to consider the complexity of decision making (Daniels, 2016). Thus, training leaders to navigate complex resource allocation decisions is an opportunity to increase the likelihood of generating public value.

Multisector and Multipartner Engagement

The federal government is increasingly recognizing the importance of interagency collaboration to streamlining interventions and better using resources. Multisector, multiagency approaches have also been identified as a priority in public health. One of the central tenets of Public Health 3.0, a framework for public health practice, is to engage multiple sectors and community partners to generate collective impact (DeSalvo et al., 2017). Public sector entities are increasingly recognizing the importance of having many different partners with seats at the table. This is due in large part to the shift in focus to social drivers of health, including upstream factors that impact health, like housing, employment, education, and access to care.

As was stated in an article on the importance of public health partnerships, “Partnership is an essential ingredient of public health for tackling the key determinants of health; as a shared responsibility, to avoid overlap and duplication…. The multisectoral approach requiring action by all agencies and individuals which have an impact on health strengthens approaches to address health problems and inequalities” (Adetunji, 2013, p. 94). Forming these partnerships requires public health leaders to be innovative and creative in thinking about systems solutions to complex adaptive public health challenges.

This study found that there was no explicit process for initiating or implementing interagency initiatives. This created a certain level of ambiguity in the process and informal communication channels and personal relationships were used to navigate and expedite the process. This was due in large part to the program originating at the grassroots level, which is an aberration from typical program development. If grassroots initiatives continue to be encouraged, it may benefit VA and other public sector agencies to develop a framework for these types of initiatives and communicate the framework and processes to program organizers.

A conceptual framework for interagency collaboration, like that developed by Ward et al. (2018), can be used to understand the context and antecedents of the collaboration; the design of the collaboration; the rules and processes in place to govern the partnership; and finally, outcomes and evaluation to understand what was accomplished. This more defined process would also consider necessary stakeholders and their desired level of involvement. Following and defining a process for collaboration can ensure traceability and set a precedent for future interagency collaborations.

Conclusion

This study explored the decision-making factors that impacted the adoption of the VHA Women’s Health Transition Training as a permanent program. The study, in alignment with the study’s hypothesis, found that decision making is complex. In the first of its kind study examining program adoption decision making within an interagency pilot, we found that decision making was particularly complex in this grassroots initiative that targeted an underserved population that received a great deal of attention organizationally and societally.

This study confirms that context and timing matter. The decision to adopt this pilot as a permanent program demonstrates that implementations are impacted by the broader environment and societal influences. It is not sufficient to have a well-coordinated program—there has to be a match of the need, with a viable solution, at the right time.

This study demonstrated that there is room to better operationalize program implementation and decision-making processes, and in that, better designing programs with multilevel interventions and evaluations to make it so success can be more objectively identified and substantiated. There is an opportunity to make often implicit processes explicit to continue to increase the use of implementation science and evidence-based decision-making practices.

Decision making matters because public sector resources are scarce and there is an increased emphasis on reducing overlap and program duplication. Decision making is the most downstream leadership activity that reflects the state of organizations and leaders in those organizations. The art of decision making involves understanding complexity and evaluating multiple factors simultaneously. Even though not all issues will be alike, organizations can benefit from frameworks to make more standardized, equitable program implementation decisions. However, frameworks alone cannot replace the leadership artistry required of our 21st century public sector leaders in this increasingly connected and complex world. Systems and training will be necessary to support the public sector and their leaders.

Footnotes

Appendix

Appendix A.

Factors That Facilitated the Adoption of the Pilot as a Permanent Program.

Factor	Factor definition
Collaboration	Reference to the effectiveness of collaboration, including intra-agency or interagency.
Congressional influence	Reference to congressional influence.
Coordinated implementation plan	Reference to the pilot implementation team having a coordinated and organized implementation plan, including facets of how the pilot was delivered.
Organizational characteristics	Reference to organizational characteristics, including culture, values, size, structure, and risk tolerance.
Passion and persistence	Reference to the passion, persistence, and perseverance related to the pilot implementation team.
Pilot data	Reference to early signs of success or pilot data, including qualitative, quantitative, and anecdotal data. Reference to the strength of the pilot evidence.
Priority alignment	Reference to organizational priority alignment or alignment with strategy.
Readiness for change	Reference to organizational members’ resolve to implement a change and shared belief in their collective capability to do so.
Senior leader support	Reference to a senior leader specifically, by name, or references to senior leader support more generally.
Sense of urgency	Reference to the need to do something about the issue soon.
Understanding of the need	Reference to the extent to which the individual/group/organization is aware the issue exists and values it as a problem.
Visibility	Reference to the media, press, or publicity around this pilot or visibility of women veterans’ issues.

Acknowledgements

Thank you to Dr. Patricia Hayes for approving this study and to Dr. Nancy Maher and Major Alea Nadeem for their practice partnership. Thank you to my dissertation committee for their guidance and support—Dr. Elizabeth Jarpe-Ratner, Dr. Christina Welter, Dr. Karen Peters, Dr. Eve Pinsker, and Dr. Halima Ahmadi-Montecalvo.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Ethical Approval

The University of Illinois-Chicago’s Institutional Review Board deemed this non-human subjects research for the purpose of quality improvement.

ORCID iD

Kelly Sanders

Data Availability Statement

Data sharing not applicable to this article as no datasets were generated or analyzed during the current study.

References

Adetunji

O. J.

(2013). Partnership: An effective approach to public health. Journal of Natural Sciences Research, 3(7), 90–95.

Agency for Toxic Substances and Disease Registry (ATSDR). (2015). Models and frameworks for the practice of community engagement. Retrieved November 8, 2024, from https://www.atsdr.cdc.gov/communityengagement/pce_models.html

Anderson

S. C.

Jakobsen

M. L. F.

(2018). Political pressure, conformity pressure and performance information as drivers of public sector innovation adoption. International Public Management Journal, 21(2), 213–242.

Bouckenooghe

Devos

Van den Broeck

(2009). Organizational change questionnaire-climate of change, processes, and readiness: Development of a new instrument. The Journal of Psychology, 143(6), 559.

Braun

Gilardi

(2006). Taking ‘Galton’s problem’ seriously: Towards a theory of policy diffusion. Journal of Theoretical Politics, 18(3), 298–322.

Brownson

R. C.

Chriqui

J. F.

Stamatakis

K. A.

(2009). Understanding evidence-based public health policy. American Journal of Public Health, 99(9), 1576.

Brownson

R. C.

Fielding

J. E.

Green

L. W.

(2018). Building capacity for evidence-based public health: Reconciling the pulls of practice with the push of research. Annual Review of Public Health, 39, 27–53.

Castaneda

S. F.

Holscher

Mumman

M. K.

Salgado

Keir

K. B.

Foster-Fishman

P. G.

Talavera

G. A

(2012). Dimensions of community and organizational readiness for change. Progress in Community Health Partnerships: Research, Education, and Action, 6(2), 219–226.

Castro

C. A.

Kintzle

Hassan

(2014). The state of the American veteran: The Los Angeles county veterans study. USC School of Social Work Center for Innovation and Research on Veterans & Military Families.

10.

Cooper

Caddick

Godier

Cooper

Fossey

(2018). Transition from the military into civilian life: An exploration of cultural competence. Armed Forces & Society, 44(1), 156–177.

11.

Daniels

(2016). Resource allocation and priority setting. In Barrett

D. H.

Ortmann

L. H.

Dawson

Saenz

Reis

Bolan

(Eds.), Public health ethics: Cases spanning the globe (pp. 61–94). Springer.

12.

Department of Veterans Affairs. (2015). Study of barriers for women veterans to VA health care—Final report. Retrieved July 12, 2023, from https://www.womenshealth.va.gov/docs/Womens%20Health%20Services_Barriers%20to%20Care%20Final%20Report_April2015.pdf

13.

Department of Veterans Affairs/Department of Defense. (2017). United States Air Force and Veterans Affairs Women’s Transition Assistance Initiative—VA/DoD Joint Incentive Fund (JIF) Proposal Business Case [Unpublished manuscript].

14.

Department of Veterans Affairs National Center for Veterans Analysis and Statistics. (2016). Profile of women veterans: 2015. Retrieved July 12, 2023, from https://www.va.gov/vetdata/docs/specialreports/women_veterans_profile_12_22_2016.pdf

15.

Department of Veterans Affairs National Center for Veterans Analysis and Statistics. (2017). Women veterans report: The past, present, and future of women veterans. Retrieved July 12, 2023, from https://www.va.gov/vetdata/docs/specialreports/women_veterans_2015_final.pdf

16.

Department of Veterans Affairs Office of Mental Health and Suicide Prevention. (2018). Facts about suicide among women veterans: June 2018. Retrieved July 12, 2023, from https://www.mentalhealth.va.gov/suicide_prevention/docs/Final_Facts_About_Suicide_Among_Women_Veterans_508.pdf

17.

Department of Veterans Affairs Office of Mental Health and Suicide Prevention. (2020). 2020 National veteran suicide prevention annual report. Retrieved July 12, 2023, from https://www.mentalhealth.va.gov/docs/data-sheets/2020/2020-National-Veteran-Suicide-Prevention-Annual-Report-11-2020-508.pdf

18.

DeSalvo

K. B.

Wang

Y. C.

Harris

Auerbach

Koo

O’Carroll

(2017). Public health 3.0: A call to action for public health to meet challenges of the 21st century. Preventing Chronic Disease, 14, 170017.

19.

Government Accountability Office (GAO). (2018). GAO—Duplication and cost savings. Retrieved July 12, 2023, from https://www.gao.gov/duplication/overview

20.

Greenhalgh

Robert

Macfarlane

Bate

Kyriakidou

(2004). Diffusion of innovations in service organizations: Systematic review and recommendations. The Milbank Quarterly, 82(4), 581–629.

21.

Holt

D. T.

Vardaman

J. M.

(2013). Toward a comprehensive understanding of readiness for change: The case for an expanded conceptualization. Journal of Change Management, 13(1), 9–18.

22.

March

(1994). A primer on decision-making: How decisions happen. Free Press.

23.

Moir

(2018). Why is implementation science important for intervention design and evaluation within educational settings? Frontiers in Education, 3(61), 1–9.

24.

Moore

M. H.

(1997). Creating public value: Strategic management in government. Harvard University Press.

25.

Padgett

D. K.

(2012). Qualitative and mixed methods in public health. Sage.

26.

Patton

(2015). Qualitative research & evaluation methods (4th ed.). Sage.

27.

Qiao

Zhou

Shen

Tang

Stanton

(2015). Factors influencing the decision-making of parental HIV disclosure: A socio-ecological approach. Aids, 29, S25–S34.

28.

Rhodes

R. A. W.

(1997). Understanding governance: Policy networks, governance and accountability. Open University Press.

29.

Satterfield

J. M.

Spring

Brownson

R. C.

Mullen

E. J.

Newhouse

R. P.

Walker

B. B.

Whitlock

E. P.

(2009). Toward a transdisciplinary model of evidence-based practice. The Milbank Quarterly, 87(2), 368–390.

30.

Stein-McCormick

(2013). Career development for transitioning veterans (1st ed.). National Career Development Association (NCDA).

31.

Strong

J. D.

Crowe

B. M.

Lawson

(2018). Female veterans: Navigating two identities. Journal of Clinical Social Work, 46(2), 92.

32.

Taylor

V. H.

(2018). Air Force partners with VA for Women’s Health Transition Pilot Program. Retrieved July 12, 2023, from https://www.hill.af.mil/News/Article-Display/Article/1599357/air-force-partners-with-va-for-womens-health-transition-pilot-program/

33.

Thompson

J. M.

Maclean

M. B.

Vantil

Sweet

Poirer

Pedlar

(2011). Survey on transition to civilian life: Report on regular force veterans (VAC Research Directorate Technical Report). Veterans Affairs Canada.

34.

Turner

D’Lima

Hudson

Morris

Sheringham

Swart

Fulop

N. J.

(2017). Evidence use in decision-making on introducing innovations: A systematic scoping review with stakeholder feedback. Implementation Science, 12(1), 145.

35.

VERBI Software. (2019). MAXQDA 2022 [computer software]. VERBI Software. https://www.maxqda.com

36.

Ward

K. D.

Varda

D. M.

Epstein

Lane

(2018). Institutional factors and processes in interagency collaboration: The case of FEMA Corps. American Review of Public Administration, 48(8), 852–871.

37.

Yanovitzky

. (2002). Effects of news coverage on policy attention and actions: A closer look into the media-policy connection. Communication Research, 29(4), 422–451.

38.

Yin

R. K.

(2009). Case study research design and methods. Sage.