Abstract
This article presents an analytic approach to assessing the relationship between policy advocacy activities and outcomes for discrete episodes of advocacy, using a holistic array of factors that affect the relationship between advocacy and policy change. The factors pertain to not only advocacy organizational structure and strategies, but also the policy environment, the policy issue and available options to address it, and technical assistance that advocates receive. By focusing on the entire context in which advocates operate, this approach can be used to assess both advocacy campaign strategies and the broader contextual factors that contribute to advocacy outcomes. The array of factors also facilitates analytic generalizability across advocacy episodes, as evaluators can compare the role of similar factors across advocacy efforts. This analytic approach is illustrated by its application to a multiple case study evaluation of Voices for Healthy Kids.
Introduction
Policy advocacy evaluation is a relatively new and fast-growing evaluation subfield that harbors multiple challenges (Coates & David, 2002; Guthrie et al., 2005). Advocacy campaigns and outcomes are notoriously hard to measure because advocacy can be heavily context-dependent and unpredictable; policy adoption is seldom the result of a single intervention, and advocates achieve straightforward success only rarely (Chapman & Wameyo, 2001; Coffman, 2007; Devlin-Foltz et al., 2012). Moreover, advocacy campaigns can take place over months or years, consisting of multiple, discrete episodes of advocacy such as advocacy surrounding individual state legislative sessions. If the goal of the advocacy campaign is not achieved during one episode, both the activities and goals of advocacy efforts can change over time, as developments may occur during an episode that are viewed as potentially contributing to or standing in the way of the future progress of the campaign. The literature on the relationship between policy advocacy and policy outcomes largely concludes that it is challenging to causally attribute policy outcomes to advocates’ work (BetterEvaluation, n.d.; Center for Public Program Evaluation, 2011; Chapman & Wameyo, 2001; Coates & David, 2002; Levine, 2014; Roussos & Fawcett, 2000; Steinberg, 2004; Teles & Schmitt, 2011; Webster et al., 2012).
As such, advocacy evaluation often assesses intermediate advocacy outputs and outcomes. Several authors have also suggested a prospective approach to advocacy evaluation, integrating evaluation measures into advocacy planning and implementation so that there are clear intermediate benchmarks against which to assess advocacy efforts as they progress (Guthrie et al., 2005; Louie & Guthrie, 2007). These include measurable benchmarks of progress based on a campaign’s theory of change, as well as advocacy capacity, organizational capacity, community organizing and movement-building, legislative “champion-ness,” and shifts in social norms (Center for Evaluation Innovation, 2014; Coates & David, 2002; Devlin-Foltz et al., 2012, p. 584; Guthrie et al., 2005; Whelan, 2008).
Many existing advocacy evaluation methods assess advocacy campaign progress against predetermined benchmarks. This article builds on these approaches: an analytic approach using an array of factors to descriptively, but not causally, assess the relationship between policy advocacy activities and policy advocacy outcomes for discrete episodes of advocacy. To guide analysis—and in a similar vein as Chelimsky’s (2019) proposed checklist of factors for evaluators to consider when evaluating policy and program sustainability—we propose an array of factors that can contribute to advocates’ ability to influence policy change. Rather than attempting to tease out advocates’ causal influence on policy change from a sea of confounding factors, our analytic approach focuses both on factors that may affect a discrete advocacy episode’s outcome and on advocates’ contributions to policy change. The factors pertain to not only advocacy campaign structure and activities but also the policy environment, the policy issue and available options to address it, and technical assistance that advocates may receive.
This analytical approach was developed during the evaluation of the Voices for Healthy Kids® initiative (VOICES). VOICES is a collaborative initiative of the Robert Wood Johnson Foundation® (RWJF) and the American Heart Association (AHA) to reverse the childhood obesity epidemic through public education and advocacy. The initiative began in 2012 and cost more than $15 million as of 2018, awarding up to 20 grants per year and providing technical assistance to state and local advocacy campaigns active in childhood obesity to encourage evidence-based regulatory and legislative activity. Because of the ongoing nature of advocacy campaigns, AHA under the auspices of VOICES often handled technical assistance to 30–40 simultaneously active advocacy campaigns at any one time.
Our evaluation employed retrospective case studies as a data collection method, using the array of factors as the analytic approach. This approach should be applicable to other evaluation designs as well. Our evaluation of the VOICES initiative sought to answer the following questions: What factors are associated with the relationship between advocacy and policy change? What patterns of factors may emerge across advocacy episodes?
We began by synthesizing factors from the literature and then refined them empirically based on findings from a series of brief case studies of policy advocacy episodes. Used retrospectively to evaluate a concluded episode of advocacy, the array of factors is able to depict many likely influences on advocacy episodes: Which factors contributed to the episode’s progress and eventual results, and more specifically, which factors represented facilitators or impediments to progress during the advocacy episode and to policy change. This array of factors also facilitates analytic generalizability across studies of multiple advocacy episodes, allowing evaluators to compare the role of similar factors among different episodes of advocacy.
To our knowledge, this synthesis of factors that can influence the relationship between policy advocacy and policy change represents a contribution to evaluation practice. The array is intended to be an evolving framework that can be revised and improved as it is applied in further policy advocacy contexts. The array of factors allows evaluators to describe the factors that facilitate or impede the progress and contribute to the outcomes of policy advocacy episodes. Factors both internal and external to advocacy efforts can affect advocates’ contributions to policy change, as well as the policy outcome itself.
Such an array has several benefits: When used to review completed episodes of advocacy, the array can improve feedback to advocates and funders by supporting more comprehensive assessments of facilitators and impediments to the progress of advocacy episodes. When used to review multiple advocacy episodes, the array can facilitate the identification and analysis of patterns and themes across episodes. As the array is applied to post-hoc assessments in different advocacy contexts, evaluators may recognize factors that do not fit within the current array, and can continue refining it into an increasingly useful and generalizable tool. As the array is further refined, it may become useful as a prospective tool for preadvocacy and evaluation planning, to ensure awareness of important influences that should be considered in developing advocacy strategies and designing advocacy evaluations.
Influential Factors: Facilitators and Impediments to Policy Change
Developing an Array of Influential Factors
To assemble the array of factors, we referred to the approach used in Mookerji and LaFond’s (2013) multi-case-study evaluation of an immunization program. In their evaluation, Mookerji and LaFond identified drivers of program improvements in each case study, assessed the cases for replicated findings, and conducted a cross-case synthesis of the case-specific drivers, categorizing the drivers of program improvement into six overarching categories. We built on this approach by developing, a priori from the literature, an array of 26 factors that influence the relationship between advocacy and policy change (Table 1), using them to assess individual episodes of advocacy as well as patterns and themes across episodes and refining them through our evaluation practice.
Rather than combining the factors into overarching categories during cross-case analysis, we prospectively collected them into thematically consistent domains corresponding to various aspects of an advocacy episode, as discussed in the literature (see Ferrer, 2002; Jenkins-Smith et al., 2014; Klugman, 2011). The identification of domains that logically group the factors allows for discussion of and feedback on more general thematic aspects of advocacy episodes, while retaining the detail of individual factors for analysis. We identified five domains from the literature and added a sixth domain, technical assistance.
Structure of the Array
Policy or regulatory context
The first domain listed in Table 1 is the policy or regulatory context of advocacy. Factors included in this domain are the political and fiscal climate in which advocacy occurs, any previous progress on the policy issue, support (or lack thereof) from policy makers, and opposition. These factors are all features of the policy or regulatory context that can interact with advocacy efforts and can augment or thwart their ability to contribute to policy change (Center for Public Program Evaluation, 2011; Ferrer, 2002; Klugman, 2011; Teles & Schmitt, 2011). Also included are state or local social norms and cultural attitudes, which can affect the feasibility of certain types of policy change (Calancie et al., 2015; Pitts et al., 2013).
Policy issue and options
The second domain describes factors associated with the policy issue and various policy options for addressing it. Some policy issues can be more contentious or have a more challenging state or local history than others; by contrast, there can be broad areas of agreement about the importance of developing policy to address certain issues. Once a policy issue makes it onto policy makers’ agendas—if the timing is right and a policy window opens—there are still numerous approaches to addressing it, some of which are taken more seriously or are more likely to succeed than others (Kingdon, 1984; Osmond, 2010).
Grantee or lead organization
The third domain in Table 1 contains factors associated with the grantee or lead organization itself. The grantee or lead organization’s flexibility to adapt to changing circumstances, and the presence of additional funding for advocacy efforts can affect advocacy campaign outcomes (Center for Public Program Evaluation, 2011; Hsu et al., 2009; Klugman, 2011; Patton, 2008; Teles & Schmitt, 2011). The grantee’s embeddedness in the policy context and collaboration with stakeholders—that is, the grantee’s recognition that stakeholders have useful contributions to make and engaging them in deliberation and decision-making—can also affect advocacy outcomes (Klugman, 2011; Koebele, 2019). The lead organization’s ability to identify possible impediments is an important part of defining campaign goals and objectives and of structuring advocacy work from the outset, while previous experience with policy advocacy in general and of the policy issue at hand in particular can also assist with advocacy campaigns (Coates & David, 2002; Louie & Guthrie, 2007; Osmond, 2010; Roussos & Fawcett, 2000; World Health Organization, 2006).
Coalition/partnership capacity
The fourth domain represents factors associated with coalition or partnership capacity. For example, advocacy coalitions (as opposed to advocates working independently without a coalition structure) with diverse memberships who collaborate using open and frequent communication and shared decision-making can have a greater capacity—in terms of members’ skills and knowledge, shared goals, and values, and their ability to effectively work together—to work together for change (Center for Public Program Evaluation, 2011; Foster-Fishman et al., 2001; Osmond, 2010). Coalitions’ effective engagement of diverse grassroots organizations, as evidenced by the presence and activities of these organizations in the coalition, can also affect campaigns’ ability to contribute to policy change (Freudenberg et al., 2009).
Advocacy campaign activities
The fifth domain contains factors concerning advocacy campaign activities: the extent to which advocates employed concrete advocacy approaches in pursuit of policy change and how well they used those approaches (Devlin-Foltz et al., 2012). This domain includes internal advocacy activities of timely and effective planning and advocates’ use of relevant research and data (Gretchen Swanson Center for Nutrition, 2016; Patton, 2008). It also includes external-facing advocacy activities, such as engaging with government institutions, which can not only bring both new problems and solutions to the policy advocacy agenda but also increase buy-in from key stakeholders (Freudenberg et al., 2009). In our evaluation practice, we also distinguished two additional advocacy campaign activities in this domain: marshaling other advocates (the degree to which advocates mobilize other advocates and advocacy organizations) and marshaling grassroots and residents (the degree to which advocacy campaigns organize or mobilize individuals or groups most affected by the policy issue).
Technical assistance
Having compiled an array of factors from the literature, our team tested and empirically refined the array based on findings from multisite case studies of advocacy episodes of VOICES-supported policy advocacy campaigns. Based on findings from the VOICES initiative, we developed a domain for technical assistance, including factors for its availability, its pertinence to the policy issue and context, and the extent to which technical assistance included the sharing of information between policy change campaigns.
Each factor listed in Table 1 can negatively or positively affect advocates’ ability to contribute to policy change. For example, the political climate or fiscal climate in which advocates seek policy change may be favorable or unfavorable to the change they seek. If a state is facing a tight budget, the state’s fiscal climate could represent an obstacle to enacting legislation to expand physical education in public schools. Similarly, while advocates often identify and address obstacles to their efforts, they may also fail to identify key obstacles to their work or may identify them incorrectly. In the case of the factor “presence of relationship building with policy makers/decision makers,” an insufficient amount or a lack of relationship building with policy makers or decision makers could negatively affect advocates ability to contribute to policy change. Each factor can also positively and negatively affect advocates’ ability to contribute to policy change at the same time. For instance, a group of collaborating organizations and residents who demonstrate broad support for a policy may facilitate an advocacy effort—however, their collaboration may also impede the advocacy effort if key members of the collaborating organizations have conflicting interests among themselves (Butterfoss & Kegler, 2009 p. 238). Similarly, the factor “window of opportunity/timing” can simultaneously impede and facilitate an advocacy campaign if, for instance, policy makers delay voting such that advocates must stretch their limited resources to continue their efforts, but also are able to use the added time to grow their coalition of supporters.
Factors Likely to Facilitate or Impede Policy Advocacy Episode Progress.
a Indicates factor developed during the VOICES evaluation.
This array of factors allows for identification and analysis of contributors to an advocacy episode’s outcome, as well as identification and analysis of any patterns and themes in how these factors operated across multiple advocacy episodes. The conclusions drawn will not be about the relationship between advocacy and policy change writ large but about how that relationship operated in the episode or episodes evaluated.
VOICES Evaluation: Illustration of the Application of Influential Factors Array to Policy Advocacy Case Studies
We applied the array of factors to analyze a set of brief case studies of policy advocacy episodes in the evaluation of the VOICES initiative. We share our methods and results here to illustrate the approach in action: How data about advocacy episodes were collected, what it looks like, and how the factors can be used to analyze it. These case studies involved the selection of advocacy campaigns and episodes within campaigns, data collection, data analysis, and reporting, followed by additional steps for analysis and reporting across episodes/campaigns.
Using qualitative, panel-interview data collection, we were able to describe which factors from the array facilitated or impeded the progress of 21 episodes of policy advocacy, selected from 21 advocacy campaigns from 2015 to 2017. Using panel-interview data collection also facilitated identification of patterns and themes across cases. The evaluation team analyzed 13 state and eight local advocacy episodes, with seven episodes each in the Northeast, South, and West. Ten of these sought appropriations, and 11 sought policy change that did not require appropriations; 10 focused on healthy eating, eight on physical activity, and three on both policy areas. Fifteen advocacy episodes resulted in policy change (wins), two had more equivocal outcomes (mixed), and four did not result in policy change (losses); these categories of advocacy episode outcomes were defined in a purely operational fashion, based on the advocates’ and funders’ goals (Table 2).
Characteristics of Advocacy Episodes Reviewed.
Note. n = 21.
Assess Facilitators and Impediments to the Progress of Individual Advocacy Episodes
Selection
We defined an advocacy episode as a discrete round of advocacy effort focused on policy change during a single state legislative session, fiscal year, or vote of a governing body. Our team selected advocacy episodes that were settled for the current legislative session or fiscal year, meaning that the policy outcome was clear for the current legislative session or fiscal year (depending on the campaign’s goal). Either a law was adopted, voted down, or died in the legislature or a regulation was approved or failed to gain approval by the pertinent government agency or local governing body. A policy being “settled” for the year did not necessarily mean that a campaign was completely finished, as campaigns often continued into (or were resurrected during) subsequent legislative sessions or fiscal years.
When choosing among multiple eligible advocacy episodes for review, our team’s selection process followed the example and recommendations of the literature on multiple-case methods more generally: select purposively and for variety (Allen et al., 2012; Baxter & Jack, 2008; Baldwin & Twyford, 2007; Kompier et al., 2000). For maximum variety, the team selected no more than one episode per advocacy campaign to review. The evaluation team selected advocacy episodes for review within at most 2 months of their outcome being settled, and each report was completed within an additional 2–3 months for ongoing feedback to VOICES and RWJF staff. These time constraints resulted in brief case studies rather than more lengthy ones.
Data collection
The VOICES evaluation team employed a qualitative panel-interview design using semi-structured interviews with the following categories of stakeholders: funder/VOICES technical assistance staff, grantee or advocacy lead organizations, other pertinent advocacy organizations, policy makers, and additional technical assistance providers. Interviews started with funder/technical assistance staff and grantee or advocacy lead organizations as core informants who provided an overview of the advocacy episode and background on the policy issue and advocacy around it, their views on pertinent factors, and information about and connections with other prospective interviewees (Weiss, 1994). After the core informant interviews, additional interviewees that represented other advocacy organizations, policy makers, and technical assistance providers were located via snowball sampling on the recommendation of prior interviewees. Telephone interviews were used due to distance and resource constraints.
The VOICES evaluation used specialized interview protocols for each category of informants (VOICES/technical assistance staff, advocacy organization staff, and other advocates/policy makers). Some questions were distinctive to one or two versions of the interview protocol—for instance, only policy makers were asked about their colleagues’ perceptions of the policy change sought in the advocacy episode, and policy makers were not asked about technical assistance—while other questions were asked in all three versions. For triangulation, all interview protocols included open-ended questions about facilitating and impeding factors to the policy’s progress, the recent history of the policy area and related advocacy efforts in the pertinent state or locale, the features and status of the policy, advocacy activities in support of the policy in the episode being evaluated, and upcoming plans for the policy—whether another episode of advocacy or implementation of an adopted policy. All interviewees were asked about facilitating and impeding factors in the advocacy episode using general, open-ended questions about facilitators and impediments rather than using the array of factors to structure these interview questions. By using open-ended questions, the team remained neutral and receptive to any additional factors that emerged from the advocacy episodes reviewed.
Interviews were conducted until they provided diminishing returns; this, along with the variation in complexity of advocacy episodes, accounts for variability in the number of interviews per case (Patton, 2008; Weiss, 1994). Advocacy episode case studies in the VOICES evaluation involved up to eight semi-structured interviews: Advocacy episodes that involved coalitions of numerous advocacy organizations involved more interviews; episodes that were concluded more quickly or that involved little technical assistance tended to involve fewer interviews.
The evaluation team did not collect information on any lobbying activities, as this analysis was limited to funds provided by RWJF or AHA under VOICES, and no RWJF funds were used for lobbying. As a result, assessing the availability and use of unrestricted lobbying funds was beyond the scope of the study.
Data analysis
Our team used the array of factors to analyze each advocacy episode. For each individual review, the evaluation team used issue-focused analysis—per Weiss (1994), an analysis concerned with what can be learned about specific issues and processes across respondents—to assess the collected interview data. During the analysis stage, the evaluation team compiled interview data from each respondent to elicit information about the focus, context, outcome, and advocacy approach of the advocacy episode, as well as reported facilitating and impeding factors, and next steps for the campaign or the policy itself. The qualitative process of compiling interview data across respondents also resulted in identifying whether the reported factors represented major or minor facilitators and/or impediments to policy change based on which factors respondents reported most often and identified as contributing most strongly to the advocacy episode outcome. It also allowed the evaluation team to extract key themes and representative quotations and identify any discrepancies in the data to discuss in reporting.
To illustrate the extraction of facilitating and hindering factors from features of advocacy episodes: In one case study of an advocacy episode with the goal of local-level appropriations, the interview participants agreed that there had been steadily-building public support for the policy issue for several years prior to the VOICES-supported advocacy effort. They also agreed that a diverse group of advocacy organizations had sent a unified message about the importance of the appropriations—but because the city budget originated with the mayor, it was the mayor’s decision to support the appropriation that ultimately led to its success. As a result, major facilitators in this advocacy episode included “appeal, clarity, and visibility of policy issue” (because of the broad public support), “range and number of partners” (because of the large and diverse group of advocacy organizations), and “support from policy makers/decision makers” (because of the mayor’s ultimate support for the appropriation). Minor facilitators included “presence of grassroots/residents marshalling” (because advocates organized grassroots groups to contact the mayor’s office in support of the appropriation), while “fiscal climate” represented a minor hindrance (due to the limited amount of funding in the city budget).
In another case study—of an advocacy episode with the goal of state-level policy change—the interview participants agreed that previous local-level policy victories had made a state-level success on the policy issue seem feasible and that the advocacy coalition had maintained a strong, unified presence at the statehouse during legislative committee hearings and a lobby day. However, the advocates got a late start on their efforts, and the bill they had helped to draft was both filed later than planned and was assigned to a different legislative committee than expected, leading advocates to restart relationship building with policy makers in the middle of the legislative session. There was also an unrelated bill on a similar policy issue being considered simultaneously, and advocates had a difficult time making the difference between the bills clear to legislators. Despite major facilitators that included “range and number of partners” (due to the large advocacy coalition) and coalition “cohesiveness/unity” (due to their unified presence at the statehouse), and despite minor facilitators such as “presence of previous experience with policy area/ policy advocacy” (due to prior local-level successes), the bill died in committee. “Window of opportunity/timing” was a major hindrance (due to the advocates’ late start), while “appeal, clarity, and visibility of policy issue” was a minor hindrance (due to policy makers’ confusion about the bill).
Assess Patterns and Themes Across a Group of Advocacy Episodes
Once multiple advocacy episode case studies were complete, the team used the array of factors to conduct cross-case analysis to assess patterns, themes, and variations across advocacy episodes; specifically: Which factors were present and what role did they play in the campaign outcome? What patterns of factors emerged by campaign outcome? What patterns of factors emerged by government jurisdiction and request for appropriations?
To carry out this analysis, the team used the previously discussed qualitative interview data from each individual advocacy episode that determined how many factors were present in each episode, whether each was a facilitator, impediment, or both, and whether each played a major or minor role. We then developed a comparative table, focusing on domains and factors organized according to advocacy episode outcomes, to analyze this information to determine which factors featured most heavily in episodes with different outcomes (Table 3). Combined with analysis of the qualitative interview data, the team used the table to synthesize patterns across cases into a discussion of the factors that most commonly facilitated and impeded advocacy episode progress, how the factors contributed to episode outcomes, and how the factors interacted during the episodes.
As described above, each factor could act to facilitate or impede progress in the advocacy episode, and in some instances, the same factor did both. In addition to aiding thematic analysis, this table provides a way to count advocacy episode outcomes and frequent facilitating and impeding factors and to compare patterns in factors across episodes that resulted in policy change (wins), those that had more equivocal outcomes (mixed), and those that did not result in policy change (losses). Mixed outcomes and losses referred only to the outcomes of the advocacy episodes evaluated; episodes with these outcomes could help to inform subsequent advocacy episodes, and—particularly in the case of mixed outcomes—may have served as stepping stones to subsequent wins. The evaluation team’s understanding of each advocacy episode’s outcome, and the distinction between mixed outcomes and losses, was developed in communication with advocates and technical assistance staff.
Influential Facilitating and Impeding Factors and Advocacy Episode Outcomes.
a The total number of campaigns where factors acted as facilitators and impediments is listed in bold. The number of campaigns where the factor was a major facilitator or impediment is listed in parentheses. A factor may simultaneously act as a facilitator and an impediment to the same campaign.
b F = facilitators; I = impediments.
Advocacy episode outcomes were most frequently influenced by factors in the Policy or Regulatory Context domain, which includes the most prevalent major facilitator (presence of support from policy makers/decision makers) as well as the two most prevalent major impediments (political climate and fiscal climate). As one interview participant said, “the speaker, the senate president and the governor, they go into a room and they create the budget,” which creates a challenging dynamic around the state fiscal climate because “it’s next to impossible to influence.” Advocacy episodes also commonly encountered major influence from factors in two other domains, coalition/partnership capacity and advocacy campaign activities.
As shown in Table 3, advocacy episodes were much more likely to show evidence of facilitators than impediments. There is a variety of reasons why this may be the case. The team evaluated a larger number of wins (n = 15) than mixed outcomes (n = 2) or losses (n = 4), making the larger proportion of facilitators potentially an artifact of the episodes selected for evaluation. It is also possible that we compiled a greater number of factors that are more likely to be facilitators than impediments, making the larger proportion of facilitators potentially an artifact of the array of factors itself.
It is also possible that the larger proportion of facilitators may be attributable to the impediments’ greater influence: It generally took fewer impediments to stymie a campaign, while it tended to require many facilitators to contribute to success. Facilitators supporting successful campaigns tended to both outnumber impediments (128 facilitators over 15 campaigns for a mean of 8.5 facilitators, compared to mean of 2.2 impediments) as well as to outweigh impediments relative to major factors (mean of 4.4 major facilitators compared to mean of 1.1 major impediments). In losses/mixed outcomes, on the other hand, facilitators and impediments tended to be more equally balanced, both in terms of prevalence (mean of 5 facilitators compared to mean of 5.2 impediments) and in terms of weight (mean of 3.2 major facilitators compared to mean of 2.8 major impediments). Wins were also strongly characterized by relationships with a variety of stakeholders, organizational and coalition capacity, and the strength of the available policy options. Losses/mixed outcomes were characterized both by contextual factors outside of advocates’ control, such as unfavorable political and fiscal climate, as well as by challenges over which advocates did have control, including the clarity or visibility of the policy issue, and ineffective or delayed planning.
In addition to having a less favorable balance of facilitators and impediments than wins, advocacy episodes with losses/mixed outcomes also sometimes had a “fatal flaw,” a major impediment that exerted considerable influence over the entire episode. Advocacy episodes with losses/mixed outcomes often failed to address impediments in one of two ways: a lack of capacity (including a lack of time) at the beginning of the episode that compounded over time and hampered advocates’ ability to mount strong efforts or poor identification of obstacles to success that hampered advocates’ ability to address them. The fatal flaw in the state policy change example discussed above was the advocates’ late start combined with the bill’s unexpected committee assignment, which meant that the advocates “had to scramble and start all over again,” while the legislative session was under way. In another state-level advocacy episode in which a bill died in committee, the state’s significant budget shortfall was the fatal flaw: “every single thing that would normally have a decent shot of funding became the longest shot in the history of the legislature.”
On the other hand, in many wins, specific facilitators countered the influence of specific impediments. For example, in one advocacy episode, technical support brought in later was able to offset less pertinent initial technical assistance. In another advocacy episode, policy makers’ lack of awareness of the policy issue was a hindrance because they “couldn’t see why we would be doing this.” However, the advocacy coalition had successfully marshalled grassroots involvement—with “genuine support from the communities on the ground”—and brought residents “to tell their stories in a public comment format,” which educated policy makers about the need for change and demonstrated the people power behind the advocacy effort. In another, a limited city budget initially impeded policy change but was effectively countered by both a favorable local culture and by the mayor’s support.
Cross-episode analysis also allowed for comparisons between advocacy episodes at the state versus local level and between episodes seeking policy change versus appropriations. Most basically, all six losses/mixed outcomes occurred at the state level, as did seven out of the 15 wins, while all eight local advocacy episodes ended in wins. Our analysis suggested that different types of campaign mobilization may be needed at different jurisdictional levels: At the state level, coalition cohesiveness/unity appears to be important, possibly because a unified coalition can show policy makers that a range of stakeholders have united in support of policy change. At the local level, however, the support of affected residents may be more important to convince policy makers of the need for change. Additionally, state-level advocacy tended to encounter impeding political and fiscal climates more frequently than local advocacy.
As far as the goal of advocacy campaigns, nine advocacy episodes seeking appropriations ended in wins, as did six seeking policy change. Two seeking appropriations ended in mixed outcomes/losses; four of the mixed outcomes/losses were seeking policy change. Episodes that sought appropriations tended to encounter fiscal climate as an impediment more frequently than those that did not seek appropriations. In episodes with this challenge, there was greater need for the grantee’s previous experience with the policy area and with policy advocacy to navigate complex budgetary situations, as well as for greater demonstration of broad-based coalition and grassroots support to convince policy makers. Regulatory campaigns, on the other hand, tended to encounter opposition from private industry that viewed the potential policy change as disadvantageous to its fiscal interests.
Provide Feedback to Advocates and Funders
The evaluation team returned findings from each individual case to VOICES and funder staff in concise, three- to five-page reports that included information about each advocacy episode’s purpose, the focus of the larger advocacy campaign of which the episode was a part, the policy context, the status of the policy change or appropriations sought after the conclusion of the advocacy episode, the advocates’ approach, key influential factors, and next steps in the advocacy campaign. The team completed an interim cross-case review report after evaluating 10 advocacy episodes, and a final cross-case review report after all 21 advocacy episodes had been evaluated, both of which assessed patterns in the distribution of factors across cases. The final report was able to go into more depth about the relationship of factors to each other, and patterns across advocacy episodes with different outcomes, at the state versus local level, and that sought policy change versus appropriations. VOICES and funder staff were responsible for sharing these findings back to the advocates. Feedback from VOICES and funder staff indicated that our findings were instrumental in shaping how VOICES delivered its ongoing technical assistance.
These findings can be used to inform both advocates and funders in their ongoing advocacy efforts. They can be used to encourage advocates to conduct a clear, cogent initial assessment of the state or local context pertinent to their policy change goal, including the fiscal and political climate and likely opposition, and to update their assessment during the advocacy episode. The findings can also help advocates determine how to improve their efforts in subsequent episodes of advocacy, including adjustments of their advocacy activities and tactics. Likewise, these findings can help funders better tailor their support to be most beneficial to different advocacy campaigns and hone their ongoing technical assistance by informing them about the types of technical assistance that have either had shortcomings or have been pertinent and useful for advocates.
Discussion
As the example of the VOICES evaluation illustrates, the array of factors is a tool that supports the identification and description of facilitators and impediments to advocacy episode progress, analysis of patterns and themes in those factors across a group of advocacy episodes, and development of feedback for use in ongoing advocacy campaigns. In cross-case analysis, the array is also useful for comparing which factors appeared and how the factors operated—as facilitators or impediments or both, playing a major or minor role—among advocacy episodes with different policy goals, at different levels of government, and with different types of outcomes.
In our use of the array of factors in the evaluation of the VOICES initiative, we found that different factors tended to play a major role at different levels of government and in advocacy efforts that sought appropriations as opposed to those that sought policy change but did not seek appropriations. Chelimsky (2019) has pointed out that different political, economic, and social contexts come with different considerations of the sustainability of a program or policy change, and our findings from the VOICES evaluation can be understood to echo this point on the advocacy front. Advocacy at different levels of government and with different policy goals may encounter different challenges, and the array of factors provides a way to describe and analyze those differences.
The array of factors represents an addition to the tool box of analytic approaches specifically for policy advocacy evaluation. In a field often focused on intermediate outcomes and on advocacy structure and activities, the array of factors provides an approach for analyzing concluded advocacy episodes in terms of the facilitating and impeding factors that contributed to their outcomes and the broader context in which advocates operate. As advocacy is often heavily context-dependent, the array of factors as an analytic approach can also allow us to understand how similar factors that impact the relationship between advocacy and policy change may operate in different contexts. Rather than focusing on the causal impact of advocacy, the array allows for analysis of the contributions to policy outcomes of advocacy episodes as well as the effects of the larger contextual environment and other factors generally outside advocates’ control. The value of this approach is its ability to illuminate patterns and themes within and across multiple advocacy efforts—and it is a starting place for continued development of a comprehensive tool for assessing facilitators and impediments to policy advocacy.
Limitations and Future Directions
As developed here, the array of factors faces several limitations as an analytic approach. Most basically, it is likely not comprehensive: We may have missed important factors from the literature or—when testing the factors empirically—factors that would have appeared in types of policy advocacy that our team did not encounter in practice. Moreover, our methodology—using open-ended questions, rather than asking interviewees to comment on a predetermined array of factors—also may have contributed to the small incidence of factors across advocacy episodes, which in turn could have limited our ability to make cross-case comparisons. However, this does not represent a concluded project, but a start, to which other evaluators can add in order to make it more complete.
There are also limits to the types of conclusions that can be drawn using the array of factors for analysis. The array is solely descriptive and does not constitute a validated program theory or logic model: It does not posit causal relationships among the factors, among the factors and policy change, or about advocates’ impact on policy change. Positing a theory of change before more robust testing of the array of factors would have been premature and is a potential avenue for future development of the array. Additionally, the factors dealing with advocacy activities could be more precise. They can be used to assess both the extent to which advocates engaged in a given advocacy activity and how well they used it; they do not uniquely measure either one but a mix of both. While the takeaways for advocates and funders can be clarified when sharing findings in writing, this is a limitation that should be addressed in future applications of the array.
So far, the array of factors as an analytic approach has only been tested in a limited way—selecting only one episode of advocacy per campaign, using brief case studies as the data collection method, and in an evaluation of advocacy campaigns led by professional advocacy organizations rather than by organized grassroots groups of people who would be affected by the policy change. Selecting only one episode of advocacy per campaign meant that factors could not be compared across advocacy episodes that built on each other over time within the same larger campaign. Further, the preponderance of successful advocacy episodes versus mixed or unsuccessful ones (15 vs. six) available to the evaluation may have skewed the results.
Using brief case studies for data collection risks selection bias: Collecting data from a limited number of interviewees may lead to missing details or perspectives, especially if interviewees with unique information, expertise, or points of view cannot readily be reached. Including a small number of interviewees may also have made it difficult to triangulate across interview participants or assess the trustworthiness of the information provided. It is also likely that limiting the data collection to a small number of interviewees systematically, and possibly inequitably, missed entire groups of informants—namely, community stakeholders most directly affected by policy change, including residents and grassroots groups. This is especially likely because the advocacy episodes evaluated were led by professional advocacy organizations (who are likely to direct researchers to other, similar organizations as part of snowball sampling) rather than by organized groups of residents or community stakeholders. Community stakeholders may have unique perspectives on any number of factors, including how well the coalition has engaged them, whether the coalition accommodates shared decision-making, whether data reflect the community, and what other partners may be important to engage. Although professional advocacy organizations’ work often included developing and supporting grassroots and resident participation and advocacy, and although the evaluation team interviewed grassroots-led organizations, the direct application of this method to grassroots- or resident-led campaigns has yet to be tested. Applying the array to grassroots-led advocacy efforts or to multiple rounds of the same advocacy campaign represents avenues for further testing and refinement.
Additionally, during the VOICES evaluation, we were often unable to talk to groups or individuals opposed to the proposed policy change due to our lack of connections with them. It was therefore not possible to test whether opponents of a policy change identified similar facilitating and impeding factors to supporters and may have led to missing factors that could have been included. Relatedly, while the successful advocacy episodes largely identified the same impediments as the unsuccessful ones—especially contextual impediments—having evaluated a disproportionate number of wins relative to losses limits any conclusions drawn about factors that impede advocacy campaign progress. It would be worthwhile to incorporate interviews with opposition into any future data collection using this analytic approach and to evaluate a greater proportion of unsuccessful episodes of advocacy.
Similarly, it was not possible to collect information on lobbying activities, which were systematically excluded due to the funding source for the evaluation. While we assessed the presence of opposition from our interviewees, we were not able to determine the full extent of any professional lobbying activities or money aimed at advocating for or against a policy change. Both of these limitations may have contributed to missing possible factors or may have affected how we analyzed the effects of specific factors, such as those related to political climate or window of opportunity/timing.
The least tested aspect of the factors’ contribution to evaluation practice is their utility in providing feedback directly to advocates. The evaluation team in the VOICES evaluation was not permitted to share findings directly with advocates. While we provided real-time feedback to funders and technical assistance providers, who then could provide feedback to the advocates, the utility of this approach when used in a direct relationship with advocates is as-yet untested.
Similar to Chelimsky’s (2019) checklist for guiding evaluation of policy and program sustainability, our array of factors is intended to be iterative. In addition to the areas for further refinement discussed above, future development of this analytic approach to evaluating policy advocacy could involve testing its applicability to the evaluation of multiple campaigns that take place over a longer period of time or whether and to what degree early advocacy efforts laid essential groundwork for future policy change. Ongoing development of this approach could also examine whether the array of factors is stable across campaigns managed by different organizations or employing different advocacy techniques or advocacy evaluation data collected with a method other than retrospective case studies. Future work could also explore not only the relationship between advocacy and policy change but also advocacy and the content of policies.
Additionally, the array of factors has so far only been tested retrospectively, in evaluation of concluded episodes of advocacy. As employed in the VOICES evaluation, the array of factors was intended to be used to both learn from completed episodes of advocacy and to plan for subsequent rounds of advocacy effort within an ongoing campaign. It would be worthwhile to explore whether the array would be useful in preadvocacy and evaluation planning as well as evaluation of advocacy episodes after they have concluded.
Conclusion
This article presents an array of factors that can influence the relationship between policy advocacy and policy change, and an initial test of its application to a multisite policy advocacy initiative. The array can improve feedback to advocates and funders about facilitators and impediments to policy change and can facilitate robust assessments of patterns and themes across advocacy episodes. It will be particularly valuable for future evaluations to employ the array of factors and refine it into an even more generalizable approach. Further use and development of the array of factors will make it a stronger and more widely applicable analytic approach for evaluating policy advocacy.
Footnotes
Authors’ Note
The views expressed here do not necessarily reflect the views of the Robert Wood Johnson Foundation.
Acknowledgments
We thank Jill Birnbaum and her team at Voices for Healthy Kids for their collaborative assistance.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research and/or authorship of this article: This study received funding from Robert Wood Johnson Foundation (Grant No. 73770).
