Response to Errors and Misrepresentations in “Breaking Barriers: Addressing Structural Obstacles to Social Service Provision to Asian Survivors of Domestic Violence” by Mihan Lee

The Outcome Measures Are Not Used to Determine Funding

Throughout her discussion of this issue, Ms. Lee bases her argument on claims about a 2007 document we wrote for the FVPSA office to provide context for programs and state funding administrators in implementing newly mandated reporting of outcomes: Outcome Evaluation Strategies for Domestic Violence Service Programs Receiving FVPSA Funding: A Practical Guide (Lyon & Sullivan, 2007; hereinafter “the Guide”). ¹ Ms. Lee states early in her article that the Guide notes challenges to measuring DV program outcomes and reviews several published studies of services, then continues,

After this fairly lengthy review, although [sic] the Guide specifically highlights two short-term outcomes as the most useful surrogate measures for program success: (a) having a safety plan and (b) being aware of community resources. These outcomes are to be measured through two statements on client surveys: (a) “Because of [this service], I feel I know more ways to plan for my safety,” and (b) “Because of [this service], I feel I know more about community resources for domestic violence.” Clients check “yes” or “no” in response to each question, and service providers need to report 65% positive responses for both outcomes to receive FVPSA funding. (p. 1357)

First, it is simply not true that any percentage of positive responses is required to receive FVPSA funding. The Guide states clearly that findings are used in aggregate for a report to the U.S. Congress. Second, the FVPSA office provides its funding to states and territories, which in turn determine funding allocations in their jurisdictions. In fact, the FVPSA office does not receive reports from individual local DV programs at all; it receives aggregate reports of outcomes, as well as other documentation of services, from state and territorial administrators. Training provided to programs and state administrators has explicitly stated that local funding decisions were not to be made based on these outcomes. In addition, the figure 65% was only presented as an example, considered reasonable because DV program advocates routinely discuss safety issues and available resources that might be helpful to the individual or family seeking help; in most cases, it would therefore not be difficult to achieve. However, it would certainly never be a funding threshold and is not presented that way in the Guide. ²

Misunderstanding of the “Safety” Measure

The main problem with Ms. Lee’s critique of the outcome “knowing more ways to plan for my safety” is that she appears to equate this statement with “having a safety plan” (p. 1357), although these are clearly two different things. Later, she asserts, “To help programs meet this outcome, the FVPSA Guide’s authors recommend providing women with the numbers of police authorities, hotlines, and shelters” (p. 1358). We offer no such recommendations to programs in the Guide.

Of perhaps greater concern, Ms. Lee states that the Guide (and therefore FVPSA) assumes that knowing about safety strategies means people will be safe. She states,

However, FVPSA’s evaluation discourse unquestioningly equates such knowledge [of safety planning] with safety. (p. 1358)

In addition,

By defining program success in this way, specifically in terms of women’s knowledge, FVPSA’s discourse automatically assumes a subject for whom a functional safety plan can actually be achieved solely through the provision of information (and perhaps empowering counseling). (p. 1359)

In fact, the Guide repeatedly makes the point that knowing about safety strategies is not the same as being safe, because safety is not in women’s control. We note early on

. . . the importance of remembering that survivors are not responsible for whether or not they are abused again in the future. For some women, despite any safety strategies they employ, the abuser will still choose to be violent. (p. 11)

Again, we note in a footnote the reason we changed the phrasing of the safety item:

However, discussion with advisors concluded that it sounded too much like the women had control over their safety (and their abusive partners’ use of violence against them, in particular), so it has been changed to “I know more ways to plan for my safety” on the latest version of the form. (p. 33)

Finally, we say explicitly that increased safety over time is an example of a “long-term” outcome while “you will be measuring the short-term outcomes that we expect to lead to the longer-term outcomes” (p. 21).

Ms. Lee’s misunderstanding of this evaluation criterion leads her to conclude the following: “Therefore, FVPSA makes no provisions for minority-targeted services that report lower success rates for this outcome; such services simply get poor evaluations and less funding” (p. 1358). First, as we have already stated, FVPSA does not use these data to make funding decisions. Second, Ms. Lee provides no data to substantiate either of these claims: that minority-serving programs have poor evaluations or that they get less funding. In general, all funding cycles—state or private—are extremely competitive, based on multiple evaluation criteria that are applied to all grant applicants, and FVPSA does not instruct state administrators to utilize outcome measures as a criterion for funding.

Misunderstanding of the “Community Resources” Measure

Ms. Lee writes,

By assuming that all domestic violence survivors come from a community in which support is available and safe to use, FVPSA holds service providers to a standard of provision that is not designed in light of their clients’ actual options and needs. (p. 1359)

If this assertion were true, it would be reason for serious concern. Presumably, Ms. Lee assumes that “community resources” should be understood as those that could be found exclusively in a person’s racial/ethnic community. However, the Guide (and the FVPSA office) used the term more broadly. For example, DV programs commonly provide survivors help with access to education, employment, housing, transportation, and legal assistance—resources available to differing degrees in different communities.

The FVPSA Office Does Not Promote Individual Knowledge as the Only Measure of Success

In several places, Ms. Lee asserts that the Guide states (and therefore the FVPSA office promotes) that knowledge is the only measure of success. For example, she states,

However, because the FVPSA Guide exclusively values individual knowledge as the key to remedying abuse, it effectively confines the DV program’s role to knowledge provision. There is no discursive space in which to recognize linguistic, economic, and other structural factors that affect women’s experiences of abuse; therefore, services that address these structural factors, but do not directly lead to positive knowledge outcomes, are not legible [sic] or deemed valuable. (p. 1359)

In contrast, there are numerous examples throughout the Guide where we stress the importance of other outcomes and of variability across life circumstances. In fact, we state early that “This handbook takes the position that outcome evaluation must be designed to answer the question of whether or not women attained outcomes they identified as important to them” (p. 16, emphasis in original).

It is important to note that throughout the Guide, we discuss the range of changes (including attitudes, skills, behavior, emotional well-being, and life circumstances) that can result from brief or extensive contact with a DV program. We explicitly urge programs to use more than the two required outcome measures as they proceed with their work. For example, “ . . . we recommend that programs view the forms in the Appendix as a ‘menu’ of items from which to choose” (p. 43). And again,

You [i.e., programs] want to be accountable to all of the survivors you serve, and you want to provide all of them with an opportunity to provide safe feedback about their experiences with your program. You also want to have the most complete and comprehensive information about the work you do and its impact on survivors . . . (p. 44)

The Measurement Development Process Did Not Systematically Exclude Immigrants’ Voices

Chapter 6 of the Guide provides a description of the process that ultimately led to the creation and selection of the two required measures. It covers the variety of surveys and other instruments developed collaboratively beginning in 1998 by a diverse group of advocates, researchers, survivors, and funders in a project called “Documenting Our Work” (DOW). In 2002, this project piloted five service-related forms in four highly varied states. The chapter describes the different forms and provides examples of results. The results are used to demonstrate the value of a larger number of questions, the range of needs and potential services, and the ways programs could use the findings to improve their responses to survivors’ needs.

However, Ms. Lee has an extensive endnote (Note 16) in which she states that a low percentage of survivors indicated they wanted help with immigration-related issues. She wonders whether immigrants’ voices were systematically excluded. If they were, she continues, “they may promote a form of service provision that does not recognize immigrants’ specific needs, or reward programs that do so.” (p. 1367)

Several points are important here. First, the FVPSA office does not exclude immigrants’ voices—not now, and not in 2002 when the reported data were collected. As Lee notes in her article, it was not until 2005 that the Violence Against Women Act created a program for funding culturally and linguistically specific services. In 2002, immigration-related issues were not as widely recognized by all DV programs across the country, and noting the need for such help in writing may not have felt entirely safe for immigrant survivors, even with anonymous reporting, which may have accounted for low rates of immigration-related needs in the DOW pilot. Furthermore, the measures were intended for use by all DV programs, not only culturally specific programs in urban areas. Perhaps more important, the description of DOW results was intended only as a set of examples, with discussion of potential uses. In the immigration context, we observed, “Although help with ‘immigration issues’ was checked by a small percentage, this result is likely to vary by location. This type of response could alert a program to unknown gaps in service, and lead to increased resources” (p. 32). Furthermore, testing supported by the FVPSA office in 2006 before the measures were implemented found much higher rates of wanting help with immigration-related issues.

Finally, the Guide repeatedly notes the importance of obtaining responses from everyone and responding to them, as in the following: “ . . . survivors from all ages, races and cultural groups, sexual orientations, religious preferences, and abilities must be included” (p. 44). In a concluding list of important points to remember, we state,

Consider issues of diversity in designing your outcome evaluation. Such issues include but are not limited to literacy, language, and culture. Again, including input from the survivors who use your program is vital. . . . Design outcome questions that will answer whether or not survivors attained outcomes they identified as important to them. (p. 71, emphasis in original)

The Role of the FVPSA Office Was Misrepresented: It Is Not a Source of Structural Barriers

Lee notes that the FVPSA office offers only “palliative” services (p. 1364). In fact, the office provides funding to states and territories that fund programs that not only provide services but also engage in exactly the kinds of community outreach Lee praises. As DV program advocates work with other community agencies to help survivors obtain the services and supports they need, advocates contribute to changes in policy and practice. FVPSA also provides funding for technical assistance and training to develop the capacity of minority-serving programs to address the trends and needs of the immigrants and refugees they serve. In addition, the FVPSA office provides leadership at the federal level, in convening and guiding interagency initiatives to make visible the need for and implementation of culturally specific programming by all federal grantees. The office, in fact, has played a broad and critical role in addressing DV at the individual, family, community, and broader policy levels.

Ms. Lee’s argument linking FVPSA measures to funding decisions is inaccurate and misleading, as are several unfounded and unsupported claims throughout her article, such as

. . . these evaluations fail to capture the unique merits and successes of minority targeted programs; they repeatedly devalue them as less successful and award them less funding. (p. 1357)

As a result, FVPSA can withhold funding from these programs, posing a serious structural obstacle to making essential services available to the Asian community. In this way, uninformed and biased program evaluation leads to the very misguided allocation of time and resources that it seeks to avoid. (p. 1360)

Hopefully, this detailed critique makes it clear that the central allegations made in Ms. Lee’s article about the role of the FVPSA office are simply untrue. We felt compelled to refute these misrepresentations for two overarching reasons. First, they have the effect of maligning an office that works diligently and creatively to support a wide range of community efforts to end DV. Second, these erroneous claims took the focus off of an extremely important point Ms. Lee was trying to make: There are a wide range of structural barriers facing Asian survivors of domestic abuse. We could not agree more with this assertion, and we know that the FVPSA office shares this concern and will continue supporting community efforts designed to alleviate such barriers.