Abstract
Lesbian, gay, bisexual, transgender, queer, intersex, Two-Spirit, and other sexual and gender minority (LGBTQ+) individuals encounter numerous obstacles to equity across health and healthcare, education, housing, employment, and other domains. Such barriers are even greater for LGBTQ+ individuals who are also Black, Indigenous, and People of Color (BIPOC), as well as those who are disabled, and those who are working-class, poor, and otherwise economically disadvantaged, among other intersecting forms of oppression. Given this, an evaluation cannot be equitable for LGBTQ+ people without meaningfully including our experiences and voices. Unfortunately, all evidence indicates that evaluation has systematically failed to recognize the presence and value of LGBTQ+ populations. Thus, we propose critical action steps and the articulation of a new paradigm of LGBTQ+ Evaluation. Our recommendations are grounded in transformative, equitable, culturally responsive, and decolonial frameworks, as well as our own experiences as LGBTQ+ evaluators and accomplices. We conclude by inviting others to participate in the articulation and enactment of this new paradigm.
Keywords
The authors of this piece write from the traditional homelands of the Council of the Three Fires: the Ojibwe, Odawa, and Potawatomi peoples, as well as the Menominee, Miami, and Ho-Chunk nations, among many other peoples who used, and use, these lands as a site of trade, community, and healing. The American Evaluation Association, which produces the American Journal of Evaluation, is headquartered in Washington, D.C., an area once stewarded by the Piscataway and Anacostan/Nacotchtank peoples. Acknowledging the history of the lands we occupy and the first peoples who cared for them is a vital part of resisting the ongoing process of colonization, erasure, and genocide of Native American and Indigenous peoples.
In opening our piece with this acknowledgement, we seek to honor the history and ongoing importance of the original stewards of the land we occupy, and to reflexively and critically name the tensions that exist within the language and politics of this piece: LGBTQ + is a Western term which relies on Western constructions of sex, sexual orientation, and gender, and which is predominantly used in Western contexts such as the United States. The use of this term also reflects the authorship team's own positions within the colonial, Western institution of “the academy.” Consideration of the ways in which the material conditions of capitalism and colonialism have harmed those who identify as LGBTQ + is vital, yet so is consideration of how the term LGBTQ + is itself a colonial product. As this paper seeks to help advance a paradigm for LGBTQ + Evaluation, naming this tension is vital, and decolonizing our understanding of sex, sexual orientation, and gender will be a critical piece of how this paradigm grows and develops within the field.
Introduction
In her keynote address to the 2017 Conference of the Center for Culturally Responsive Evaluation and Assessment (CREA), Robin Lin Miller challenged the field of program evaluation to take serious and meaningful action to better serve the sexual and gender minority (SGM; i.e., LGBTQ+) community (Miller, 2018). Miller's address, which focused on the need for intersectional approaches to evaluation among LGBTQ+ people of color, emphasized the need for a “conscious effort” from evaluators to critically consider why it is important to account for LGBTQ+ populations in evaluation.
More recently, the politically volatile climate of the United States (US) has cast both anti-LGBTQ+ oppression and LGBTQ+ resistance into sharp relief. Despite notable victories such as in Bostock v. Clayton County and associated cases, which expanded the definition of sex-based protections to include LGBTQ+ persons (Bostock, 2020), the US political and social climate remains hostile to LGBTQ+ persons overall. Black transgender persons in particular continue to be murdered and abused by the police and carceral state at disproportionate rates (Reisner et al., 2014; Snorton, 2017; Snorton & Haritaworn, 2013). Meanwhile, as the COVID-19 pandemic continues to spread across the United States, little work has examined the disproportionate impacts it has had on LGBTQ+ persons (Bowleg, 2020; Cahill et al., 2020; Phillips, Felt, Ruprecht et al., 2020; Pirtle, 2020; Poteat et al., 2020). Although LGBTQ+ community organizations have stepped up to meet the gaps left by the social and academic structures that have ignored LGBTQ+ people’s needs (Brave Space Alliance, 2020; Ruprecht et al., 2021), this reality again makes clear the need for a conscious effort from evaluators to identify where and how we can best act to support LGBTQ+ health and equity.
In response to Miller’s charge (Miller, 2018), to the ongoing paucity of LGBTQ+-focused evaluation theory and practice, and to the material realities of our current climate, this manuscript seeks to identify and provide guidance on how to address key gaps in both LGBTQ+ Evaluation theory and practice. First, we provide brief necessary definitions and background information regarding LGBTQ+ community composition and known inequities. Second, based on an informal review of select evaluation-specific academic journals, we discuss the history of, recognition of, and focus on LGBTQ+ individuals in evaluation practice, highlighting extant gaps in the contemporary evaluation field. Finally, we make specific recommendations to improve the state of LGBTQ+ Evaluation in line with Equitable, Culturally Responsive, and Transformative Evaluation frameworks (Equitable Evaluation Initiative, 2017; Hood et al., 2015; Mertens, 2010). Evaluators should use the content of this manuscript as a starting point to improve the cultural responsiveness of their own work regarding the LGBTQ+ community, and as a framework to guide ongoing LGBTQ+ community partnerships in practice. Our recommendations are designed to supplement existing evaluation theories and approaches. Whether you are using empowerment evaluation principles (Fetterman, 2001), conducting a developmental evaluation (Patton, 2011), or using a CIPP (Context, Input Process, Product) evaluation approach (Stufflebeam, 2003), evaluators can and should consider steps they can take to ensure their current and future evaluations are more appropriate and responsive to LGBTQ+ communities.
As a cadre of lesbian, gay, bisexual, transgender, non-binary, queer, and genderqueer authors and accomplices, this work is informed by our own experiences and our deeply personal investment in the liberation and joy of our communities. Our ultimate goal is to imagine and to bring about a world where all LGBTQ+ people can flourish. As evaluators, we see real opportunities for our field to help create that world, and we hope you will join us in that work.
Notes on a Common Language
The term LGBTQ + is used here as a common shorthand. In other contexts, this same population may be referred to as LGBT, LGBTQIA + , 2SLGBTQ + , or SGM (sexual and gender minority), among other terms. In general, individuals who identify as gay, lesbian, bisexual, pansexual, asexual, transgender, Two-Spirit, non-binary, queer, intersex, and other non-cisgender or non-heterosexual identities—including many which do not conform to Western understandings of sex, gender, or sexuality—can be included under this umbrella. We use the term SSOGI (sex, sexual orientation, and gender identity) to refer to data regarding the individual characteristics typically associated with LGBTQ+ identity.
Disparities—Why LGBTQ+ Evaluation Is Needed
The LGBTQ+ community faces numerous well-documented challenges to health and wellbeing compared to their heterosexual and cisgender counterparts. Fundamental to these disparities are the multilevel manifestations of violence, stigma, and discrimination which have exacerbated the impact of minority stress and perpetuated substantial inequities in health and social outcomes for LGBTQ+ people (Bowleg et al., 2003; Hatzenbuehler et al., 2013; Lehavot & Simoni, 2011; Meyer, 1995). For racial/ethnic minority LGBTQ+ people, these stressors and inequities are compounded, leading to more extreme disparities (Bowleg et al., 2003; Bowleg, 2008, 2012, 2020; Bowleg et al., 2013).
The well-established and ever-growing body of knowledge on inequities faced by LGBTQ+ individuals centers Miller's call to action: if we are not invested in understanding the broad experiences of all LGBTQ+ individuals, we are missing countless impactful stories and, as a result, are doing an enormous disservice to an already marginalized population. By excluding certain groups, or not engaging with them in culturally responsive ways, we communicate that those groups are not important to us, and we fail to meet—or even understand—their needs. We can, and we must, do better.
The Evaluator's Role
Evaluators play an essential role in identifying and addressing gaps in LGBTQ+ equity. Through equitable design, cultural responsiveness, and transformative action, LGBTQ+ Evaluation can assess and address the impact of novel policies, programs, and interventions on LGBTQ+ individuals. By educating ourselves about LGBTQ+ issues, examining the experiences of LGBTQ+ people in the evaluations we conduct, conducting evaluations of initiatives designed by and/or for LGBTQ+ people, learning how to ask questions in ways that affirm the presence and dignity of LGBTQ+ people in any given evaluation, and advocating for LGBTQ+ liberation, we can collectively advance an evaluation paradigm that honors, values, and works towards justice for LGBTQ+ people. We share this work with each other, as we acknowledge that collective action supersedes what we can accomplish on our own.
State of Current Evaluation Literature
Unfortunately, there has been an extraordinarily limited focus on LGBTQ+ concerns within the evaluation field to date, and critical questions about the cultural responsiveness of extant LGBTQ+ Evaluations remain. In order to ground our future-facing recommendations within the current and historical state of evaluation's engagement with LGBTQ+ communities, in March 2020 we conducted an informal review of existing evaluation literature, with an eye towards both the presence and cultural responsiveness of current practice with LGBTQ+ populations. The purpose of our search was not to provide a systematic assessment or analysis of the history of evaluation with LGBTQ+ populations, but rather to note broad themes regarding how this population has been acknowledged and engaged to inform recommendations for the field moving forward. We first identified all mentions of LGBTQ+ populations in 18 prominent evaluation journals (Table 1) using the search terms, “gay OR lesbian OR transgender OR non-binary OR genderqueer OR LGBT OR “sexual and gender minority” OR SGM OR bisexual OR asexual OR pansexual OR queer OR MSM OR WSW OR intersex OR two-spirit.” Due to a majority of initial findings focusing on health evaluation specifically, we subsequently searched PubMed for health evaluations which included LGBTQ+ populations using these same search terms. The results and processes of the review are categorized in Table 2 and visualized in Figure 1, respectively.
Flow chart depicting our review of published evaluation literature, and inclusion decisions.
Evaluation-Specific Journals Searched.
Results from Search of Evaluation Journals.
Demographic Data Collection
The informal literature review identified several critical gaps. First and foremost, there was a paucity of evaluations that focus on LGBTQ+ individuals. Likewise, the number of non-LGBTQ+ specific program evaluations that explicitly considered LGBTQ+ populations are in the single digits. We further noted a common theme of inconsistent capture of SSOGI demographic data within the few studies that included these measures; very few studies included gender minority individuals, and often used inappropriate categorization methods to account for transgender people. Most commonly, studies that asked questions about gender modality (Ashley, 2021) asked if participants were male, female, or transgender, a confusing and inaccurate data capture attempt. Often, specific LGBTQ+ demographic identities were grouped together, and participants dichotomized to represent “heterosexual” and “non-heterosexual” groups, eliminating the ability to make meaningful inferences about known disparities within LGBTQ+ groups, for example between bisexual and gay individuals (Beach & Hall, 2020). This practice has the added concern of further erasing some of the most underrepresented members of the LGBTQ+ population, such as transgender, intersex, and Two-Spirit people. Notably, these same populations were among the least frequently mentioned in our informal review of evaluation literature. Additionally, the literature was limited in geographic scope, with almost all identified publications being from the United States.
We argue that these data-related concerns are a direct result of (1) poor, inconsistent, and limited standardized guidance on evidence-based SSOGI data capture within the field of evaluation; (2) limited culturally responsive training for evaluators working with LGBTQ+ populations; and (3) evident disinterest or apathy in the knowledge and lived experience of LGBTQ+ populations themselves. We encourage evaluators to oversample LGBTQ+ populations whenever possible to allow for a more nuanced analysis of data, in addition to following guidance on SSOGI data capture best-practices from LGBTQ+-focused institutions such as the Williams Institute (Herman, 2014); although not infallible, these standardized measures represent an effective starting place to avoid common pitfalls and non-responsive language. At the same time, evaluators must avoid over-reliance on any one standardized approach, and remain attuned to the shifting and contextual nature of LGBTQ+ identity terminology, particularly in a global context (Reisner et al., 2016), as well as to the political and social dimensions of identity itself (Thompson & King, 2015). Even the Williams Institute guidance, cited above, has been appropriately critiqued for its over-emphasis on capturing information about birth sex, as opposed to current gender identity and modality. Approaches which do not unnecessarily ask for birth sex have also been effectively used to generate nuanced demographic categories based on gender (Felt et al., 2021). We also encourage organizations that provide evaluation training, including universities and professional organizations, to offer and make mandatory ongoing learning on the topic of LGBTQ+ Evaluation, with a particular emphasis on culturally responsive LGBTQ+ Evaluation. Training such as these have been offered by members of our authorship team at prior American Evaluation Association (AEA) and CREA meetings, and have helped lay the groundwork for future workshops which we hope to see become more frequent and led by a greater number of evaluators.
Data Analysis and Interpretation
In addition to the limited inclusion of SSOGI measures, we observed numerous inaccuracies and assumptions present in the interpretation of LGBTQ+ evaluation data among reviewed studies. This is a critical concern, as data interpretation can be actively harmful if not done carefully and with historical and theoretical context in (Phillips, Felt, Fish et al., 2020). For example, when discussing elevated rates of smoking in LGBTQ+ youth as compared to non-LGBTQ+ youth, many studies interpreted these patterns to mean that LGBTQ+ youth are attempting to “rebel” or that this group has a greater than average desire to conform and may smoke to fit in. One evaluation of smoking cessation programs states, “In the LGBT community, tobacco use takes on a symbolic function for some, as it is a behavior that counteracts mainstream society's norms” (Treiber, 2011). This reflects a troubling tendency to “other” marginalized persons and to attribute the symptoms of oppression to an invented essential flaw. In other words, such interpretations, while they may seem justified by an uncritical approach to the data, result in greater stigmatization and shame attributed to the behaviors of already marginalized peoples (Phillips, Felt, Ruprecht et al., 2020). Instead, it is more appropriate to interpret these high rates of smoking through both minority stress and intersectionality frameworks (Crenshaw, 1993; Meyer, 1995). It is also critical to contextualize these rates against decades of targeted marketing from the tobacco industry aimed at exploiting LGBTQ+ youth (Stevens et al., 2004).
Once viewed through this lens, the source of this disparity is revealed as structural, rather than individual: undue discrimination and stress, combined with exploitative marketing initiatives, have created a pattern of disparate tobacco use among LGBTQ+ youth. Without a culturally responsive theoretical framework and historical context, evaluators risk reinforcing stigma and blame toward the individual rather than focusing on the systemic root causes of these disparities. There is a long history, particularly within efforts geared towards HIV elimination, of placing blame for health inequities on LGBTQ+ individuals as a moral failing or personal responsibility (Millett et al., 2006). It is our obligation to be cognizant and highly critical of this pattern in data interpretation, as it represents a form of epistemological violence with serious personal and structural consequences (Teo, 2010).
This particular point bears emphasizing, as evaluators must proceed with a cautious, critical lens when interpreting data that may later inform a praxis of resistance. Marginalized populations are often subjected to a deficit-oriented analysis (Garcia et al., 2018). When white and/or cisgender and/or heterosexual counterparts are centered as the “default” or “neutral” perspective in any given quantitative context, other groups are placed at a deficit without acknowledgement of how dominant groups have benefited from “default” social and political structures. As shown in the example of tobacco use above, this can result in biased and harmful assumptions entering into our interpretation and communication of data. To this point, evaluators must note that data cannot speak for themselves (Zuberi & Bonilla-Silva, 2008), and must be willing to become advocates for LGBTQ+ people through their use of and interpretations of data. With time and effort, we believe that our field can and will become more practiced at telling necessary stories of LGBTQ+ love, survival, liberation, and joy.
Content and Outcomes of Interest
As evaluators push to illuminate the stories of LGBTQ+ people, we must especially shine a light on the stories that have been less frequently—or never—told. Although not surprising given the impact of HIV within the LGBTQ+ community, the overrepresentation of work centering HIV was concerning. While we as HIV epidemiologists, evaluators, and researchers do not discount the importance of this work, it is essential to re-emphasize two key points. First, LGBTQ+ communities experience disparities across more than one domain. Second, there is more to the experience of being LGBTQ+ than disparities in health, sociopolitical power, and a burden of HIV. Evaluators have the opportunity to uplift stories of programs that center LGBTQ+ health, pleasure, happiness, success, leadership, and wellbeing, a direction that is not only emotionally meaningful, but also scientifically so: for example, recent work has shown that a focus on pleasure can allow us to discover new opportunities for HIV health programming (Mabire et al., 2019).
This is also an opportunity for evaluators to ensure that the necessary disparities work that does occur is not limited to HIV programming. Crucially, evaluations limited to HIV programming are also, by the nature of HIV transmission risk, frequently limited to specific sub-populations of LGBTQ+ individuals—most commonly gay, bisexual, and other cisgender men who have sex with cisgender men—although there are notable and important examples of work which present exceptions to this pattern. Broadening the scope of our work beyond HIV will also likely help the field to grow in the voices it centers and values from LGBTQ+ communities.
Community Leadership and Evaluator Reflexivity
A major point of concern also identified from the current body of literature is the lack of explicit efforts to ensure participation of LGBTQ+ people in evaluations. LGBTQ+ community perspectives and leadership were sorely lacking in a majority of papers we reviewed. To be clear: as readers, we are neither able to know nor interested in inferring the sex, sexual orientation, or gender of any authors. Rather, we seek to emphasize the value of LGBTQ+ leadership being made explicit within evaluation processes, and the importance of reflexive, reflective thought and self-critique among evaluators (van Draanen, 2017). Very few of the studies reviewed showcased meaningful inclusion of LGBTQ+ community perspectives, either as project consultants, (co-)leaders, authors, or advisory board members, nor was reflection on the authors’ own histories and social positioning included explicitly. Although a disappointing finding, this presents an opportunity for future LGBTQ+ Evaluation work to deliberately and explicitly uplift and center how LGBTQ+ lived experience and critical reflexive thought has the potential to support evaluators’ practice (Hartman et al., 2020).
Limited Definition of Knowledge and Emphasis on “Objectivity”
The gaps and opportunities we highlight above are attuned largely to products of colonial and Western epistemologies (Adas, 2008). Although we believe it is important to speak to and provide suggestions to improve these methods, we also seek to critique these flawed structures. It is necessary to name that our above review is specifically a review of knowledge produced through mechanistic colonial and Western methodologies and, perhaps ironically, conducted using the same accepted mechanistic methods—thus, largely a measurement of solely what is known by colonialist measures themselves (Chambers et al., 2018). Although we have intentionally incorporated our own perspectives and lived experiences as LGBTQ+ authors and accomplices in this manuscript, the body of literature we are speaking to is reflective only of the knowledge deemed valuable by “the academy.”
Relatedly, we note that none of the articles identified discussed the necessity of LGBTQ+ liberation—that is, the removal of structures and systems that have long enabled and enforced the subjugation of LGBTQ+ people. This pattern was largely unsurprising. Generally speaking it is uncommon for peer-reviewed literature to take what could be viewed as an “advocate's perspective”—that is, to position itself as “non-objective,” and open itself for critique associated with having a “political agenda.” This tendency, rooted in Western, colonial understandings of what is considered science, and which forms of knowledge are considered valid and valuable (Iaccarino, 2003; Krause, 2016; Mazzocchi, 2006; Sahai, 2013) is both unproductive and untenable for LGBTQ+ Evaluation. Acknowledging the reality of oppressive structures and their impact on programs, projects, or interventions is good scientific practice, and allows evaluators to engage with the state of our world more meaningfully and honestly. This framing also encourages evaluators to use our work to support necessary change, and to contribute to the removal of structures and systems which are fundamentally unjust. Positioning ourselves firmly in support of LGBTQ+ populations is not only morally and ethically imperative, but scientifically so as well.
Our challenge as evaluators then is to view our work as restorative, but also transformative; to resist harm that is a result of the colonial frameworks nested within the aforementioned historical and theoretical contexts, and to see our work as an opportunity to envision a better world together. Interpretation of data is a critical component of evaluation that, when done carelessly, can result in alienating and inaccurate depictions of the marginalized “other.” However, critical and reflexive approaches to data interpretation can allow the process to become a tool for justice and equity (Garcia et al., 2018). Moreover, approaching data relationally, reciprocally, and as autonomously owned by the community to which it belongs, can help foster better relationships between evaluators and our partners, and can be powerful grounds for collaboration and change (Waapalaneexkweew & Dodge-Francis, 2018).
Concerns and Recommendations for Professional Standards
Although published evaluation literature is an important metric of the state of LGBTQ+ Evaluation, it is not the only one. We must also look to how our governing organizations and the professional guidelines that define the duties of evaluators account for LGBTQ+ experiences. Here again, however, we find an unfortunate lack of content, clarity, and consideration of LGBTQ+ voices. While the AEA guiding principles specifically discuss respect for persons—including persons defined by constructs of gender, power, and underrepresentation—and also explicitly cite the importance of evaluators working towards common good and equity, there are no specific guidelines available which explain what this means in theory, practice, or praxis when it comes to working with LGBTQ+ people (American Evaluation Association, 2018). Similarly, AEA's current statement on cultural competency (American Evaluation Association, 2011), though inclusive of sexual orientation and gender, does not provide a definition or outline for culturally competent, responsive, and humble work (Greene-Moton & Minkler, 2020) with LGBTQ+ persons, nor, notably, does it specifically indicate an inclusive definition of gender encompassing, trans, non-binary, or non-Western experiences of gender. AEA and other professional evaluation organizations can and must do better to ensure LGBTQ+ Evaluation and aspects of LGBTQ+ cultural responsiveness are explicitly defined and included in all statements and policies.
Time for a Change
In sum, the field of evaluation finds itself with a notable challenge. Lacking both a clear moral commitment to equity for LGBTQ+ persons, as well as any materials—either tools or theory—that are explicitly designed to guide this work, many evaluators may reasonably feel trapped. On the one hand, the lack of evaluation work done with LGBTQ+ persons leaves many opportunities for future evaluators. On the other, evaluators are left without support or guidance to determine the development or implementation of LGBTQ+ Evaluations. Given the current non-inclusive, non-affirming, and problematic state of the field of program evaluation with regard to LGBTQ+ people, work which does not intentionally and appropriately push back against these patterns of exclusion, erasure, and ignorance runs the risk of perpetrating harm against LGBTQ+ persons. What then can be done?
Transforming the LGBTQ+ Evaluation Paradigm
Based on our observations above, and on our experiences as LGBTQ+ evaluators and co-conspirators, we call for the development of a new paradigm of “LGBTQ+ Evaluation” to guide the field's restorative work in honoring and uplifting LGBTQ+ voices and experiences. We consider LGBTQ+ Evaluation in this context as encompassing any evaluations that explicitly examine the implementation, outcomes, and impacts of programs and policies on LGBTQ+ populations, whether the programs specifically target all or some segments of the LGBTQ+ population or not.
To achieve this goal, we must begin by stating that the patterns we observed in our review of evaluation literature and standards were not produced in a vacuum. It is not by chance, nor by default, nor by some universal inevitability that LGBTQ+ evaluations are rare or flawed. It is the direct result of pervasive, systemic anti-LGBTQ+ bias that allows cisgender, heterosexual dominated spaces to erase or ignore the experiences and needs of LGBTQ+ persons, and it is pervasive and systemic racism and colonialism that lead to the limited work in the sphere focusing almost exclusively on white LGBTQ+ persons.
Historically, LGBTQ+ populations have been oppressed by state-sanctioned and institutional violence that produces and sustains injustices observable in any number of outcomes related to health, wellbeing, or achievement. Engagement with this history and the current state of anti-LGBTQ+ oppression is critical to Culturally Responsive and Transformative Evaluation with LGBTQ+ populations, particularly given that evaluation's limited attention to and engagement with LGBTQ+ communities has taken place within and was enabled by this history of violence. At an individual level, all evaluators must be aware that the systemic and historic nature of LGBTQ+ oppression further renders evaluators likely to hold unconscious biases against LGBTQ+ populations which have the potential to manifest in our work.
Simultaneously, this reality demands that we work not only towards LGBTQ+ inclusion within specific evaluations, but also see evaluation itself as a means of working towards LGBTQ+ liberation. This central tenet of LGBTQ+ Evaluation is dedicated to creating a world that is more loving, just, and safe for LGBTQ+ people, and all people oppressed on the basis of sex, sexual orientation, and gender. We center this work in an understanding of love as both an intention and an action (Hooks, 2000); a practice grounded in community and geared towards the liberation of all peoples (Teng & Nuñez, 2020). In transforming the current paradigm of LGBTQ+ Evaluation, we must not simply imagine what a world without injustice looks like, but also what a world actively shaped by equity, justice, and love could be, and how evaluation and evaluators can be part of realizing that world.
In this imagining, we embrace reflexivity, reflection, and personal growth. We do this by not only asking ourselves questions, but asking them of each other, too. We turn inwards, and outwards, and ask: “What opportunities exist that I can use to educate myself?” “How can I show solidarity to those who are/have been doing the work I wish to support?” “What are the resources that are needed? What are the resources that I have to offer? How can I contribute them?” “Where in my evaluation work can I uplift mutual aid efforts, solidarity-building, and consciousness raising with peers?” Within the recognition that transforming the paradigm for LGBTQ+ Evaluation requires self-reflection, we also challenge ourselves to think critically about the extent to which we can reflect our way out of oppression—in this fashion, we stress the importance of collective consciousness-raising, dialogue, and action alongside personal, inward work.
Guiding Theoretical Frameworks
In envisioning a new LGBTQ+ Evaluation paradigm, the frameworks of Transformative, Culturally Responsive, and Equitable Evaluation stand out as valuable signposts to guide us. Under a transformative paradigm, members of distinct cultural groups are engaged within the evaluation or research process to produce work that contributes to human rights; it can be applied to resolve questions of ethics, methodology, interpretation, and application that arise during the research or evaluation process (Greene, 2000; Mertens, 2007, 2010; Mertens & Wilson, 2019). Herein, science and politics are inherently inseparable, and thus objectivity is an impossible and dangerous pursuit. As Mertens to Wilson write in a summary of Greene (2000): “What distinguishes evaluation from other forms of social inquiry is its political inherency; that is, in evaluation, politics and science are inherently intertwined” (Mertens & Wilson, 2019). As such, we view the paradigm of transformation as appropriate in articulating a framework to guide evaluation with LGBTQ+ populations.
Culturally Responsive Evaluation (CRE), meanwhile, prioritizes the experiences of traditionally marginalized cultures in evaluation (Bledsoe & Hopson, 2009; Hood et al., 2015). Initially centering the voices of Black communities, CRE has since been adapted and tailored by Indigenous evaluators (Bowman et al., 2015), Latinx evaluators (Escamilla & Nathenson-Mejía, 2003), and other evaluators of color to better center the unique experiences and needs of these communities and to account for intersectional, multicultural experiences within evaluation practice (Kirkhart, 2010). The CRE framework not only promotes meaningful engagement with cultural context, but also empowers marginalized evaluators to lead initiatives within their own communities. As a result, CRE has transformed the evaluation landscape to better reflect racial and ethnic minority evaluators and evaluands. Given this strength, there is substantial opportunity for CRE to address gaps in the evaluation literature and practice regarding LGBTQ+-responsive and LGBTQ+-centered evaluation, particularly for work in partnership with LGBTQ+ communities of color.
Further, the framework of Equitable Evaluation offers valuable pathways through which to approach a praxis of LGBTQ+ Evaluation. Equitable Evaluation holds as core principles that work should be done in pursuit of equity, in response to critical questions about historical and contemporary cultural context, and in a way that is in line with the values of equity work (Equitable Evaluation Initiative, 2017, 2020). Thus, it is well-positioned to guide evaluations which are fundamentally oriented towards addressing the inequities produced by historical and contemporary structures which impact LGBTQ+ communities. In particular, Equitable Evaluation's focus on addressing the systems of discrimination which produce inequity is vital to the creation of a more equitable and just future for LGBTQ+ populations (Dean-Coffey, 2018). Moreover, given its emphasis on multicultural validity (Equitable Evaluation Initiative, 2017, 2020), it is highly capable of accounting for the unique contextual, cultural, and intersectional dynamics of LGBTQ+ identity and experience.
Embedded in our articulation of LGBTQ+ Evaluation through these frameworks is the drive to decolonize evaluation methods that have been given epistemological power by the academy. The phenomena that there is an absolute, objective truth that guides traditional research and evaluation practices is a haunting of settler colonialism which affects how we measure, draw, and define reality. We also note that LGBTQ+ Evaluation must reject epistemically exploitative practices (Berenstain, 2016) that result in taxing labor performed by marginalized people to the benefit of those in positions of power. By disrupting the colonial academic lexicon of capital, profit, and power, we can fill its absence with the presence of community through co-creation, reciprocal evaluation practices, and the re-integration of love and humanity into the knowledge systems we reach for in evaluation. In imagining this new paradigm, we particularly look to and honor the teachings of our Indigenous evaluation family, and their vision in seeing a path for evaluation to grow beyond the Western doctrine (Waapalaneexkweew & Dodge-Francis, 2018).
These guiding perspectives are not the be-all-end-all of LGBTQ+ Evaluation, and instead represent well-established approaches to evaluation and epistemological framing whose core principles lend themselves well to evaluation with LGBTQ+ persons. Thus, they represent what could be considered an effective “entry point” into the articulation of this new paradigm. As stated above, however, it is our intention that LGBTQ+ Evaluation develops in a way that is both flexible and clear enough to guide work under any overarching theory or framework, while still adhering to a set of core principles and values consistent with work in pursuit of LGBTQ+ liberation. For example, we see great potential for LGBTQ+ Evaluation to flourish within a framework of feminist evaluation, given this framework's emphasis on gendered critiques of social arrangements. We further see this as an important opportunity for feminist evaluation to honor its radical feminist roots and reject a feminism tainted by white supremacy and historically harmful or violent towards LGBTQ+ people. Embracing Black, Indigenous, Queer, Transgender, intersectional, and other such traditions of feminism can make feminist LGBTQ+ Evaluation a space where the growth and amalgamation of liberatory evaluation theory could truly flourish.
Priority Action Items
Articulating a body of LGBTQ+ Evaluation theory will require time, and the voices of many evaluators beyond our own. In the immediate term, we propose the following actions to improve the state of evaluation practice with LGBTQ+ persons. This list is incomplete, and will be developed as LGBTQ+ Evaluation develops in the future. The work we can do together includes, among other things the imperative to address the following domains:
Field Standards, Guidelines, Scholarship, and Pedagogy
Update existing professional guidelines to include LGBTQ+ cultural responsiveness and respect explicitly, with an eye towards the national and global diversity of LGBTQ+ experience. Invite evaluators with lived and professional expertise to contribute publications articulating LGBTQ+ Evaluation theory and practice in peer-reviewed and other media. Immediately integrate education on LGBTQ+ identity, equity, history, and liberation movements into evaluation curricula, in order to cultivate an emerging evaluation workforce capable of challenging dominant, hegemonic systems of oppression (Thomas & Madison, 2010). Ensure LGBTQ+ content is featured at all professional evaluation organization gatherings and professional development spaces; reach out directly to experienced LGBTQ+ evaluators to provide this content and include specific requests for such content in calls for submissions. Compensate experts appropriately. Bring LGBTQ+ evaluators into classroom spaces in instructional roles, with an emphasis on those with both personal and professional expertise. Professors may invite guest lecturers or encourage student leadership, while Deans or Department Chairs may appoint professors with such expertise. Compensate instructors appropriately. Appoint LGBTQ+ evaluators to the editorial boards of evaluation journals to ensure higher quality of published work relevant to LGBTQ+ populations. Remember that LGBTQ+ experiences are not homogeneous and an evaluator with personal or professional expertise in, for example, gay men's or bisexual women's health, may not have relevant personal or professional expertise in, for example, transgender or intersex health. Ensure AEA and other evaluation organizations have LGBTQ+ representation in leadership positions and acknowledge (and define) the need for LGBTQ+ cultural responsiveness within all evaluations. Highlight voices from LGBTQ+ evaluators at evaluation conferences, not just through paper presentations, but through workshops specifically designed to provide LGBTQ+ cultural competency skills to all evaluators.
Evaluator Reflexivity and Subjectivity
Critically reflect on how your own power and relationships to dominant social institutions, not just in relation to sex, gender, and sexuality, but also race, ethnicity, ability, class, and culture, have impacted and continue to impact your evaluation work and the broader production of knowledge (Hartman et al., 2020). Challenge others around you to do the same. Become unafraid to ask ourselves questions, and ask them of others when we feel anger, grief, and isolation. Ask ourselves questions in the name of reflexivity, and collective action. Share knowledge and communal resources, resisting privatization and an emphasis on individualism or exceptionalism. Bridge the gaps between theory and practice. Question the process of knowledge creation as much as we question what is known.
Demographic Data Collection and Interpretation
Engage with the work of decolonizing our own understanding of SSOGI, and the ways in which these socially constructed categories show up within evaluation. Pay attention to the assumptions the program(s) you are evaluating make about sex, sexual orientation, and gender. Do they assume gender is binary and static? Are they considering non-white or non-Western conceptions of SSOGI? How can you help the program grow beyond these assumptions? Critically interrogate the idea that any standardized demographic measure represents a “best practice” in demographic assessment—LGBTQ+ identity terminology is contextually and politically specific (Thompson & King, 2015) and should be engaged with as such. Balancing the tensions of ensuring demographic data which is usable, generalizable, accurate, and appropriately respectful of a community's own experiences and autonomy regarding how they are referred to must be our guiding philosophy (Robinson et al., 2020). LGBTQ+ community engagement and LGBTQ+ stakeholder measure validation are necessary elements to any evaluation which hopes to achieve these aspirations. Whenever appropriate and safe to do so (i.e., when you can ensure the anonymity and/or confidentiality of the data shared with you and of the individuals who provided them), include contextually based and community-validated measures of SSOGI within an evaluation to understand the program's impact on LGBTQ+ people. In some contexts, such as within smaller organizations, it may not always be safe to expect people to disclose LGBTQ+ identities, as there is a chance you may inadvertently “out” them to their colleagues (Robinson et al., 2020). Thus, it is incumbent upon evaluators to ensure that this information will be appropriately safeguarded. Ensure that your interpretation of data is attuned to the larger structural forces of power and oppression at play. Theories of minority stress (Meyer, 1995), intersectionality (Crenshaw, 1993), and stigma as a fundamental cause (Hatzenbuehler et al., 2013) are all valuable frameworks to help understand disparities as the products of injustice and social power structures, rather than inherent deficits.
Community Engagement
Meaningfully include populations you are studying at every step of the evaluation process, not just as community advisors, but also as part of the evaluation team (Miller et al., 2013, 2016). This may mean challenging traditional academic hierarchies and traditional modes of knowledge curation, but it is essential for effective, transformative LGBTQ+ Evaluation. Create a work environment that is actively and explicitly committed to LGBTQ+ liberation. Make spaces safe for LGBTQ+ evaluators, evaluands, and guests without them having to ask for it. Invest in LGBTQ+ competent trainings, policies, and a commitment to workplace culture shifts that decenter cisgender, heterosexual dominance. Work with LGBTQ+ communities, not for. Using “with” instead of “for” is to acknowledge that we must be critical of the language that we use. To act for is to assert dominance and superiority. By aligning with, white, cisgender, heterosexual evaluators retract ownership over the liberation of LGBTQ+ people and relinquish any attempt to maintain power over these communities. Engage with LGBTQ+ communities and organizations in your immediate context. Identifying LGBTQ+-led organizations and the work they do will provide you greater insight into the immediate needs of these communities and can supplement the process of relationship building.
Liberation and Becoming an Accomplice
Understand that oppressed groups have historically been told that change will come—eventually. Eventual change has been promised over decades, but this is not enough. Evaluators must commit to enacting change now. We must be responsive to the demands and needs of LGBTQ+ communities, not vice versa. Familiarize yourselves with the history and present state of LGBTQ+ oppression and discrimination, as well as the history and present state of LGBTQ+ liberation movements. Situate your work within the struggle and prepare yourself to join it. See and feel the joys and successes of LGBTQ+ liberation. Celebrate them. Seek out opportunities within your evaluation to advocate for social and structural change in favor of LGBTQ+ populations. Do not stop there. Seek out the same opportunities outside of your evaluation, and outside of your day job entirely. Allow the internal and communal work to be futuristic. Let our imaginations lead this life-long practice in the deconstructions of harm and violence, and the reconstruction of a better world. When you find yourself thinking from a place of deficit, encourage yourself to start instead from a place of presence. We shift from “This isn't possible. This will never work. This isn't feasible.” to “What if this works? What are the conditions needed to make it work? How can I experiment, and let go of the need for this to “work” or “be successful”?”
Remembering You Are Not Alone
For LGBTQ+ evaluators, the work of pressing for competent LGBTQ+ Evaluation can feel isolating, frustrating, and even painful or traumatizing. For non-LGBTQ+ evaluators, it can often feel confusing, challenging, and unclear. Through working together and centering teachings of love as a community practice, we remember that we are not alone, and that we are powerful together (Teng & Nuñez, 2020). LGBTQ+ evaluators have been working in the field for a long time. Though our work is not always celebrated or uplifted, it is never absent. In moving towards a new future for LGBTQ+ Evaluation, we must look to and learn from work which is already putting the principles we care about into practice. For example, recent work by Miller et al., (2021), which takes a transformative approach to LGBTQ+ Evaluation, is worth looking to as a source of inspiration. Reach out to those doing the work of LGBTQ+ Evaluation if you find yourself unsure of where to start, or in need of more resources. If you are unsure who you can contact, the AEA LGBT Issues Topical Interest Group (TIG) is a great place to start.
Conclusions
The multidimensional oppressions that LGBTQ+ populations face are essential to name and understand, and the failure of our field to consider them is necessary to make explicit. It is also vital to understand the multidimensionality of liberation itself, and even more vital to do the radical work of (re)imagining a better future. In other words: there is a significant amount of work to be done, and it is incumbent upon all of us to do it. The state of LGBTQ+ Evaluation will not be changed overnight, nor is it feasible to expect individual evaluation agencies, teams, individuals, or community organizations to independently take on the burden of leading such a change. Rather, effective LGBTQ+ Evaluation will require a genuine, transformative paradigm shift within the evaluation field, encompassing everything from pedagogy to practice, and all activities in between. Each of us must be active in this work and engaged in direct resistance to complicity.
Like so many things, transformation is non-binary and cannot be conceptualized as an either/or function wherein we are either failing or succeeding. The journey of growth, reflection, and action towards LGBTQ+ inclusion and liberation for evaluators, and evaluation more broadly, will take time, energy, and uncomfortable conversations. We embrace that discomfort and encourage all readers not to shy away from it. We invite other evaluators, community members, and LGBTQ+ leaders to lend their voices to this conversation, to the difficult conversations to come, and to the ongoing struggle for LGBTQ+ liberation.
Readers are encouraged to contact all authors directly to discuss this piece, particularly those who are invested in helping to articulate a new paradigm of LGBTQ + Evaluation. In addition, we encourage LGBTQ + evaluators, evaluands, and our co-conspirators to get involved with the LGBT Issues TIG at AEA; those interested in the TIG may reach out directly to the corresponding author, Dr. Gregory Phillips II.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship and/or publication of this article.
