I Can’t Even Think Count Straight: A Queer Revolution in Survey Research?

Assumptions/Attachments/Disinvestments

To illustrate this point, it’s helpful to consider the similarities and differences in how social scientists in various disciplinary contexts generally conduct survey research. In American sociology, the historical gold standard of survey research has been the nationally representative, probability sample (Luker 2008). One advantage of representative datasets that are collected annually is that this allows for the study of change over time; we can look to the General Social Survey, for example, for evidence of how abortion attitudes have shifted (or not) across several decades. These kinds of datasets are rarely produced by individual researchers or small labs. They are instead the product of massive research operations at major research centers, such as NORC or the U.S. Census Bureau. In this research context, individual researchers who may want to probe these datasets to test hypotheses and/or ask questions about sexualities (or gender, or race) are beholden to data they did not create and questions they did not ask. The sloppy and often crude ways gender identity and sexual orientation—not to mention race and other relevant identity variables—are asked on these national surveys artificially limit what can be learned about the population and arbitrarily constrain, in particular, how nonnormative sexual and gender identities are represented (Baumle 2018). Survey methodologists informed by queer and trans perspectives have made many important scholarly interventions in the past several decades (Westbrook and Saperstein 2015). But the impact of these developments in queer and trans survey research is limited by the extent to which these insights are incorporated into the datasets most commonly used by sociologists, demographers, and other social scientists who are concerned with data that represents demographic trends in the broader population.

In psychology, whose disciplinary norms are less organized around representative datasets, researchers are far more accustomed to creating their own survey research instruments even if they use existing and standardized measurement tools. ¹ In other words, because psychologists often rely on convenience (i.e., nonprobability) samples of undergraduate students or samples collected using online surveys distributed via social media or collected via crowdsourcing networks (e.g., Amazon’s Mechanical Turk [MTurk], Prolific, etc.), they often exercise far more power over the type of data they collect. This does not mean their surveys are invariably or even typically higher quality, but it does mean that they can nimbly tailor questions to suit specific research aims. One advantage of this approach is that surveys can be adjusted to address contemporary and highly dynamic concerns, such as the emergence of a term (e.g., “pansexual”), or its decline in use or cultural salience (e.g., “heteroflexible”). Previously validated scales, such as measures of internalized heterosexism, can also be easily tweaked to function in different populations or replaced to help answer new questions. For example, an item such as “I feel discriminated against as a gay man” used in one study could become “I feel discriminated against as a bisexual man” in a follow-up study. A drawback, however, is that it is difficult to make meaningful comparisons across nonequivalent surveys, which hinders the capacity to confidently assess changes over time for items or questions that are inconsistently asked. Even slight differences in the format (e.g., “Check all that apply” versus “Please choose one”) or content (e.g., “Homosexual” versus “Gay”) in a demographic item can make comparisons spurious or impossible. Though convenience samples are not statistically representative of the population of interest, that may not be the goal of the study, and therefore not a fundamental limitation. Instead, the findings may be framed as logically or analytically representative (Luker 2008) insomuch as they illuminate the substance and contours of a dynamic or construct under investigation (akin to the goals and strengths of qualitative inquiry).

So, on the occasion of the inaugural volume of Sex & Sexualities, an important moment in the history of the sociology of sexualities, I suggest that our commitment to “better” methods would benefit tremendously from sexualities scholars marking and naming our tacit investments (Pfeffer 2018) and disciplinary assumptions. These are practices queer theorists have taught us for generations (Stein and Plummer 1994), but they can be hard to hold onto in the epistemic currents of our disciplines, which sometimes function like a riptide (Grzanka and Cole 2021). Employing relatively straightforward statistics, two recent examples from quantitative sociology (Mittleman 2023; West et al. 2024) suggest the dividends of such a move, especially in a time of increased enthusiasm for the inclusion of sexuality and gender variables in national and international datasets (Wang 2023).

Though statistics are regularly positioned as more objective tools than qualitative methods (Luker 2008), the angle at which one approaches quantitative data can obscure or illuminate what’s actually going on statistically. To illustrate this point, Mittleman (2023) quantitatively explored a theoretical axiom of queer studies that is under-investigated in mainstream population research: sexual fluidity, or the extent to which individuals’ sexual identities and attractions change over time (Diamond 2009). Using five waves of annual data collected since 2013 among a nonclinical population of U.S. adults (N = 17,712 from the Population Assessment of Tobacco and Health [PATH]), Mittleman explored the extent to which respondents’ reported identities and attractions changed over time. He found that among the over 2,500 total identity changes reported in the data, the vast majority (86%) of changes in identity were between three categories: straight, bisexual, and “something else.” Conversely, he found far fewer identity changes among those who identified as gay/lesbian, which was a more “stable” category.

Mittleman (2023) then tested whether accounting for sexual fluidity might change our conclusions about observed health outcomes for sexual minority adults; it did not. However, sexually fluid adults reported significantly worse outcomes than those who consistently reported that they were straight. Sexually fluid respondents were far more like stable sexual minorities than those who were stably straight. Moreover, the failure to account for sexual fluidity grossly underestimated the overall size of the population of sexual minorities in the data, which Mittleman notes is a prime goal of demographic research. So, while accounting for sexual fluidity did not drastically change the conclusions drawn from the PATH survey regarding health inequities, measuring sexual fluidity did unsettle assumptions about what demographic knowledge can be gleaned from cross-sectional (i.e., single timepoint) assessments of who counts as LGBQ+. Mittleman concludes that no single survey administration can accurately quantify how many people “are” queer. Instead, he argued demographers should dedicate attention to the dynamics of social complexity that are already well-established in the sexualities literature. So, the problem for Mittleman isn’t in how we ask about sexuality per se, but how we cut the data and the way we seek knowledge from it.

Relatedly, West et al. (2024) found that national survey data may misdirect our conclusions about sexual minority health outcomes, particularly if measures arbitrarily constrain options for sexual orientation identity. West and colleagues used five consecutive years of the National Survey of Family Growth (2015–2019) to test whether differences in how sexual identity was asked would significantly affect tests of subgroup differences (straight vs. everyone else) on various health outcomes, including substance use and reproductive health. This national survey was randomly distributed with two different versions of the sexual identity question: one 3-question option (gay, straight, bisexual) and one with four choices, including “something else.” They found that, for both cisgender women and men, there were statistically different conclusions on 15% to 16% of the outcome measures between the three- and four-category versions of the survey questionnaire. When using the four-category version, they found significant differences in a variety of dependent variables (three each for women and men), including wanting a/another child, cigarette and marijuana use, and having ever contracted a sexually transmitted infection. Whereas the exclusive straight/gay/bisexual triad reflects heteronormative ideas that “bisexual” captures anyone who is not monosexual in their attraction or identity, the simple addition of a fourth “something else” category allowed West and colleagues to draw more nuanced and statistically meaningful conclusions from the data. Echoing Mittleman (2023), West et al. (2024) do not suggest the four-category sexual identity question is a fix-all solution for measuring sexual identities. They instead underscore the need for all survey researchers to consider “fluidity and complexity when it comes to the construct of sexual orientation identity” (p. 123).

Mittleman (2023) and West et al. (2024) point toward the wily issue of construct validity—or the confidence we have that we’re measuring what we think we’re measuring—when it comes to sexual identity and related constructs. While perhaps a bit “unsexy” as a research topic, construct validity is of profound consequence for the knowledge we create about sex and sexuality with our survey instruments. A perennial problem across the social sciences is that virtually all commonly used survey instruments are developed with and normed upon majority- or dominant-group populations. Take the case of the three-category (gay, straight, and bisexual) sexual identity item West et al. (2024) examined: ask any person under age 25 if that would be sufficient, and you are likely to receive an eye roll. Survey instruments have historically been developed by (older) experts who are historically included in exclusionary institutions. The epistemic exclusion (Settles et al. 2021) of people of color and other marginalized social groups from knowledge production in social science has meant that the social life we’re assessing with our science is largely based on measures reflecting White, cis, straight, male, able-bodied, older, educated, and American perspectives. How might this systematic exclusion undermine our aspirations to better understand both normative and nonnormative sex and sexualities?

In feminist psychology, Galupo et al. (2014) asked sexual minorities (N = 285) what they thought of the Kinsey Scale and the Klein Sexual Orientation Grid, two of the most widely used survey measurement tools to assess sexual orientation/identity. Their respondents offered an extensive qualitative critique of both tools, especially the bipolar Kinsey Scale, along a number of structural and content-based dimensions. They found the scales (1) lacked comprehensiveness in terms of capturing relevant information about sexuality (including sexual fluidity), (2) were dichotomously organized in ways that effaced sexual experiences outside of binaries, and (3) failed to accommodate the ways sexual desires, identities, and behaviors are motivated by things besides sex and gender (such as personality and other ephemeral characteristics). These findings suggest that, at least in this convenience sample, both Kinsey and Klein’s measures lacked construct validity.

Similarly, critical methodologist McClelland (2017) demonstrated how survey tools can (mis)lead researchers into a false sense of measurement validity. Consistent with Cantril’s self-anchored ladder approach popularized in the 1960s in his nationwide survey of American well-being, ² McClelland asked respondents to rate themselves on a 10-point scale of “overall sexual satisfaction” and to define the anchors, as well as the midpoint. In her purposively derived sample of racially and sexually diverse undergraduate students (N = 34), McClelland found some degree of consistency in sexual satisfaction ratings. For example, many respondents used the term “mind-blowing” to describe the best sexual satisfaction, and respondents commonly organized the ladder in terms of emotions and sexual outcomes. However, she also observed meaningful differences in how respondents conceptualized and structured the anchors and midpoint of the scale. Some respondents rated their sexual satisfaction in terms of partner type, time, and who was satisfied; others stressed orgasm (e.g., present or not) or the kind of sex (e.g., oral and anal). For some, a lack of sexual satisfaction involved pain, whereas for others it was the absence of pleasure. McClelland’s findings suggest that while a standard sexual satisfaction rating scale may be reliable, it may be invalid to the extent that respondents mean very different things when they rate their level of satisfaction. By ceding power to respondents, she suggests the self-anchored ladder encourages “conceptual disruption”: a skepticism toward extant research tools and taken-for-granted practices, as well as the embrace of disruptive methods that displace normative standards in each exchange for tools that foreground those perspectives and experiences that have been historically marginalized—in social science, as well as in society.

Galupo and McClelland’s respective projects underscore the extent to which survey research tools can give us false impressions of both (a) the ideological distance between us, as researchers, and our participants, as well as (b) the homogeneity of perspectives among our participants. In other words, if a measurement tool is initially found to be reliable and valid, it can lead us to think we share an understanding of a term or construct—such as sexual pleasure—with most or even all our respondents. However, our respondents (i.e., sample of the population of interest) may actually define this construct in ways that are quite different from how we have conceptualized it. There also may be critical distinctions among our participants such that “sexual pleasure” to one subset or respondents is completely orthogonal to another subgroup’s understanding of the term. This is not the same as measuring actual variance in responses, such as different rates at which individuals report experiencing sexual pleasure, which is something we want to know if it exists. If we think we’re measuring “sexual pleasure” but we’re really measuring something else entirely, this is error, whereby we claim to know something we do not know.

Another way to mitigate these threats to construct validity is to develop survey instruments that are rooted in the lived experiences and actual language of stigmatized minority groups, who are experts in their own lives. In my own research on sexual orientation beliefs—or what everyday people believe sexual orientation is, where it comes from, and the extent to which it can change—my colleagues and I inverted the typical way that prejudice and discrimination measures are developed. Rather than base our Sexual Orientation Beliefs Scale (SOBS; Arseneau et al. 2013) on the worldviews of straight and cis people, we began developing and validating the scale using ideas from queer and feminist theory and the thoughts and perspectives of queer and trans people. This meant that the items we proposed for the scale incorporated social constructionist themes—such as the notion that sexual orientation is culturally and historically contingent—with more depth and frequency than they appear in heteronormative discourse about sexuality. We also used exclusively queer and trans respondents to conduct the initial validation of the instrument and then moved on to study how the measure might perform psychometrically with straight respondents. Every aspect of the SOBS’s development was designed to buck the commonplace, yet methodologically suspect, ways most psychometric work on prejudice is conducted. We wanted to start with the marginalized groups who are the targets of discrimination and anchor the instrument in their subjectivities, rather than those of the dominant group (Grzanka 2016). Ultimately, this allowed us to develop an instrument that moves beyond a reductionist “born this way”-or-not binary and to capture elements of essentialist and social constructionist beliefs that are not exclusively about how straight people perceive or discriminate against sexual minorities.

From Queer Methods to Queer Data?

Though none of the empirical projects I described in the previous section explicitly invoke the term “queer method,” they circulate in a time where increased scholarly attention has been paid to the potential for queer theory to catalyze consequential methodological innovations and contestations, as well as to destabilize the divide between “theory” and “method.” Four celebrated volumes by Browne and Nash (2010), Ghaziani and Brim (2016, 2019), and Compton, Meadow, and Schilt (2018) represent the gestalt of the queer methods movement, which has importantly been led by several key figures in the sociology of sexualities. These contributions have challenged social scientists and humanists to reconsider the role of empiricism in the study of all sexualities (including heterosexualities; see Silva 2017), as well as to deepen the field’s engagement with other critical movements, including intersectionality and anticolonial theories. Though the queer methods literature certainly implicates questions of measurement and the refinement of survey tools, the thrust of this work has more commonly been to address the epistemic contradictions of subjecting the messiness of all that is “queer” to the disciplinarity and order of “method” (Ward 2016). Ghaziani and Brim (2019), for example, echoed sociologists Stein and Plummer (1994) in their call for a queering of method as opposed to a method for studying queer (people, things). Plainly, they write, “Methods are queered when we use the tenets of queer theory to tweak or explore what is possible with our existing procedures” (Ghaziani and Brim 2019:15; emphasis original). The literature on queer methods is littered with familiar rhetoric of queer studies, including terms such as displacing, disrupting, rethinking, interrogating, and resisting—all of these typically directed at violent disciplinary norms and social institutions. But it would be a strong mischaracterization of the queer methods discourse to suggest that the thesis of such a project is the rejection of method, particularly social scientific methods, as fundamentally at odds with the transdisciplinary project of queer studies. As literary scholar Love (2019) notes, “To see one’s practices as beyond method and utterly undisciplined is a failure to reckon with queer scholars’ position in the university; it fails to recognize the violence of all scholarly research—even its most insurgent and intimate forms” (p. 30). Instead, I read and suggest using queer methods as a means to cautiously and even optimistically approach knowledge production with an informed skepticism—to return to McClelland’s (2017) use of the term—toward inherited tools and practices.

Queer data studies is an exciting complement to the queer methods movement, drawing on critical science and technology studies approaches to consider the travels and consequences of knowledge about queer lives. Similar to queer methods, Keilty (2023) notes in the introduction to Queer Data Studies that scholars working in this emerging field are not pro- or anti- data, but instead take data as a site of potential that is always mediated throughout relations of power (see also Budnick, Pao, and Velasco [2025] in this issue). Wang’s (2023) contribution is helpful for considering how sociologists of sex and sexualities might navigate the international zeitgeist for counting up queer bodies in various forms of surveillance, including governmental and activist projects. Rejecting the notion of a data “gap” as a colonial, extractive metaphor for describing what we do not know about queer people and their lives, Wang suggests “situated indications” as an alternative analytic: “data and narrative practices that bounce between situations in an uneven topography to elevate positioned LGBTI experiences and find common contexts of struggles” (2023: 176). Wang evades the question of “good” or “bad” data—and avoids data nihilism or what he calls “scholarly narcissism” (p. 168)—and reframes the stakes of “global datafication” in terms of a dynamic process of indications: “What’s queer is not to be found within the apparatuses and tools (for enumeration or interpretation) but in their multiple, unintended, perverse and thus queer uses” (p. 175). For Wang, provocatively, the question for queer data studies is not so much “What do we think we (can) know?”, but “What do we hope to do with this knowledge?”

I began this brief meditation by naming my skepticism toward scholarly talk about refining survey tools and building better instruments, even though, admittedly, that is a lot of what occupies my time and thoughts when it comes to the sociology of sexualities. Queer methods and, now, queer data studies offer critical resources for avoiding the twin trapdoors of enlightened positivism or universal pessimism. In conclusion, rather than desire the perfect tool, analyze ourselves into statistical oblivion, or leverage our political futures with data we can never fully control, I think a methodological calculus informed by queer methods can reframe our investments thusly: