Abstract
This article presents design principles and practical steps for web-based Q methodology surveys. Drawing on the experience of two online Q studies, we discuss theoretical concerns, sort and survey design, software programs, and issues in researcher–participant engagement. We argue that opening Q methodology to online modes of data collection is important to capture greater diversity in social perspectives and geographies.
Introduction
Since 2000, social scientists have increasingly used Q methodology as a tool to investigate social perspectives and human subjectivity. Q methodology (hereafter Q) is a mixed-methods technique that seeks to identify shared affinities or divergences about a particular topic (Brown 1980; Eden at al. 2005; Ramlo 2016; Watts and Stenner 2012). Despite the growth of online surveys in general, Q methodologists tend to emphasize face-to-face methods of data collection (known as the “Q sort”), which potentially limits its uptake as a social science methodology. We ask: What does it mean—theoretically and practically—to collect Q sort data at a distance?
Our goal here is to present design principles and practical steps for implementing an online Q sort, the data collection technique at the heart of Q methodology. We note theoretical concerns, sort and survey design, software programs, and issues in researcher–participant engagement. Ultimately, we argue that opening Q methodology to online modes of data collection is important to capture greater diversity in social perspectives and participant geographies.
Details of Q Methodology Studies by Authors.
Theoretical Concerns and Survey Design
Early examples of distance-based Q data collection include postal surveys (Watts and Stenner 2012:87). In general, the Q community strongly encourages face-to-face Q sorts, which they argue facilitates “engagement with the items” and allows direct observation and interaction with the researcher (Besika et al. 2018; Jeffares and Dickinson 2016; Watts and Stenner 2012; Wolf 2020). In a Q method training workshop at the 2018 ISSS conference, for example, we received the advice to “do Q in person, because that human connection is imperative.” Online data collection is gaining acceptance by some Q practitioners (Besika et al. 2018; Jeffares and Dickinson 2016; Watts and Stenner 2012), but it tends to be the exception and not the norm.
In part, the face-to-face preference is rooted in how Q methodologists theorize and measure subjectivity, reflecting its epistemological origins in the discipline of psychology. Subjectivity, in Q terms, is the sum of behavioral activities that constitutes a person’s point of view, which is revealed in the act of the “Q sort” (Watts and Stenner 2012). The sorting procedure asks participants to rank-order selected stimuli (typically statements) relative to all others, resulting in a Q sort grid. Researchers then use completed sorts to “measure individuals’ affinities with those views, as well as similarities and divergences amongst individuals” (Eden et al. 2005:414; see also O’Neill et al. 2013; Robbins and Krueger 2000).
Software Programs for Online Q Sorts (Trialed as of July 2018).
Second, there is evidence to suggest that asynchronous online sorts may result in a more honest expression of subjective preferences, since participants are not under the direct gaze of the researcher (Besika et al. 2018). Third, previous validation studies demonstrate that computer-based Q sorts are equally reliable and valid as traditional paper methods (Exel et al. 2015; Reber et al. 2000).
Fourth, in instances where privacy and confidentiality are concerns, online sorting may allow for greater anonymity. Fifth, online sorts may be more convenient for participants, especially if they allow participants to pause and return at their own schedule. Finally, online Q sorts may reduce the costs of data collection (e.g. labor time, travel), which opens the method to researchers with less funding support. Despite these advantages, we recognize limitations and necessary adjustments, which we discuss below.
Practical Steps for the Online Q Sort
Several design issues are paramount in online Q sorts. First, the number of items to sort should be conservative, because the fixed screen size of computer screens (or phones/tablets) can make the holistic arrangement of too many stimuli frustrating or impractical for participants. A well-designed Q sort should represent the sum of discourse on the research topic (Eden et al. 2005). In practice, the number of stimuli selected depends on the study and can be as low as 25 or high as 80 (Watts and Stenner 2012), but 35–40 is typical (Jeffares and Dickinson 2016). Practically speaking, online Q sorts should not exceed 36–40 stimuli (Jeffares 2015). In Study 2, for example, our pilot testers reported fatigue and frustration in the sorting process. We reduced our item count (from 48 to 43 items in a 9-point Q grid) after two pilot rounds. In our experience, keeping sort items minimal, pilot testing with a variety of users, and visualizing the Q grid in different software programs were critical design steps.
Second, written instructions for online Q sorts must be clear, accessible, and replicable (Watts and Stenner 2012). We found this step to be especially important with self-administered online Q sorts (like ours), because the researcher is not immediately available to explain procedures. Alternatively, researchers could (1) provide short videos or animations that explain the steps of Q sorting, including captions in different languages or to aid hearing-impaired participants; or (2) arrange a synchronous meeting (e.g. using Zoom, Skype, WebEx, telephone) to explain sorting procedures.
Third, after the initial correlation and factor analysis is complete, post-analysis interviews with participants can improve internal validity of results, as done in Study 1. Interviews allowed participants to elaborate on their reasoning and provide alternative interpretations of factors in a way that does not rely only on researcher knowledge or statistical norms and averages to evaluate factor validity (Brown 1980; Ormerod 2017, 2019; Watts and Stenner 2012).
We expect Q sort software will evolve in terms of capabilities and cost. Table 2 lists the programs we assessed and trialed as of 2018. When evaluating Q sort software options, consider the following questions: How is the Q sort presented holistically? (e.g. forced or free distribution?) How easy is it to make changes? Can the participant see their rank-order and make changes relative to other statements? What options exist for pre- or post-sorting questionnaires? (e.g. participant reflections or demographic data) Which web browsers work best? What measures are used to protect participant identity? Who owns the rights to the sort data? Which programs comply with data management guidelines? What is the cost? Language compatibility? Data output?
In sum, the online Q sort holds some strategic advantages. In our experience, we recruited participants from very different geographies (68 people from eight different countries and five U.S. states), an advantage in exploring diverse discourse coalitions. Costs and environmental impact of data collection were minimized; convenience was enhanced for participants. Data management was streamlined: Sorts, demographics, comments, and metadata were immediately recorded, reducing human error in manually collecting and inputting data.
At the same time, a major disadvantage is the inability to directly observe the sorting process—a barrier that could be overcome with synchronous video conferencing (e.g. Skype, WebEx) and appropriate permission. Other considerations include limited access to technology and internet service, additional time needed in the pilot process, and data management/privacy compliance.
Conclusion
The use of online Q sorts is a promising approach to uphold Q methodology principles while resolving some of the more persistent challenges of face-to-face research: including cost, convenience, timing, researcher influence, and the suspension of in-person data collection following COVID-19 public health guidelines and university regulations.
In moving forward, we suggest that online Q methodology also demands new reflexive strategies of research. A virtual Q sort cannot remove or eliminate research bias; it simply implicates the researcher in new ways (Sneegas 2020). Careful attention to recruitment, user instructions, software choice, piloting, sorting, data privacy, and closing the loop of researcher–participant engagement are important design considerations. Finally, the development of open-source software for Q sorts should be a major priority, to remove cost barriers and democratize uptake by the research community. Ultimately, online data collection of Q sorts should not preclude the researcher’s capacity to engage with participants; and moving forward with online Q is an individual choice aided by technologies that we hope benefit the research community at large.
Footnotes
Acknowledgments
We thank Dylan Brady, Kate Shields, and Gretchen Sneegas for helpful feedback during pilot phases of this research.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This material is partially based on research supported by the U.S. National Science Foundation under Grant #1655038. The usual disclaimers apply.