Short Take: Sorting at a Distance: Q Methodology Online

Abstract

This article presents design principles and practical steps for web-based Q methodology surveys. Drawing on the experience of two online Q studies, we discuss theoretical concerns, sort and survey design, software programs, and issues in researcher–participant engagement. We argue that opening Q methodology to online modes of data collection is important to capture greater diversity in social perspectives and geographies.

Introduction

Since 2000, social scientists have increasingly used Q methodology as a tool to investigate social perspectives and human subjectivity. Q methodology (hereafter Q) is a mixed-methods technique that seeks to identify shared affinities or divergences about a particular topic (Brown 1980; Eden at al. 2005; Ramlo 2016; Watts and Stenner 2012). Despite the growth of online surveys in general, Q methodologists tend to emphasize face-to-face methods of data collection (known as the “Q sort”), which potentially limits its uptake as a social science methodology. We ask: What does it mean—theoretically and practically—to collect Q sort data at a distance?

Our goal here is to present design principles and practical steps for implementing an online Q sort, the data collection technique at the heart of Q methodology. We note theoretical concerns, sort and survey design, software programs, and issues in researcher–participant engagement. Ultimately, we argue that opening Q methodology to online modes of data collection is important to capture greater diversity in social perspectives and participant geographies.

We discuss insights based on our combined experience with two separate studies that utilized online Q sorts (Table 1). The first study, conducted in 2014, examined the subjectivity of people with interests in planned potable water reuse in the southwestern United States. The second study, conducted in 2018, queried participants from across North and South America about international collaboration in global change science. This article also benefits from insights gained at two Q annual conferences (in 2016 and 2018) and the ISSS/Q Methodology discussion list (q-method@listserv.kent.edu).

Table 1.

Details of Q Methodology Studies by Authors.

	Study 1	Study 2
Number of statements (Items)	30	43
Number of participants	41	29
Survey method	Online, POET Q	Online, QsorTouch
Questionnaire (Before or after survey)	Before and after	After
Analysis package	PQMethod	PQMethod
Number of factors	2	4

Theoretical Concerns and Survey Design

Early examples of distance-based Q data collection include postal surveys (Watts and Stenner 2012:87). In general, the Q community strongly encourages face-to-face Q sorts, which they argue facilitates “engagement with the items” and allows direct observation and interaction with the researcher (Besika et al. 2018; Jeffares and Dickinson 2016; Watts and Stenner 2012; Wolf 2020). In a Q method training workshop at the 2018 ISSS conference, for example, we received the advice to “do Q in person, because that human connection is imperative.” Online data collection is gaining acceptance by some Q practitioners (Besika et al. 2018; Jeffares and Dickinson 2016; Watts and Stenner 2012), but it tends to be the exception and not the norm.

In part, the face-to-face preference is rooted in how Q methodologists theorize and measure subjectivity, reflecting its epistemological origins in the discipline of psychology. Subjectivity, in Q terms, is the sum of behavioral activities that constitutes a person’s point of view, which is revealed in the act of the “Q sort” (Watts and Stenner 2012). The sorting procedure asks participants to rank-order selected stimuli (typically statements) relative to all others, resulting in a Q sort grid. Researchers then use completed sorts to “measure individuals’ affinities with those views, as well as similarities and divergences amongst individuals” (Eden et al. 2005:414; see also O’Neill et al. 2013; Robbins and Krueger 2000).

In our experience, the online Q sort holds true to the underlying principles of Q—including adhering to an ontological commitment of holism and relational interpretation—while providing strategic advantages for research design and broader uptake of the methodology. First, we retained the structured form of the Q sort, which satisfies the methodological assumption that the act of holistic sorting captures a snapshot of human subjectivity (Sneegas 2020). In our studies, participants independently sorted and ranked statements within the typical grid structure using web-based software (Table 2). During the sorting process, participants could visualize their grid, reflect on their rankings, and discuss their impressions in a post-survey questionnaire—which, alternatively, could be completed in a remote video call or face-to-face.

Table 2.

Software Programs for Online Q Sorts (Trialed as of July 2018).

Program name	Developer/administrator	Pros	Cons
Q-SorTouch	Alessio Pruneddu	● Reasonable cost ● Good user interface ● Satisfactory user experience ● Straightforward output method for post-survey data ● Accessible to participants online ● No IT experience needed ● Utilizes grid	● Difficulties providing on screen instructions ● Interface has limited capabilities with adjusting to varying screen sizes ● Does not support data analysis

POET Q	Steven Jeffares	● Free● No IT experience needed● User-friendly interface● Utilizes grid	● Closed software requiring permission from does the health services management centre at the University of Birmingham● Does not support data analysis

Q-Assessor	Stan Kaufman	● Can perform data analysis● Accessible to participants online● No IT experience needed● Utilizes grid	● High cost● Limited subscription time● Limited user interface

Flash Q	Christian Hackert and Gernot Braehler	● Free and open source● No statement limits	● Requires computer programming knowledge● Limited program documentation● Limited user interface● Set-up time can be extensive● Requires additional steps to make survey accessible to participants● Does not support data analysis

Second, there is evidence to suggest that asynchronous online sorts may result in a more honest expression of subjective preferences, since participants are not under the direct gaze of the researcher (Besika et al. 2018). Third, previous validation studies demonstrate that computer-based Q sorts are equally reliable and valid as traditional paper methods (Exel et al. 2015; Reber et al. 2000).

Fourth, in instances where privacy and confidentiality are concerns, online sorting may allow for greater anonymity. Fifth, online sorts may be more convenient for participants, especially if they allow participants to pause and return at their own schedule. Finally, online Q sorts may reduce the costs of data collection (e.g. labor time, travel), which opens the method to researchers with less funding support. Despite these advantages, we recognize limitations and necessary adjustments, which we discuss below.

Practical Steps for the Online Q Sort

Several design issues are paramount in online Q sorts. First, the number of items to sort should be conservative, because the fixed screen size of computer screens (or phones/tablets) can make the holistic arrangement of too many stimuli frustrating or impractical for participants. A well-designed Q sort should represent the sum of discourse on the research topic (Eden et al. 2005). In practice, the number of stimuli selected depends on the study and can be as low as 25 or high as 80 (Watts and Stenner 2012), but 35–40 is typical (Jeffares and Dickinson 2016). Practically speaking, online Q sorts should not exceed 36–40 stimuli (Jeffares 2015). In Study 2, for example, our pilot testers reported fatigue and frustration in the sorting process. We reduced our item count (from 48 to 43 items in a 9-point Q grid) after two pilot rounds. In our experience, keeping sort items minimal, pilot testing with a variety of users, and visualizing the Q grid in different software programs were critical design steps.

Second, written instructions for online Q sorts must be clear, accessible, and replicable (Watts and Stenner 2012). We found this step to be especially important with self-administered online Q sorts (like ours), because the researcher is not immediately available to explain procedures. Alternatively, researchers could (1) provide short videos or animations that explain the steps of Q sorting, including captions in different languages or to aid hearing-impaired participants; or (2) arrange a synchronous meeting (e.g. using Zoom, Skype, WebEx, telephone) to explain sorting procedures.

Third, after the initial correlation and factor analysis is complete, post-analysis interviews with participants can improve internal validity of results, as done in Study 1. Interviews allowed participants to elaborate on their reasoning and provide alternative interpretations of factors in a way that does not rely only on researcher knowledge or statistical norms and averages to evaluate factor validity (Brown 1980; Ormerod 2017, 2019; Watts and Stenner 2012).

We expect Q sort software will evolve in terms of capabilities and cost. Table 2 lists the programs we assessed and trialed as of 2018. When evaluating Q sort software options, consider the following questions:

How is the Q sort presented holistically? (e.g. forced or free distribution?) How easy is it to make changes? Can the participant see their rank-order and make changes relative to other statements?

What options exist for pre- or post-sorting questionnaires? (e.g. participant reflections or demographic data)

Which web browsers work best?

What measures are used to protect participant identity?

Who owns the rights to the sort data? Which programs comply with data management guidelines?

What is the cost? Language compatibility? Data output?

In sum, the online Q sort holds some strategic advantages. In our experience, we recruited participants from very different geographies (68 people from eight different countries and five U.S. states), an advantage in exploring diverse discourse coalitions. Costs and environmental impact of data collection were minimized; convenience was enhanced for participants. Data management was streamlined: Sorts, demographics, comments, and metadata were immediately recorded, reducing human error in manually collecting and inputting data.

At the same time, a major disadvantage is the inability to directly observe the sorting process—a barrier that could be overcome with synchronous video conferencing (e.g. Skype, WebEx) and appropriate permission. Other considerations include limited access to technology and internet service, additional time needed in the pilot process, and data management/privacy compliance.

Conclusion

The use of online Q sorts is a promising approach to uphold Q methodology principles while resolving some of the more persistent challenges of face-to-face research: including cost, convenience, timing, researcher influence, and the suspension of in-person data collection following COVID-19 public health guidelines and university regulations.

In moving forward, we suggest that online Q methodology also demands new reflexive strategies of research. A virtual Q sort cannot remove or eliminate research bias; it simply implicates the researcher in new ways (Sneegas 2020). Careful attention to recruitment, user instructions, software choice, piloting, sorting, data privacy, and closing the loop of researcher–participant engagement are important design considerations. Finally, the development of open-source software for Q sorts should be a major priority, to remove cost barriers and democratize uptake by the research community. Ultimately, online data collection of Q sorts should not preclude the researcher’s capacity to engage with participants; and moving forward with online Q is an individual choice aided by technologies that we hope benefit the research community at large.

Footnotes

Acknowledgments

We thank Dylan Brady, Kate Shields, and Gretchen Sneegas for helpful feedback during pilot phases of this research.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This material is partially based on research supported by the U.S. National Science Foundation under Grant #1655038. The usual disclaimers apply.

References

Besika

Collard

P. C.

Coogan

2018. Attitudes of therapists towards people with learning disabilities. Counselling and Psychotherapy Research 18:89–101.

Brown

S. R.

1980. Political subjectivity: Applications of Q methodology in political science. New Haven, CT: Yale University Press.

Eden

S. A.

Donaldson

Walker

2005. Structuring subjectivities? Using Q methodology in human geography. Area 37:413–422.

Exel

Baker

Mason

Donaldson

Brouwer

Team

2015. Public views on principles for health care priority setting: Findings of a European cross-country study using Q methodology. Social Science & Medicine 126:128–137.

Jeffares

2015. Online Q-set for PhD research query, Q method list [Electronic mailing list message]. Available to subscribers at: q-method@listserv.kent.edu (accessed October 19, 2020).

Jeffares

Dickinson

2016. Evaluating collaboration: The creation of an online tool employing Q methodology. Evaluation 22:91–107.

O’Neill

S. J.

Boykoff

Niemeyer

Day

S. A.

2013. On the use of imagery for climate change engagement. Global Environmental Change 23:413–421.

Ormerod

K. J.

2017. Common sense principles governing potable water recycling in the southwestern US: Examining subjectivity of water stewards using Q methodology. Geoforum 86:76–85.

Ormerod

K. J.

2019. Toilet power: Potable water reuse and the situated meaning of sustainability in the southwestern United States. Journal of Political Ecology 26:633–651.

10.

Ramlo

2016. Mixed method lessons learned from 80 years of Q methodology. Journal of Mixed Methods Research 10:28–45.

11.

Reber

B. H.

Kaufman

S. E.

Cropp

2000. Assessing Q-assessor: A validation study of computer-based Q sorts versus paper sorts. Operant Subjectivity 23:192–209.

12.

Robbins

Krueger

. 2000. Beyond bias? The promise and limits of Q method in human geography. The Professional Geographer 52:636–48.

13.

Sneegas

2020. Making the case for critical Q methodology. The Professional Geographer 72:78–87.

14.

Watts

Stenner

2012. Doing Q methodological research: Theory, method and interpretation. London: Sage.

15.

Wolf

. 2020. Conducting Q methodology during coronavirus, Q_method List [Electronic mailing list message]. Available to subscribers at: q-method@listserv.kent.edu (accessed October 19, 2020).