Abstract
This method note presents Q methodology as a useful tool for evaluators to add to their practice toolbox. Q methodology, which involves both quantitative and qualitative techniques, can help researchers and evaluators systematically understand subjectivity and the communicability of opinions and perspectives. We first provide an overview of Q methodology, followed by a brief summary of how evaluators are using Q, and an explanation of the steps for implementing Q methodology. Either by itself or with other methods, the potential uses of Q methodology in evaluation are diverse. For practical demonstration, we describe how Q methodology was used in a recent evaluation in the UK to understand stakeholder subjectivity within the program. We then reflect upon the pros and cons of using Q in program evaluation, concluding that it constitutes a worthwhile tool for evaluating complex programs.
This method note presents Q methodology (hereafter referred to as Q for brevity) as a useful tool for evaluators to add to their practice toolbox. Q is a system of assumptions and procedures to investigate and analyze a phenomenon that may encompass multiple coexisting perspectives among different people (Stephenson, 1993). According to Brown (2019), at the heart of Q is its focus on “the communicability of opinions and perspectives on any topic” (p. 1). This is important for evaluation because subjectivity (i.e., things we may express internally and externally; Brown, 2019) is central to our perceptions of programs and the decisions and actions that determine their relative success or failure.
Building on the contributions already made by the likes of Brown (2019) and Militello et al. (2016), we present Q as a methodology for understanding human subjectivity and describe its place in evaluation for measuring perception. We first provide an explanation of Q, followed by a brief overview of how evaluators are using it. Next, we walk through the steps for implementing Q. Then for practical demonstration, we describe how Q was used in an evaluation within the UK. We conclude by considering the benefits and challenges of employing Q within evaluative contexts.
Overview of Q
Q was developed 85 years ago by physicist-psychologist William Stephenson (1935a, 1935b). Stephenson’s work correlated and factor analyzed people (Stephenson, 1935a, 1952) or, more precisely, people’s perspectives or viewpoints on a topic of interest at a certain moment in time. At the center of Q is its attention and focus on subjectivity (Brown, 2019), which is at the heart of human volition and decision making. As a student of Stephenson, Brown (1980) has long contributed to the conceptual foundations of Q. In regard to subjectivity, Brown (2019) states that through internal and external conversations within and between individuals, opinions and conjectures manifest themselves.
Appreciation of subjectivity forms the fundamental premise of Q and its foundational concept of a concourse. Brown (2019, p. 2) highlights concourse as the body of talk and thought about any topic or situation, whereby many tributaries feed in. In Q, the concourse is commonly captured by gathering communications that represent the opinions, attitudes, and beliefs about a particular phenomenon (Hensel, 2017). In evaluation, there will be a concourse in existence for any program or intervention, such as prevailing views, recent reports, academic literature, and even photos and music. This is crucial for evaluation work, as the concourse forms the basis for measuring perception and understanding programs.
Participants in Q reveal their subjectivity by sorting a representative subset of the concourse, usually in the form of written statements. Sorting patterns are then statistically analyzed across individual sorts to examine the degree of consensus and divergent viewpoints within a group (Ramlo, 2015a). In this way, Q is thought to allow researchers to make subjectivity operant and thus available for scientific measurement (Ramlo & Newman, 2011).
Q entails a unique and philosophically consistent combination of qualitative and quantitative research approaches that brings benefits, but also leads to controversy, within social science research communities (Brown, 1980; Ramlo & Newman, 2011). The focus of Q is on interpreting meaning through the participant’s perspective, traditionally the domain of qualitative methods. At the same time, it involves statistical methods that lend rigor to its resulting factors or operationalized viewpoints. This combination permits researchers to measure subjectivity and empirically group people based on unique sorts that represent their own nuanced perspective (Ramlo, 2016).
On the other hand, critics have dismissed Q as insufficiently scientific, especially from a traditional quantitative research perspective, because it involves elements of subjective interpretation on both the participant’s and the researcher’s parts (Ramlo, 2016). Not surprisingly then, the more procedural components of Q, such as the sorting process or factor analytic techniques, have been adopted by quantitative researchers piecemeal without appreciation for the full methodology.
Nevertheless, Q has been proposed for use in a wide variety of applied social science disciplines as an approach for uncovering divergent or marginalized perspectives, clarifying complex constructs, grouping stakeholders, and facilitating dialogue and planning processes. Specific endeavors for which Q has been recommended include participatory policy design and appraisal (Baker et al., 2006; Militello et al., 2016; Ockwell, 2008; Robbins & Krueger, 2000; Wolf, 2013; Zabala et al., 2018), tailoring counseling and social work interventions to individual needs (Ellingsen et al., 2010; Shemmings, 2006; Stickl et al., 2018), informing reaccreditation processes in higher education (Pruslow & Owl, 2012; Ramlo, 2015b), and promoting leadership development in the workplace (Woods, 2011).
Uses of Q in Program Evaluation
Q has been proposed for use in program evaluation as well, most commonly to enhance authentic stakeholder engagement by identifying and classifying stakeholders’ needs, implicit program theories, and changing viewpoints on an issue (Harris et al., 2019; Militello et al., 2016; Ramlo, 2011). In line with these suggestions, evaluators have most often employed Q to gather stakeholder perceptions of a program—including its feasibility, fidelity of implementation, perceived outcomes, and critical elements—and suggest formative improvements (Butler et al., 2014; Corr et al., 2003; Lazard et al., 2011; McPherson et al., 2016; Militello & Benham, 2010; Pike et al., 2015).
While stakeholders queried through Q are most often program participants, they at times include program staff in order to identify and compare perceptions of the program across different stakeholder groups. In one participatory evaluation (Lazard et al., 2011), program staff administered the Q-sorts to program participants, promoting process use. Evaluators have also used Q to explore stakeholder engagement in participatory evaluation in order to identify participants’ initial motivations (Militello et al., 2016; Thompson, 2015), perceptions of the process (Cotton & Mahroos-Alsaiari, 2015; Danielson et al., 2009), and their methodological preferences (Kelly et al., 2016).
Indeed, Q has been widely used by evaluators to improve their own evaluation processes and systems. Q has been employed to gather stakeholders’ differing perceptions of performance measurement systems and resulting data use (de Jonge et al., 2017; Frayne, 2014; Militello et al., 2013; Velez, 2006), identifying divergent priorities for reconciliation. Evaluation researchers have used Q to explore stakeholder perceptions of process use resulting from participation in evaluation (Akanban et al., 2013; Baptiste, 2010; Baptiste, 2011) and stakeholder self-assessment on different aspects of evaluative thinking (Mumford, 2018; Nunns, 2016; Pham, 2018).
Q has been used for summative program evaluation, albeit less commonly. For instance, reported in a series of articles, Cuppen (2012, 2013) employed pre and post Q-sorts as part of a quasi-experimental evaluation of a dialogue intervention. The initial Q results were used to select stakeholders for participation in dialogue, structure dialogue sessions, and match participants to a comparison group that did not receive the dialogue intervention. Then Q was used as a repeated measure at the end of the evaluation with both dialogue participants and the matched comparison group to compare changes in each group’s Q-sorts over time and to one another. The evaluator found that dialogue participants changed their sorts (and thus their perspectives) more than the comparison group and showed broader awareness of the issue’s complexity after the dialogues.
Q offers practical benefits and opportunities related to these diverse aims. Because it inverts traditional factor analysis (i.e., variables are people, and items comprise the sample that in turn represents the concourse), Q can be employed with samples of 30 or fewer participants. Dziopa and Ahern (2011) suggest that a 1:1 ratio of people to items is appropriate. Q is seen as more engaging and sensitive than a traditional Likert-type scale survey for eliciting participants’ complex and idiosyncratic viewpoints, and it may help attenuate participant bias and enhance validity of the resulting perspectives (Fluckinger, 2014; Ho, 2017), although it is also potentially more time-consuming to implement than traditional survey methods.
Q is focused on the measuring of individual perceptions, but its contributions to generalizability are theoretical rather than statistical (Watts & Stenner, 2012). That said, Q-sort results have frequently been used to strategically sample participants for follow-up interviews and compare perceptions of program implementation across multiple sites. Q results can also form the basis for developing traditional surveys (Danielson, 2009), such as on the effectiveness of partnerships and collaborations (Jeffares & Dickinson, 2016).
In summary, Q has been employed in evaluation both formatively and summatively and as an approach to developing new programs. Evaluators have also used Q to gather stakeholder feedback and reflections on the evaluation processes and performance measurement systems they produce and their construct definitions and measurement tools. There is thus strong precedent for adopting Q within evaluative contexts, and perhaps with additional awareness and guidance, its popularity among evaluators will continue to grow.
Q Procedures
An overview of Q procedures is provided in Table 1. In summary, Q explores subjective meaning via statistical procedures. Participants in a Q study typically sort statements (or pictures or objects) about a specific topic, which provide a representative view of all the discourses related to that topic. Participants are asked to sort or rank the statements according to a specific instruction to reflect on their subjective interpretations of the statements in relation to the instruction. The completed sorts are then compared and grouped via factor analysis using specialized (often free) software (e.g., PQMethod 2.35).
Overview of Q Procedures.
The resulting factors represent statistically distinct groups of participants’ perspectives about the topic of interest (Brown, 1980; Watts & Stenner, 2005). Using abductive reasoning, the evaluator interprets each factor to understand what, how, and why differences exist in points of view, attitudes, or opinions about the topic. For more complete treatments of the steps to implement Q, including sample conditions of instruction and sorting grids, see, for example, Watts and Stenner (2012), Militello et al. (2016), and a host of other tutorials on Q cited in this article and elsewhere.
Demonstration Case: Use of Q in an Evaluation From the United Kingdom
What now follows is an illustration of a UK-based evaluation that implemented Q. The evaluation was conducted (by two of the authors) to evaluate a UK public and voluntary sector partnership, pseudonymized as Partnership4People (P4P) to protect confidentiality. Funded by a large organization in the UK, the P4P program was implemented via a hub and spoke approach whereby a key agency triaged referrals and coordinated a partnership of stakeholders across the public and voluntary sector to deliver tailored support to people living in crisis (e.g., financial poverty and unemployment). The project focused on achieving outcomes both at an individual level for the person in crisis and at an interorganizational level to operate effectively and coherently as a network of providers in partnership to support the person(s) in taking action to address the crisis. There was motivation among the P4P stakeholders to obtain insight and understanding about the program outcomes, the causal processes that lead to them, and how they could use the findings to inform future practice to enhance effectiveness in responding to crises.
The evaluation focused on mechanisms of change, and therefore, it was essential to investigate how the network’s resources (mechanisms) provided through P4P partners were helping network stakeholders foster the intended outcomes (Pawson & Tilley, 1997). To capture this, Q was mobilized to tease out what worked, and how, for the different stakeholders. The overriding evaluation aims were addressed across three key phases: Phase 1: Developing the concourse (program theory), Phase 2: Selecting participants and Q-sorting exercise, and Phase 3: Interpreting and planning for improvement based on evidence synthesis. These phases are discussed below with reference to Table 1.
Phase 1: Developing the Concourse (Program Theory)
This phase focused on developing the concourse about P4P, which we did collaboratively with practitioners. They were not comfortable with the term concourse because it confused them. To address this, we referred to this phase as theories about how and why the P4P program could work to achieve the outcomes envisaged. We made sure that while we referred to this as the program theory development stage, we were capturing all of the important aspects of concourse central to Q. Developing program theory usually involves engaging with key stakeholders to gain their own perspectives about how they think the program works, alongside the analysis of existing literature relevant to the program at multiple levels. This usually leads to the creation of candidate program theories (Pawson & Tilley, 1997) that are open to investigation.
Harris et al. (2019) have argued that the development of program theory and the concourse in Q are mutually compatible because they both place significant emphasis on context. Pawson (2013) categorizes context into four layers: (1) individuals (their characteristics and capacities), (2) interpersonal relations (stakeholder relationships), (3) institutional settings (rules, norms, and customs), and (4) infrastructure (social, economic, and cultural setting). This overview of context identifies the key circumstances, boundaries, and backgrounds, which are synonymous with establishing the concourse in Q.
Specifically, we explored the volume of discussion (Stephenson, 1982) about P4P, making use of primary and secondary data to establish the concourse. This involved interviewing key stakeholders involved in delivering the P4P program and examining industry and academic literature associated with P4P and similar programs. This concourse phase helped establish a host of if/then and because narratives about the P4P program.
This process informed the development of the Q statements (Table 2, consisting of 40 statements from the concourse). We constructed the statements iteratively and carefully to ensure that each one reflected the program theories about P4P. Collectively, the statements focused on key contexts and outcomes associated with the P4P program. The statements were validated in two ways: first, among the evaluation team through our reflexivity and engagement in Phase 1 of the evaluation, and second, the statements were presented to a small sample of practitioners leading P4P to confirm that they made sense and reflected the program.
Q Statements Prioritized From Concourse.
Note. P4P = Partnership4People; PAP = partnership around the person.
Phase 2: Selection of Participants and Q-Sorting
Our evaluation questions focused on gaining insight into how practitioners from different organizations (including referrers to P4P) understood P4P, how it worked as a network, and what it contributed toward supporting people in crisis. Given the reliance on good interorganizational partnerships, including for learning and knowledge transfer, it was necessary to select a range of practitioners with varying roles and experiences to complete the Q-sorting exercise. Participants in the P-set were made up of 18 practitioners, consisting of stakeholders from various government departments and nongovernmental organizations.
Each participant ranked 40 statements (Table 2) into the Q-sort grid, completing the Q-sort activity individually at times that were convenient for them. This was important because the Q-sorting procedure required a lot of explanation and was very new to most of the participant sample. While the participants completed the Q-sort, the evaluator made notes of questions asked, comments made, and relevant contextual information about the participant’s role in relation to P4P, as these data could inform understanding and explain why participants sorted statements in the way they did.
Once all practitioners had completed their Q-sorts, we entered each sort into PQMethod, a free statistical software program designed for Q, and selected by-person principal components analysis. This analysis identified eight possible groups for qualitative interpretation across the 18 completed Q-sorts. Having examined the statistical relevance of each group, we selected four groups for full qualitative interpretation based on how many people loaded onto each factor, study variance, eigenvalues, and feel for the data (Table 1). This was made up of Group 1 with four participants, Group 2 with three participants, Group 3 with six participants, and Group 4 with three participants.
Phase 3: Interpretation and Planning for Improvement in P4P
This final phase consisted of interpreting each resulting Q-sort score for each of the four groups to produce a “realist holistic narrative” (Harris et al., 2019) of how and why P4P works. In order to foster use of the evaluation findings, the evidence needed to be presented in a way that was easy to interpret and understand for stakeholders and funders. In considering the different ways to interpret and communicate the findings of the Q-sort (Table 1), Watts and Stenner’s (2012) approach was adopted, as it allowed the Q-sorts within each Q group, and between the different Q groups, to be qualitatively interpreted to explain the statistical relationships in narrative form. Watts and Stenner’s (2012) approach was followed by drawing upon the data presented from the comprehensive factor analysis output file for the four subgroups. Their approach (crib sheet method) specifically involved identifying where participants in each subgroup had ranked their statements and distinguishing statements (from other subgroups) which we used to build our abductive holistic narratives.
These narratives were produced around the different subjective viewpoints, drawing comparisons with each partner’s organizational role and context. This was important given that we were exploring subjectivity at the organizational level of operation where a lot of interagency networking was taking place. This helped the P4P funders make sense of the different ways P4P worked in relation to the different views of the stakeholders’ Q-sorts and subsequent groupings. In doing so, we provided insight into how to work with different stakeholders based on their interpretation of P4P to better target, communicate, and integrate P4P and achieve better outcomes for people in crisis.
What now follows is the presentation of findings from two of the subgroups that emerged from the Q-sort activity (we have selected Subgroups 2 and 3). As indicated in the previous section, the stakeholders who completed the Q activity loaded across four subgroups. We presented these factors using the Watts and Stenner (2012) format, where the story of the narrative is supported by the statement rankings. For example, 25 +3 would mean that those in this particular subgroup would have rated statement 25 as plus 3 on their respective Q-sort. This use of statement rankings helped us to justify the relevance and credibility of the story behind the narratives, which was important to the client commissioning the evaluation. For the purpose of this method note, we present Subgroups 2 and 3 in the narratives below to illustrate the data informing the establishment of the subgroups (we have not supplied the full factor analysis output data file due to length and size). This is then followed by Table 3, which captures the distinction between each subgroup. At the top of each column, a summary descriptor is provided about the subgroup. Underneath the descriptor are key bullet points that represent the subjective shared viewpoints of the practitioners who loaded into each subgroup.
Description of Select Resulting Subgroups.
Note. P4P = Partnership4People.
Subgroup 2 narrative
“Collaboration and smart working facilitate individual- and system-level outcomes, but is the approach sustainable?”
This subgroup represented a total of three participants and is formed by a manager of an organizational partner of P4P, a key partner in supporting people referred to them and a volunteer at an organization that supports people to access P4P. This subgroup strongly agrees that through the adoption of tailored action plans, the P4P program helps those who are in genuine hardship (5 +4). These action plans help to motivate individuals to take control of important aspects of their lives in the longer term (12 +5), including their own finances (11 +1). Furthermore, this subgroup feels the expert advice from the P4P program helps people to make immediate changes to their lives (14 +3), which leads to less dependency on resources in the longer term (21 0). Interestingly, this motivation does not extend to empowering people to take positive action against the cause(s) of their crisis (13 −3). Subsequently, like most subgroups, this subgroup feels that more work needs to be invested in ensuring the P4P model is a sustainable approach to preventing crisis from reoccurring in people’s lives (10 −4). However, this is also dependent on factors outside of the network’s control, such as broader problems in line with the sociopolitical context.
The ability of P4P to take time to listen and understand the causes of why people are in crisis is well recognized within this subgroup (31 +4), which places importance on the role of listening to understand individual circumstances. However, this subgroup does not feel the program has the capacity to effectively build people’s skills and knowledge of the appropriate services (4 −4) around them to successfully move out of crisis. In addition, in contrast to any other subgroups, this subgroup did not feel that the financial support provided through the P4P program provides the opportunity for people who have fallen into crisis to meet their basic needs (2 −2). Thus, this subgroup feels the program does not ensure benefit issues are dealt with in an efficient and reactive way to avoid individuals falling into further crisis (32 −3). In addition, the theory of individuals accessing the program and sharing knowledge and solutions with others who may be experiencing a similar crisis (8 −3) is not something that regularly happens.
Overall, this subgroup feels that the support P4P provides for individuals reduces pressure (22 +2) and strain on system resources (37 +3). This subgroup distinctively recognized that the ability for the P4P network of partners to work within a collaborative environment (24 +2) led them effectively working together to facilitate the holistic needs of every person in crisis being met (23 +3). This subgroup shows evidence of the programs achieving a range of individual-, organizational-, and system-level outcomes.
Subgroup 3 narrative
“The key stakeholders are informative experts and provide immediate support.”
This narrative represented six participants (more than any other subgroups), including staff working within departments across the system and organizational partners of P4P. This subgroup is comprised of staff who are connected to P4P and understand how it works to support the system at a strategic and operational level.
This subgroup feels that the P4P network consists of all the key organizations (25 +3) necessary to alleviate crisis. The capacity of these organizations to have the knowledge and understanding of the wider system results in an effective and efficient model that deals with people in immediate crisis (30 +4) within 24 hr (19 +4) by signposting people to the appropriate services (9 +3). This subgroup shares commonality with other subgroups that P4P helps people in genuine hardship (5 +2) and provides help in an empathetic way to support people (3 +2). Similarly, this subgroup identifies the contribution of P4P in being able to take time to listen, understand the causes of individual crisis, and work with the individual to make changes to resolve the issue(s) causing crisis (31 +3). In addition to this, this subgroup shares viewpoints around how P4P works, through being able to signpost people to the appropriate services that they need (9 +3), providing financial support that helps to build individual capability (1 +1), and more strongly creating an opportunity for people (2 +5) who have fallen into crisis to meet their basic needs.
This subgroup identifies how the current eligibility criteria result in the program working with people who are most in need of their support (6 +2) and operate in a responsive way to people in crisis within 24 hr (19 +4). As such, this supports to ensure benefit issues are dealt with in an efficient and reactive way to avoid individuals falling into further crisis (32 +1).
There is skepticism about how the P4P model is a sustainable approach to preventing crisis from reoccurring in people’s lives (10 −2), supported by how P4P helps people to stay out of crisis longer term (15 −4). This is linked to why this subgroup disagrees that the expert advice provided through P4P program will not always help people to make immediate changes to their lives (14 −4), which prevents crisis from reoccurring. Further, some causes or crises are by larger system changes such as the introduction of universal credit, which this subgroup feels strongly that P4P is constrained in helping to reduce complexity for people in crisis (38 −3). In addition, this subgroup is apprehensive about how P4P ensures housing issues (33 −1) and domestic living issues (35 −1) are dealt with in an efficient and reactive way to avoid individuals falling into further crisis. The reason for why this subgroup feels less strongly about the capacity of P4P to achieve a wider positive impact with system partners could be due to the current funding model and perceived sustainability of the P4P network, which impacts on how it is valued by other service areas (28 −1).
Building on this, like Subgroup 2, this subgroup does not feel that people who have been supported by the P4P program currently have the capacity to give something back through volunteering with P4P partners (7 −5), and therefore, unsurprisingly, the program does not provide individuals with opportunities to share knowledge with others who may be experiencing similar crisis (8 −3). This could be because within the context of attempting to reduce the complexity around the new universal credit system (38 −3), people are still concentrating on taking control of their own finances in the shorter term (11 −2). However, this is not to say that P4P could not achieve this, if it was financially resourced in an improved way.
Overall, this subgroup feels it is the financial support provided through the P4P program which helps people who have fallen into crisis to meet their own basic needs (2 +5), as opposed to the expert advice from the P4P program which helps people to make immediate changes to their lives (14 −4). This is an important finding and places emphasis on financial support as the key mechanism to ensuring benefit issues are dealt with in an efficient and reactive way (32 +1). Generally, this subgroup captures the program as a simple and efficient program which uses a collaborative model effectively to address short-term outcomes.
Reflections on Using Q in Evaluation
The above demonstration case provided a snapshot of the steps taken and findings in using Q to evaluate P4P. On reflection, the evaluators mobilizing this approach found the process of conducting Q iterative and clear. The methodological steps it requires enabled us to capture a program theory through concourse-building, which drove the creation of the statements, all the way through to the sorting and analysis, where we were able to build a story with depth as to how and why each stakeholder saw P4P working and why. This comprehensive process instilled confidence in the evaluation, given how much time we afforded to the use of Q.
Essentially, carrying out a Q study enabled us to assess the merit and the worth of the program under study because it was able to showcase the subjective viewpoints of those integral to the program concerning how they saw it working. Further, the quantitative factor analysis procedures, followed by the qualitative interpretation, allowed for the groupings of shared viewpoints, which helped us to tell a story about certain individuals and specific contexts around the program across multiple levels. The synergy of qualitative and quantitative approaches also helped us to mitigate the ongoing quandary a researcher or evaluator finds themselves in when choosing between qualitative and quantitative methodologies.
Moreover, when conducting any type of evaluation, it is always a danger that in the face of any academically respected and innovative evaluation methodology, the client and stakeholders become alienated (Harris et al., 2019). For example, in the knowledge translation discourse, it is crucial that the clients requesting and paying for the evaluation understand what is being done, and why it is being done in the way it is, to avoid alienation and subjugation (Harris & Adams, 2016).
Therefore, in accordance with Shula et al. (2016), there should be motivation for collaboration on the part of the stakeholders from the beginning, and the findings should foster use to inform improvements and refinements to practice in the future. This was important for us in relation to the use of Q in this evaluation, and we did not see or experience any confusion on behalf of the client throughout. In fact, there was a lot of interest and motivation to be part of the process, as we highlighted in the generation of the statements.
In terms of fostering use (Harris, 2020; Shula et al., 2016), Q is useful for clients because the holistic narratives (the Watts & Stenner approach we took) can provide a succinct story and reflection about those shared viewpoints that can share practice. In a realist sense, it was crucial to be able to communicate to the client that P4P works for different individuals in different ways, taking into account varying subjective viewpoints (Harris et al., 2019; Pawson & Tilley, 1997).
Therefore, the findings informed P4P both at a practical level—in how to tailor and communicate the benefits and address misunderstandings between partners in delivering P4P—and at a program level that changes or adaptations to the program should not be generalized in one way and should be adapted and refined in accordance with what is illustrated within the holistic narratives we presented. We argue that this can only be positive for program refinement and development, informing practice moving forward.
Challenges to Using Q
However, when using a new, innovative methodology within program evaluation such as Q, there will be limitations and/or implications that arise in practice. We present here some practical challenges that occurred when conducting Q with participants and in following the methodology.
It is likely that participants will not have heard of Q, as was the case in this evaluation. This created our first challenge to support the participants in feeling capable and willing to do something different to what they are used to. Secondly, when explaining the process of Q to participants, they perceived Q as a simple, process-oriented task (as shown in Table 1). In practice, participants became confronted with a series of challenges in deciding between statements (as shown in Table 2). For example, when conducting the Q-sort, some participants did not feel comfortable with ordering statements within the negative (−) scoring of the Q grid as they felt they were being negative about the program.
Within the demonstration case, participants generally agreed with more statements than they disagreed with and therefore did not have space to fit them all neatly within the “+” spaces available on the Q grid. This presented a challenge to the evaluator in encouraging the participant to choose the statements that were most prominent to them, without negatively impacting on their motivation or influencing sorting of the statements. Further, the time it takes to conduct a Q-sort can vary: For some participants, it took 30 min, whereas for others, it took close to 60 min. Participants took longer when they reordered their statements as their reasoning changed while conducting the Q or as a result of having to choose between statements when (re)ordering. This can result in participant fatigue.
Beyond practical challenges with participants completing the Q-sort, the methodological process of Q—from generating the concourse to conducting the Q-sort, factor analysis, and interpretation—can span a duration of months, with each phase requiring considerable time to ensure there is a smooth transition between the phases described in Table 1. Further, in considering the technical requirements of Q, the evaluator should be competent in the use of statistical software packages such as PQMethod in order to conduct the factor analysis (Table 1). This is a technical phase in the Q process and requires evaluators to be careful with their data entry and competent with the software to conduct the factor analysis in order to then proceed to identify subgroups and build the holistic narrative.
Conclusion
This article has sought to demonstrate the diverse potential uses of Q in evaluation, either by itself or with other methods. The case study example has illustrated the value of Q in evaluation, crucially the capacity of Q to quantify participant viewpoints, meaning participant voices are more than anecdotal evidence in an evaluation. It is valuable when working with relatively small sample sizes and offers practitioners an appropriate methodology for evaluations in which the human factor and/or context may influence an intervention’s success. This holds promise in fostering use through better informing program funders and decision makers on where to target, adapt, and refine the program in relation to the holistic narratives.
In reflecting on using Q within the case study example, the authors highlight practical challenges in particular phases, especially when participants are not familiar with Q and are challenged by the number of statements to rank and having to place statements into negative (−) scoring on the grid. As such, the authors recommend appropriate participatory opportunities to be promoted with program practitioners and other stakeholders throughout the Q process: supporting concourse development, refining the Q statements, interpretation of Q factor analysis, and how findings can be best presented to foster evaluative thinking and use. We accept that the approach we have demonstrated in this method note is not the only way to mobilize Q. Indeed there are a number of different ways to carry out the interpretation of the factors (see Militello et al., 2016), and as reflective practitioners, we are keen to explore these different approaches in the future. Nevertheless, we anticipate that the approach we have followed and demonstrated can make a suitable contribution to the evaluation community.
Interested readers can learn more about Q through the International Society for the Scientific Study of Subjectivity, a professional community of Q researchers. They offer an annual conference, a website (qmethod.org), an active LISTSERV, and a peer-reviewed journal, Operant Subjectivity. In addition, the “Sue-Z Q” YouTube channel, created and maintained by Prof. Susan Ramlo of the University of Akron, provides detailed tutorial videos on every step of Q, including use of software for factor analysis. Finally, we hope that evaluators interested in collaborating on Q will reach out to the authors, so that we might establish our own community of practice around this innovative methodology and promote its broader adoption. After all, subjectivity is central to program evaluation, and Q provides a useful methodology for measuring it.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
