Member checking: A brief metalogue of a career interview

Metalogue as a method of inquiry

Staller (2007) argued that the interactions among authors, editors, and referees during the peer review process form the “site” where the limits of acceptable scholarship are defined. These metalogues—written conversations that maintain the distinct voices of participants while highlighting points of disagreement—offer a valuable approach to examining how scholarly work is crafted (Staller, 2007). As these behind-the-scenes conversations unfold, they highlight valuable aspects of a manuscript and identify areas that need further discussion and clarification. This process not only refines the work but also provides fascinating insights into how knowledge is crafted and agreed upon within the academic community.

Despite their potential, metalogue analyses are infrequent, perhaps due to the nature of the peer review process itself, which is seldom considered an object of inquiry. From a critical realist perspective, the peer review process possesses several characteristics that make it suitable for study. Critical realism posits a reality that is both layered and emergent, while recognizing that our understanding of it is socially constructed (Bhaskar, 1978; Boost et al., 2023). This view suggests that events or outcomes, such as decisions made during peer review, are influenced by underlying causal structures within specific contexts. Peer review is defined by its participants—authors, reviewers, and editors—and the outcomes they generate, such as editorial decisions. It is influenced by the prevailing norms and values of the time, which means it evolves with changing academic and cultural contexts. Additionally, peer review wields significant causal power, as it plays a crucial role in shaping the publication landscape and establishing scholarly standards.

However, engaging in metalogues presents significant challenges. The standard practice of maintaining anonymity in peer review inhibits open and transparent dialogue among the key participants—authors, reviewers, and editors. Initiating a metalogue requires not only a breakdown of this anonymity but also a foundation of transparency and mutual respect, which can be difficult to establish in such a traditionally private and confidential environment. Unless editors make a concerted effort to identify and obtain consent from researchers and reviewers to participate in a metalogue and disclose their identities for analytical inquiry, engaging in this type of reflective dialogue remains uncommon.

Moreover, authors and reviewers are typically exposed only to one manuscript, making it challenging for them to know if issues they encounter are isolated or indicative of broader patterns. In contrast, editors oversee numerous manuscripts and interactions, placing them in a unique position to identify recurring issues that may facilitate or hinder publication. Though seldom explicitly acknowledged as metalogues, the insights garnered from these observations often find expression in editorials, which can significantly influence peer review practices. For instance, recent editorials published in Affilia: Feminist Inquiry in Social Work have involved the editorial board in discussions about principles and practices (Goodkind et al., 2021) and the role of positionality in feminist scholarship (Jackson et al., 2024). These editorials effectively serve as consensus statements on what the journal considers to be best practices.

At QSW, we have published various editorials that are arguably “metalogues in disguise” aimed at addressing common pitfalls in the publication process and providing advice to enhance the likelihood of successful submissions (Piedra, 2019, 2020, 2022, 2023; Staller, 2015, 2019a, 2021, 2022; Staller and Krumer-Nevo, 2013). These editorials serve dual purposes: they raise the quality of manuscripts reviewed and published, and they ensure equitable access to information that can give authors a competitive advantage. By publishing these insights through editorials rather than peer-reviewed articles, we make them accessible without the need for a journal subscription, which is particularly beneficial for scholars at smaller institutions or in countries with limited resources.

The career interview explained

To fully appreciate this brief metalogue, it is important to understand the concept of a career interview. QSW has a tradition of publishing career interviews with established scholars who have significantly contributed to qualitative social work research throughout their careers. In these instances, the editorial team at QSW plays a matchmaking role by identifying a scholar and securing their agreement to participate. We then find a seasoned researcher to conduct the interview, oversee its submission for review, and handle its eventual publication. The specifics of the interview, such as the questions and the format, are negotiated directly between the author and the interviewee.

This practice has led to a variety of representational styles in published interviews. For example, some interviews feature a pre-set list of questions with the scholar’s responses published as they are provided (Sinclair, 2012). Others are presented as edited transcripts, capturing the dynamic ebb and flow of the conversation (Hardy, 2019; Interview with Noel Timms, 2014). Alternatively, some publications involve detailed analyses of the interviews that highlight thematic developments throughout the scholar’s career (Riessman and Staller, 2016; Staller, 2019b). For these in-depth explorations, the author collaborates closely with the interviewee on the planning and execution of the interviews, including the number of sessions and their settings.

A key aspect of this process is member checking, where the final interpretation of the interview is reviewed by the interviewee to ensure the accuracy and authenticity of the representation. This practice is widely regarded as enhancing the trustworthiness of the published work, making the career interview a valuable tool for both the scholar’s legacy and the academic community’s understanding of their contributions.

Member checking

Member checking is an essential component of qualitative research, where researchers return their findings to the participants for review and feedback. This process enables participants to confirm, substantiate, verify, or correct the researchers’ interpretations, ensuring that the analysis authentically reflects the participants’ perspectives and the complexity of their experiences (Birt et al., 2016; Buchbinder, 2011). However, this practice can also lead to differences in how evidence is interpreted. While it might seem logical to always regard the participant’s perspective as the definitive “correct” view, giving them the final say, there is room to consider that the researcher might provide a valuable alternative interpretation. This outsider perspective, or experience-distant viewpoint, introduces a dynamic akin to the differences between autobiography and biography, where each offers unique insights and nuances.

Initially, member checking ensures that interview transcriptions accurately reflect the information participants shared. Reviewing these transcripts allows participants to correct factual errors or clarify misunderstandings. For example, a participant might clarify a point by saying, “I didn’t mean it that way,” and provide a correct interpretation, often yielding new empirical evidence that can clarify or enrich the original material (Bloor, 1978). Feedback from member checking can also lead to minor adjustments like correcting transcription errors or refining quotes to better convey their intended meanings. At times, participants may request the removal of sensitive remarks previously made, now deemed inappropriate for public disclosure. More substantial feedback can include challenges to analytical categories or contributions of new insights that augment original interpretations. This extensive feedback can generate significant new empirical evidence, potentially necessitating another round of analysis (Bloor, 1978). Thus, member checking evolves from a mere procedural step to a collaborative and integral part of the research process, shifting it from a “data” analysis phase to an active “data collection” phase. This transformation actively shapes research outcomes, positioning interviewees as co-creators of knowledge rather than merely sources of empirical evidence.

The necessity and extent of further analysis after member checking depend on various factors, including the participants’ engagement with the research. While some may show casual interest, others could be deeply invested in the outcomes. Researchers must then assess the value of the new empirical evidence against the constraints of time and resources, determining what is “good enough” to conclude the study. Indeed, the iterative nature of member checking raises a compelling question in interview-based studies: When is an interview truly “over”? This issue highlights the inherent limitations of such studies—they capture only a snapshot in time, yet the perspectives and insights of the interviewees can continue to evolve afterward. As participants reflect on their responses and possibly gain new perspectives, their understanding and narratives might change, suggesting that an interview may never be conclusively “over.” This ongoing evolution of thought and experience challenges researchers to consider how best to depict and represent the fluidity of human experiences within the seemingly static framework of a study. These considerations were particularly pertinent during the final review of the career interview and commentary sent to Gilgun for her evaluation. Her feedback not only influenced the trustworthiness of the analysis but also highlighted the iterative and participatory nature of qualitative research.

The Gilgun interviews and their interpretation

Several years ago, Debra Nelson-Gardell, with the assistance of Kelly Bolton towards the project’s conclusion, conducted three interviews with Jane Gilgun. These interviews were carefully transcribed and reviewed by Gilgun herself, and they formed the foundation for two distinct scholarly interpretations published in this issue (Bolton and Nelson-Gardell, 2024; Staller, 2024). In their work, Bolton and Nelson-Gardell employed content analysis to categorize Gilgun’s contributions into four main pathways: American pragmatism, human resilience under adversity, the challenges of qualitative evaluation in community practice, and the revitalization of the social work research legacy (Bolton and Nelson-Gardell, 2024). Their analysis presents Gilgun as a pivotal figure whose scholarly endeavors and personal insights have significantly advanced the field of social work through qualitative research.

Conversely, Staller used the same empirical evidence to delve into a recurring theme in Gilgun’s career: the presence of evil in a world Gilgun believes is fundamentally good. Staller’s analysis places Gilgun’s extensive work within the historical and intellectual framework of the Chicago School, emphasizing how these connections equipped Gilgun to tackle complex topics using qualitative research methods. This portrayal emphasizes Gilgun’s dedication to rigorous inquiry and her commitment to addressing significant societal challenges.

Both manuscripts underwent review by Gilgun before publication. While she approved both, her response to Staller’s interpretation—“I feel understood”—signifies a level of validation that goes beyond mere factual accuracy. Capturing the deeper essence and motivations behind a scholar’s work often impacts the subject more profoundly than just getting the facts right. Consider how autobiography and biography can indeed reveal different facets of a person’s career or life, especially in how patterns and themes are recognized and articulated. Gilgun might not have identified or emphasized the same patterns as external scholars like Bolton, Nelson-Gardell, or Staller, highlighting the distinction between descriptive and interpretative scholarship.

In descriptive scholarship, Gilgun might focus on a factual recounting of her experiences and the direct impacts of her work, aligned closely with her personal perceptions and experiences. In contrast, interpretative scholarship, as demonstrated by Bolton, Nelson-Gardell, and Staller, involves a deeper exploration of themes such as intellectual influences and underlying philosophical debates. These interpretations provide a broader understanding of her contributions through the lens of academic and historical contexts, which might not be as evident from Gilgun’s own recounting. This illustrates how varied perspectives can enrich our understanding of a subject, blending subjective (experience-near) insights with objective (experience-distant) analysis.

However, Gilgun noted some gaps in the portrayal, due to topics not discussed in the original interviews. She expressed a desire to add new content to the completed analyses, which were based on the initial interviews. This addition would alter the body of evidence upon which the career interview was based, complicating the fidelity of the original analyses.

The role of saturation

When Jane Gilgun suggested incorporating new content, it raised questions about whether true saturation was achieved during the original interviews. Saturation occurs when no new information is found that would further develop the properties of the category under study (Saunders et al., 2018). Essentially, it is reached when additional data gathering does not yield new insights related to the research question. This point is pivotal because research inherently requires setting boundaries for data collection and analysis, and deciding to extend these boundaries to include new empirical evidence is an analytical choice that can enhance the rigor of the research. While this is a worthwhile endeavor, it is not the sole factor to consider.

The decision for further data collection also depends on the research goals. The quality of research is determined by how well it answers the posed research question, influenced by decisions made throughout the project—from formulating the question to collecting and analyzing data. The research question itself may evolve during this process (Staller, 2022). These decisions aim not for objective accuracy or perfection but for the best possible explanation within the given time and resource constraints. Thus, there was no “right” answer about whether to add content to the analysis.

This situation also underscores a practical challenge in research—balancing the need for comprehensive data with the limitations of time, resources, and the original scope of the study. Deciding whether to reopen data collection after declaring saturation involves careful consideration of what additional empirical evidence would contribute and whether it would significantly affect the interpretation or understanding of the subject matter. In Gilgun’s case, adding new content after the review could indeed enrich the understanding but it could also complicate the integrity and boundaries of the initial analytic framework.

The fact that interviews capture a person’s perspective at various times always presents a challenge in determining the optimal number of interviews needed for a comprehensive understanding. This issue extends to member checking, which typically takes place after the interviews have been conducted and analyzed. People’s perspectives and circumstances change over time, which may affect how they perceive and react to the analysis of their interview data, regardless of the number of interviews conducted. While conducting more interviews may offer a richer, more nuanced perspective, a smaller number of well-analyzed interviews can also provide sufficient insight. This evolving context underscores the complexities of conducting and interpreting qualitative research, emphasizing the need for flexibility and adaptiveness in the research process. It also humbly acknowledges that any interview-based study captures only small snippets of a person’s lived experience or views.

The resolution

As a reflective scholar, Jane Gilgun’s understanding of her work and its significance has evolved since the initial three interviews. This evolving perspective is crucial for fully appreciating her contributions to qualitative inquiry. However, a career interview, which typically relies on limited evidence, aims to capture critical reflections and significant highlights rather than detail every aspect. There is also a temporal element to capturing a “career interview” of a living scholar. Interviews conducted immediately post-retirement may offer a perspective that differs from one obtained many years into retirement, as the scholar continues to reflect on her past and present work. While it may seem reasonable to add content if significant gaps are identified, it is essential to consider the feasibility and potential impacts on the study’s credibility and coherence. Thus, Staller and I concluded that adjusting the scope of a career interview by adding new material could potentially initiate an ongoing biographical project, extending beyond the typical scope of a career interview. Such a change would fundamentally alter the nature of the project, requiring a reevaluation of its objectives and methods. Nevertheless, I wanted to ensure our readers had access to Gilgun’s latest insights.

Ultimately, in my editorial role, I chose to allow Gilgun the opportunity to address the missing elements from the original interviews by incorporating her feedback into a response to Staller’s article (Gilgun, 2024). In her rejoinder, Gilgun acknowledges the thoroughness of Staller’s analysis but notes that certain topics not discussed in the initial interviews led to omissions in her portrayal. She elaborates on areas not fully explored in the initial sessions, highlighting the contrast between the spontaneous nature of interviews and the meticulous, often revised, process of writing and editing scholarly articles. This approach not only enriches the existing analysis but also emphasizes that scholarship involves an ongoing dialogue—an engagement with the processes that underpin the creation of knowledge.

This brief metalogue on Gilgun’s career interview sheds light on the dynamic nature of qualitative research and the complex considerations that form the foundation of knowledge creation, particularly emphasizing the role of member checking in interview-based studies. It illustrated how fresh insights from interviewees can transform previously concluded analyses, highlighting the critical need for adaptability and thoughtful decision-making in scholarly work. Moreover, the detailed exploration of this particular career interview provides broader insights into methodological approaches applicable to all forms of qualitative research, thereby enhancing our understanding of the nuanced decisions that researchers and editors make in their scholarly endeavors. This dialogue underscores the continuous and collaborative nature of scholarly research, where every conversation can lead to new understandings and refined perspectives.