The Ethical and Methodological Dilemma of Questioning the Truthfulness of a Participant’s Story: Using “Circulating Reference” to Enhance the Validity of Qualitative Research

Study Background

Although the study in which I encountered The Almost Too Incredible Story is not the main point of this article, some details might nevertheless provide the reader with useful context. The study was part of an evaluation of a national health care reform in Norway that increased citizens’ freedom of choice of secondary health care providers (Ervik et al., 2021). For the qualitative sub-study that I conducted, the purpose was to gain insight into how patients experienced the assumed increased freedom of choice (Holter, 2021). The research question was pursued from a critical realist perspective (Houston, 2001), and I was interested in what participants had experienced (i.e., what had happened and how this had affected them), mindful that these experiences would be filtered through the participants’ and my own subjectivity. Data were collected through individual interviews and focus groups, some online. These online focus groups took place on a safe technological platform and were text-based and asynchronous; that is, they resembled a discussion forum in which I posted new interview questions every morning, while the participants logged on at their convenience to answer and engage with each other’s replies. This type of focus group lasted for 1 week. The data were analysed with thematic analysis (Braun & Clarke, 2006), seeking patterns across individual stories.

The Ethical Problem of the Almost Too Incredible Story

It was in one of the online focus groups that I encountered The Almost Too Incredible Story. The interview had run smoothly for a few days, when suddenly a participant told (wrote) – almost as a side note to the question he was answering – a story that seemed almost too incredible to be true. This participant had been in need of extensive and specialized health care services, and he wrote that at one point, the leader of one institution had threatened him. The story was incredible and shocking – but not impossible or completely unbelievable. Up until that point, the participant had seemed engaged and credible, but I questioned the story. Could he have misinterpreted the interactions he had had with the health care provider? Could he have made it up for ulterior motives? This posed an ethical dilemma. I have never before felt inclined to doubt what a participant has told me. I respect participants and the time and effort they put into participating in research. I often find that they care deeply about the issues under study and that they participate because they want to contribute to the common good. Often, their stories describe times when they have faced someone more powerful. Therefore, I have an ethical obligation towards them, to trust the truthfulness of their experiences: The research process should be one where their voice is heard. On the other hand, I also have an ethical responsibility not to swallow whole an Almost Too Incredible Story. If I report The Almost Too Incredible Story, I vouch for it. People (should) trust research. Furthermore, the purpose of research is for it to do something once it is “out there”: to be built on and used by decision makers, in research, in education, or for other purposes. It is therefore of utmost importance that our research is trustworthy, and that we do not mindlessly bear the voice of untruthfulness.

So, what could be done with this Almost Too Incredible Story and the participant who told it? Several options came to mind. One option could have been to conduct a follow-up interview with the participant, respectfully checking and testing his story to decide whether it could be trusted. However, this solution was not available to me for practical reasons. Another option could be to leave the participant out of the study to protect the integrity of the research – however, this would be at the expense of a potentially truthful participant who has experienced something incredible and has invested time and energy to contribute through research. Alternatively, including the participant and the story in the study would mean risking the truthfulness and integrity of the study, of me as a researcher, and (if taken to the extreme) of research in general. Another option would be to leave this specific story out of the analysis but keeping the remaining data from this participant – leaving the question of the story’s truthfulness open. However, that leads to the even more troubling question: Without answering the question of this story’s trustworthiness – could anything this participant had said be trusted? I realized that if I did not trust the story, I could not trust the teller, and I could not trust anything he had said. If I did not trust the story, I must leave it all out of the analysis, along with all the time and effort this participant has invested in this research project. I realized: Ethically, there was no easy way out.

Methodological Issues: Validity, Rigour, Trustworthiness, and Intergrity

In addition to being an ethical question, the question of The Almost Too Incredible Story and the person who told it is a methodological question of research quality. Quality in qualitative methods – defined as integrity, rigour, trustworthiness, or simply validity and reliability – are issues that have been thoroughly outlined and debated in the literature (Guba, 1981; Levitt et al., 2017; Morse, 2015; Whittemore et al., 2001). The question of The Almost Too Incredible Story goes into the heart of a study’s validity and credibility. ¹ Validity or credibility involves the truth value of the findings (Guba, 1981; Whittemore et al., 2001) – these would be compromised if some of the data the study builds on is untrue. A more recently proposed quality standard, “fidelity to subject matter”, is defined as “an intimate connection that researchers can obtain with the phenomenon under study” (Levitt et al., 2017, p. 10). One way in which fidelity is increased is by choosing data sources and data that are deemed “adequate”, meaning “the quality and sufficiency of information as it provides close access to the richness of the subject matter” (p. 12). Again, an Almost Too Incredible Story should raise concerns because including it might diminish the data adequacy and thus the study’s fidelity to the subject matter.

There are a few relevant methodological strategies for addressing the problem of The Almost Too Incredible Story methodologically. One is through building a trusting relationship between interviewer and researcher, enabling both candid and rich descriptions (Hamilton, 2020; Morse, 2015). Building a trusting relationship between interviewer and interviewee might safeguard against the problem of The Almost Too Incredible Story, because a participant would presumably be less inclined to be untruthful in a trusting relationship, and the researcher is more likely to know that the participant can be trusted. However, it does not solve the problem if an Almost Too Incredible Story already exists in the data corpus. Another possible strategy for addressing the problem of The Almost Too Incredible Story is member checking; that is, returning to participants for verification to check the accuracy of the researcher’s account (Guba, 1981). This might seem an ideal solution to shed light on the researcher’s doubts of the story. However, returning to participants for verification is not always possible for practical reasons, and member checking as a validity strategy has been highly criticized (Morse, 2015). Further, if the participant has knowingly lied to the researcher, member checking seems almost the antithesis to the topic of this article, where the problem is reversed: What if it is not the researcher’s account that inadequately describes the participant’s report, but the participant’s account that inadequately describes what he has experienced? Thus, it seems that there is a need for a strategy if the matter cannot be resolved with the participant. Negative case analysis might seem applicable. With negative case analysis, deviations from the normal are pursued, saturated, and included in the analysis (Morse, 2015). However, pursuing and saturating negative cases is not possible if data collection is completed, as in the current example. Further, the problem with The Almost Too Incredible Story is not that it lies outside the norm established by the other participants; the problem is that the researcher doubts its truthfulness. Another potentially relevant analytic strategy is to try to distinguish between what a participant does and what he says he does. For example, the documentary method of interpretation (DMI; Philipps & Mrowczynski, 2019) is an analytic approach for uncovering what participants do and what drives their actions (implicit knowledge), which might differ from what they say they do (theoretical knowledge). This might be a useful approach for some Almost Too Incredible Stories; however, only if the problem with the story is a potential deviation between theoretical and implicit knowledge – not if it is told for other reasons or motivations (e.g., the participant’s misinterpretation of a situation, impression management, or for achieving recognition or sympathy).

Thus, although there are techniques available for handling some situations in which one encounters an Almost Too Incredible Story, none of these are applicable if data collection is completed, if member checking is not an alternative, and if the story may be a result of the participant’s misinterpretation or ulterior motives. Further, although a study’s validity undeniably is the responsibility of the researcher, participants certainly contribute with data. Despite this, I have not seen any discussion of how a study’s validity also relies on the participant’s truthfulness (or lack thereof). Here seems to be a missing piece: Although we as researchers by default trust participants, it seems naïve to believe that none can have ulterior motives or conflicting interests that might influence their inclination to telling the truth.

One way of approaching this dilemma might be through a closer look at the knowledge production involved in qualitative inquiry. In an interview study, we may think of each story told or reply given as a data point and each participant a collection of these data points (Figure 1). The question is: What role does each data point play in the presented result: the final analysis? And how will the result be affected if one of these data points is untrue?OPEN IN VIEWER

In answering these questions, I turn to Bruno Latour’s concept of circulating reference (Latour, 1999) and its usefulness for thinking about qualitative research – specifically, the relationship between qualitative findings and the messy, chaotic world which they attempt to describe or explain. Armed with his theory of knowledge production, and the application of this theory to qualitative inquiry, I will return to the ethical and methodological problem of The Almost Too Incredible Story.

Qualitative Inquiry as a Process of Circulating Reference

In his book “Pandora’s hope: Essays on the reality of science studies”, Latour (1999) describes how natural scientists move from matter (the world in all its complexity) to sign (the final representation or description of that world in a scientific publication). His project is to demonstrate the trustworthiness of this process, and to denounce the accusation sometimes made towards science: that the final sign is an arbitrate construction of reality, made in an abrupt leap from matter to sign. Rather, he shows how natural scientists move gradually from matter to sign in meticulous, sideways movements – which, if each movement is done correctly, result in a final representation of the world that is truthful (although decontextualized). He offers the term circulating reference to describe this process. Circulating reference can be understood as a ladder: For each step, the scientist draws the most essential aspects from the previous step to create a new representation. This representation is more abstract and contains less context than the previous step, but also provides the scientist with more condensed information concerning the question (s)he seeks to answer. The representation then becomes the next reference, from which the scientist repeats the process of drawing the most essential aspects and leaving more context behind. This process of circulating reference repeats until the scientist reaches the final representation – for example, a figure summarizing the results in the final report. For each step, the scientist tries to make the difference between reference and representation as small as possible, retaining only what (s)he considers to be essential elements. However, for each step, there will always be a small “jump” between reference and representation – they are not the same, the second is qualitatively different from the first, with less context and more condensed information.

What Latour shows is that there is no one “grand leap” from thing to representation, but that there are many, small leaps, linked together in a chain (or ladder), until the final reference that appears in the paper or report. The truth-value of this final reference depends on the continuity of the chain – that the difference in reference and representation is kept as small as possible and that all essential information is retained – and that one may, through the chain, go back from reference to original matter. Latour argues that this circulating reference is absolutely necessary for gaining knowledge about the world, because through discarding context and retaining essential elements the scientist is able to bring together, in a very literal sense, similar examples of the same element under study, in a way that enables the discovery of patterns that otherwise would have been impossible to see. In other words, for each step, abstraction enables the scientist to see patterns and draw insights that would not be possible in a previous, more contextualized step.

I suggest that Latour’s concept of circulating reference can be applied to the qualitative research process within the social sciences. Just as in quantitative research and the natural sciences, there is no “grand leap” from experience to representation. Rather, there are a series of small leaps, moving from written (or visual) raw data through different forms of organization and abstraction until the final representation in the published work: A set of themes, a theory, a phenomenological description (Figure 2). The validity of this final representation rests on each of the small leaps having been made as truthfully and accurately as possible.OPEN IN VIEWER

The process of circulating reference in qualitative inquiry can be exemplified with the research project referred to at the beginning of this article: An interview study of patients’ experiences of health care services based on thematic analysis (Braun & Clarke, 2006). In the chaotic messiness of real lives, people seek the help of health care services when they experience illness or disturbing symptoms. Embedded in their respective contexts, it is difficult to say something meaningful about how “patients” experience “health care”; the sum of many patients’ experiences is not immediately available. Instead, to achieve such a feat, their stories must go through a series of abstractions to remove nonessential context and arrive at increasingly condensed information: the ladder of circulating reference, the steps of which will be specified in the following.

First, Person X has a health issue. He goes to see his medical doctor. He has a very specific story about his complaints, the initial encounter with his doctor, and potential subsequent tests, follow-up appointments, or referrals to secondary care, all set within the complex and chaotic context of his life. (This is also true for every other patient involved in the study.) As the researcher of the study, I meet with Person X for a qualitative interview and ask him about his experiences with the health care services. These experiences are inherently tied to a range of aspects of his life – where he lives, the health issues he has, his way of reasoning, the choices he has. He nevertheless recounts his experiences with health care, constructing a more linear, tidy story by answering my questions. The words he uses are the best representations he can find of his inner experience – his recollections of what has happened – and they form a story, “Story 1” (Figure 2). As a researcher-interviewer, I try to imagine the inner experience the words are representing. Then I pull out the parts I find most interesting, and try to describe the inner experience I imagine he is referring to, testing my understanding by asking: Did you mean this? He agrees or corrects me, and we continue until I have no further questions and he has nothing to add. This process repeats for each interview with each participant, resulting in the first representation of people’s experiences with health care: the collection of recorded interviews (“Reference A”, Figure 2).

Reference A is then transformed for easier analysis by transcribing the audio-taped interviews. In transcribing, I make decisions that will affect how much of the information in Reference A that will be retained – but some of the context from the audio recording of an encounter between two people will undoubtedly be lost. The next representation in the chain of circulating reference is therefore the transcribed interviews (“Reference B”, Figure 2). Collating all the transcriptions in front of me, I now have access to something completely new: written representations of one specific aspect of the otherwise contextualized lives of different people – people separated by time, space, and circumstances, but sharing the experience of seeking health care services. These words are uttered at different times (weeks, months, even years apart) and in different locations, but all are uttered as responses to my interview questions, in the effort of making sense of experiences taking place under similar circumstances (seeking health care because of illness or disturbing symptoms).

Moving from the transcripts of Reference B I start coding for information relevant to the research question, resulting in the next representation of how people experience health care services: a provisional list of inductive codes (“Reference C”, Figure 2). Looking at these codes, I acknowledge that the context of each transcribed interview is now lost. Even more so, other research questions would have led to a completely different set of codes, as the material could have been coded in a multitude of ways. The interpretative process I have undertaken also means that another researcher would have arrived at a different set of codes. Nevertheless, the codes in front of me represent my best effort of summing up the essential elements of how people experience health care. Together, they form a new level of abstraction with condensed information about the topic under study.

Next, the initial codes of Reference C are organized into higher-order codes, categories, or themes. I use mind-maps to organize codes that are similar or appear to be related, and I use tables to collate codes together with data excerpts. Whether mind-maps, tables, or other tools are used for this process, these intermediary steps of analysis are the next representation of how people experience health care (“Reference D”, Figure 2). Much context is now lost – there is no longer the richness of a person-to-person conversation, with all its words, non-verbal behaviour, in its social encounter between two people. Not to mention that I have never had direct access to the actual experiences under study, but rather, the participants re-worked compilations of these experiences as a response to the interview questions. However, I now have access to something no-one could ever have accessed in any other way – I can see before me, simultaneously, words uttered by people who have never met, about the same feature of a common experience. Reference D therefore contains more condensed information than the transcripts of Reference B, or even the disjointed list of codes of Reference C. Looking at the collated material on the screen before me I can discern patterns that might constitute a more substantial element in how people experience health care.

Working with the collections of codes and data excerpts in Reference D, I move to the final step of the ladder of circulating reference for this example: a list of themes that describe how the participants in the study experience health care (“Final internal reference”, Figure 2). This final internal reference is made possible through the small steps of decontextualization and abstraction from reference to reference. To keep this process trustworthy, each step must be made as carefully and diligently as possible. It should also be possible to retrace my steps through consecutive representations, moving from the list of themes (final internal reference) to mind-maps and tables (D) to initial codes (C) to coded data segments within the transcript (B) to (if possible) the audio recording (A). This is as far as the trail can go – it cannot go back to the actual conversation, or to the actual experiences of the participants.

Another significant point in qualitative inquiry: Throughout this process of abstraction, a parallel process has run, which might be called the Me Transformed: Each step has transformed me as a researcher and changed my understanding of the topic under study, bringing it closer to the experiences of the people I have encountered. This changes the way I understand the next story, and the next. Thus, the Me Transformed also influences the next step in analysis. As such, each step has informed and been informed by the Me Transformed until the final reference.

This final reference has lost all the original context of individual, situated experiences. But in its abstraction, it has made it possible to describe and discuss patterns of experiencing health care that would never be visible in the original, messy contexts of a multitude of people’s lives. This is how qualitative knowledge is produced within a critical realist framework.

Using Circulating Reference to Handle the Almost Too Incredible Story

The principle of circulating reference depends upon each step of abstraction – from reference to representation – being as small and truthful as possible. That means that the information going from Reference A to Reference B must be true (Figure 2). This poses a problem when it comes to The Almost Too Incredible Story: If the story is not true, the following abstractions up the ladder of circulating reference will equally not be true. Further, if the participant has told one untrue story, we must consider the possibility that nothing this participant has told is true. However, keeping in mind our ethical responsibility towards this participant – and the possibility that his story might be true – we do not light-heartedly discard his interview.

I propose an approach using the ladder of circulating reference to first decide whether The Almost Too Incredible Story poses a problem, and only if it does, to consider which option to choose. The approach involves four steps: First, to analyse all the data (including The Almost Too Incredible Story, as if the story were believed). Then, with a completed analysis, to systematically trace the role of The Almost Too Incredible Story (and the rest of the data of the participant who told it) throughout the process: Through the steps in the ladder of circulating reference and in forming the researcher’s understanding – the Me Transformed (Box 1).

1. Analyse all the collected data as if it all were believed to be true.

2. Retrace the role of any data you doubt the truthfulness of through the steps of circulating reference (Figure 2).

3. Retrace the role of any data you doubt the truthfulness of through your transformed understanding of the phenomenon under study.

4. A) If the data you doubt have not been decisive for the results, the analysis can be kept as it is.

B) If the data you doubt have been decisive for the results, the validity may be threatened, and you may choose to:

i) Return to the participant to clarify (if possible).

ii) Trust the story and state the concerns you have about truthfulness as a potential limitation.

iii) Discard all the data from this participant from the analysis.

iv) Discard only the data that does not align with other participants’ accounts.

Box 1: Proposed strategy if a researcher doubts the truthfulness of a data segment in the data corpus.

If The Almost Too Incredible Story, and other data from this participant, follow the movements of the other participants’ stories in all important ways in relation to the research question, the question of Too Incredible or not need not be raised – because the story was not decisive for the results. The data can be kept, and the researcher can rest assured that the final representation is as truthful as possible. However, if tracing the role of The Almost Too Incredible Story and the participant who told it through the steps of circulating reference reveals that either of them have been decisive for the resulting analysis, the researcher needs to consider what measures to take: Keeping in mind the possibility that the participant has not been telling the truth, there is a chance that the final reference is not true. The researcher now has several options (Box 1):

One option is to return to the participant to clarify. However, as stated at the beginning of this article, this may not be possible. This option also brings about a different ethical problem, namely, how to conduct such a follow-up interview in a respectful way when the reason for conducting this interview is that you question the participant’s truthfulness. Another option is to trust the participant, and report the results as they are, including The Almost Too Incredible Story. This runs the risk of using your authority as a researcher to validate an untrue story. Once “out there”, you do not know the possible individual and societal consequences of presenting this story as “true” (if it is not). Further, if it should later turn out that the story was untrue, it will certainly affect your reputation and career, and potentially the trust in research and social science in general as well. Nevertheless, in some situations this may be the option that the researcher chooses. If so, the researcher’s concerns about the truthfulness of certain data segments should be reported as a study limitation. An option in the other extreme may be to discard all the data from this participant from the analysis and re-analyse the remaining data corpus. As stated earlier, this runs the risk of distrusting a truthful participant who has experienced something incredible and invested time and effort in participating in your study. It also runs the risk of dismissing important aspects of the phenomenon under study.

A final option is to look at how data sequences from this participant align with data from the other participants. Data sequences that align with the other participants’ pattern are retained; however, data sequences that stand out from the rest of the data material are taken out of the analysis and the analysis is adjusted accordingly. In this option, the researcher’s ambiguity is left unresolved, and the participant is not left completely out of the analysis, but nor is his story allowed to be decisive for the results. This approach might be criticized: it runs the danger of bordering data construction, and indeed, removing data that has importance for the results is a huge responsibility for the researcher and therefore should be performed with caution. To ensure transparency, the final report should include information about this methodological choice. For example, the researcher may state that one participant shared a story that at certain points seemed almost unbelievable and differed significantly from the other participants’ stories, causing him or her to disregard the most extreme elements of this participant’s story in the analysis to ensure the credibility and trustworthiness of the research. However, exactly what was incredible and how the story differed from the other stories should not be disclosed, both to support the participant’s anonymity and to keep the participant from identifying himself as someone who’s truthfulness was doubted.

As this review of options reveal, there still is no easy answer to the question of what to do if you as a researcher encounter an Almost Too Incredible Story. None of the options are inherently superior to the other, and all involve pros and cons. Further, most of these options were available before going through the steps of circulating reference with a particular focus on this specific participant. However, I argue that going through these steps – even though it entails extra labour – has two advantages. First, it may reveal that The Almost Too Incredible Story and the participant who told it has not been decisive for the results, and that the researcher can safely refrain from doing anything more about it. Second, if it turns out that this participant has played an important part in the analysis, using the ladder of circulating reference to decide exactly where and how much can aid in deciding what to do, and in undertaking this process with thoroughness and transparency.

In the study in which I first encountered the dilemma of The Almost Too Incredible Story, the approach outlined above was used. Initially, the data from the focus group of this participant was kept in its entirety in the analysis, including this specific story. Guided by the research question of the study, “How do people experience and make use of the increased freedom of choice that they have as a result of the reform?”, the analysis led to four themes: many patients were unaware of their freedom to choose or got to know this right by coincidence (theme 1); practitioners who had referred patients to secondary care acted in very different ways, from supporting the patient’s right to choose, to choosing for them, to actively working against them (theme 2); patients differed in their ability to manoeuvre through the system, depending on their knowledge, experience, technical skills, mental energy, and support (theme 3); and patients who did experience increased freedom of choice used it to tailor their choice to their specific life circumstances and found this freedom valuable in its own right (theme 4) (Holter, 2021). With a completed analysis I returned to the steps of circulating reference and the role played by The Almost Too Incredible Story. I concluded that this participant had not influenced the final themes or transformed my understanding in relation to the research questions in any significant way. His story had followed the general pattern: He was one of several participants who had been well informed of their right to choose (theme 1); he had experienced referring practitioners who were not helpful, as several others (theme 2); he was one of several resourceful patients who managed quite well in manoeuvring through the system (theme 3); and he used his freedom of choice to find what he considered the best treatment option (theme 4). Hence, his data could be kept as part of the results and the analysis need not be changed.

It should be noted that this methodological approach is limited to (critical) realist research studies aimed at describing the actual or causal level of reality (Houston, 2001). The critical realist perspective holds that it is possible to gain knowledge about “a reality out there independent of our thoughts and impressions” (Houston, 2001, p. 850), although this reality is always seen through the subjectivity of both participant and researcher. Social constructionist research would not fit the suggested approach, because the “reality” of interest is that which is constructed through culture and language, and a participant’s story will, from this perspective, always be “true”. Moreover, whether the approach fits a research question addressed with a critical realist perspective depends on what level of reality is the focus of enquiry. According to Bhaskar, the originator of the critical realist perspective, an incident has three levels of reality: What happened (the actual level); why and how it happened (the causal level, consisting of a multitude of interacting mechanisms); and how the incident is experienced (empirical level) (Houston, 2001). The current approach is appropriate if the research is concerned with the actual or causal level, but less relevant to the empirical level of reality, because, as stated above, a person’s experience will always be true. However, the approach may be relevant also for this level of reality if a discrepancy is suspected between what the participant has experienced and what he or she claims to have experienced. Finally, when considering the issue of The Almost Too Incredible Story and the participant who told it, the researcher should also be mindful that what is tellable in an interview depends on memory, and that memories of an event may change retrospectively ² . Further, that the memories themselves depend on “cultural categories that shape what is thinkable and what is not, what is counted as appropriate, what is valued, what is noteworthy, and so on” (Atkinson & Coffey, 2003, p. 424). Nevertheless, the responsibility of the researcher remains the same: To judge what to do when a participant tells an Almost Too Incredible Story.

In conclusion, for those of us sometimes working within a critical realist framework, situations may arise in which we doubt the truthfulness of a participant’s story. In these situations, we are best served by considering this issue carefully and systematically, and not jumping to any extreme conclusion; neither blindly trusting the participant on default nor automatically discarding his or her data from the data corpus. The alternative, systematic approach hereby suggested is first, to review the steps of decontextualization and abstraction in the ladder of circulating reference; next, to trace how the participant has influenced each step. This will help the researcher establish whether the truthfulness of the story is important for the validity of the results. If the truthfulness of the story is decided to be of importance, the researcher must decide what he or she believes to be the best choice, which might include returning to the participant to clarify; trusting the story and keeping it as part of the data corpus; discarding all the participant’s data from the analysis; or discarding only the data that does not align with other participants’ accounts. The chosen approach should be described in publications along with the rationale behind the choice and potential related limitations. In the best of situations, by using this approach, the result may be that both participant and process are honoured: the data is kept (honouring the participant) and the trustworthiness of the final representation does not stand or fall with this participant’s account (honouring one’s responsibility as a researcher).