Evaluating Complex Health Interventions With Randomized Controlled Trials: How Do We Improve the Use of Qualitative Methods?

Abstract

Qualitative methods are underutilized in health intervention evaluation, and overshadowed by the importance placed on randomized controlled trials (RCTs). This Commentary describes how innovative qualitative methods are being used as part of RCTs, drawing on articles included in a special issue of Qualitative Health Research on this topic. The articles’ insights and a review of innovative qualitative methods described in trial protocols highlights a lack of attention to structural inequalities as a causal mechanism for understanding human behavior. We situate this gap within some well-known constraints of RCT methodologies, and a discussion of alternative RCT approaches that hold promise for bringing qualitative methods center stage in intervention evaluation, including adaptive designs, pragmatic trials, and realist RCTs. To address the power hierarchies of health evaluation research, however, we argue that a fundamental shift needs to take place away from a focus on RCTs and toward studies of health interventions.

Keywords

methodology intervention evaluation research design qualitative global health

Introduction

Qualitative methods offer enormously important insights for understanding the impacts that interventions can have on improving human health. A vast array of qualitative methods have been developed over the past 50 years. They have generated better understandings of people’s perceptions of their own health needs and hopes, have improved knowledge of the social and structural drivers of health behaviors (Blankenship, Bray, & Merson, 2000), and have contributed to the involvement and participation of individuals in research about their own health (Blumenthal & DiClemente, 2013). However, qualitative methods have been underutilized in the evaluation of health interventions, and overshadowed by the importance placed on measuring the impact of interventions using randomized controlled trials (RCTs).

The limited utilization of qualitative methods alongside RCTs is evident in a rapid review we conducted of qualitative methods currently being used alongside RCTs of complex health interventions. The aim of our review was to better understand the “state of the art” in this specific area of mixed-methods health evaluation research for this Commentary. Our focus in the review was on innovative methods, which we defined as the use of qualitative methods beyond standard interviews and focus group discussions that ask trial participants about their impressions of the intervention. We generated a list of 654 qualitative research methods from a search of 25 qualitative methodology journals using the terms “innovat* new novel emerg*,” and then used this list as key terms in a search of protocols published since 2012 and registered with the International Standard Randomised Controlled Trials Number (ISRCTN) trial database. Our search showed that while 1,452 of the registered trial protocols mentioned some form of qualitative research, only 34 of these discussed more innovative qualitative methods.

Qualitative methods were most often employed during trial process evaluations or as formative research in the pre-trial phase. Protocols rarely went into depth about the details of the methods used or explained how their analysis would contribute to the trial results. This finding is consistent with other scholars who have recognized an enormous gap in the use of theoretically informed qualitative methods alongside RCTs of complex interventions (Lewin, Glenton, & Oxman, 2009; Rapport et al., 2013). This gap significantly undermines the potential learning that could be gained from better understandings of the successes and failures of interventions in different contexts using qualitative inquiry.

Despite widespread acceptance of RCT design as the “gold standard” for evaluating interventions, RCTs are fairly limited in their explanatory power (Mowat, Subramanian, & Kawachi, 2018). This is particularly true for the evaluation of “complex” or “structural” health interventions, in which multiple components or causes of poor health outcomes are addressed as part of the intervention (Craig et al., 2008; Petticrew, 2011). Trial designs produce limited insights about the reasons why interventions succeed or fail in changing health outcomes, something that many qualitative methods have been explicitly designed to do (Grypdonck, 2006). Moreover, only 13% of RCTs currently use qualitative methods as part of intervention evaluation (O’Cathain et al., 2014). Our starting point in this Commentary is that producing excellent evaluation requires the full integration of qualitative methods used alongside RCTs. This means moving beyond reflections of how qualitative methods can be included in trials and toward questions of how qualitative methods can best be used to understand interventions and their effects in health research.

In an effort to move toward this integration of qualitative and quantitative methods in health evaluation research, this special issue draws on articles that all aim to produce “better” interventions. They do this through suggesting potential improvements to trials and interventions brought about by the integration of robust theoretically informed qualitative methods. The overarching aim of the articles included in this special issue is to contribute to current methodological discussions of how to best capture the effectiveness of complex health interventions for the populations they serve. Through engaging in the intersection of quantitative and qualitative research cultures in pursuit of this goal, these articles represent what Goertz and Mahoney (2012) refer to as “the most exciting social science in coming years” (p. 230). In this introduction, we draw on our synthesis of the articles included in the special issue and our broader “state of the art” review of research protocols to summarize three areas of conceptual development needed to fulfill this potential for the use of qualitative methods alongside RCTs.

Improving Trials Versus Improving Interventions

The five articles included in this special issue of Qualitative Health Research address improving health evaluation research through the uptake of qualitative methods alongside trials from different standpoints. On one hand, there are those who are interested in improving current methodological approaches to the evaluation of health interventions and who prioritize the potential for qualitative methods to improve trial protocols. For example, Turner, Percival, Kessler, and Donovan (2018) describe the process of triangulating qualitative data from different related studies to inform the development of trials by informing decisions about treatment arms and factors hindering participant engagement. However, other scholars are interested in how qualitative methods can improve the intervention being evaluated through offering insights about the context or everyday realities of participants’ lives. This is displayed in the article by Bernays, Paparini, Namukwaya, and Seeley (2018), which describes the use of audio diaries as a means of capturing the everyday experiences of adolescent participants involved in the trial outside of the clinical setting. In this article, the authors take account of the potential for audio diaries to give this inside account of people’s inner life worlds, but also describe the failure of the audio diaries method for not accounting for the lack of secrecy that is significant to adolescents in Uganda. These articles each reflect quite different starting points for highlighting the value of qualitative methods within RCTs: One focused on maximizing the value of the trial protocol and the other maximizing the effectiveness of the intervention.

This is an important distinction to make. Reynolds and colleagues (2014) have synthesized lessons learned from the practice of evaluation across nine different complex health interventions, and point to the ways in which interventions are often conflated with their evaluation in practice. This occurs where participants change their behavior, in ways that either improve or reduce effectiveness of the intervention, because they feel they are being observed by evaluation tools such as surveys. Practitioners may also experience this “Hawthorne effect” (Wickström & Bendix, 2000), changing their implementation of the intervention simply because it is being evaluated. Clear boundaries between interventions and evaluations are certainly needed, as highlighted by Reynolds and colleagues, but there may also be the need for greater recognition and explicit inquiry around the limitations of strict evaluation procedures in practice.

At the heart of the tension between researchers wanting to maintain strict adherence to evaluation standards and those calling for adaptation and flexibility for complex interventions (Leeming, Marshall, & Locke, 2017; Wells, Williams, Treweek, Coyle, & Taylor, 2012) are two fundamentally different research cultures. The prioritization of an adaptable intervention implicitly takes a causes-of-effects approach of qualitative researchers more broadly (Goertz & Mahoney, 2012) by emphasizing the importance of understanding how the intervention works and focusing on potential improvements given broader individual, inter-personal, and contextual dynamics. In contrast, Goertz and Mahoney (2012) argue, the prioritizing of the rigor of the evaluation procedures implicitly takes an effects-of-causes approach by emphasizing the importance of ensuring good measurement through appropriate governance. These differences can raise tensions between qualitative and quantitative researchers. For instance, the idea of blinding both participants and researchers to information about which allocation arm they have allocated to is perceived quite differently in the two research approaches outlined by Goertz and Mahoney (2012). An effects-of-causes approach inherently perceives blinding as a critical component of good governance procedures to ensure that the effects of an intervention are “clean” and not influenced by the preconceptions or biases of either participants or researchers. In contrast, a causes-of-effects approach perceives blinding as attempting to remove a critically important source of information by trying to erase the contextual causes of social interaction implicit in both interventions and their evaluation. This perspective is evident in scholarly arguments about the inherently socially constructed nature of intervention and its measurement (Adams, 2016). Researchers embedded in either of these approaches can find it difficult to understand the alternative or to see its value as a research strategy.

Paradigms of Qualitative Methods Used for Trials (and Gaps)

Since the emergence of qualitative methods, there have been persistent efforts to define the epistemological paradigms underlying different research traditions and their respective usefulness in understanding aspects of structure and agency (Denzin, 2010; Flick, 2018). It is useful to return to this debate in thinking about the potential for qualitative methods to be included as an essential component of evaluation research method, particularly for non-clinical trials focused on behavioral interventions. Parsons (2007) suggests four types of scientific explanations of human action, which can be used to identify different paradigms underpinning evaluations of complex health interventions, and include; Structural, institutional, ideational, and psychological. This typology is helpful in identifying the assumptions made by researchers when evaluating complex interventions, particularly in identifying and evaluating the fit of the intervention with a “theory of change” or the causal mechanisms that bring about improvements in health as part of an intervention. Structural mechanisms refer to causal explanations that derive from the social and material structures of power, such as the distribution of wealth or gender inequality. Institutional mechanisms refer to the constraints or rules imposed on individuals by organizations or legal frameworks. Ideational mechanisms pay attention to what people do because of affective beliefs, cultural, and historical thought patterns that organize their way of thinking. Psychological mechanisms refer to cognitive or instinctual elements that also organize thought patterns, but that are “hard-wired” into the brain and generalizable across all human beings.

Of these causal explanations/mechanisms, the most commonly used within RCTs of complex health interventions is the psychological because of its clear affiliation with cultural and medical approaches to health and illness (see Table 1). The assumption of generalizability for psychological explanations for intervention outcomes is reassuring for those who see the potential intervention as something that should be transferrable across contexts. This has contributed to the uptake of qualitative tools alongside RCTs that explicitly try to capture psychological processes of behavior change. For example, in the article by Coventry et al. (2018), a longitudinal qualitative study of a health coaching intervention undergoing an RCT is used to identify the “mechanisms of impact and contextual factors” (p. 2) that may influence a telephone health coaching intervention aimed at changing the health-related behaviors of those with mild chronic obstructive pulmonary disease (COPD). This is a prime example of the use of qualitative methods to identify psychological thought-patterns to address specific health behaviors as part of an intervention.

Table 1.

Protocols reporting qualitative methods according to Parson’s (2007) causal explanations for human action.

Causal Explanations for Human Action	Definitions From Parsons 2007	Protocols Reporting Qualitative Methods by ISRCTN Registration Number	Research Articles Included in This Special Issue of QHR
Structural	Social and material structures of power, such as the distribution of wealth or gender inequality	(n = 2) ISRCTN50036837 ISRCTN64692465
Institutional	Constraints or rules imposed on individuals by organizations or legal frameworks	(n = 15) ISRCTN15330120 ISRCTN27989430 ISRCTN11112933 ISRCTN12279556 ISRCTN20414435 ISRCTN47349949 ISRCTN51780222 ISRCTN15949118 ISRCTN15551042 ISRCTN32174198 ISRCTN56270821 ISRCTN77334588 ISRCTN10443908 ISRCTN24905314 ISRCTN17107760	Improving recruitment to randomized trials by qualitatively investigating recruitment strategies: the QuinteT approach (Rooshenas et al., 2019) Synthesizing qualitative datasets to improve the design of clinical trials evaluating complex health interventions: a worked example (Turner, Percival, Kessler, & Donovan, 2018)
Ideational	Affective beliefs, cultural, and historical thought patterns that organize an individual’s way of thinking	(n = 4) ISRCTN42717948 ISRCTN65922679 ISRCTN14058106 ISRCTN97447076	Value and limitations of Broad Brush Surveys used in Community Randomized Trials in South Africa (Bond et al., 2018) A failed method? Reflections on using audio diaries in Uganda with young people growing up with HIV in the Breather Trial (Bernays, Paparini, Namukwaya, & Seeley, 2018)
Psychological	Cognitive or instinctual elements that organize thought-patterns, are “hard-wired” into the brain, and are generalizable	(n = 13) ISRCTN22393598 ISRCTN14526380 ISRCTN34354086 ISRCTN12464275 ISRCTN74331412 ISRCTN80222865 ISRCTN10148247 ISRCTN21656568 ISRCTN71109104 ISRCTN26184967 ISRCTN14282228 ISRCTN58861782 ISRCTN12826685	The push and pull of self-managing mild COPD: a longitudinal qualitative evaluation of a nurse-led telephone health coaching intervention (Coventry et al., 2018)
TOTAL		N = 34

Note. QHR = Qualitative Health Research; COPD = chronic obstructive pulmonary disease; ISRCTN = International Standard Randomised Controlled Trials Number.

Institutional explanations for action are also evident in a significant number of RCTs focused on changing the response of health organizations to health issues, such as hospitals and health clinics. Fifteen of the 34 protocols (44%) in our 2018 review focus on institutional activities. Attention to institutional processes can be seen in two of the studies published in this special issue, both of which suggest the triangulation of qualitative methods at different stages of the trial to improve aspects of the evaluation. In the first instance, Rooshenas and colleagues (2019) draw on their involvement with 15 “challenging” RCTs to discuss how multiple qualitative methods (including observation, document analysis, pathway mapping, and interviews) can be triangulated to improve recruitment to trials. The proposed methods aim to overcome many of the challenges that recruiters with strict trial protocols face when they encounter the realities of institutionally embedded practices, rules, and power dynamics. Understanding these realities from different perspectives, using different qualitative methods, allows for better recruitment practices to be developed in alignment with the institutional environment in which the trial is taking place. A second example of triangulation to address institutional practices appears in the article by Turner and colleagues (2018) who describe the valuable insights gained through synthesizing qualitative datasets across different primary care depression interventions. The authors highlight how this unique approach to synthesizing qualitative data can improve the trial protocol by designing more appropriate interventions and strategies for improving the retention of participants. This focus on triangulation in both articles is perceived to improve the robustness of the data. However, triangulation as a means of improving the objectivity of the researcher’s interpretation of the data is implicitly aligned with a post-positivist paradigm, dominant within public health research (Flick, 1992). It therefore stands apart from more structural research paradigms that instead emphasize the need for critical or participatory approaches to disrupting unequal power structures, which we return to.

Ideational explanations for people’s health behaviors are also considered as part of RCTs; however, this is most frequently done as part of a formative stage in RCT research. Although many social scientists would take an interest in describing the health perceptions of a particular population, when placed in relation to intervention evaluation this type of rich description is often perceived to be most valuable for designing the intervention itself. A diverse range of qualitative methods are available to do this early formative work, including participant observation (Dahlke, Hall, & Phinney, 2015; Peacock, Khumalo, & McNab, 2006), spiral walks (Ngwenya et al., 2018), audio diaries (Mupambireyi & Bernays, 2018), life history interviews (Harris & Rhodes, 2018), narrative interviews (Vindrola-Padros & Johnson, 2014), and illustrated story cards (Karnieli-Miller, Nissim, & Goldberg, 2017). The Broad Brush Survey method described by Bond et al. (2018) in this special issue of Qualitative Health Research outlines a comprehensive means of providing rich description of the social context to inform trial design decisions early on using participatory qualitative methods. They suggest a streamlined and practical approach to analyzing the huge amount of data that can emerge from these participatory activities through generating “community profiles” that can then be used to inform a trial.

Using in-depth qualitative research to inform the design of a trial is no doubt tremendously valuable, however, limiting the use of qualitative research to the formative stage of trials may also be undermining its inherent value for developing better understandings of interventions more broadly. From a social science perspective, the notion of whether or not we can intervene in social dynamics is often questioned due to the complex social and political dynamics that influence human action and behavioral changes (Howarth et al., 2013). Qualitative methods can play a role in investigating these complexities, as demonstrated in the article by Bernays and colleagues (2018). In this article, the authors discuss the “failure” of using audio diaries with young people living with HIV in Uganda to understand their experiences of adhering to treatment outside of the clinical setting, and the difficulties participants faced in maintaining the privacy they need to engage with the method. Although the method may have failed in generating data for the trial, the insights gained about the home environment of young people in Uganda is rich and insightful, and enormously valuable for developing better understandings of both adherence and intervention design for the future.

This raises an important concern for trialists who are interested in using qualitative methods to understand an intervention while a trial is in progress. Namely, that the qualitative research may highlight how an intervention should be changed to improve its potential impact on the primary outcome while the trial is taking place. This raises both a practical and an ethical challenge for RCTs. Practically, a strict clinical trial approach must ensure that the intervention is delivered consistently throughout the testing phase of the RCT and that the outcomes are directly related to the effects of the intervention and not confounded by other factors. However, in dealing with complex health interventions there is a growing interest in trial designs that are either adaptive (Montgomery, 2017) or pragmatic (Schwartz & Lellouch, 2009). Both designs allow for changes to interventions to be made midway through a trial as part of a broader recognition that interventions need to be adapted to socio-cultural context and insights that might arise. This demonstrates some of the ways in which methodological discussions about RCTs are being influenced by the increased use of qualitative approaches and the insights they provide.

However, persistent gaps remain. In contrast to institutional, psychological, and ideational explanations for action, structural explanations are far less frequently represented in the qualitative research currently utilized alongside RCTs of complex interventions. One reason for this absence is the constraints of RCT methodologies and the need to control for the influence of the social context on the intervention to appropriately measure its effects (Ravallion, 2009). Moreover, as structural explanations are inherently contextual, these are often excluded from the interests of trialists beyond their role as potentially confounding factors. Qualitative researchers interested in structural explanations may in turn find that the depth required to make structural causal claims has been stripped from the project design, and that an alternative or add-on research project is needed to investigate the role of social structures in the intervention’s success/failure. There are some excellent examples of research that has explored structural mechanisms as part of RCTs through the use of qualitative methods such as photovoice (Sims-Gould, Clarke, Ashe, Naslund, & Liu-Ambrose, 2010), and arts-based methods including community mapping (Falb et al., 2016). However, the studies published using these methods are rarely published together with the trial results. This trend is also apparent in this special issue of Qualitative Health Research, where the articles submitted often referred to the quantitative trial findings that had been published, but do not include these findings within the qualitative article. The consequence of this is that qualitative research is often positioned as a secondary output of the research, with the quantitative trial findings taking center stage.

How Do We Move Toward a Mixed Methods Approach?

The articles included in this special issue and our rapid review of trial protocols both reveal significant gaps in the ways qualitative methods are currently being used alongside trials of complex health interventions. Qualitative methods have emerged from many different research paradigms (Flick, 2018); however, their uptake alongside trials has been largely constrained by a biomedical post-positivist paradigm interested in the objective observation of phenomenon. This radically limits the methods that are currently being used to evaluate interventions, and side-lines much of the structural and participatory approaches that have emerged from critical schools of thought within health, anthropology, psychology, and sociology. Instead, qualitative researchers are being increasingly asked to demonstrate how their methods produce objective observations of the world, and asked to bring in post-positive approaches to triangulation as part of qualitative inquiry (Farmer, Robinson, Elliott, & Eyles, 2006).

This is the result of significant power imbalances between quantitative and qualitative research paradigms and cultures. The contradictory research approaches outlined by Goertz and Mahoney (2012) create barriers to understanding the perspective of the other, but are also deeply characterized by a hierarchy that positions quantitative results as more valuable, policy-oriented, and actionable than qualitative findings. From the amount of funding to the number of papers published in top journals this hierarchy is explicit and tangible in the disadvantages it creates for career recognition and advancement among qualitative researchers working in the health sciences.

More implicitly, we can see the power hierarchy of quantitative over qualitative research playing a role in the cooptation of numerous concepts originating from qualitative research and their re-interpretation through the dominant post-positivist lens of health and medical sciences. For instance, reflexivity has become a means of interpreting quantitative results rather than unmasking the power relations embedded in the perspective of academic researchers as it was originally intended. A key example of this is Reynolds and colleagues (2014) discussion of the use of reflexivity within trials as a process of considering and reconsidering “how evaluation activities can impact meaningful interpretation of trial results” (p. 10). This is a radical departure from how reflexivity has been conceptualized by researchers such as Koch and Harrington (1998), and Dowling (2006) who use reflexivity within their research to shed light on the power relations that are embedded within research and in the socially embedded exchange between researcher and participants (Finlay & Gough, 2008; Rae & Green, 2016).

The combination of qualitative and quantitative research as part of mixed methods designs is widely considered best practice within health evaluation research (Johnstone, 2004; Morgan, 1998). However, the real benefits of qualitative research for trials will only be realized if opportunities exist for the historical and epistemological lessons of qualitative research to be fully and equally taken account of in current research and evaluation practice. Integrating a handful of qualitative methods into large-scale quantitative trials will not accomplish this, and trials will be left with only limited explanations of the successes or challenges of the interventions they are evaluating. Using innovative methods alongside trials requires far more than an openness of quantitative researchers to some interesting methods beyond interviews and focus groups. It requires a radical shift or innovation in current methodological approaches to understanding the value of qualitative methods for answering health’s most pressing research questions.

So, what does this radical shift look like in practice? A completely new vision of trials is needed, one which acknowledges the complexities of intervening in real-world health problems and that ensures the methods being used to evaluate these interventions are appropriate to investigating these complexities. Pragmatic trials and adaptive designs are a start in acknowledging the potential need for interventions to be adapted and changed while the trial is underway. Strict protocols for RCTs, such as blinding prior to recruitment, are also being questioned by quantitative methodologists, particularly when it is considered not to be feasible such as in cluster RCTs. However, none of these go far enough. Far more promising is the move toward realist RCTs, which attempt to take account of the mechanisms of human action by examining “which interventions work, for whom and under what circumstances” (Bonell, Fletcher, Morton, Lorenc, & Moore, 2012), rather than constraining evaluation to whether an intervention carried out in one specific context works to bring about a clearly defined outcome or does not work at all. A realist approach to RCTs assumes that social causation is a complex process that needs to be explored, but relies on the value of RCT methods for providing evidence with strong internal validity (Jamal et al., 2015). Realist RCTs are not without their critiques and many scholars see scientific realism as epistemologically opposed to the principles of RCT methods (Marchal et al., 2013; Van Belle et al., 2016). Despite the promise of mixed methods that realist RCTs offer, even realist approaches to trials do not fully address the power hierarchy between qualitative and quantitative research that currently shape and define health research practices. Instead, we argue that there needs to be a fundamental shift in thinking that moves away from implementing trials altogether (which implicate a focus on quantitative evaluation) and toward conducting studies (which include a mixed-methods approach to understanding the impact of interventions).

In building the way forward for this kind of transformation to take place, there are several things qualitative researchers can do as interlopers and agents-of-change in RCTs. First and foremost, qualitative researchers can play a role in ensuring that their findings are used in the interpretation of trial results. This is a bare minimum. If the qualitative findings are not used to understand the reasons behind the changes in behavior or health outcomes that have resulted from an intervention, then the interpretation of the trial is either limited in scope (e.g., “this intervention works in this setting if all parameters are the same”) or left to the interpretation of the investigators’ own subjective ideas about why the trial worked rather than actual evidence.

Achieving this integration of qualitative findings into the interpretation of trial findings however leads to a second more substantial shift that is needed: Publishing regimes. As described previously, qualitative findings are often perceived as less robust and therefore less valid than quantitative research findings. This contributes to a bias against qualitative research findings by health journals and a reluctance by researchers to publish anything qualitative alongside their “clean” trial. This publishing bias is reinforced by small word counts by quantitative journals that do not allow for the inclusion of robust qualitative findings as part of trial results, and contributes to the loss of knowledge about complex mechanisms, social causations, and lived experiences that qualitative research brings to intervention evaluation. Academic publishers need to support a shift in conventional ideas of how trials should be reported to foster the inclusion of qualitative data alongside trial data. Moreover, the registration of trial research protocols also has a part to play in establishing a mixed-methods approach to intervention evaluation simply by asking the authors of protocols how they are using qualitative methods and how this is contributing to their results.

The third and final shift required is a shift in research culture and academic reward structures. The majority of studies published in high ranking health journals are quantitative and only a handful of these journals publish any qualitative studies at all (McKibbon & Gadd, 2004). Academics publishing primarily qualitative or mixed methods research are therefore at a disadvantage when metrics, such as number of publications and the rank of journals, are used for promotion and career advancement. Qualitative studies are more time-consuming to write and more challenging to publish, which contributes to a research culture that undervalues the contribution these studies make to advancing knowledge about interventions, how they work and why they fail. This special issue is a first step in trying to disrupt this trend and raises important questions about how we can fully utilize qualitative methods to improve the research we do.

Conclusion

The question posed by Toye, Williamson, Williams, Fairbank, and Lamb (2016) in a previous issue of Qualitative Health Research is telling of the dominance of quantitative methods and the “add-on” status of qualitative. Toye asks, “What value can qualitative research add to quantitative research design?” The fact that this question still gets asked is a clear reflection of the current state of evaluation research design. We would encourage health researchers to push the methodological boundaries of their discipline, consider what mixed-methods approaches that fully integrate qualitative and quantitative research as equal partners look like in practice, and redefine the “gold standard.” Addressing the “add-on” status of qualitative research will require creative approaches that establish what it means to do good health evaluation research in new ways and that reimagine the limits of what is possible.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research included in this article: Funding support was provided by The Academy of Medical Sciences and Wellcome Trust.

ORCID iD

Jenevieve Mannell

Author Biographies

Jenevieve Mannell is a lecturer in the Institute for Global Health at UCL, and advocate for the use of qualitative and participatory methodologies in global health. She specializes in participatory research on violence against women and gender relations with indigenous communities in Afghanistan, the Amazon and Rwanda.

Katy Davis is a PhD student at the Priestly International Centre for Climate in Leeds, working on participatory climate modelling and health in the Arctic. She has a background in global health and an interest in participatory research.

References

Adams

(2016). Metrics: What counts in global health (Reprint ed.). Durham, NC: Duke University Press Books.

Bernays

Paparini

Namukwaya

Seeley

(2018). A failed method? Reflections on using audio diaries in Uganda with young people growing Up with HIV in the Breather trial. Qualitative Health Research, 29, 719–730. doi:10.1177/1049732318813534

Blankenship

Bray

Merson

M. H.

(2000). Structural interventions in public health. AIDS, 14, S11–S21.

Blumenthal

D. S.

DiClemente

R. J.

(2013). Community-based participatory health research, second edition: Issues, methods, and translation to practice. New York: Springer.

Bond

Ngwenya

Murray

Ngwenya

Viljoen

Gumede

. . . Seeley

(2018). Value and limitations of broad brush surveys used in community-randomized trials in Southern Africa. Qualitative Health Research. Qualitative Health Research, 29, 700–718. doi:10.1177/ 1049732318809940.

Bonell

Fletcher

Morton

Lorenc

Moore

(2012). Realist randomised controlled trials: A new approach to evaluating complex public health interventions. Social Science & Medicine, 75, 2299–2306. doi:10.1016/j.socscimed.2012.08.032

Coventry

P. A.

Blakemore

Baker

Sidhu

Fitzmaurice

Jolly

(2018). The push and pull of self-managing mild COPD: An evaluation of participant experiences of a nurse-led telephone health coaching intervention. Qualitative Health Research, 29, 658–671. doi:10.1177/1049732318809679.

Craig

Dieppe

Macintyre

Michie

Nazareth

Petticrew

(2008). Developing and evaluating complex interventions: The new medical research council guidance. British Medical Journal, 337, Article a1655.

Dahlke

Hall

Phinney

(2015). Maximizing theoretical contributions of participant observation while managing challenges. Qualitative Health Research, 25, 1117–1122. doi:10.1177/1049732315578636

10.

Denzin

N. K.

(2010). Moments, mixed methods, and paradigm dialogs. Qualitative Inquiry, 16, 419–427. doi:10.1177/1077800410364608

11.

Dowling

(2006). Approaches to reflexivity in qualitative research. Nurse Researcher, 13(3), 7–21.

12.

Falb

K. L.

Tanner

Ward

Erksine

Noble

Assazenew

. . . Stark

(2016). Creating opportunities through mentorship, parental involvement, and safe spaces (COMPASS) program: Multi-country study protocol to protect girls from violence in humanitarian settings. BMC Public Health, 16(1), Article 231. doi:0.1186/s12889-016-2894-3

13.

Farmer

Robinson

Elliott

S. J.

Eyles

(2006). Developing and Implementing a triangulation protocol for Qualitative Health Research. Qualitative Health Research, 16, 377–394. doi:10.1177/1049732305285708

14.

Finlay

Gough

(2008). Reflexivity: A practical guide for researchers in health and social sciences, Chichester: John Wiley.

15.

Flick

(1992). Triangulation revisited: Strategy of validation or alternative? Journal for the Theory of Social Behaviour, 22, 175–197. doi:10.1111/j.1468-5914.1992.tb00215.x

16.

Flick

(2018). An introduction to qualitative research. London: Sage.

17.

Goertz

Mahoney

(2012). A tale of two cultures: Qualitative and quantitative research in the social sciences. Princeton, NJ: Princeton University Press.

18.

Grypdonck

M. H. F.

(2006). Qualitative Health Research in the era of evidence-based practice. Qualitative Health Research, 16, 1371–1385. doi:10.1177/1049732306294089

19.

Harris

Rhodes

(2018). “It’s not much of a life”: The benefits and ethics of using life history methods with people who inject drugs in Qualitative Harm Reduction Research. Qualitative Health Research, 28, 1123–1134. doi:10.1177/1049732318764393

20.

Howarth

Campbell

Cornish

Franks

Garcia-Lorenzo

Gillespie

. . . Tennant

(2013). Insights from societal psychology: A contextual politics of societal change. Journal of Social and Political Psychology, 1, 364–384. doi:10.5964/jspp.v1i1.64

21.

Jamal

Fletcher

Shackleton

Elbourne

Viner

Bonell

(2015). The three stages of building and testing mid-level theories in a realist RCT: A theoretical and methodological case-example. Trials, 16, Article 466. doi:10.1186/s13063-015-0980-y

22.

Johnstone

P. L.

(2004). Mixed methods, mixed methodology health services research in practice. Qualitative Health Research, 14, 259–271. doi:10.1177/1049732303260610

23.

Karnieli-Miller

Nissim

Goldberg

(2017). “It’s in the cards”: The contribution of illustrated metaphor cards to exploring values within narratives. Qualitative Health Research, 27, 138–151. doi:10.1177/1049732315609897

24.

Koch

Harrington

(1998). Reconceptualizing rigour: The case for reflexivity. Journal of Advanced Nursing, 28, 882–890. doi:10.1046/j.1365-2648.1998.00725.x

25.

Leeming

Marshall

Locke

(2017). Understanding process and context in breastfeeding support interventions: The potential of qualitative research. Maternal & Child Nutrition, 13(4), e12407. doi:10.1111/mcn.12407

26.

Lewin

Glenton

Oxman

A. D.

(2009). Use of qualitative methods alongside randomised controlled trials of complex healthcare interventions: Methodological study. British Medical Journal, 339, Article b3496. doi:10.1136/bmj.b3496

27.

Marchal

Westhorp

Wong

Van Belle

Greenhalgh

Kegels

Pawson

(2013). Realist RCTs of complex interventions—An oxymoron. Social Science & Medicine, 94, 124–128. doi:10.1016/j.socscimed.2013.06.025

28.

McKibbon

Gadd

(2004) A quantitative analysis of qualitative studies in clinical journals for the 2000 publishing year. BMC Medical Informatics and Decision Making, 4, 11.

29.

Montgomery

C. M.

(2017). From standardization to adaptation: Clinical trials and the moral economy of anticipation. Science as Culture, 26, 232–254. doi:10.1080/09505431.2016.1255721

30.

Morgan

D. L.

(1998). Practical strategies for combining qualitative and quantitative methods: Applications to health research. Qualitative Health Research, 8, 362–376. doi:10.1177/104973239800800307

31.

Mowat

Subramanian

S. V.

Kawachi

(2018). Randomized controlled trials and evidence-based policy: A multidisciplinary dialogue. Social Science & Medicine, 210, 1. doi:10.1016/j.socscimed.2018.05.003

32.

Mupambireyi

Bernays

(2018). Reflections on the use of audio diaries to access young people’s lived experiences of HIV in Zimbabwe. Qualitative Health Research, 29, 680–692. doi:10.1177/1049732318780684

33.

Ngwenya

Gumede

Shahmanesh

McGrath

Grant

Seeley

(2018). Community perceptions of the socio-economic structural context influencing HIV and TB risk, prevention and treatment in a high prevalence area in the era of antiretroviral therapy. African Journal of AIDS Research, 17, 72–81. doi:10.2989/16085906.2017.1415214

34.

O’Cathain

Thomas

K. J.

Drabble

S. J.

Rudolph

Goode

Hewison

(2014). Maximising the value of combining qualitative research and randomised controlled trials in health research: The QUAlitative Research in Trials (QUART) study–a mixed methods study. Health Technology Assessment, 18(38), 1197. Retrieved from http://eprints.whiterose.ac.uk/95273/

35.

Parsons

(2007). How to map arguments in political science. Oxford, UK: Oxford University Press.

36.

Peacock

Khumalo

McNab

(2006). Men and gender activism in South Africa: Observations, critique and recommendations for the future. Agenda, 20(69), 71–81.

37.

Petticrew

(2011). When are complex interventions ‘complex’? When are simple interventions “simple”? European Journal of Public Health, 21, 397–398. doi:10.1093/eurpub/ckr084

38.

Rae

Green

(2016). Portraying reflexivity in health services research. Qualitative Health Research, 26, 1543–1549. doi:10.1177/1049732316634046

39.

Rapport

Storey

Porter

Snooks

Jones

Peconi

. . . Russell

(2013). Qualitative research within trials: Developing a standard operating procedure for a clinical trials unit. Trials, 14(1), Article 54. doi:10.1186/1745-6215-14-54

40.

Ravallion

(2009). Should the Randomistas Rule? The Economists’ Voice, 6(2), 15. doi:10.2202/1553-3832.1368

41.

Reynolds

DiLiberto

Mangham-Jefferies

Ansah

E. K.

Lal

Mbakilwa

. . . Chandler

C. I.

(2014). The practice of “doing” evaluation: Lessons learned from nine complex intervention trials in action. Implementation Science, 9(1), Article 75. doi:10.1186/1748-5908-9-75

42.

Rooshenas

Paramasivan

Jepson

Donovan

J. L.

(2019). Intensive Triangulation of Qualitative Research and Quantitative Data to Improve Recruitment to Randomized Trials: The QuinteT Approach. Qualitative Health Research, 29, 672–679. doi: 10.1177/1049732319828693.

43.

Schwartz

Lellouch

(2009). Explanatory and pragmatic attitudes in therapeutical trials. Journal of Clinical Epidemiology, 62, 499–505. doi:10.1016/j.jclinepi.2009.01.012

44.

Sims-Gould

Clarke

L. H.

Ashe

M. C.

Naslund

Liu-Ambrose

(2010). Renewal, strength and commitment to self and others: Older women’s reflections of the benefits of exercise using photovoice. Qualitative Research in Sport and Exercise, 2, 250–266. doi:10.1080/19398441.2010.488032

45.

Toye

Williamson

Williams

M. A.

Fairbank

Lamb

S. E.

(2016). What value can qualitative research add to quantitative research design? An example from an adolescent idiopathic scoliosis trial feasibility study. Qualitative Health Research, 26, 1838–1850. doi:10.1177/1049732316662446

46.

Turner

K. M.

Percival

Kessler

Donovan

J. L.

(2018). Synthesizing qualitative data sets to improve the design of trials and complex health interventions: A worked example. Qualitative Health Research. doi:10.1177/1049732318803894

47.

Van Belle

Wong

Westhorp

Pearson

Emmel

Manzano

Marchal

. (2016). Can “realist” randomised controlled trials be genuinely realist? Trials, 17, Article 313. doi:10.1186/s13063-016-1407-0

48.

Vindrola-Padros

Johnson

G. A.

(2014). The narrated, nonnarrated, and the disnarrated: Conceptual tools for analyzing narratives in health services research. Qualitative Health Research, 24, 1603–1611. doi:10.1177/1049732314549019

49.

Wells

Williams

Treweek

Coyle

Taylor

(2012). Intervention description is not enough: Evidence from an in-depth multiple case study on the untold role and impact of context in randomised controlled trials of seven complex interventions. Trials, 13(1), Article 95. doi:10.1186/1745-6215-13-95

50.

Wickström

Bendix

(2000). The “Hawthorne effect”—What did the original Hawthorne studies actually show? Scandinavian Journal of Work, Environment & Health, 26, 363–367.