Abstract
Can psychoanalysis address the press for empirical evidence of effectiveness, which our contemporary culture appears to demand, without compromising or relinquishing our ways of knowing and working with patients? Can we develop research methods that are consistent with a psychoanalytic view of treatment and change? And can we use research in ways that honor both the scientific method and the psychoanalytic method?
My opinions on these questions are framed by the contexts within which I work—academic psychology and private practice. As a member of the faculty of an undergraduate psychology department, I am often confronted with colleagues’ and students’ questions, misconceptions, and doubts about psychoanalysis, including whether our treatments are effective. I see that psychoanalysis is routinely dismissed in academic psychology as outdated, irrelevant, and even potentially harmful. As a practitioner, I am disconcerted that the form of treatment I know, believe in (dare I say it “speaks” to me?), and practice is not considered validated. I am concerned by recent movements, such as that undertaken by the American Psychological Association, to develop guidelines for practice that may exclude forms of treatment that are effective and unique. I believe systematic research is one way to bolster the standing of psychoanalysis in the field and in the wider world, in that it provides a type of evidence that is increasingly valued and even deemed essential.
This optimism regarding the potential of systematic research to address this task is informed by my knowledge of research methodology and strong general interest in research. My research training has taught me that the scientific method is a set of principles that can be used creatively to learn about complex phenomena; it is not a set of procedures or techniques (e.g., treatment manuals, random assignment). The scientific method is surprisingly like psychoanalysis, and the common stereotypes of both psychoanalysts and scientists do no justice to the creativity, passion, or spontaneity of either. Like psychoanalysts, researchers navigate a terrain marked by inherently conflicting demands. Psychotherapy researchers must find a resolution to the conflict between internal validity, or the degree to which an experiment entails the tightly controlled conditions necessary to support the inference that the treatment is the likely cause of any change in patients’ conditions, and external (ecological) validity, or the degree to which the findings can be extended to patients, therapists, and conditions beyond those of the experiment itself. Internal validity is often gained at the direct expense of external validity, and vice versa, yet both types of validity are crucial. As with the dynamic conflicts with which psychoanalysts are familiar, effective resolutions of conflicts of validity acknowledge the demands of both and repudiate neither.
In my view, contemporary psychoanalysis faces a tremendous twofold challenge. First, we must work to develop a research literature that accurately documents the effectiveness of our clinical work; this requires not only the resources to conduct such research but also the development of research methods that appropriately assess our treatments. Second, we must be attentive to our uses of research and how they shape our discipline, particularly if and as we move more vigorously into the empirical validation world. This involves being mindful not only of the conditions and limitations of research findings as we attempt to use research (of any type) to advance our discipline and inform our work, but also of the changes (some inevitable, others potential) that this endeavor imposes on our discipline, such as those Irwin Hoffman articulated in his 2009 JAPA paper.
Demonstrating Effectiveness
Outcome research offers us a way to demonstrate the effectiveness of our treatments in the quantitative, scientific mode that is valued and expected in our current cultural climate. Even beyond the fields of mental health treatment, quantitative assessment of outcomes is increasingly sought. In undergraduate education, for example, institutions are under increasing pressure to demonstrate the “added value” of the educational experiences they provide. This numbers game is not new to higher education, of course. Quantitative indicators (e.g., student-faculty ratio, percentage of courses taught by full-time faculty, number of graduates who enter advanced degree programs or secure employment) have been used for decades as largely unquestioned measures of “quality” as colleges compete for prospective students. Now the emerging push for quantitative assessment of the outcomes of higher education raises questions similar to those that confront psychoanalysis: What are the desired outcomes? Who should define these outcomes? How do we know when such outcomes have been achieved?
I agree with Eagle and Wolitzky (2011) and others that the advancement of our field rests, to some degree, on our ability to demonstrate effectiveness in this quantitative mode. I do not consider this an issue of accountability, as Eagle and Wolitzky do, perhaps because I do not question the value of the treatments we offer or the accumulated evidence of their value. Specifically, the anecdotal evidence of our clinical experiences does tell us whether or not our work is effective with the individuals we treat, despite the fact that the evidence is limited from a scientific standpoint; notably, changes in patients cannot be unequivocally and specifically attributed to the treatment in any individual case. When Eagle and Wolitzky assert that clinical experiences and case study descriptions of those experiences are inferior sources of evidence of treatment outcome (see, e.g., pp. 793, 794), they engage in the privileging of controlled studies that prompted Hoffman’s essay to begin with. Although such privileging may be warranted in theory based on the tenets of the scientific method, their reasoning disregards the crucial considerations of external validity; specifically, when experiments investigate conditions (e.g., therapists, patients, treatments) that are quite different from those of the real world, those experiments do not inform us about “general treatment effectiveness” any more than case studies do. In both cases, the methodology provides an insufficient basis for extrapolating the observed results to situations beyond the study itself.
The problem, as I see it, is not that we lack solid evidence of effectiveness, but that we do not have enough evidence (or do not appear to have enough, according to Shedler [2010]) in the quantitative form that is currently considered most cogent. The challenge facing psychoanalysis is to adapt to the press for quantitative evidence in a thoughtful and creative way, without either abdicating to the pressures or refusing to abide them. I do not think we need to engage in doublethink to provide this form of evidence to the wider world, as there are ways to research psychoanalytic treatments that are true enough (despite their limitations) to our therapeutic methods and epistemologies. That said, the gold standard of psychotherapy research (i.e., patient selection based on DSM diagnoses, random assignment of patients to treatments, use of treatment manuals, limited treatment duration, outcome measured as symptom improvement) was developed over the last few decades when psychoanalytic researchers played a minor role. Consequently, these standard procedures are not well suited to the study of psychoanalysis, even if we are willing to accept all the general constraints imposed by the scientific method. For instance, Hoffman rightly notes both the inadequacy of symptom inventories as measures of outcome and the real challenge of identifying outcome measures when most of the treatments we provide are not designed to bring about a specific, pre-identified outcome with a specific, predetermined type of patient. Moreover, much of the existing database, including research cited both by Eagle and Wolitzky and by Shedler (2010), investigates treatments (manual-driven, problem-focused, and/or short-term) that are not representative of the work most of us do.
It is noteworthy that many practitioners of CBT tend to ignore psychotherapy research and do not practice the specific treatments that have been tested in controlled research (Gunter and Whittal 2010). Concerns about low external validity of outcome studies, which cut across theoretical orientations, are as old as the psychotherapy research movement itself. In a now-classic paper, Martin Seligman (1995) reversed his unequivocal support of controlled efficacy studies as the best evidence of psychotherapy outcome and detailed the many ways in which the standard procedures of most outcome research place constraints on therapists (manuals) and patients (random assignment) that hinder the effectiveness of the therapy and lead to underestimation of its efficacy. He concluded that “the efficacy study is the wrong method for empirically validating psychotherapy as it is actually done, because it omits too many crucial elements of what is done in the field” (p. 966; emphasis in original). Hoffman offers the same critique, though in different terms. The refusal of Eagle and Wolitzky to engage with this critique is disappointing to me as one who would like to see well-reasoned arguments for the empirical validation of psychoanalysis that attend to the challenges and limits both of specific methods of research and of the endeavor more generally. Some of these problems are not mere methodological shortcomings that can be overcome with better experimental design.
That said, I am generally in agreement with Eagle and Wolitzky’s statement that “a constructive way of dealing with the shortcomings of [systematic empirical] research is to do better, more creative, and more ecologically valid research . . .” (p. 797). This requires real ingenuity in developing ways to identify and assess the processes and outcomes that are fundamental to our work, even when we know those processes to be complex, unpredictable, and sometimes even ineffable, and when we experience our ways of working as more akin to art than science. Fortunately, the processes of science need not be mirrored in the objects of its study; psychoanalysis need not be a science to be investigated scientifically. (We might recall that Carl Rogers was among the first psychotherapy researchers.) Rather than conforming to current models of psychotherapy research, psychoanalysis must strive to be in the vanguard of efforts to develop quantitative means to assess psychoanalytic treatments as they are actually practiced. The critiques of psychotherapy research offered by practitioners like Hoffman can and should inform these ongoing efforts.
In addition to developing experimental methods more in keeping with our values and therapeutic processes, we must continue to appreciate the evidential merit of other methods, including correlational survey research (see Seligman 1995), as well as case studies. Rather than operating according to a methodological hierarchy with the randomized clinical trial at the pinnacle, we should endeavor to build a database of studies using different methods and to integrate different types of evidence based on complementary methodological strengths. For instance, case studies can be used to complement efficacy studies when they describe application to the actual clinical situation of treatments or processes identified as efficacious by controlled research. Moreover, I believe we must articulate our discipline’s principles for determining the effectiveness of treatments based on the appropriate use of a range of accepted scientific methods, including experiments. In these ways, we will strengthen the literature that demonstrates the effectiveness of our treatments as they are actually practiced, using a range of research methods appropriate to our work. And we will demonstrate to the world beyond psychoanalysis that we take the demand for quantitative validation seriously.
Using Science Scientifically
Development of more appropriate research methods and thus a more solid research database is one part of the challenge we face. The other part is equally important and perhaps more difficult—to use research in a way that respects the scientific method on the one hand and the psychoanalytic method on the other.
If we value research on its scientific merits, then we must interpret and apply research results in ways that are consistent with the scientific method and must guard against the tendency to misconstrue and overextend the results of research. How easily findings can be taken out of context and imbued with exaggerated and methodologically unwarranted meanings! When we engage in this type of thinking about research findings, we are likely to undermine rather than enhance the scientific foundations of psychoanalysis. In practice, using science scientifically is difficult to do. I have written about unscientific reasoning in recent discussions of mirror neuron research in our literature (Vivona 2009), and I agree with Hoffman that the tendency to reify and overextend research results is both common and potentially dangerous. Even Eagle and Wolitzky, who clearly understand and respect the scientific method, provide an example of such overextension. I note this example here not because it is so egregious or so important to Eagle and Wolitzky’s argument, but because it exemplifies a common tendency to disregard the principles of the scientific method that dictate the types of situations to which research findings can be generalized.
In arguing for the potential relevance of research to an individual treatment, Eagle and Wolitzky mention an accepted finding from psychotherapy research: “There is a good deal of evidence that exposure to feared objects and situations is frequently an important factor in the treatment of phobias, including agoraphobia” (p. 794). They remark that this finding is consistent with an observation of Freud’s and might be useful to an analyst working with an individual patient. But the (uncited) research that provides the evidence investigates behavior therapy for patients diagnosed with phobias in which prolonged exposure to the phobic stimulus typically occurs in one three-hour session, although treatment may last as long as five sessions (see, e.g., Choy, Fyer, and Lipsitz 2007).
This research provides no scientific basis for reasoning that exposure to feared objects or situations would be useful in psychoanalytic treatment. The speculation offered by Eagle and Wolitzky may be true, but it is not a methodologically warranted extension of the research findings in that it does not respect the limits of generalizabilty. Speculating about research findings in this way may be a source of inspiration for us, but it is not the scientific use of research that Eagle and Wolitzky advocate when they extol the merits of the scientific method over anecdotal evidence. Ironically, perhaps, Hoffman’s idea that research findings are something we might “keep in mind” is often more consistent with the limits of generalizability and the scientific method in general than are Eagle and Wolitzky’s assertions that research results can be relevant in particular ways and in particular situations. In the end, this example does not demonstrate the relevance of the findings of controlled research studies to psychoanalysis. It does demonstrate the very real potential, articulated by Hoffman, for the objects of scientific study to become reified because they appear to bear the stamp of scientific legitimacy and then to be held up as validated “things” that we might (and perhaps even should) incorporate in our practices. Against such unscientific reasoning, we must be ever vigilant.
Certainly, Eagle and Wolitzky mention other studies that can be linked more scientifically to psychoanalytic practice. Nevertheless, between research findings and any specific clinical moment there is always a gap of unknown width that must be acknowledged. Indeed, this is by design; as Eagle and Wolitzky remind us, the purpose of outcome studies is to investigate how treatment works in general. The gap is present regardless of the degree of statistical significance or the effect sizes reported in a study. Hoffman discusses this gap in terms of the “consequential uniqueness” of the psychoanalytic dyad; that is, every dyad will be different from those who participated in any study in unknown and potentially significant ways. This gap does not signify a limitation of research that can be overcome with methodological ingenuity or creativity; neither does it supply a reason to abandon or ignore efforts to document our effectiveness with research. It is the reality about what we can expect research to teach us about individuals, including ourselves.
Tolerating Inevitable Losses and Safeguarding psychoanalysis
Can anyone doubt that psychoanalysis has already been changed by the movement toward quantitative evidence, despite the fact that so many psychoanalysts ardently resist it? The overall influence of systematic research on psychoanalysis is greater still if we consider our increasing engagement with extraanalytic research in fields such as cognitive science and developmental psychology. Our involvement in research is not neutral, nor is it simply an exercise in public relations with only beneficial consequences. Indeed, these engagements with research have already begun to infuse our ways of working, our identities, our worldviews. We are changing indeed.
Like Hoffman, I believe there to be both ethical and practical consequences of any involvement in quantitative validation efforts that we must attend to lest we precipitate the very desiccation of psychoanalysis we all wish to avoid. I am surprised that Eagle and Wolitzky seem not to take the threats seriously, instead attributing many of Hoffman’s concerns to either methodological naiveté or radical epistemology. We need look no farther than the experiences of our CBT counterparts to find an alarming example of the dangers. CBT treatment manuals were originally developed as aids for psychotherapy research, a means to ensure that therapists in experiments faithfully provide the target treatments. Over time, however, those manuals were increasingly promoted as a foundation for training and practice in CBT, and even in clinical psychology more generally (see, e.g., Baker, McFall, and Shoham 2008), based on the reasoning that the manualized treatments should be both taught and used more extensively than they are because they have been validated by research. At first glance, this may appear reasonable. However, few studies have compared the manualized treatments to their unmanualized counterparts or to standard treatments offered in the field (Garland et al. 2010), particularly in private practice settings. Thus, although we may know that the manualized treatments are effective, we do not know (and it is much debated) whether manualized versions of treatment are an improvement over standard treatments. Thus, the trend to use research manuals for clinical training in CBT is founded on a striking circular logic and not on evidence.
The empirical validation movement in CBT, which began as an effort to validate the treatments that were practiced in the field, has led to increasing pressures on therapists to practice treatments that have been validated with efficacy studies. If we wish to avoid this fate for psychoanalysis, we must develop a representative research literature that demonstrates the effectiveness of our full range of treatments as they are actually delivered in the field, and we must be on guard for uncritical overvaluation of controlled research as a guide to practice. The threats to our ways of working are not inevitable, but they may be likely if we do not actively resist the seductive illusion of scientific certainty.
In the end, I advocate for psychoanalysis a path of unavoidable difficulties and dangers. On the one hand, I believe we must vigorously engage in outcome research and move to the vanguard of efforts to enhance the external validity of psychotherapy research. We must decide for ourselves what types of evidence we can use for what purposes, rather than accede to a view of knowledge that may not accord with our own. We must use the scientific method creatively and flexibly, just as we use the psychoanalytic method. On the other hand, we must remain vigilant to the threats this engagement poses to our ways of knowing and working, threats any increased engagement in research is likely to exacerbate. In particular, we must guard against an uncritical attitude about research findings, including the common tendency to downplay the limitations inherent in all systematic study. We must not compromise our openness to uncertainty in the wake of efforts to know.