Abductive Reasoning: How Innovators Navigate in the Labyrinth of Complex Product Innovation

The challenges of complex product innovation

The first challenge is that innovators working on complex new products have to deal with an enormous amount of information, and they struggle to focus their search for solutions to the poorly defined problems they face. Uncertainty increases the amount of information that must be processed, and much of the information is noisy, surprising, and seemingly random (Tsoukas, 2005). In the case of drug discovery, large amounts of noisy information arise in part from the huge search landscape. Nightingale (1998) quotes one discovery scientist who suggested that there may be 10¹⁸⁰ possible molecular entities, while there are only 10⁷³ particles in the universe. Scannell et al. (2012) explain that there could be between 10²⁶ and 10⁶² chemotypes that meet the criteria for an oral drug, each with many derivatives. In addition, each compound is intended to bind to a single protein, but there are anywhere from one to twenty million proteins in the human body that interact in unknown ways (Pisano, 2006). To overcome this challenge, innovators need a way to focus their search that encompasses and leverages, rather than ignores, noisy information.

The second challenge is the tendency to become stuck in competency traps or local optima. As Baumann and Siggelkow (2013, p. 129) explain:

The challenge raised by complex systems is inherently perilous: because complex systems create vast and multi-peaked spaces of potential alternatives, the risk of ending up with a set of choices that create low performance, i.e., a low local peak, is significant.

People tend to land on local optima because they have a much harder time interpreting “alternatives, their consequences, and their possible impact on problem solving when conducting distant search” (Afuah & Tucci, 2012, p. 359). Weick (2005) uses the Challenger disaster and 9/11 to illustrate local thinking. The debris strike that caused the Challenger disaster was categorized as a familiar problem that could not cause serious harm to the vehicle while for 9/11, intelligence experts could not imagine that airplanes would be used as weapons. Familiar categories abstract the actual phenomena by stripping away details, but details are important for understanding new possibilities. Drug discovery has shifted from random screening to more “rational” or guided processes (Henderson, Orsenigo, & Pisano, 1999), which should help look beyond local knowledge. However, unless all these new search techniques are effectively integrated into discovery, they may only “stretch the landscape,” as Pisano (2006) suggests. The very long development time for new drugs introduces additional local optima from temporal complexity (Garud, Dunbar, & Bartel, 2011a). Asynchronies in developing different aspects of a product may make intermediary models look like useless mistakes to be weeded out, or like finished efforts to be pushed forward prematurely. To overcome the tendency to get stuck on local optima and search more broadly for good models, innovators need a way to examine more alternatives and build more effectively on intermediary models.

The third challenge for innovators concerns how to untangle the “knots” that complex systems continually create, and articulate the problems of search in manageable ways (Perin, 2005). The common method for untangling problems is to decompose them into separate parts. Decomposition presumes that parts are nested in an orderly hierarchy with defined connections, which is not the case in complex systems. Drug discovery is characterized by many unknown interdependencies among chemical compounds, disease processes, and the rest of human biology (Collins, 2011; West & Nightingale, 2009). Even a minor change in a compound (e.g., to make it more soluble) can create damaging side effects, but these consequences may not be discovered for some time. Because of the unknown interdependencies, the necessary untangling needs to be done holistically, so people can keep the possible interactions in mind even as they work locally (Pisano, 2006). But efforts to scale up steps impose some hierarchical decomposition on drug discovery (Scannell et al., 2012). To overcome the tendency to decompose the problem into separate steps, innovators need a way to keep the whole in mind as they work on different facets.

Most innovation practices that innovators typically draw on to deal with the challenges they face in their work were developed for incremental innovation processes and do not fit complex systems. For example, innovation management literature emphasizes the need to develop a very sharp and complete product definition at the outset of a project, which is used as a blueprint for integrating the various functions and for bringing the product to fruition and launch into the market (Cooper, 1998). This sharp, complete definition is a conceptualization of how the product might work, a holistic hypothesis. However, this practice overlooks the challenges of complex innovation because in complex contexts innovators cannot know at the outset of a project what specific functionalities are possible. Further, research suggests that innovators’ inability to get their arms around the full set of uncertainties involved in a project contribute to the ad hoc, crisis-oriented management practices found to dominate radical innovation processes (Leifer et al., 2000).

Studies of radical innovation (Leifer et al., 2000; Van de Ven et al., 1999) highlight practices that may be important in the context of complexity. Van de Ven et al. (1999) suggest that innovators cycle repeatedly between divergent searching that explores a broad product vision and a convergent search process to choose among elements via trial and error learning. Leifer et al. (2000) find that radical innovators iterate among a mosaic of organizational, technological, market, and resource uncertainties, and not in any linear or sequential fashion. While innovation management research does not comprehensively address the challenges outlined here, these few studies of radical innovation do provide some insight into practices that may be important in the context of complex innovation.

Abductive reasoning: Potential and current limits for dealing with these challenges

Organizational scholars also suggest that abductive reasoning fits complex situations and can potentially help address these three challenges (Abolafia, 2010; Garud et al., 2011a; Grandori, 2010; Dunbar et al., 1996; Simon, 1977; Weick, 2005). Abductive reasoning was defined at the outset of this paper as “reasoning that forms and evaluates hypotheses in order to make sense of puzzling facts” (Thagard & Shelley, 1997; quoted in Weick, 2005, p. 433). Others define abductive reasoning in similar ways. According to Magnani (2001), abductive reasoning or abduction is a process of forming an explanatory hypothesis for poorly defined phenomena. He defines it as “the process of reasoning in which explanations are formed and evaluated” (Magnani 2001, p. 18). Grandori (2010, p. 490) says:

Abduction or ‘retroduction’ [as called by the original proponent of the concept, Charles Sanders Peirce, 1935] can in fact take two forms: ‘empirical’ (recognize patterns in data and posit laws that can regulate them) (Simon, 1977) or ‘theoretical’ (formulate theory based, casual hypotheses from which the observed or sought action/consequence chain would follow) (Hanson, 1958).

Locke, Golden-Biddle and Feldman (2008, p.907) also draw on Peirce, explaining that “deduction proves that something must be; induction shows that something actually is operative; abduction merely suggests that something may be” (emphasis in original).

Innovation, by definition, requires that people not only hypothesize a novel idea, but also bring that novel idea into use (Schön, 1967). To use abductive reasoning for innovation, we build on the understanding that it is not a single step of hypothesis formation, but the process of using that hypothesis to understand a puzzling situation, consistent with Magnani (2001), Locke et al. (20087), and Nesher (2001). Magnani (2001), Paavola, Hakkarainen, and Sintonen (2006), and Mantere and Ketokivi (2013), among others, argue that abductive reasoning also underlies the processes scientific discovery, which is our empirical focus. Other studies of science do not use the term abductive reasoning, but highlight scientists’ practices of knowing that are consistent with abductive reasoning (Grinnell, 2009; Latour & Woolgar, 1979). Nightingale (2004) argues that scientists cannot confirm hypotheses deductively when knowledge is limited and fragmented, because experiments will likely fail and the results provide no indication of where else to explore. Instead, scientists rely on a style of research based on discovery and understanding, not on prediction and testing. Scientists tinker with experimental conditions to identify and fine-tune more promising alternatives (Pavitt, 1987), and compare divergent implications of competing explanations.

Despite the potential of abductive reasoning for dealing with the challenges of complex innovation, the existing literature is mostly conceptual, and provides limited insights for how people actually use this form of reasoning for particular problems like complex innovation. Based on definitions of abductive reasoning, we expect that innovators would use abductive reasoning to formulate and evaluate hypotheses about possible complex products. We use the term “hypothesis” to refer to an unproved theory or supposition that explains, in the case of product innovation, how a possible new product might work, and provides the basis for further investigation. While the literature does not explain how innovators might formulate and evaluate these hypotheses, we synthesize conceptual ideas from organization scholars about abductive reasoning to specify some important contributions and limits to existing literature.

With regard to formulating hypotheses, Denrell et al. (2004) suggest that “mental models” of the problem are necessary to navigate because they seed the search with possibilities while constraining the search from less likely arenas. However, they do not explain how people can develop such mental models. Grandori (2010, p. 484) suggests that the problem should be defined as a performance potential. The problem is formulated as a theory-based causal hypothesis, with causes understood as available resources with potential for action (Penrose, 1959), and effects understood as useful consequences. “Alternatives are generated according to cause-effect hypotheses: certain resources (e.g. polyphenols) are hypothesized to be put to certain causes (e.g. food production), which in turn should produce consequences that may be evaluated as desirable (health) and valuable also in economic terms” (Grandori, 2010, p. 484).

Weick (2005, p. 433) provides a different approach for formulating a hypothesis, one that starts with a clue and then discovers or invents a world in which that clue is meaningful:

Current use of this broadened sense of abductive reasoning is found in the work of people such as Ginzburg (1988), Harrowitz (1988), and Patriotta (2004) who argue that the conjectural paradigm, grounded in abductive reasoning, is the foundation of inquiry. The basic idea is that when people imagine reality, they start with some tangible clue and then discover or invent a world in which that clue is meaningful. Imagination “conceives a whole design almost at once, which it then fills out and gives body to by particular association…. The mind thinks simultaneously of specific parts and of their one organizing principle” (Engell 1981, 82–83). This act of invention is an act of divination that has a close resemblance to detective stories.

In Weick’s (2005) thinking, the hypothesis that is formed is a world or a whole design, which is different from Grandori’s (2010) hypothesis about potential, but not inconsistent. Garud et al. (2011a) use abduction to explain how people in organizations identify possibilities in narratives of unusual events that are relevant to their own situations. The narrative that is formed through abductive reasoning is similar to Weick’s imagined world, it entails a coherent set of events with a beginning, middle, and end. This research suggests that abductive reasoning involves formulating a hypothesis, one that would build on clues to a whole world and reflect potential, yet little research explains how a hypothesis might be formulated in the context of complex product innovation where the requirement is not only developing a novel idea, but also bringing that idea into use.

The next aspect of abductive reasoning is evaluating hypotheses “in order to make sense of puzzling facts.” According to Denrell et al. (2004), actors generate the feedback they need to evaluate their progress by making predictions from their intermediary models, and adjusting the models to accommodate deviations from those predictions that they discover as they navigate in the labyrinth. Grandori (2010) suggests that, rather than expecting results based on parameters that are fixed ex ante, people empirically inquire into the hypothesis with actual results, and look for surprises. Weick (2006, p. 786) quotes Harrowitz (1988, p. 88) who says that by using abductive reasoning, people are able to “leap from apparently significant facts which could be observed to a complex reality which—directly at least—could not.” Nightingale (2004) explains that scientists working in complex conditions create something new to learn from, and use that knowledge to move from simplified laboratory experiments that isolate particular mechanisms to increasingly complex settings. Knorr-Cetina (1999, p. 92) finds that microbiologists do not try to understand the numerous problems that arise in their experiments because “their attempts to understand a living organism, of which little is known, quickly reached its limits.” Instead, they treat problems by “varying components of the experimental strategy until things worked out, not by launching an investigation of the cause of the problem.” Abductive reasoning suggests that innovators need to leverage intermediary models to navigate systematically, examine alternatives, and build more effectively on intermediary models. Innovators should look for surprises, leaps, and varying experimental strategies to make sense of puzzling facts, but research has not studied how these general ideas are applied to the context of complex innovation.

Several scholars directly or indirectly extend the abductive reasoning process by suggesting a third aspect that would help accumulate insights—reframing. Tsoukas (2005) cites Orr’s study of copy repair technicians, in which solutions were discovered through reinterpretation of known facts and then following the new interpretation with new investigations. Denrell et al. (2004) also emphasize changing the mental model to create a fresh perspective and see new aspects of the problem. Grandori (2010) warns against lowering aspiration levels when results are not as expected. Instead, decision makers should reformulate their model and create new performance objectives that reflect new resource alternatives and consequences. Dunbar et al. (1996) suggest deframing by using abductive reasoning to reconsider assumptions and open up the reasoning process to alternate explanations. Majchrzak, Logan, McCurdy, and Kirchmer (2006) describe “spurts” of innovation by reframing a problem qualitatively, e.g., from building a bridge to affecting the flow of traffic.

Reframing would cycle back to the hypothesis formulation and evaluation aspects of abductive reasoning, and may be an important but challenging aspect in the context of complex innovation. Dunbar et al. (1996) summarize the extensive literature on how difficult reframing and deframing can be in organizations. It is not surprising that research on abductive reasoning does not explain how innovators might reframe their understandings, gather up and synthesize what they know, while they are in the thick of the innovation process, but it may be important to understanding the role of abductive reasoning in complex product innovation.

To better leverage abductive reasoning in the context of complexity, research needs to examine how these types of hypotheses are formulated, evaluated, and potentially reframed. Innovators need to understand how to apply ideas such as clues, intermediary models, leaps and surprises in their work on complex systems, if they are going to navigate effectively in the labyrinth. In the next section we explain the methodology that we use to address our general research question and these limits to the existing research on abductive reasoning.

Data gathering

We focus on drug discovery, which is the first six years of an approximately thirteen-year process. Our primary data comprise 85 interviews with scientists, technologists, and managers. As Table 1 outlines, we interviewed people from a variety of large, integrated pharmaceutical firms and small biotechnology firms. We deliberately sought insights from people with diverse roles, experiences, and organizations, because sorting through differences enables the comparative analyses grounded theory-building uses to explore alternatives and sharpen the theory (Strauss & Corbin, 1998). These interviews contain a variety of insights and form an adequate basis for building theory for complex product innovation. By level, interviewees include vice presidents of R&D, scientific directors, project team leaders, and bench scientists. By role, some work on “therapy teams” that focus on developing drugs for particular disease areas (e.g., pain, metabolism, cancer). Others work as supporting technologists, and carry out high-throughput screens and assays, structural biology, genomics, bioinformatics, combinatorial chemistry, and pre-manufacturing. Ten people focus on R&D management and nine others work on business development.Of the 85, 82 hold a PhD in a chemistry, biology, physiology, pharmacology, biochemistry, or chemical engineering, so almost all interviewees are scientists.

OPEN IN VIEWER

Table 1.

People Interviewed Distinguished by Category.

Company	VPs of R&D	Directors	Team Leaders	Scientists	Total # of Interviews
Ph 1	3	7	8	4	22
Ph 2	0	6	1	3	10
Ph 3	0	6	2	6	14
Ph 4	1	3	0	1	05
Ph 5	1	3	6	1	11
Pharma 6, 7, 8, 9	1	2	1		04
Biotechs	4	8	3	4	19
Total	10	35	21	19	85

We are not distinguishing between people with PhDs and people without because 82/85 people interviewed had PhDs, and 81/85 people had PhDs in life sciences or life science-related fields.

In addition we interviewed 14 consultants.

The 85 interviews are people’s stories of how they do their work of drug discovery. The authors carried out 20 interviews together, the first author carried out 34 more alone, while the second author carried out another 26 interviews alone. We received 5 transcripts from colleagues working on a related project, who followed the same interview approach. These interviews show patterns that are very similar to the interviews we conducted. The similarity of the process across researchers provides additional support for the three social mechanisms that we find in these data. Of the 85 interviews, 79 were held at the person’s worksite and averaged about an hour, and 6 were done over the telephone. Of the 79 in-person interviews, 68 were machine-recorded and transcribed, and 11 were hand-recorded, filled in right after the interview, and transcribed.

We asked people to describe what they do, the approaches they take, problems they encounter, and differences across projects in their experience. We asked for concrete examples to keep people grounded in the phenomenon, and to aid our understanding of what they were describing. Our interview questions shifted over time as we learned more about some things and moved to probe other things more fully. Because the work is difficult for non-experts to understand, we used a “cheat sheet” of relevant scientific and technical terms, and built up an understanding of processes over the time we collected data. Throughout, we looked up terms in the interviews, discussed findings with colleagues who are also studying biopharmaceuticals, and read industry reports as well as many articles in scientific, industry, and business magazines on drug discovery. The stories also contain instances of problematic events, failed efforts, and concerns about the process.

These interview data are limited in several important ways that constrain the inferences we can draw from them. First, they reveal what people chose to reveal, and reflect their rationales and understandings for why things unfold as they do, rather than “objective” depictions of what they actually do. Second, the data do not comprise direct observations, compared to ethnographic studies where researchers observe what is happening. Third, the data are cross-sectional, depicting a snapshot of an evolving process, and we have no outcome measures of successful versus unsuccessful approaches. We rely on these experienced scientists’ insights regarding what seems more or less effective. However, the interviews comprise people’s rationales for how they work, and reflect their reasoning processes, which is what we build theory about. In the discussion section we suggest additional research to overcome these limits.

Data analysis leading to the grounded theory

We analyzed our data following the process described by Strauss (1987) and Strauss and Corbin (1998), which involves movement from data, to theory, to data collection, and back to theory over time. The analysis is an iterative process, involving extensive refinements and revisions to our theory as it emerged over time (Bailyn, 1977; Elsbach, 1994; Miner, Bassoff, & Moorman, 2001). The authors performed the initial analyses separately and together throughout the process of theory development, and included a number of doctoral students over time to code the data and provide additional input. Figures 1a, 1b, and 1c provide an overview of our coding process, initial open codes that eventually led to categories that became the three social mechanisms that make up the framework we develop in this paper.

OPEN IN VIEWER

Figure 1a.

Overview of the Coding Process for the First Social Mechanism.

OPEN IN VIEWER

Figure 1b.

Overview of the Coding Process for the Second Social Mechanism.

OPEN IN VIEWER

Figure 1c.

Overview of the Coding Process for the Third Social Mechanism.

Our theory about the essential role played by abductive reasoning in complex innovation emerged after many cycles of hypothesizing, examining the data, and rethinking. Through the coding process (open, axial, selective) we identified a number of categories that captured facets of the data and traced them through the data set (Denzin & Lincoln, 1998; Strauss & Corbin, 1998). One category is the use of clues. A second category is the content of what innovators are looking for—a configuration of interactions or what we described in earlier stages of analysis as a plausible pattern of interactions among product elements. A third category is a dynamic of “elaborating” and “narrowing”: e.g., focusing in on a category of compounds versus looking at diverse structures; narrowing in on a protein target versus elaborating different ways to express that protein. Additional categories include the surprising (to us) complexity of the work, and a tension between the ambiguities of navigating and the need to commercialize products for profit.

We iterated over time through several different theories to try and explain the deliberate way of proceeding with innovation that we saw, but each captured only some of the core categories and did not adequately explain this different form of reasoning. One early theory was that discovery scientists follow the logic of complexity, while managers prefer the logic of incremental innovation, but this highlighted differences in proceeding, not similarities that we were attempting to reveal. Then we hypothesized that scientific reasoning conflicted with the engineering reasoning. This too did not hold because, while the technologists in our data differ from therapy scientists, they also talked about the same underlying way of proceeding. We then centered on the clues and tried to describe different kinds and different approaches to using them. Several different colleagues who patiently listened to early presentations told us that we seemed to be talking about abductive reasoning, a concept that was unfamiliar to us. After iterating again with the literature on abductive reasoning, we figured out that abductive reasoning defined the way of proceeding that we saw. We compared the use of abductive reasoning by discipline and role, and while there were different degrees of use, everyone seemed to rely on abductive reasoning in some ways. The less successful events showed less abductive reasoning.

The first social mechanism of using clues to imagine a configuration of interactions between a drug possibility, a disease, and the human body developed from our initial focus on clues. We could see that they were using clues as guidelines to hypothesize a pattern of interactions among molecules, diseases, and human biology. We use the term “imagine” because, as Weick (2005) described, the scientists use clues to invent or discover a world in human biology where a particular molecule could moderate a disease. This first social mechanism brought together two of our initial categories of clues and what the innovators were looking for. As noted earlier, Figures 1a, 1b, and 1c provide additional details about our coding process.

The second mechanism of elaborating and narrowing around the interactions in the imagined configuration to examine alternatives and build on intermediary models emerged as we looked closely at descriptions of asking questions and “in vivo” discussions such as “trolling” in different areas or “going down a path you don’t know what it will be.” We contrasted these with examples where people did not elaborate but instead zeroed in, or emphasized “shots on goal.” To develop the social mechanism of iteratively iterating across disciplinary boundaries to reframe the configuration of interactions we examined people’s discussions of figuring out interactions, which typically involved boundary crossings. Many people used “iteration” to describe this process, so this in vivo code became part of the label for it. We compared descriptions of not surfacing interactions or not working across disciplines with those that do, to sharpen the process. We explain our findings next.

The framework: Using clues, elaborating and narrowing, and iteratively integrating

Clues are the foundation of the first social mechanism of abductive reasoning, using clues to imagine a configuration of interactions. Discovery scientists begin the innovation process with a hypothesis about the most critical interactions, which focuses their search and also encompasses possible variations that may be relevant. Scientists develop a hypothesis by leveraging clues from their science and experience, and imagining how those clues point to a possible configuration of interactions. As Weick (2005) suggests, abductive reasoning starts with some tangible clues that people use to discover or invent a world in which those clues are meaningful. We find that this “world” is the possible drug product in action.

The dictionary definition of a clue is something that leads out of perplexity; it is a fact or object that helps to solve a problem or a mystery. We find that clues synthesize otherwise noisy information about human biology and medical chemistry into implications for how a disease process works, and how a molecule can be distributed through the bloodstream to affect that disease process, and the scientists’ ability to manipulate some of the processes involved. Drug discovery teams pull these clues together to suggest a configuration and they explore the dynamics that the configuration suggests. Many scientists said that they were working with clues.

Our first story begins to explain the role of clues in imagining a configuration of interactions and comes from a biology team leader who was working on a drug to overcome a serious side effect of kidney disease. She and her team began with the natural ligand (the natural ligand normally binds to the receptor but is missing in people with the disease she is studying). Her team developed some clues from the ligand about receptors in the same family to imagine a configuration in which a smaller molecule could bind in the same way as a very large molecule (the natural ligand is very large). This was a breakthrough because they were imagining a new kind of drug. She describes their reasoning process:

We wanted to try and find other molecules like that [pointing to a graphic on the wall of the natural ligand binding to a protein], but they must be smaller. The natural ligand that sticks to the receptor is much larger… Everyone said you can’t do it, you can’t find a smaller molecule because there are too many contact points. Like a basketball fits into your hand and your hand covers a good part of the surface of the ball, but what if you tried to attach a golf ball to the surface of the basketball? It does not bind with the same strength.

As she explains, the received wisdom was that it would be impossible to create a drug that could bind in the configuration that her team was imagining. But they used a clue to make an analogy (Gavetti, Levinthal, & Rivkin, 2005) and “find other molecules like that”—to figure out possibilities (Dunbar, 1995). She continues:

We had some clues that we thought would be important… There were other receptors in the same family, and other clues about what could make a connection. Molecules are bumpier than a flat surface, and they have a lot of points of contact and different strengths of contact points.

Clues from similar receptors sugges ted plausibility, while clues from chemistry about the strength of contact points suggested that they could create a smaller biologic molecule (a peptide) that might bind like the natural ligand. Clues about combining amino acids, the specific contact points needed for binding with the receptor, and how that binding interacted with the disease process led them to imagine a configuration of interactions that might work. The team also drew on multiple perspectives to imagine a plausible configuration, and constantly negotiated possibilities. People in different units or disciplines see different aspects of the same configuration, so when they collectively negotiate the meaning of what they are looking for, they are less likely to become trapped in existing competencies.

The second social mechanism of abductive reasoning is elaborating and narrowing around the interactions in the imagined configuration to examine more alternatives and build more effectively on intermediary models. Scientists evaluate the imagined configuration of interactions to see how it might actually work, and to discover what else they need to know. They use their hypothesis to sift through all the noisy information and explore whether or not, and if so how and why, their configuration might actually work as a possible drug. The process of elaborating and narrowing forms the foundation of this social mechanism. Scientists use elaborating and narrowing to anchor on one interaction in the configuration and then reach out and around that to surface more information by exploring the various details that they find. Elaborating and narrowing enable the scientists to delve into details yet stay open to unexpected insights as they systematically look at related facts. Evaluating in this way contextualizes the innovation process, and captures some of the particulars (e.g., of disease processes, and of the diversity in genetic make-up in the human population). The exploration digs in to find the core dynamics of the possible drug in action, and opens up to more alternatives. Scientists in complex systems are not just searching, they are continually configuring and contextualizing as they elaborate out and narrow in on facets of the imagined configuration of interactions.

Scientists need to proceed without closing off potentially valuable pathways, because they do not know what might work. To examine their configuration, the team described above by the biology leader had to see if they could find this new peptide that was part of their imagined configuration of interactions:

We had a collaboration with a biotech start-up. We used their technology to screen for peptides. You can go from 165 amino acids down to 20. That [the final new molecule in the picture] is a 20 amino acid peptide and that is the ribbon here. It is cyclic in on itself, and it binds to the molecule. This peptide has no identity at all with the original ligand. The peptide is not known in nature, we had to create it. We worked with this company looking through libraries with millions of possible ways of how to arrange peptides, and screened them to bind to this receptor. I remember very vividly thinking it was a waste of time. Also we were trolling in a couple of other areas like natural products, antibodies. Once we got the peptide and it [combined with the receptor], we did our first crystal structure [with a famous academic lab].

They empirically evaluated their imagined configuration of a much smaller peptide by trying many ways to build a small peptide that would also bind to the receptor, but they were also open to alternate possibilities such as natural products or antibodies. The team elaborated around many possible peptides and also narrowed down to the needed binding. They expanded existing understandings to see new contingencies and learn more about possible interactions. She doubted the searching, and was worried that all this empirical evaluation was a waste of time. However, according to Locke et al. (2008, p. 908), doubt is the engine of abductive reasoning because it drives and energizes people to generate possibilities, try them out, modify, and so on, until new concepts are generated that satisfy the doubt. This doubt also highlights the serendipity involved in complex innovation like drug discovery, because many drug possibilities never pan out. With no immediate feedback they did not know if they were on the right track, but they persevered.

The next story also emphasizes the second social mechanism of abductive reasoning, elaborating and narrowing around the interactions in the imagined configuration, because only through a systematic sifting of related insights can they identify and explicate the relevant biological mechanisms. Opening up around the configuration to look for unexpected contingencies keeps the scientists poised to spot new possibilities as well as emerging perturbations that might escalate into major problems or opportunities. A director of biology at another pharmaceutical company explained how they elaborated and narrowed around a configuration to evaluate their imagined configuration of interactions. They had imagined that a certain kind of molecule would bind to an enzyme and inhibit its ability to speed up a cellular process that may lead to a form of cancer. He explains how they proceed to examine this configuration:

You want to know OK—you have now picked the most likely inhibitor of this enzyme. Now, I want to understand what it does in a cell—in a hepatocyte—a liver cell or a cancer cell.

He emphasizes the need “to understand” how the potential drug molecule behaves in the cancer cell and in the liver. He outlines a variety of next steps that they deduce from the configuration:

One of the sets of experiments that I need to do to be able to understand what this drug does in a cell, then the next question is OK, I now understand what it does in a cell and it does what I want it to do—it lowers the ability of the liver to generate lipids or sugar or a cancer cell to grow.

They generated understandings through several sets of experiments that create new clues regarding “what the drug does in a cell.” He evaluates the configuration of interactions by opening up to explore several possible mechanisms. He identifies good questions to work on, and he reaches out around these questions to look for clues. He finds support for his developing understandings because “it does what I want it to do.” Then he narrows in on another question, and works to figure out if the prediction is actually operative. His questions keep the search open yet also provide focus. Elaborating and narrowing bounds the work systematically, so they do not amplify out endlessly and they also do not focus in on a single feature prematurely.

Elaborating and narrowing balances the processes of opening up to investigate new possibilities with closing in on some aspects to examine them more fully. Scientists might elaborate out around a certain interaction, for example, by exploring a variety of possible effects between the molecule and the cardiovascular system, and then narrow in on particular effects they discover to consider how to address them. The configuration of interactions is an important tool for navigating, because it keeps innovators open to new alternatives about how a molecule might behave in the body against a disease but also enables them to dig in to situated possibilities.

Next, the same scientist outlines the kinds of additional clues to look for:

Now how do we understand how that [the effect they find] is going to translate into additional studies?… Is the drug metabolized in a way that it all gets broken down to something that is no longer effective, or can it deliver [results in] the right models? I want to study it in systems that seem to predict activity in humans, so what kinds of models do I need to understand [if the compound works in humans]? I need to develop, build, create and begin to test new compounds…

His clues suggest new questions and new experiments to be tried, new ways to think about the configuration.

Elaborating and narrowing around the interactions in the imagined configuration also draws on different parts of the organization. He explains how the project “touches on a lot of different expertise.” Different groups in the organization are actively engaged in the evaluation of this possible drug, integrating new knowledge into the project, and leveraging different perspectives:

For chemists it is a different vision, for protein biologists it is a very much different vision but the morphing a program goes through—most of the steps go through kind of biology hands and then touches on a lot of different expertise until it ultimately gets toward the stages of becoming a drug that is going to go into people…

He explains how they accumulate knowledge. Different groups have a different “vision” of the possible configuration, so they can contribute unique insights.

This example also illustrates the third social mechanism of abductive reasoning, iteratively integrating across disciplinary boundaries to reframe the configuration of interactions. The imagined configuration of interactions goes through “morphing” as it goes through the hands of different experts, suggesting that they reframe the project over time as they move toward testing on people (clinical trials). The “morphing” is iterative, not linear, as it goes back and forth. Everyone involved in the project across the organization may be thinking of a slightly different version of the configuration of interactions, but they think about it as a whole rather than in discrete pieces. Iterating brings in diverse views and juxtaposes them to test and push ideas.

The project shifts and changes as they find new clues to the configuration, open up to evaluate and to create new understandings, and reframe as they leverage different perspectives. This scientist goes on to explain how the cycling continues and returns to the first abductive reasoning mechanism of imagining a configuration of interactions. The configuration they imagine in the next cycle is developed from the clues of the previous cycle. He emphasizes the need to interpret well. They are interpreting how well they have imagined a configuration of interactions:

If they interpret well, then they have done the right experiment [and] that proves we have met the hurdle—again some of these are technical and some of these are more philosophical.

He explains further that “technical” hurdles refer to not being able to express the protein that is involved in the cancer (to carry out more tests). Philosophical hurdles arise when they need to decide whether or not to abandon the project because they cannot find a way to generate the protein to test the mechanism of the compound. He explains the ongoing navigating as additional surprises can arise:

The trick is to understand what the problem is and to get it into the hands of people who know how to solve the problem for you. That is just one problem. The next problem is that you have expressed the protein and it has no activity and it does not do what you thought it did. How did you study that, did you do the right assays, did you have it under the right conditions and have the right substrate for it? … Every step of the way there is something.

The navigating involves encountering additional surprises and getting those problems “into the hands of people who know how to solve the problem for you.” They depend heavily on a network of internal and external experts to figure out, for example, how to generate enough of the protein for tests, or if they can work around possible toxic interactions they discover. They also continually question not only their findings but also their methods for developing those insights as they interpret the plausibility of the imagined configuration. This director of biology finishes his example by explaining a major event that suggests that they are on the right track:

If you knock the gene out and the cell completely changes, reverts to a normal form of cell, we get real excited. Now let’s make the protein that the gene codes for and study it.

Experimentally knocking out a gene examines whether inhibiting the protein that the gene expresses in the cancer cell with a drug might stop the cancer. The cycle continues.

These innovators revise or reframe their imagined configuration of interactions by iteratively integrating across different disciplinary boundaries, to explore various facets of information from different perspectives. The iterative integrating accumulates the insights generated by the discovery process into a revised hypothesis about the configuration of interactions that animate the drug’s functioning in the body against the disease. The mechanism of reframing connects formulating and evaluating hypotheses into ongoing cycles of discovery that, over time, build up a clearer and clearer understanding of how this molecule will behave in the body against the disease.

The third story also illustrates how the three social mechanisms work but emphasizes the third mechanism, iteratively integrating across disciplinary boundaries to reframe the configuration of interactions. The scientists use this reframing to assess their work and identify next steps. They collectively determine what makes the possible configuration of interactions they are working on good enough to continue with, and what they should change as they proceed. For ease of exposition we start the third example with the second social mechanism of elaborating and narrowing. A chemistry team leader describes how he and his team are opening up and reaching out for clues by drawing on people in other parts of the organization:

We can all make a prediction as to what kind of potency we think we need, but it is a reiterative process where the biologists will not only provide the data but can tell us a lot about our molecules that we could not foresee, such as how do they look when they dissolve? Do they dissolve? How did the cells look after they saw the molecule?… It is reiterative—it demands a lot of creativity and it is a very competitive area so we have to work well together and again try to develop molecular chemistry…

Notice the deep contextualization as he asks how do the cells look after they saw the molecule—they are looking at what animates the disease process, and delving into the specifics of possible mechanisms the molecule seems to have in the cells and how the cells react to the molecule. The potency of the potential drug is used to evaluate its effectiveness, but they elaborate out around that potency since “what kind of potency” they need is judged collectively. A similar level of intensively situating the evaluation and reframing is evident in the examples above as well. To generate these deep insights, the scientists work iteratively, back and forth.

As they work back and forth, they reframe their project and continue the cycle. The chemistry leader explains the reframing:

[Early molecules] will be screened in an in-vitro setting against activity within the cell or it could be the cell membrane or an enzyme or whatever and we will get a quick readout on that. That kind of screening might take a day or two and then from there we pick our best molecules…and we try to build upon that activity. And as we improve the potency of the molecules, they got into tissue-based and then eventually [animal]-based models and so it is a reiterative process that really drives the science forward… You try to get potency there [in a tissue assay] because that is a little more complex of a system. So as you go to animals you hit different roadblocks or challenges so you go back and try to redesign your molecules and try to figure why the molecule is not doing precisely what you wanted it to do. Does it have good bio-availability in a body? Is it really potent or maybe it is also toxic because it is hitting some other targets that you don’t know.

Now they hit different roadblocks from evaluating their configuration in animals, and have to redesign the molecule to figure out why it does not do what they wanted it to do. They ask a variety of new questions to explore the configuration and go back and redesign the molecules. Throughout they are trying to get a “quick readout,” to indicate “our best molecules” and to indicate “potency.” They do this by iterating from one experimental context to another, “an in-vitro setting,” in a “tissue base,” and other models, he says it is a “reiterative process that really drives the science forward.”

These three stories suggest how the three social mechanisms cycle together over time to imagine, examine, and reframe a viable configuration of interactions. We have discussed all three social mechanisms together since they constitute the abductive reasoning process that we find in this complex innovation context. However, each mechanism is important in its own right and cannot be skipped. As noted at the outset of this section, the appendix contains a number of examples of each mechanism.

While we find that these three social mechanisms of abductive reasoning work in the context of complex product innovation, we have fewer examples of the third process of reframing. We agree with scholars summarized in our introduction that reframing is necessary, and think that the discovery scientists do not reframe their imagined patterns as often or as thoroughly as they might need to do. We speculate that the relatively limited use of reframing in our data might arise because scientists and managers alike are reluctant to rethink the project and start off on another trajectory. Instead, they seek to progress through the discovery and development process to get products commercialized.

Barriers to the three social mechanisms

While these three social mechanisms advance existing research on abductive reasoning and address the challenges of complex innovation, they may be limited or hindered in practice because abductive reasoning generates only weak inferences about plausibility with indeterminate outcomes, and provides no guarantees that a viable new product will result. The scientists and managers we study need to create new products to generate revenues and that requires new drugs be materialized in concrete, efficacious forms (synthesized, mass produced, packaged, and distributed safely and effectively), and in a timely manner. Many managers struggle with the desire for more discipline, better planning, and better decision-making.

Our findings point to three specific barriers to abductive reasoning that, if removed, might enhance its implementation. One major barrier to using abductive reasoning is its unfamiliarity. No one we spoke to said they were using abductive reasoning, and some instead emphasized intuition and luck and spoke of clues, guesses, many stumbling blocks, and so on—all of which would certainly worry managers who are responsible for the expenditure of so much money as part of the product development process. If we are correct in our inference that abductive reasoning is essential for drug discovery, then improving its implementation will depend on more research that shows how and why it can work.

A second barrier is the limited use of reframing, which is necessary for the entire cycle of abductive reasoning. Reframing provides closure, either by pulling ideas together more fully for another round or by deciding to stop a project. Additional research is needed to understand why reframing is limited, but theory tells us that reframing is very hard to do (Dunbar et al., 1996). We speculate that scientists do not want to shut down projects prematurely, while reframing disturbs managers who need to show significant pipeline progression. The fact that many projects fail in late-stage clinical trials suggests that the “stopping rules” to shift or end the abductive reasoning for a given project are inadequate. Future research might address how these rules incorporate judgments about the viability of the configurations of interactions that projects are developing: are these configurations becoming clearer, what are the unknowns, and can we address them? Existing ideas about stopping rules that address information search in other contexts (Browne & Pitts, 2004) may also be useful for developing stopping rules that can be applied to complex situations. Innovators need to figure out how to shut down projects without shutting down projects that are going to be successful.

A third barrier to abductive reasoning is the preference for conventional or mechanistic decision-making that is based on confirmatory studies, or only on strict deductive logic. Conventional decision-making focuses on clearly identifiable goals and laying out the means to achieve those goals. In complex innovation there are not clear goals or a clear path to achieving those goals; the process is messy. Managers are worried and they are not familiar with abductive reasoning so, instead of searching for clues, they search for answers, instead of searching for clues to configuration they search for single facts. They try to simplify something that is complex. Instead of evaluating by elaborating and narrowing they hone in on particular results. Instead of iterating across disciplines to reframe they chop the process into parts and they do not cycle back through the process.

These barriers also suggest additional ideas for future research on the use of abductive reasoning for complex innovation. For example, future research might focus on how people can become more comfortable with the use of clues, and how they might better balance elaborating and narrowing to generate insights. Imagining a whole configuration seems to belie the calls for serendipity and creativity. However, new products are novel arrangements of parts, not a collection of separate parts, and we infer that innovators cannot create new products in complex domains working part by part, so future research could focus on different ways to focus on the configuration of interactions.

Future research might also continue to develop ideas about reframing. We can only speculate about reframing because our data are limited. For example, why may reframing not occur as often as it should? We suggest ethnographic study of complex innovation to explore why reframing may be difficult and how to reframe more effectively. We infer that the commercialization goal of drug discovery redirects attention from discovery to pushing products out into the market. People working with the intent of launching new products to generate revenue may feel pressured to move products forward linearly rather than cycle iteratively around possibilities. How innovators and their managers can more effectively balance the need for progress and revenues with the need for new products in complex domains needs more focused attention. The usual answer is to balance exploration with exploitation, but that may not be possible in complex domains like drug discovery because most new products will be complex (Pisano, 2006; Scannell et al., 2012).

Over the long term, producing continuous streams of innovation requires capabilities to support this process that are in line with an organization’s overall strategy. Garud and colleagues (2011b) find that the entire organization at 3M engages in and manages complexity to develop long-term technologies and strategies to continually support innovation. We suggest that this involves integrating the role of abductive reasoning. Further research into if and how organizations are able to do this would advance ideas for abductive reasoning and complex innovation.