Abstract
Systematic reviews of academic research have not impacted management practice as much as many researchers had hoped. Part of the reason is that researchers and managers differ significantly in their knowledge systems—in both what they know and how they know it. Researchers can overcome some of these challenges by including managers as knowledge partners in the research endeavor; however, doing so is rife with challenges. This article seeks to answer, how can researchers and managers navigate the tensions related to differences in their knowledge systems to create more impactful systematic reviews? To answer this question, we embarked on a data-guided journey of the experience of the Network for Business Sustainability, which had undertaken 15 systematic reviews that involved researchers and managers. We interviewed previous participants of the projects, observed different systematic review processes, and collected archival data to learn more about researcher-manager collaborations in the systematic review process. This article offers guidance to researchers in imbricating academic with practical knowledge in the systematic review process.
Keywords
Researchers often aim to advance knowledge by systematically reviewing a body of work. These systematic reviews synthesize what is known, highlight what is not known, and expose new opportunities for further research. During the review process researchers follow a set of well-prescribed protocols, which ensure that the answers to these research questions are based on the body of evidence and not just intuition or opinion (Briner et al., 2009).
Systematic reviews have the potential to impact practice because these types of reviews can answer the broad questions that managers ask by including a large body of work, rather than focusing on a single, often narrowly scoped, research study (Briner & Denyer, 2012; Briner et al., 2009; Rousseau et al., 2008; Rousseau & McCarthy, 2007). 1 However, systematic reviews often do not deliver on the potential to impact practice (Reay et al., 2009). This is at least, in part, because researchers and managers have different, seemingly incompatible, knowledge systems. Managers seek forward-looking, prescriptive insights, whereas systematic reviews offer abstract, descriptive findings based on historical data.
In this article, we argue that researchers can overcome these limitations and produce systematic reviews that impact both management research and management practice by involving managers in the review process. However, involving managers directly in the review process can be challenging and fraught with tension because of the very reason that systematic reviews miss the mark with managers—the differences in knowledge systems between researchers and managers (Bartunek & Rynes, 2014; Kondrat, 1992; Van de Ven, 2007). Although systematic review protocols guide researchers carefully through the review process (e.g., Briner & Denyer, 2012; Tranfield et al., 2003), they offer little guidance on involving managers (and other stakeholders) in the review process and navigating the tensions that arise. Hence, in this article, we ask, how can researchers and managers navigate the tensions related to differences in their knowledge systems to create more impactful systematic reviews?
To answer this question, we studied the 15 systematic reviews facilitated by the Network for Business Sustainability (NBS) since 2005. NBS is a nonprofit that brought together researchers and managers to collaborate in the systematic review process on topics of mutual interest. These NBS reviews were different from the traditional systematic reviews because they used a systematic process that synthesized the available body of research and prescribed specific practices for managers. The two authors were closely associated with NBS, which provided an excellent research opportunity. Bansal founded the Network of Business Sustainability (NBS) and was the Executive Director at the time of this study. She asked Sharma, who was a postdoctoral research fellow, to suggest improvements to NBS’s systematic review process. Sharma approached this request as a research project, seeking to gain deep insights into the systematic review process.
To draw on NBS’s experiences, we conducted 49 interviews with researchers and managers who had participated in NBS projects. We also observed a systematic review project over its entire 8-month life cycle and collected related documents, such as PowerPoint slides and emails. These data revealed that researchers and managers often experienced tensions between their knowledge systems, but over time they figured out how to navigate those tensions with the help of NBS staff. The practical insights that we present in this article complement the theoretical insights that we published on knowledge cocreation in the Academy of Management Journal (Sharma & Bansal, 2020).
Our article contributes to prior work on systematic reviews and the research-practice gap. To the literature on systematic reviews, we show how researchers can involve managers in the systematic review process and navigate the tensions that arise. We endorse the idea that management research needs to impact management practice, and especially on issues related to sustainable development. Furthermore, we contribute to the literature on the research-practice gap by suggesting that the impact of research on practice need not always be significant and measurable, but can appear as small changes in managerial thinking and practice throughout the process.
Literature Review
Systematic Reviews: Potential for Impact
Briner and Denyer (2012, p. 112) describe the systematic review process as one in which the researcher “addresses a specific question, utilizes explicit and transparent methods to perform a thorough literature search and critical appraisal of individual studies, and draws conclusions about what we currently know and do not know about a given question or topic.” Unlike literature reviews, systematic reviews aim to synthesize all, not just summarize selected, studies to provide a complete picture of the research landscape.
The issue of impact is important in the discussion of systematic reviews, given that the very aim of systematic reviews is to encourage managers to base their practice on accumulated evidence and not just some selected evidence (Briner et al., 2009). Because systematic reviews offer “systematic accumulation, analysis and reflective interpretation of the full body of relevant empirical evidence related to a question” (Rousseau et al., 2008, p. 475), they can provide the guidance that managers seek to improve or change their practices. Indeed, many protocols—such as the Cochrane Model (Schunemann et al., 2008)—clearly state that the conclusions of a systematic review should provide readers with practical implications for making decisions.
There has been considerable discussion of the challenges of measuring the impact of research on practice. Aguinis et al. (2014) recognize that research should impact a broad range of stakeholders—such as managers in business, government, and the nonprofit sector—and that any measure of impact should involve a wide range of context-specific measures, such as through teaching or publishing research findings in practitioner journals. What makes measurement even more challenging is that the impact of research on practice is a recursive process, so that researchers impact managers, who impact researchers, and so on, making it difficult to disentangle the impact of research on practice (Schulz & Nicolai, 2015; Simsek et al., 2018). Astley and Zammuto (1992) suggest that the impact of research is difficult to see because research ideas are abstract and do not translate into direct action; rather, research impact is seen when managers make the abstract ideas relevant in their idiosyncratic practice. MacIntosh et al. (2017) extend the discussion by suggesting that research impact is evident not only in practice, but also in the change in managers’ and researchers’ sense of identity.
Despite the potential for systematic reviews to impact management practice, there is little evidence that they have actually done so (Reay et al., 2009). Some researchers have translated their systematic reviews into managerial language once the review is completed, while others have tried to mobilize the reviews in business school classrooms (Rousseau & McCarthy, 2007). However, this one-way, end-of-pipe approach to reach managers is part of the problem because the final product may not be relevant to managers and insights may be lost in translation (Rynes et al., 2018; Shapiro et al., 2007).
In addition, most research studies are based on historical data, and individual studies can take quite some time to be published. Systematic reviews drawing on these studies, especially in dynamic contexts, can quickly seem outdated and irrelevant. Also, many issues relevant to practice are context-specific and require an understanding of the lived experience and knowledge of managers and other stakeholders. The researchers who conduct the systematic review can sometimes be out of touch with the needs of the people who use the review, such as managers, policy makers, and regulators. Even though such people are the target audience of systematic reviews and may even be mentioned in the outcomes, they often do not provide input into the actual research or review process. Consequently, the findings reported in systematic reviews may not reflect the lived experiences of managers or their day-to-day challenges.
One way to alleviate these concerns and create more impactful systematic reviews is for researchers and managers to collaborate in the systematic review process. Researchers and managers could work hand-in-hand in all stages of the process. However, by doing so, the tensions between the two different knowledge systems are brought into sharp relief.
Research-Practice Tensions
Scholars recognize not only the value of collaborating with managers to generate knowledge, but also the challenges in navigating the tensions that arise between the two communities (Bansal et al., 2012; Bartunek, 2007; Bartunek & Rynes, 2014; Van de Ven, 2007). Moisander and Stenfors (2009) argue that the tensions reflect differences in epistemic cultures, which is “a system of shared symbols and meanings” (p. 231). Epistemic cultures can highlight differences among communities regarding the knowledge that is considered salient, who is considered credible in creating knowledge, and the audience that is considered relevant to the knowledge being created (Nielsen, 2018). The actors involved in producing knowledge organize their skills, strategies, and apparatus in a way that is unique to their epistemic culture (Knorr Cetina, 1999; Latour & Woolgar, 1986). Hence, differences in epistemic cultures reflect differences in the knowledge that is valued, based on the conditions under which the knowledge is created.
We draw on this prior work to describe tensions experienced between researchers and managers because of differences in their knowledge systems, specifically what they know and how they know what they know. We recognize that the categories that we describe below are caricatures and not reflective of all researchers and managers. However, by polarizing the differences, we can show how the tensions can be navigated to foster new knowledge (Bartunek & Rynes, 2014).
Differences in What They Know
Managers generally value concrete knowledge, whereas researchers prefer abstract knowledge. Concrete knowledge is often implicit and specific to the context. The knowledge varies by the resources, actors, and other conditions that are unique to each context (Astley & Zammuto, 1992; Kondrat, 1992). Managers need to work productively in fast-changing and often uncertain environments, generating concrete knowledge to solve immediate, context-specific problems. Managers validate their knowledge through these actions. Knowledge that helps solve problems is validated and deepened, and knowledge that does not help solve problems is adjusted or dismissed.
Researchers, on the other hand, value abstract knowledge, which refers to the principles and relationships that can be generalized beyond a particular context. When patterns emerge, researchers seek to formulate generalized principles or theories that explain why or how things work that extend beyond the immediate context (Dewey, 1910; James, 1907). Hence, abstract knowledge can be published in journals, research reports, and textbooks, because it can be made explicit and transferred across multiple contexts (Kondrat, 1995). Researchers purposefully seek knowledge that can apply to a broader range of contexts and endure for a longer period of time than the knowledge pursued by managers.
Differences in How They Know What They Know
Managers and researchers create knowledge in different ways, reflecting differences in how knowledge is created and by whom and how quality is judged (Bartunek & Rynes, 2014; Kondrat, 1995). Prior research has described these differences as differences in institutional logics—researchers and managers differ in how they define and tackle problems. Researchers and managers also differ in the criteria of quality of knowledge; specifically, researchers judge quality by the rigor of scientific practices, and managers judge quality based on the relevance of the knowledge (Bartunek & Rynes, 2014).
A manager gains knowledge through practice, by responding to changes in their context as opposed to one generalizable, clearly articulated problem. The manager moves from context to context, acquiring and adapting knowledge across contexts. Managers often have to grasp for meaning, making choices between several equivalent courses of action based on prior experience and new surprises, often because of the short time horizons for decision making (Bartunek et al., 2003; Bartunek & Rynes, 2014). They act based not only on their own experience but also on the opinions of other managers. The exercise of knowing is about retaining one’s own point of view while considering others’ points of view and being attentive to how others’ views affect one’s own conclusions (Kondrat, 1995; Van de Ven & Johnson, 2006). Managers learn as they go, using their own experiences to judge the validity and quality of the knowledge being offered.
A researcher, on the other hand, seeks to eliminate personal biases and build on a body of credible knowledge. A researcher answers precise questions, offers precise answers, so the knowledge can be applied across contexts without distortion. The researcher seeks to create knowledge detached from the research context, even when the knowledge is created from the perspective of a participant observer or ethnographer (Evered & Louis, 1981). Maintaining distance or objectivity is considered good science (Rhodes & Carlsen, 2018). A researcher draws on widely agreed upon protocols that are deemed valid by the research community. Answers to questions are considered worthy of attention only when the research processes and outcomes are replicable, reliable, and valid, as judged by academic peers. When knowledge is created by the principles of good science, a researcher is rewarded by their institution through journal publications, tenure, and high-status appointments, an incentive system that does not align with that of managers (Bartunek & Rynes, 2014).
Opportunities to Create Impactful Systematic Reviews
These tensions between researchers and managers hold the opportunity for creating more impactful knowledge. Indeed, Bartunek and Rynes (2014) remind us that the metaphor “research-practice gap” may have outlived its value. The differences between the research and practice communities are sources of creativity that offer opportunities for deeper theorizing. Similarly, Dipboye (2014) proposes an alternate metaphor of a “map” instead of a “gap.” He argues that the difference in the map made by the cartographer and its interpretation by the traveler can be generative since the traveler will always make the map her own by taking side trips, losing herself in places, and finding new places. The tensions we describe above are the starting point to generate impactful systematic reviews and motivated the question that grounds this article: how can researchers and managers navigate the tensions related to differences in their knowledge systems to create more impactful systematic reviews?
Method
Research Context
To answer our research question, we took a data-guided approach to our inquiry. We used neither an inductive approach to generate substantial theory nor a deductive approach to test theory. Rather, our objective was to identify a set of protocols, based on our observations and inquiries that could help researchers engage productively with managers to create more impactful reviews. We did not limit ourselves to the aspirations of tightly regulated prescriptions to build theory systematically, but rather sought to uncover more practical answers to the perennial and important problem of impacting practice by involving managers in research.
Our source of insights was the systematic review process used by NBS, a Canadian nonprofit organization seeking to bridge sustainability research and management practice (www.nbs.net). NBS was established in 2005 and had conducted 15 projects when we started writing this article (see Table 1 for a list of all systematic review projects). The first systematic review started in 2005, and the final one concluded in 2015. NBS discontinued conducting systematic reviews because business leaders felt that many of their management-related questions had been answered and much of the sustainability literature had already been reviewed under prior topics. They were wanting to engage in new research, rather than reviewing existing research.
NBS Systematic Reviews.
a. NBS projects produced systematic review reports and other materials such as executive reports, one-page summaries, and checklists.
b. These figures are from August 26, 2017, to August 25, 2019. Total page views is the total number of times the page has been viewed. It includes repeated views of a single page.
c. For this project, separate numbers are not available for full systematic review and the shorter executive report.
NBS was an appropriate site for our insights for a few reasons. First, it had significant experience in conducting systematic reviews in an untraditional context, that is, outside of the medical sciences or industrial psychology. NBS was focused on macro-level sustainability issues and organization-level practices. Second, Sharma and Bansal could take an outsider-insider perspective: Sharma was the outsider invited to study the process; Bansal was the insider because she founded NBS and was motivated to create impact through its work. This arrangement provided us excellent access to a process, people, and archival materials. Furthermore, both authors had developed a critical eye and deep interest in impacting management practice to advance sustainable development.
Each year, NBS released a call to researchers for proposals to conduct a systematic review of a topic identified by its Leadership Council—a group of approximately 15 to 20 cross-sectoral (private sector, NGOs, government) sustainability business leaders representing noncompeting organizations. The NBS project manager and Leadership Council members collectively chose the research team from the proposals submitted in response to the call. They chose the research team following a process in which they ranked proposals based on the researchers’ topic expertise and experience in conducting relevant research. The team often comprised one or two researchers and sometimes a student. In addition, three to four Leadership Council members with functional experience related to the topic and an academic advisor offered guidance to the research team throughout the systematic review process. An NBS project manager facilitated the interactions among researchers and managers and ensured the project met its major milestones. The research team received a modest remuneration from NBS to help cover costs such as research assistance. NBS disseminated the reports and frameworks from the systematic projects free of charge through its website and newsletter. The primary funding for NBS’s activities was secured through a federal government grant. As well, Leadership Council members paid a nominal annual membership fee. Sharma and Bansal did not receive any remuneration for their NBS work.
The project team met approximately once a month for the systematic review process, which extended over 8 months. The meetings followed a set protocol (based on other similar protocols; Tranfield et al., 2003) involving different steps of the systematic review process, as described below. 2
Systematic reviews begin with a clearly defined question or set of research objectives.
After the question is defined, the researchers conduct an extensive search of prior studies which have answered the question. These studies can come from a variety of published and unpublished sources, including electronic databases and journals, recommendations from experts, and snowballing references through cross-referencing (Briner & Denyer, 2012; Rousseau et al., 2008). Whereas some protocols recommend focusing only on high-quality scholarly work, Adams et al. (2017) recommend that gray literature (published in high-quality, but not peer-reviewed, journals) be included if it fits the review’s purpose, that is, fills in blind spots.
Once researchers accumulate the corpus of studies, they apply clear criteria for filtering studies. The criteria need to be explicit to ensure that the review is “impartial and balanced” (Briner et al., 2009, p. 26). Different filtering techniques are offered by different protocols. For example, Rousseau et al. (2008) ask researchers to screen the studies by applying criteria such as construct validity, effect size, measurement quality, and so on.
In the next step, researchers code and synthesize the findings from prior studies. Rousseau et al. (2008) remind researchers to use the research question to guide the coding process, so that the studies are synthesized in such a way that answers the question. Tranfield et al. (2003) recommend that researchers use a data extraction form, which can help researchers systematically code the studies. The synthesis could be narrative or quantitative (meta-analysis) depending on the nature of studies coded.
In addition to the work completed during the meetings, researchers and managers on the project team also worked on project tasks between the meetings. Researchers conducted various tasks such as distilling, coding, and synthesizing the studies. They brought the results of their analysis to each meeting. The work that managers completed during the meetings was less explicit, such as conducting thought experiments within the context of their work to test if the ideas and framework developing from the systematic review were relevant in their work context. A few managers also performed ad hoc tasks between meetings, such as providing evidence from their organizations to support the claims they made in the meetings or connecting the researchers to organizations outside of the project team to help the researchers gain further insights.
Data Collection
To build a body of knowledge that is based on systematic reviews (such as the Cochrane Library), Bansal developed NBS’s systematic review process. Even though she was not involved in the day-to-day management of the systematic review projects, she sought to continuously improve the process, which motivated this data-based inquiry and Sharma’s involvement.
We collected data from several sources, including interviews, observations, and documents. We both conducted 20 interviews with 17 researchers who had participated in NBS projects over the years. The interviews focused on the researchers’ experience of working with the managers, specifically the moments of connection and moments of tension, and how the tensions were navigated. We also both conducted 29 interviews with 14 managers involved in the projects. We asked them about their experiences working with researchers, and moments in which they were on the same page and moments they were not. We also asked the managers how they navigated the tensions or saw the researcher / project manager navigate these tensions.
In addition to these retrospective insights, Sharma observed a systematic review project from January to September 2014. The project’s topic was related to long-term decision making for sustainability. The project team comprised two researchers, five managers, an academic advisor, and an NBS staff member who coordinated the project and facilitated the meetings. There were 7 meetings, 4 to 6 weeks apart. The meetings followed the systematic review process stages described in Figure 1.
Stages in the Systematic Review Process.
Furthermore, Sharma reviewed documentation associated with prior projects such as PowerPoint slides, meeting minutes, and reports circulated before meetings. From these data sources, we were able to glean insights around concrete practices that helped to overcome the tensions between researchers and managers. Finally, Sharma joined NBS staff meetings to discuss the progress of ongoing systematic review projects, among other issues. The team shared challenges and offered solutions for the ongoing projects.
Data Analysis
We coded the tensions and practices to navigate the tensions at each step of the review process. Sharma coded the data and discussed the emerging insights with Bansal. We used our insider-outsider positions (Dwyer & Buckle, 2009; Louis & Bartunek, 1992) to extract novel insights that may not be evident if both of us were distant to the data (outsiders) or embedded in the context (insiders).
We began our data analysis by dividing the data into the stages of the systematic review method described by Tranfield et al. (2003), which was a rather detailed and actionable account of what needed to be done and broadly mirrored other systematic review protocols (e.g., Rousseau et al., 2008). As well, NBS’s process had been initially imprinted by Tranfield et al.’s (2003) protocol.
Within each stage, our analysis focused on the actions researchers took at each step of the systematic review project. For example, researchers extracted data from the existing studies based on a well-defined data extraction form, a step prescribed by Tranfield et al. (2003) and followed by the researchers in NBS systematic review projects. Similarly, researchers synthesized insights from prior studies by building descriptive and analytical frameworks, also a step described by Tranfield et al. (2003). In sum, the researcher actions that we coded mapped onto the steps laid out in the systematic review protocol.
We also coded the actions managers took at each step of the systematic review process. For example, when researchers extracted data from studies using a prescribed form, we coded what managers did during this step. In this example, we found that the managers sought to add/remove codes from the data extraction form based on the outcomes of the project. Finally, we coded the actions NBS project managers took at each step of the review process—such as distributing the summary of the researcher’s work before each meeting.
We also coded the tensions that researchers and managers faced in the course of generating knowledge. Tensions were evident in our data when researchers and managers disagreed on how to conduct a particular systematic review step, such as adhering to the guidelines of a rigorous systematic review or adding dimensions to the emerging framework based on the manager’s experience. In such moments, researchers and managers offered contradictory approaches to accomplishing a task, such as using white papers and organizational experience to build the framework (managers) versus drawing insights only from the literature (researchers). For example, we coded for tension at the initial stage of the systematic review process when the researchers defended their list of keywords for selecting studies to include in the review, while managers wanted to expand the list. We found similar tensions throughout the systematic review process.
Finally, we identified the practices for navigating these tensions. Once we identified the tension, we analyzed the data for the actions that researchers, managers, or the NBS project manager took to navigate the tension. We coded for “navigate tension” when specific practices helped both parties to continue forward. Some practices involved more than one party, such as the researcher and NBS project manager, and affected more than one tension. For example, we found that researchers involved managers in writing the recommendations (including case studies that illustrated specific aspects of the theoretical framework) to navigate the tensions in the report and recommendations task.
Findings
Systematic Review Process
In this section, we describe how researchers and managers came together at each of the three stages (planning the review, conducting the review, and report and dissemination) of the NBS systematic review project and the underlying tensions at each stage. Also, we explain the practices that helped both parties address these tensions (see Table 2). We end by describing the impact that NBS systematic reviews have had.
Tensions in Researcher-Manager Knowledge Systems.
Stage 1: Planning the Review
Each year, the managers on NBS’s Leadership Council suggested to NBS a broad topic they believed was important to advance the sustainability agenda. NBS then issued a worldwide call (including topic details as offered by the managers) for proposals from researchers related to the topic. The call for proposals was sent through NBS’s newsletter and academic outlets.
The call for proposal guidelines required researchers to briefly summarize the salient literature and their plan for disseminating study findings. A small group of managers (4–6) from the Leadership Council volunteered to join each systematic review project team. None of the managers had a contractual obligation to stay involved in the project, nor were they looking for advice specific to their organization. Instead, the managers joined the project to develop knowledge with the researchers, knowing that the project outcomes would be distributed widely to the business and academic communities. The managers who joined the project volunteered their time, but there was an informal expectation that these managers held functional expertise on the topic. For example, if the systematic review focused on global supply chains, the managerial team often had experience with their organization’s supply chain.
NBS’s project manager along with this group of 4 to 6 managers reviewed the proposals, shortlisted candidates, and chose the researchers for the systematic review project team after a round of interviews. The researchers were chosen based on the rigor in the proposal they submitted, their publication record on the topic, and their openness to working with managers. NBS also chose an academic advisor—a researcher with expertise on the topic. In sum, the project team included managers who volunteered their time, the researcher(s) selected through a competitive process, an academic advisor invited to the project, and an NBS project manager.
Step 1.1: Identifying the Need for a Review
NBS projects began with defining the research question for the review. Researchers reviewed the literature based on the topic of the review and their own research expertise. For example, topics addressed in NBS projects included innovating for sustainability and orchestrating social change. By scoping the literature, researchers crafted a research question which could help them address an important theoretical puzzle and contribute to the existing knowledge on the topic.
Managers in the project team participated at this stage to define what they thought was the relevant question based on their experience. Managers crafted a question based on the salient issues in their organizational life, or what they believed was a salient issue for the larger business community. So while researchers wanted to solve theoretical puzzles, managers wanted to find answers to problems of practice.
Research-practice tension
Such contrasting approaches to defining a question yielded tensions, which have been documented by other scholars in terms of the differences in the logics of problem definition and solution. Research-practice tensions in problem definition and solution arise because science and practice systems are self-referential to their own knowledge system with idiosyncratic logics (Bartunek & Rynes, 2014; Kieser & Leiner, 2011). A researcher described her frustration to us. Managers in her project wanted to know whether consumers would pay for sustainable products. However, the researcher wanted to focus not on whether but on why consumers would pay and to unpack the underlying mechanisms: Managers would push for things like what percentage increase premium can be charged? And I’d say well, based on the 13 studies, it’s like 1–2%, whatever. But such statements were based on a very small set of studies. It would’ve been much more useful to ask okay, why do consumers respond positively to these products.
Practices to navigate this tension
To navigate the tension at this step, the NBS project manager often reminded both parties of their common interest in the topic. In some projects, the NBS project manager encouraged researchers and managers to list salient issues related to the topic, which encouraged dialogue. For example, in a project focused on competitor collaboration for sustainability, researcher and managers could not agree on a research question because they had diverging views of “competitor.” The researcher wanted to define competitor in clear terms so as to draw boundaries of the study. Managers, on the other hand, provided a variety of definitions. To break the gridlock, both parties began listing what competition meant in a sustainability context. Once the ideas were listed, researchers and managers looked for common ground to cocreate the question. The researcher realized that she had not considered “competing for inputs and not just for selling outputs” in the definition of competition. Managers saw the variety in the definition of “competitor” across their peers in the meeting and hence were open to converging to a tighter definition, resulting in a research question which met the needs of both parties.
Step 1.2: Reviewing Proposal and Protocol
Once both parties agreed on a question, the next step was to put together a proposal and research protocol. In the proposal, researchers briefly summarized the state of the literature to motivate the question. The protocol explicitly described each step, which the researcher would take in conducting the review. Researchers curated a list of keywords to search for existing studies and stated the criteria for inclusion, that is, the threshold that the studies have to pass to be included in the study pool for review.
Researchers presented the list of keywords to the managers. Managers suggested changes to the list by adding a word salient to the manager’s organizational experience or removing a word if the managers believed it was outside their organizational experience.
Research-practice tension
Since keywords in a protocol determine the scope of the review, adding keywords could significantly increase the pool of studies included in the review. As well, managers used keywords more broadly than did researchers. Hence, tensions arose as managers wanted to increase the project scope, which researchers resisted.
Bartunek and Rynes (2014) describe this as a tension of communication where the knowledge representations of one party do not work for the other. In our study, such tension was evident, for example, on the topic of civic dialogue for sustainability; managers wanted to add “nationwide conversations” as a keyword, yet the researchers could not find an equivalent concept in their literature, especially since managers could not define precisely what they meant by “nationwide conversation.” A researcher described her frustration in discussing the keywords list with the managers: Those conversations [with managers] were important but they had an underlying tension about are we going to expand [the keywords list] or are we going to hold the fort. That was a bit anxiety-producing for us at times, especially when we found out exactly how many articles we were going to have to review.
Practices to navigate this tension
NBS navigated this tension by involving an academic expert who was not part of the team but an external third party. She or he could identify when the project scope became too broad and steered the conversation back to the most important issues related to the topic. Several researchers mentioned that the academic expert played an important role in deciding the final list of keywords. Since this expert was not part of the research team, both parties saw her or his suggestions as objective and could accept them.
In a few cases, researchers addressed this tension by asking managers for examples for each keyword suggested. By identifying themes across the examples, researchers were able to cluster new keywords such that the list did not expand significantly beyond the project’s scope. For example, in the project on long-term thinking, researchers included “sustainability” as one of the keywords since the concept implicitly included the temporal element. However, a manager insisted that the researcher must also include “triple-bottom line.” The researcher did not agree and asked the manager, “How do you see the time element in ‘triple-bottom line’?” In response, the manager provided a variety of examples. By discussing each example, it became evident that the manager’s disconnect was with the focus on the natural environment in the word “sustainability” and that researchers would miss studies that focused on social impact. The researchers agreed to include “triple-bottom line” as a keyword, but only in the intersection with the keyword “time horizon.”
Stage 2: Conducting the Review
Step 2.1: Identifying and Selecting Studies
Once the keywords list and inclusion criteria were finalized, the researchers followed the protocol to identify studies meeting the criteria. For the researchers, the review’s rigor stemmed from closely following the protocol.
Research-practice tension
Managers often found it challenging to contribute at this step of the process. They could not access relevant tools, such as the Web of Science database needed for collecting studies for the review. Such tools are either behind a paywall or are provided for a fee by organizations such as the Center for Evidence-Based Management. Furthermore, the researchers’ updates were often a summary of the research process, for example, databases used and a number of studies meeting the criteria. Such summaries were abstract in the sense of general principles. Managers found it challenging to contribute to such abstraction, as evident in the silence we observed in one of the meetings after the researchers finished summarizing their research process. The NBS project managers corroborated this insight. In one NBS team meeting, a project manager said that managers in her project claim to “trust NBS and the researchers” to follow the due research process, hence the managers did not actively contribute at this step.
To exacerbate the disconnect, managers were interested in research evidence which validated and extended their experience. Peer-reviewed research often did not include the ideas managers encountered in practitioner conferences or interacted with their peers because the timelines for research and practice were widely different. By the time an idea that was potentially useful in practice made its way to research publications, managers found little value in the idea, and commensurately, the ideas that were currently relevant to managers were seldom found in research publications. The researchers found it difficult to include suggestions such as white papers or other similar suggestions made by the managers. One researcher summarized this tension as follows: “Some of what you [managers] are looking for may not actually be written about or not written about rigorously.”
Practices to navigate this tension
Researchers navigated these tensions in a few ways. Some researchers intentionally went beyond summarizing the research process, and instead brought broad themes or emerging findings to their conversation with the managers. Such a practice provided an opportunity for managers to contribute, since the discussion was no longer about the research process. However, many researchers were hesitant to “jump the gun”; as one researcher said, most did not want to make conclusions before systematically extracting the data.
Other times, researchers navigated the tension by reconsidering their decision to include articles from the gray literature, which was closer to the manager’s lived experience. This decision was seldom easy for researchers. Researchers needed to balance the relevant ideas in the gray literature with the rigor available in the peer-reviewed papers. A researcher vividly recalled his decision to include gray literature: It was hard at first, but the managers on the project shifted our thinking. We tried to find ways to accommodate current practices not reflected in the academic literature, but at the same time remain faithful to what the systematic review tries to do in terms of capturing the best available evidence.
Step 2.2: Assessing Quality of Evidence
Once researchers identified a pool of studies, the next step was to assess the quality of the studies. Researchers in some NBS projects included the “weight of evidence” as an important analysis step but adjusted this weight based on the methodology followed by the studies included in the systematic review. They followed Briner and Denyer (2012) in assessing the weight of evidence by methodological quality; that is, did the study follow the commonly accepted protocols of the particular research method? In other words, was the research design appropriate to answer the research question? In contrast, we found that the quality for managers was in whether a piece of evidence was relevant for an organizational problem the managers were facing.
Research-practice tension
The managers’ focus on relevance did not often align fully with what researchers deemed as high-quality research. Others have described this as the tension of rigor and relevance (Bartunek & Rynes, 2014; Gulati, 2007; Vermeulen, 2005). Researchers can find that focusing on relevance compromises rigor, and yet managers often find rigorous research not very relevant to their work (Kieser & Leiner, 2009). In our study, a researcher described this tension in her project on social change. Managers in her project pointed to social media as a force for social change. She explains, “This was in 2012, social media was still a new phenomenon. We got a sense that there is some literature on social media that shows that social media is a force for social change, but it was clear that the literature wasn’t developed enough.” However, for the managers, various blogs, articles, and email exchanges about social media should have been included because they represented an idea of relevance.
Similarly, in the project that we observed, the discussion on assessing quality of evidence was clearly demarcated between quality in terms of research rigor and quality in terms of practical relevance. Managers in a meeting shared many examples of why an idea such as “Obama’s latest report on climate change” should be included in the pool of studies for the systematic review. After receiving suggestions that ranged from ideas in consulting reports to anecdotes from the managers’ organizational experience, one of the researchers exclaimed, “It is almost anything you want to pick, somebody has written something about it, but that does not mean they have any evidence to support their claims.” He reiterated that there needs to be rigorous data to support someone’s beliefs; otherwise, it is challenging to include opinions in a systematic review.
Practices to navigate this tension
A few researchers in NBS projects navigated this tension by including relevance to practice as one of the criteria for assessing the quality of the studies. However, such an approach was challenging. Researchers often felt stuck when faced with trade-offs between high-quality research and ideas relevant for practice.
We found that there was no one moment where researchers transitioned to a solution. One researcher described his gradual approach: “It took me a long time to be happy with what we were saying as both a faithful reflection of data and something that might be relevant to managers.” Relevance within each project was an evolving concept and contextual to the project participants. Researchers who could effectively include relevance as one of the quality criteria were the ones who were willing to repeatedly and openly discuss the disagreements with the managers instead of reiterating the research process or standards of rigor in response to managers’ suggestions.
One researcher described her many conversations with managers in her project on stakeholder collaboration. A manager insisted on including studies on eco-labeling as one way for stakeholders to collaborate. The researcher was not convinced; she could not find research at the intersection of eco-labeling and stakeholder collaboration that also met the quality criteria. The manager connected the researcher to peers in the retail industry working on eco-labeling for the industry. The researcher described speaking with the managers from the retail industry and seeing the relevance of the idea: “I could see how eco-labeling could morph into collaboration between fisheries, retails stores, and other stakeholders.” She returned to the next project meeting willing to continue the discussion with the managers on eco-labeling and the quality of available evidence. Along the same lines and continuing the social media example from above, the researcher described that “in the end [they] included two studies which weren’t really complying fully by our criteria [of quality of evidence]” because after many conversations, researchers and managers agreed that “social media is a big, emerging area and has potential to kind of create large scale social change.” Researchers, in this case, used the relevance of the concept as a quality criterion.
Step 2.3: Extracting Data and Monitoring Progress
Once the studies passed the quality threshold, they had to be coded; this step was known as data extraction. The researchers used a data extraction form to code the studies. They also employed multiple coders to increase the reliability of their coding.
Research-practice tension
Researchers shared the data extraction form with the managers and explained the codes used. Managers understood that the kind of data extracted affected the project outcome achieved, so they often asked to modify the codes. However, because the researchers were immersed in the data, they were often not open to modifying the codes. Other scholars have described this tension as one of “usefulness,” that is, whether research insights must be useful for practice, where usefulness is described in many ways, including informing the managerial practices, going beyond the obvious or the common sense of the user, and focusing on the predictive outcomes rather than the explanation of why the outcome is achieved (Brief & Dukerich, 1991). A researcher in our study explained this tension. In one of the projects, the managers were dissatisfied with the codes because the codes did not include the business case to support a phenomenon. A manager explained that the study should provide quantitative data to establish the business case, in this instance, for cooperatives: “Being able to demonstrate that cooperatives and mutuals outperformed investor firms allows me to provide some evidence that there is something to this cooperative model; something other than we’ll say the governance structure or the touchy-feely belief that cooperatives are more about people.” However, the researcher stated that coding for a business case was not feasible since success was not “black and white.” Another researcher expressed similar frustration. Managers in her study were looking for answers by industry. She elaborated, Managers were not happy with the way the project was going. They wanted it broken down by industry; for example, “I want forestry, I want whatever the main sector.” First of all, academics don’t do by industry. And second of all, there will not be enough studies [to code] if broken down by industry.
Practices to navigate this tension
We found that some researchers navigated this tension by replacing data extraction forms with frameworks and figures. The data extractions forms had extensive text and provided a linear layout of the ideas. Therefore, the focus of the discussion between the researchers and the managers remained stuck on the semantics. Each party interpreted the data extraction codes in their own way, unable to communicate with the other. However, when the NBS project managers encouraged the researchers to put the interim findings/codes into a framework, researchers found a concise way to communicate the ideas to the managers. The framework contained fewer words, and hence the dialogue focused around the ideas instead of the semantics.
For many researchers, producing a framework without coding all the studies was challenging because it diverged from the rules of systematic analysis. At the same time, researchers who were able to present the emerging ideas in a framework were the ones who moved beyond the tension of incrementally adding/removing codes from the extraction form, and instead could have a dialogue with managers around the blind spots in existing research. A researcher summarized her experience: They [managers] didn’t fully grasp [in the framework] what we thought was clear. There was definitely a learning moment for both of us because once they grasped what we were trying to convey, that changed their thinking about it. Their questions and their interpretations of what we had put on that paper also provoked us to say, oh, I see, you were thinking of it in this way. That would be another alternative way of modeling what we were trying to talk about.
Step 2.4: Synthesizing and Interpreting Data
Synthesis and interpretation involved combining the individual studies to arrive at new knowledge not evident otherwise (e.g., Briner & Denyer, 2012). The synthesis could be narrative or quantitative (meta-analysis) depending on the nature of the studies coded (Tranfield et al., 2003). NBS projects focused on narrative syntheses. The researchers looked for convergent findings and inconsistent results. In accordance with Rousseau et al. (2008), when faced with inconsistent results, researchers explored whether the inconsistency was methodological or substantive, and defined the theoretical and empirical boundary conditions accordingly.
Specifically, researchers undertook two kinds of narrative synthesis, following the guidelines provided by Tranfield et al. (2003). In the descriptive synthesis, the researchers provided an overview of the state of the research on the topic. Such a summary revealed the focus of the existing research and blind spots, if any. This summary also provided the context for a more analytical synthesis. The analytical synthesis comprised more abstract generalizations yielding one or more conceptual frameworks.
Research-practice tension
Managers were forthcoming with examples to illustrate the framework emerging from the synthesis. However, tension at this stage arose when a manager identified parts of the framework as dissonant with her or his own experience, and hence suggested changes. The managers suggested changes in a few ways. They would suggest adding a new concept or idea to the framework such as adding drivers of a phenomenon, but this suggestion was rare if the initial steps of defining the codes were conducted effectively. A more common suggestion was to add dimensions to the existing concepts. For example, in one of the projects, managers suggested longevity (number of years in existence) of a collaboration as one of the dimensions of the outcome. Researchers at this point were faced with an important choice: should they defend the content in the framework since the framework was built on systematic coding and interpretation of the studies; or should they modify the content such that the framework could be more relevant for practice? Other scholars have described this tension as one related to the independence of researchers for generating knowledge, which precludes ad hoc changes based on opinions instead of data (Grey, 2001). A researcher in our study expressed his frustration when faced with this choice: “I cannot change the framework, just because you [manager] said it. It cannot be done arbitrarily.”
Managers were aware of the researcher’s discomfort and some saw it as resistance. One manager explained, “You felt the researchers had a theory that they were going to be pushing and they were trying to get the research or our thinking to fit into it.”
Practices to navigate this tension
Researchers in the NBS projects addressed this tension in several ways. Some found this tension to be an opportunity to triangulate. Researchers would revisit the literature to explore whether the manager’s insights are represented in the literature. However, given the issue of published studies lagging behind practice (Adams et al., 2017), usually the manager’s inputs were not reflected in the existing studies. Other times, researchers considered the manager’s inputs as equivalent to the data from coding existing studies. The researchers in this case were open to modifying the framework. For example, a manager in a project insisted that “organizational level” is an important dimension under the construct of “drivers” in the framework for long-term thinking. The researchers explained that “organizational level” was not evident in published studies as a driver. One of the managers collected job descriptions from her organization, which she shared with the researchers to show that the job descriptions emphasized long-term thinking differently based on the organizational level. Researchers saw the relevance of the idea and included it in the framework, noting in the publication their process of adding the dimension to the construct.
We also found that the quality of the dialogue around the frameworks was critical to the changes made to the frameworks. Some of the indicators of high-quality dialogue included both parties taking turns to listen and speak and building ideas together, instead of a bilateral question-answer approach. For example, researchers asked managers what they thought of the ideas, inviting them to build on the ideas. Similarly, managers asked other managers in the meeting to share their experience, instead of always looking to the researchers for answers (also see, Sharma & Bansal, 2020).
This kind of dialogue was possible when the researchers presented—and managers understood—the frameworks as incomplete. A manager explained such incompleteness: “The researcher kind of had a draft framework. But it wasn’t like she was presenting to us work that was completely done. I would say both sides had the ability to be influenced by it.” Visually, such frameworks were sparsely drawn, something one would draw quickly on a piece of paper. Rhetorically, these frameworks were presented by the researchers as work in progress. Researchers used phrases such as “this [framework] is not set in stone” or “these ideas will continue to evolve,” inviting managers to complete the frameworks. The ensuing dialogue allowed for deeper insights and preempted the resistance researchers felt in changing the frameworks based on the manager’s inputs.
Stage 3: Report and Dissemination
Step 3.1: Reporting Recommendations
Existing scholarly articles on systematic reviews (e.g., Briner & Denyer, 2012; Rousseau et al., 2008) provide little guidance on the outcomes of a systematic review, that is, outcomes for management practice, in addition to publishing an academic paper. And what little guidance there is can be found in sources such as blog posts instead of traditional management journals. Moreover, researchers are seldom trained to envision a variety of outputs beyond academic papers or book chapters. As a result, several differences emerged in how researchers and managers approached this step.
Research-practice tension
The researchers often took a passive approach to outputs. They saw the main project outcomes as a long report summarizing the question, systematic review process, and the frameworks. NBS appointed staff members for creating practitioner-friendly outputs such as executive reports, checklists, and case studies. These materials were created in collaboration with researchers and managers involved in the systematic review projects. However, some researchers focused on the time it would take to have these discussions with NBS, accounting for it as the time over and above what they expected to spend on the project. Others were dissatisfied with the simplification of their research findings. Bartunek and Rynes (2014) identify this tension as one of the interests and incentives. Researchers are not incentivized to produce outputs that are not academic articles. One researcher from our study explained, Sometimes editing felt like overkill…. They [NBS staff members] did their best but at times it felt like well, your question is coming from an uninformed place and these texts need to be said this way because this is what the research is all about. No, it doesn’t make sense to say it the way you are proposing because that violates the [research] construct.
Practices to navigate this tension
We saw a few ways to navigate this tension. NBS staff members played an active role in translating the systematic review findings based on the researchers’ long report into a format more useful to the managers. In essence, NBS produced two kinds of reports: One was the traditional systematic review report; and the other was an executive report, a short summary with the research findings translated into language accessible to managers, in addition to other outputs for practice explained above.
In a few projects, we found that the researchers leveraged the managers’ interest in prescriptive outputs. For example, some researchers used the report’s table of contents (instead of the detailed text) to discuss with managers the main ideas and the flow of the larger narrative. Some researchers asked managers to write the recommendations with them and share organizational examples to illustrate the concepts in the report. Other researchers used the outputs as discussion tropes throughout the systematic review process, often to move beyond their disagreements. For example, in response to a manager’s suggestion, instead of saying “this is what the research says,” the researchers often reframed the discussion to “how would your suggestion change the final output?” Creating outputs in this way was not a onetime, post hoc event. Instead, it was an idea which guided the entire systematic review process.
Step 3.2: Getting the Evidence Into Practice
In most of the existing guidelines for systematic review, this step is where managers (and other stakeholders) are heeded again.
Some researchers expressed that there was unmet potential for the systematic review reports to create impact on practice. As one researcher mentioned, “I remember delivering a webinar but that was it. We left the project there. It’s hard for me to think [about] how we could do better with the practitioner side of this because we’re not practitioners.”
Within NBS, getting the evidence into practice was a constant ambition, but often elusive. NBS used various media to deliver outputs, such as webinars, social media, and podcasts to get the evidence into practice. However, NBS staff members described most success in using more intensive approaches. For example, in a few projects, NBS called upon the managers involved in the systematic review process to “own the knowledge” (as described by an NBS project manager) and act as change agents for disseminating the review findings. Many managers involved in NBS projects presented the findings on the researcher’s behalf within their organization and in external forums such as industry association meetings, increasing the chances that their peers would find the evidence meaningful. For example, a manager described her experience of presenting at the World Bank the findings of NBS systematic review on decision making for sustainability: I was invited to speak at the World Bank on this topic and I used it [systematic review] as a core part of my talk. I talked about the body of research that was used and here’s three models that I kind of just touched on the three models, but I said the one that we [manager’s company] used. I was able then to draw my experience from that and say here’s what we do with it. Here’s our lessons learned in applying it.
Impact of NBS Systematic Reviews
Impact as Outcomes
We found evidence that NBS systematic reviews impacted both research and practice. In terms of impact on research, NBS projects were often published in books and academic journals, such as articles in the annual reviews in the Journal of Management (Reilly et al., 2016; Stephan et al., 2016) and the International Journal of Management Reviews (Adams et al., 2016). NBS also garnered awards related to impact on practice, such as Canada’s Clean50 Award. Over the two-year period from August 26, 2017, to August 25, 2019, NBS’s systematic review reports had been viewed on its website 15,840 times. A shorter version of the systematic review reports, called as “Executive Reports,” had been viewed 35,288 times over the same period (see Table 1).
In terms of impact on practice, NBS systematic reviews resulted in actionable frameworks that managers could use in practice. For example, the systematic review on embedding sustainability in organizational culture yielded a framework juxtaposing abstract dimensions built on prior evidence from the literature with concrete practices that managers in the systematic review project teams brought to the table.
In our interviews, we heard several examples of the systematic review reports being used in management practice. A manager explained that frameworks in systematic review reports were his “go-to answers” for when a peer asked a question about say “sustainability-oriented innovation.” Another manager who was part of the systematic review team on “Decision Making for Sustainability” described that she presented the framework from the systematic review report to her peers in industry conferences. Yet another manager explained how he worked with a consultancy to design a social change project driven by his organization and used the framework from the systematic review report on “Business-Driven Social Change” as the scaffold for the project design.
The impact of NBS’s systematic reviews was also evidenced by managers’ decision to remain involved in NBS’s Leadership Council. As we explained above, the Leadership Council shaped the topics to be systematically reviewed each year and guided the systematic review process. Over the 10 years in which the systematic reviews were conducted, 26 different organizations were represented on NBS’s Leadership Council. Organizations committed to a three-year term, which they could renew. Only 4 organizations did not renew their membership over those 10 years, indicating they saw value in the systematic review outputs and the network.
In addition, in terms of long-term impact, two NBS projects led to further collaboration in the form of working groups between researchers and managers for addressing sustainability issues, as described above. These working groups produced outcomes such as practitioner reports and checklists such as for monitoring a company’s chemical use in manufacturing. 4
Impact as Changes to Self
NBS’s impact was also evident in the changes in the people involved in the systematic review process. Managers believed that, in addition to gaining knowledge on the topic, they drew personal satisfaction and personal growth out of these projects. The projects challenged the managers’ assumptions and provided them with newer ways of looking at things. A manager described the projects as an opportunity for learning about new ideas on topics that she thought she already understood. A researcher described such a moment as, There’s this light bulb moment where the people in the room go to a different place because they could leave the alternative ideas behind with all the baggage that it had…and they could see what they were doing in a different light.
Impact as Understanding the Other
Both researchers and managers explained that through the systematic review process, they developed a deep insight on the differences between researchers and managers. A manager framed the project as a rich experience because “I got to hear multiple opinions and throughout the project see how those opinions developed.” One manager explained being intrigued by how the researcher develops a question by drawing on what has been published, and the systematic process through which the researcher answers the question.
Similarly, the researchers described learning more about the managers in the course of these projects. One researcher expressed her understanding of managers by dividing them into various categories. For her, some managers were “closeted academics” who extended their hand out to meet her “across the research-practice divide.” They were willing to stay uncomfortable as she worked her way through explaining abstract concepts. In contrast, others were strongly embedded in their own community. Many other researchers similarly described a new and nuanced understanding of the managers.
Discussion
In this article, we argued that systematic reviews can impact practice more significantly than they have previously done if researchers include managers (and other stakeholders) as partners in the systematic review process. Systematic reviews have often been criticized for not being as impactful as they could be, in part because the content of the reviews are often not relevant to practitioner interests and also because the evidence often lags behind current management practice (Adams et al., 2017). By involving managers in the systematic review process, managers can bring their experience into the process in real time to ensure both relevant and timely insights. However, the knowledge systems of researchers and managers are so different that tensions often arise in the collaborative process (Bartunek & Rynes, 2014; Kondrat, 1992; Van de Ven, 2007).
Prior literature on systematic reviews offers protocols for conducting reviews, such as how the review question should be developed and what articles are relevant for synthesis (Campbell Collaboration, 2019; Petticrew & Roberts, 2006; Rousseau et al., 2008; Tranfield et al., 2003). However, this prior literature offers little guidance on how researchers can navigate the tensions when managers are included in the process. In this article, we sought to understand how researchers and managers can navigate the tensions related to what they know and how they know, to create more impactful systematic reviews.
We sought to inform this question by studying the work of the Network for Business Sustainability (NBS), a Canadian nonprofit that brought researchers and managers together to conduct systematic reviews that impact both research and management practice. We interviewed researchers and managers involved in NBS systematic review projects, observed a systematic review project, and gathered documentary evidence. In the remainder of this section, we describe some practices that can help alleviate tensions between researchers and managers to generate impactful systematic reviews.
Practices to Navigate Tensions
This article builds on and takes forward the insights we found in an earlier paper on cocreating rigorous and relevant knowledge (Sharma & Bansal, 2020) by providing concrete practices that researchers can adopt to involve managers in conducting systematic reviews. Researchers and managers experienced tensions at each step of the systematic review process. Reinforcing prior work on research-practice tensions (Bansal et al., 2012; Bartunek & Rynes, 2014; Van de Ven, 2007), we found that researchers and managers approached the systematic review process differently from each other. For example, managers did not hesitate to bring their own self and their organizational experience in interpreting data (i.e., results from studies included in the systematic review), while researchers sought to separate their own “biases” from the research endeavor. Researchers argued for maintaining distance and sticking to scientific principles. Hence, researchers resisted the recommendations advanced by the managers, such as revising keywords, adjusting frameworks, or including gray literature that was not published in peer-reviewed journals. Along the same lines, managers sought concrete guidance at the end of the systematic review project, such as success factors for a stakeholder collaboration, yet researchers resisted giving such prescriptive advice.
Existing protocols of systematic reviews do not describe the process of including managers in the review process, which is a shortcoming because doing so can generate significant tensions yet yield more positive outcomes and impacts. Often the interests of the managers that use systematic reviews and the researchers that produce them can be misaligned. Rousseau et al. (2008) suggest that researchers must ask stakeholders, including practitioners who may use the findings, to assess the systematic review’s question from their idiosyncratic perspectives. Yet, as we found, managers’ perspectives may not necessarily align with researchers’ perspectives. Simply suggesting that a researcher must consult different stakeholders is not enough, as there is a high likelihood that there will be no resolution. Without clear guidelines, the tensions may be insurmountable, and researchers may miss the valuable insights that can generate impactful systematic reviews.
Our study describes practices that can help researchers navigate these tensions (see Table 2). The practices that we uncovered and describe in Table 2 complement existing protocols for conducting systematic reviews. Researchers will be able to use this practical guidance in creating reviews that impact both management practice and academic research. These practices will allow researchers to connect the historical evidence of systematic reviews with forward-looking prescriptions. And by involving managers in the systematic review process, systematic reviews will be relevant to users while synthesizing the complete body of evidence.
The practices we describe respond to the call from Bartunek and Rynes (2014) to recognize the tensions between the communities of research and practice and offer suggestions for leveraging the interdependencies. Nielsen (2018, p. 61) points to the challenges of getting stuck in these tensions: It is also possible to get so involved with one or the other communities that one loses one’s way. That is, one can get so busy with the practice side that one neglects to do the theory-building side and the reverse.
Even though this article is focused on systematic reviews, the practices we identify are generalizable to other research settings where researchers and managers have the opportunity to generate knowledge together. For example, the practice of researchers and managers building a framework together is replicable in the data analysis stage of any research project.
At the same time, there are a few contextual factors that are specific to systematic reviews that must be considered in generalizing these insights. First, NBS projects were focused on sustainability issues such as stakeholder collaboration and climate change. It is arguably easier for managers and researchers to have the same outcomes in mind when it comes to sustainability issues than issues with divergent interests. We recognize that not every project is suited for collaborations among managers and researchers. Much depends on the type of research and the preferences of the people involved in the process. The researchers that NBS invited were willing to work with managers, and managers who joined NBS projects were committed to generating knowledge that was freely available to everyone. Managers working with NBS were not looking to solve an immediate organizational problem; rather they were interested in generating knowledge that can address ongoing sustainability issues. Other research methods, such as action research (e.g., Jay, 2013; Lüscher & Lewis, 2008), may be more relevant for finding answers to an immediate organizational problem.
As well, NBS projects were designed to place the responsibility of executing the various steps of the systematic review process with researchers. The manager’s role was to guide the researchers and provide feedback. As a result, we identify practices for researchers to navigate tensions, but a good way forward would be to share this responsibility equally. Each step of the systematic review could be even more cocreative; for example, creating a corpus of studies and coding the studies are executed by both researchers and managers. Future systematic reviews could explore such approaches.
Impact of Systematic Reviews
Evidence of the impact of management research on practice is exhibited in a number of ways: its influence on organizational practice and on the manager’s understanding of herself or himself or of others and in changes to systems, such as policies and collective action (Adams, 2005; Aguinis et al., 2014; MacIntosh et al., 2017). We found a few ways in which NBS systematic reviews were seen as impactful, beyond academic publications. For example, the metrics of web visitors and page views showed the frequency with which managers and researchers, beyond the project team, accessed the systematic review reports. As well, in the interviews, researchers and managers mentioned that the systematic review projects helped them learn new skills and gain new insights, including a better understanding of the other party. Managers in the projects presented the review findings at industry conferences, so they impacted not only the managers who read the review but also other peers. And two NBS projects resulted in long-term working groups where researchers collaborated with managers to generate further knowledge and effect change in organizations and industries.
Despite these indicators, measuring the impact of research on practice remains elusive, as seen in the variability in our data on how managers used the systematic review reports, in contrast to more concrete indicators of impact on research such as the number of citations. We found it difficult to establish direct causal links of the systematic reviews on individual, organizational, and systems change. 5 However, instead of lamenting the lack of standard, reliable, agreed-upon measures, our study concurs with the view of others that it is important to assess impact broadly (Aguinis et al., 2014), including seeing old topics in new ways because of the dialogue between research and practice communities.
Our study’s insights also have implications for translating research for impact on practice. Research translations describe research insights in a language and through media that is accessible to practitioners (Shapiro et al., 2007). We show that translation does not necessarily happen after the research is done, or in formulating the research question. We found translation was evident throughout the research-practice collaboration. Others have alluded to a similar phenomenon. For example, Paula Jarzabkowski (Jarzabkowski et al., 2017, p. 136) offers an anecdote from her work with managers from the insurance industry: “The problem was that I was used to differentiating between my academic findings and my engagement with practitioners; only translating findings into something that practitioners valued or understood after the fact. Now integration was necessary, as we worked together in developing the research questions and conducting the work.” Similarly, in our study, we show that impact is not only in the knowledge produced at the end of the systematic review process, but along the way. We found that tense moments can be useful and positive. They serve to change each party’s understanding of themselves and the other, move the collaboration forward, and ultimately result in greater knowledge for both researchers and managers.
Concluding Remarks
Scientific evidence now shows that temperatures have risen to over 1.5°C above the postindustrial average (Intergovernmental Panel on Climate Change, 2019), insect populations are decreasing rapidly including pollinators such as bees (Hoff, 2018), and the gap between the very rich and the middle class is growing (Deprez, 2018). Prudence would demand organizations to act, yet management research in this area has failed to give clear and strong guidance to managers. Despite the desire of many researchers to impact practice, few feel they have (Haley et al., 2017; Rynes et al., 2018).
In a recent editorial, Simsek et al. (2018) argue that researchers can see a phenomenon in new ways when they take seriously the assertion that practical and academic knowledge should be integrated. By generating knowledge with managers and other stakeholders, researchers can assess the efficacy of their ideas in real time. Like canaries in coal mines, managers offer contemporary, real-time insights of changes on the ground. The diversity of perspectives can generate a synthesis of prior research that can help to separate meaningful patterns from noise.
Systematic reviews hold the potential to bring research into use and advance knowledge by involving managers at each step of the review process. In this article, we show that by navigating the tensions between the two communities, researchers conducting systematic reviews can more significantly impact both academia and practice.
Supplemental Material
Footnotes
Acknowledgments
This article benefited tremendously from the comments of our handling editor, Jean Bartunek, and three anonymous reviewers. Jean raised the quality of our article through her amazing eye for detail, wisdom in structuring ideas, and deep knowledge of the literature. We also appreciate the support of NBS staff (Maya Fischhoff, Chelsea Hicks-Webster, and Ghaid Saadaldin) who not only supported our research endeavor but have demonstrated unwavering commitment to bridging research and practice to advance business sustainability.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Notes
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
