Mutual Distrust: Perspectives From Researchers and Policy Makers on the Research to Policy Gap in 2013 and Recommendations for the Future

New Contribution

This qualitative study provided an in-depth assessment of the commonalities and differences between applied health policy researchers and policy makers in 2013, focusing particularly on their perspectives toward each other. The results illuminate sources of potential friction between researchers and policy makers and identify recommendations for improved processes of research translation that incorporate broader perspectives on how scientific evidence can be used across the policy-making process, not just at the point of policy decisions.

Study Participants

In the summer of 2013, we collected data from researchers and policy makers at the AcademyHealth’s Annual Research Meeting (ARM) and the National Conference of State Legislatures annual meeting, respectively. AcademyHealth is the largest gathering of US health policy and health services researchers (more details on recruitment and methodology are available elsewhere ²⁰ ). The AcademyHealth sample included 215 health services and health policy researchers recruited from a random sample of 325 preregistrants for the 2013 AcademyHealth’s Annual Research Meeting in Baltimore, Maryland, in June (66% response). We limited our sample to researchers whose primary affiliation was an academic institution. We sent each of the 325 registrants an invitation to be interviewed at the meeting along with a $2 incentive. We sent a follow-up e-mail to schedule interviews with responders, and we contacted nonresponders 2 additional times by e-mail. Interviews were conducted by trained research assistants in a designated meeting area. Twenty-eight participants were unable to complete the interview during the meeting and were interviewed by telephone.

The second sample included legislators and legislative staff attending the National Conference of State Legislatures’ annual Legislative Summit in August 2013 in Atlanta, GA. The Summit is the largest national gathering of state-level legislators. Our sample was limited to legislators and legislative staff from the United States who focused on health and/or human services issues. Inclusion in the study was based on confirmation of role as legislator or legislative staff by interview, and a total of 43 policy makers participated. Because these were intercept interviews solicited as they attended conference sessions (ie, no formal sampling frame), we report no response rate.

Consent was obtained prior to all interviews. Participants were entered in raffles for a $200 retail gift card. The study was approved by the University of Pennsylvania Institutional Review Board.

Interview Instruments

The 2 instruments were designed to identify researchers’ and policy makers’ professional activities related to the dissemination of and use of research in the policy-making process. For the researcher sample, researchers first answered a structured (quantitative) survey instrument (with results previously reported ²⁰ ). Then they were asked several open-ended interview questions concerning activities they engaged in related to the dissemination of their own research. The interview instruments for policy makers were solely qualitative. Policy makers were asked to describe specific activities they engaged in related to finding and using research evidence in the policy-making process, with probes related to their use of social media. The open-ended question that is the focus of this analysis for the researcher sample was, “What is the biggest challenge as a researcher to influencing policy?” The open-ended question that is the focus of the analysis for the policy maker sample was, “What do you see as the biggest challenge to using research evidence in health policy?”

Analysis

All interviews were transcribed and de-identified. First, all members of the author team discussed a small set (N = 5) of transcripts from both groups. Through this inductive process, we identified the main emergent themes, mentions of challenges and barriers to translation, best practices for finding and/or communicating research (how it is accessed, how it is communicated, and by whom), characteristics of the policy-making process, and differences between researchers and policy makers. While many themes were identified in both researchers’ and policy makers’ transcripts, others were unique to one group (such as “credibility of science,” which emerged only in the policy maker sample). These themes comprised the initial codes in our codebook; additional themes were identified and added to the coding scheme as needed in a pilot coding process led by 3 of the authors. Before coding all transcripts, a subset of transcripts were double-coded to ensure interrater reliability. Disagreement was reconciled and agreement exceeded 90%. Next, 2 authors coded all transcripts using the final codebook in NVivo 10.0. After all codes were entered, the study team created analysis memos for all codes, describing the major themes under each code, key outliers, and a comprehensive list of illustrative quotes. Through a comparative method, the team analyzed all the researcher and policy-maker memos for consistency and disagreement. We also examined differences in themes for researchers by their academic rank (assistant, associate, full professors) and for policy makers by whether they were elected legislators or staff. Differences in these characteristics did not stand out as important and so we report the findings at the level of the group as a whole. For the purposes of the present analysis which seeks to illuminate the role of relationships in research translation, we focused our results on the barriers and facilitators of research translation and attitudes relevant to relationships between researchers and policy makers.

Consistent Barriers

Researchers and policy makers were consistent in their acknowledgment of the main barriers that prevent research from being communicated to, and used by, policy makers. Specifically, researchers most commonly identified competing demands on their time that limited their ability to communicate science to policy makers. Many researchers noted that the current expectations of the tenure and promotion process are not compatible with taking the time to engage with policy makers. As one respondent said,

if you want to do research there’s time constraints, so if you’re working on dissemination then you’re not doing research, and if you want to do research, which is why you’re in the business probably, it’s a time cost, and there’s no promotional incentive within the university promotions structure to do that.

For their part, policy makers also most frequently identified being very busy as a reason they do not access research information more routinely, saying, for instance, “As citizen legislators, a time constraint is our biggest problem.” Many felt overwhelmed by the amount of information that they were expected to process, receiving reports and other information daily. Like researchers, they faced hard constraints in their ability to absorb research into the policy work that they do. Several policy makers, in addition to commenting on the deluge of information they receive, also mentioned difficulty finding the “right” information for their needs.

Another consistent barrier cited by both groups was that researchers do not understand the policy process sufficiently. Researchers acknowledged that the linear model they may have been taught whereby research gets created in academia (detached from the policy context) and then the results subsequently communicated to the policy setting to be used at the point of decision making is wrong and potentially misleading. A few researchers explicitly stated that some of their peers erroneously presume that policy makers are “rational decision makers working in a very logical stepwise fashion.” This misunderstanding of the policy process contributes to researchers’ feelings of frustration about or alienation by the policy process, and also leads them to misjudge their likelihood of influence. As one stated, “A lot of researchers, especially early career researchers, think that all they have to do is tell the truth and powers will listen and that just doesn’t happen.”

Policy makers also expressed frustration with researchers’ seemingly naive perceptions of the policy process, which, they argued, leads to ineffective communication approaches. As one policy maker indicated,

I’ve seen a lot of academics come in prepared to teach. No, no. This is not a teachable moment, okay? You’ve got to explain, got to develop credibility, but much more quickly than that. And then you’ve got to hit the ground with things that legislators can really figure out and understand.

Policy makers also mentioned that researchers commonly use strategies that belie their lack of understanding of policy-making processes, such as not using constituents as messengers of research and sending too lengthy reports.

Attitudes About the Value and Credibility of Research in Policy Making

More than half of policy makers stated explicitly that they believed research has value in their policy work. They think research has value because it “can cross party lines” and that they have a responsibility to approach tough problems using the most effective mechanisms. In contrast, a single outlier (an elected official) explicitly questioned whether research is valuable, commenting that policy makers may just want to please academics by saying that research has value. He reacted against the notion that research should have an elevated influence in policy making over other factors, commenting that this was antidemocratic: “Who elected those institutions of higher education to do the research? Who elected them to tell us what to do?”

Policy makers considered research evidence to be a component of politics, not removed from it. They did not perceive research evidence as neutral, but considered it potentially politicized just as any information in the political process might be. As one stated, “Most of the information that comes to legislators is not pure science data. It’s typically biased, and so you have to take the time to figure out who is publishing the article and what their agenda might be.” Another indicated that data were used for political persuasion: “Both sides have statistics.”

Most policy makers expressed concern about potential bias among researchers, sponsors, and funders; 25 elected officials and 9 staffers raised the point that science is not always credible. This concern mostly stemmed from unease about funding and interest groups; for instance, citing an experience with research about indoor tanning, a policy maker stated, “You have to be a critical reader and think about who’s really driving this. Are you gonna trust that the people who are just studying this etiology of this increase in melanoma don’t have anything against tanning booths?” Others mentioned that research is necessarily a mix of higher and lower quality work, with demands on researchers to be prolific. In assessing research credibility, policy makers mentioned details as specific as study sample size and generalizability issues as well as more general concerns about researchers’ interpretation and advocacy for a particular perspective. As one policy maker stated, “The numbers can say what you want them to say.” Concern over the credibility of science contributed to hesitancy about the misuse of research in decision making, as a policy maker voiced:

I see the biggest challenge is [policy makers] not knowing how to read and judge research. They don’t know: if someone comes up with something that’s very politicized, but someone can say anything, right, it’s a free country. But [policy makers] don’t ask, is it in a peer reviewed journal? Has the result been replicated? Can you tell me where to find that?

Researchers perceived that policy makers may not seriously care about research, but unlike the policy makers—who frequently discussed the politicization of science and evidence—researchers infrequently identified concerns about bias in research. In contrast, they often characterized the nonresearch-related factors that go into policy as “biased.” As one described,

I think there are a lot of biases, that people approach policy making from the perspective of what seems most accessible, what seems most important to you, based on what your constituents are telling you, based on what’s in the media, and it can be very biased.

Researchers were cynical about policy makers’ authenticity in their consideration of research in their decisions, relative to political considerations like lobbyists’ and constituents’ influence; several noted this concern has only gotten worse in recent years with heightened politicization of health policy issues as well as more politically polarized legislatures. As one said,

I would say it’s, the political process is . . . it feels like a completely separate world and it’s very much out of our control. So the best we can do as researchers is do the best science possible and though we have very little control over how it will be used.

Some researchers acknowledged that this cynicism about the political process could drive researchers away from wanting to even engage in research translation, because “it might seem like a waste of time to get too closely involved.” Few researchers shared the dominant view among policy makers that research should be carefully critiqued. An outlier, one researcher made the point (as did the policy makers) that research can be “good, bad, and ugly” and that researchers sometimes push out research that aligns with a particular perspective; this researcher suggested that policy makers not “jump to every little thing that researchers find.”

Mutual Mistrust

Some policy makers’ and researchers’ cautious attitudes about research use in policy making went beyond a “healthy cynicism” of research influence in policy making (such as scrutinizing research funding or sources, and being aware of the nonevidentiary yet important influences in the political process). In fact, a strong and vocal mistrust emerged from some policy makers in their perceptions of academic research; a strength of mistrust that was matched in some researchers’ perceptions of policy makers.

Some policy makers believed that research is manipulated to be persuasive:

I would also say that it is a challenge where people are trying to persuade other people, you get into cases where some people are presenting information that is out of context or not properly supported by a well-designed research, and it’s more zealotry than factual conclusions.

Policy makers’ distrust of research was based in concerns regarding the academic setting. A small number of policy makers made statements that academic institutions care more about seeking publicity or funding than promoting the common good. One participant noted,

I think sometimes universities will, you know, affiliate themselves with—or allow the relationship to continue because there’s publicity involved for the institution. Although it may not—the underlying level of credibility of the work that’s actually being done may not be all that good.

Two mentioned a concern about “liberal bias” of universities.

Some policy makers negatively noted the influence of funding on research. As one policy maker said, “You will see articles published, and at the end, even though it doesn’t say it in the article . . . you’re left going, all they’re looking for is additional funding for whatever project they’re looking for.” One researcher also described this perspective, acknowledging that academic institutions and the researchers within them often face incentives to generate fundable research, not actionable research:

If you’re in an academic setting, I think the idea that for tenure, the hurdles that are put up for tenure, they may not be that interested in research that has to do with policy change, they may be more interested in research that brings in big dollars.

On the contrary, some researchers believed that policy makers were more strongly influenced by money than scientific evidence. One researcher said, “Well, you know unfortunately a lot of policy makers don’t base policy on truth and reason. They base it on ideology and the interests of the powerful people who pay for their campaigns.” A few others suggested that politicians are not open to hearing about a particular research finding because they have predisposing policy ideas they want to pursue, which leads to a censoring of evidence (as one said: “If you’re finding something not in line with what they want, they will not even put it out there.”) Expressing a high degree of mistrust, one researcher said: “Policy makers really don’t care about what the researcher says. They just care about who’s paying their bill.” A large group of researchers, however, acknowledged that policy makers may not use research as a result of the policy makers’ busy schedule and time constraints, rather than a willful nonuse of research.

Finally, one policy maker—albeit an outlier in how vocal he was in his distrust of researchers—believed that researchers are arrogant and “belittle” policy makers for not having sufficient knowledge: “You’re saying I can’t get in the weeds in the nuances—because [researchers are] so smart that the legislators just don’t understand?” A single researcher’s comments actually supported this belief, saying, “We’re on an intellectual plane that is so different from what plane the policy makers are on.”

Mismatch Between Needs of Policy Makers and Actions of Researchers

When solutions to facilitate more research translation were offered in the interviews, researchers and policy makers both described some best practices in communicating research and in finding or using research. Both groups talked about the importance of keeping research findings brief, actionable, and to transmit them using credible messengers. Policy makers frequently noted that the best way to find “good science” is to get it from credible people and organizations. As one participant said,

You know, no one’s just gonna read someone I’ve never heard of. Now if you get to be known enough, like if maybe some professional at a university where I live that I know are very knowledgeable . . . there’s a credibility there . . . I’ve seen the track of where they’re going and I agree with the thinking.

Or as another confirmed, “So if someone I know and respect in [home state] can explain or bring me a piece of research and put into the vernacular, it’s going to be better because I can ask questions about it.” Many relied on individuals they already knew well to provide information when needed. One elected official said, “Actually, my own physician wanted to—there was a piece of legislation that he was very interested in, and so he came to the legislature and testified.”

Policy makers noted that credibility can come not just from the individual researcher but also the institution or organization that can help communicate the research, including local universities, hospitals within their district, and intermediary organizations like the National Conference of State Legislators, Pew, and Council of State Governments. Many policy makers noted that they find the most trustworthy sources of research from their internal contacts—their own staff, committees, and legislative research branches. After pursuing these internal channels, legislators and staffers noted that local researchers are the best sources of research, especially via in-person meetings (noted by half of the 24 policy makers who mentioned a specific preferred communication channel). As one policy maker noted, “I think, at least in our legislature, reaching out and making it that individual contact is probably the best but most time-consuming.”

While not a majority, some researchers also recognized that forming relationships with policy makers could be effective and some were indeed interested in building relationships and using alternative forms of communication, including face-to-face meetings. Sometimes, though, researchers did not know the best way to connect with policy makers. Some noted that traditional scholarly channels were not effective, saying, “I don’t think those high impact factor journals are making the impact they used to make because people are going so many other places. And I don’t think the policy makers are necessarily going to those journals.” However, few felt prepared to disseminate effectively using alternative mechanisms. One spoke to a lack of dissemination training: “Writing policy briefs is something that we don’t get a lot of training in and we don’t have a lot of attention towards.” With regard to forging relationships on a personal level with policy makers, researchers noted 3 major challenges. The first was that they are not necessarily working in a single field, so the people with whom they would need to cultivate relationships would frequently change (ie, across a topic area like Medicaid or long-term care); they also reinforced the lack of time they have to cultivate relationships; and they noted that the “interesting questions” that policy makers might pose may not have funding attached to them, so researchers cannot add this to their workload when they have other pressing research grants and obligations. So while most researchers understood the need to modify their typical formats and channels of communication to meet policy makers where they are, the researchers on a whole did not demonstrate much interest in doing so given their professional constraints and, as mentioned at the outset, extreme time pressures.

Social Media as a Possible Bridge Between Worlds?

Policy makers and researchers alike viewed social media with ambivalence. Policy makers, supporting their perspective that face-to-face is best, indicated that social media would be unlikely to overcome the need for trusted relationships. As one said, “I would rather speak with a person one-on-one than through Twitter.” On the contrary, there were a few positive outliers in the policy maker sample who noted using Twitter as a method for identifying topics of interest related to research:

I read linked articles off of Twitter. So if something grabs my attention that this just published about X, Y, Z topic, I’ll go read it. I read off of that more than I do out of my newspapers or anything else.

Researchers, too (as noted in more detail elsewhere), ²⁰ were ambivalent that social media like Twitter could be an effective mode for communicating research results, and were concerned that it might not actually reach policy makers, that it could force researchers to “dumb down” their message, and that it could lead to reputational consequences if messages are not carefully constructed (ideally by others who take this communication function on as a designated responsibility).

Recommendations and Future Directions

The similarity between findings in this study and the level of mistrust demonstrated in research on this topic 20 or more years ago^15,16 suggests gaps in knowledge translation are unlikely to improve organically. Indeed, while “build relationships” has been the dominant recommendation in the literature, our study suggests that this conclusion may be simplistic and insufficient. Active investment and interventions are needed to infuse evidence into multiple points in the policy process across a wider range of actors. Below, we focus our recommendations on those in the research and intermediary communities.

First, others have recommended that researchers need training in dissemination and in policy making, ²⁴ and indeed our work supports this. But specifically, training opportunities must describe the spectrum of ways that research can influence policy making: from prioritizing the importance of various health problems, to identifying what issues can/should be framed as policy problems where government has a role to play, supporting a particular type of policy approach, and evaluating the effects of policies after they are implemented (including their effects on behavioral outcomes, cost outcomes, and on more abstract outcomes such as citizens’ attitudes and beliefs). In addition, researchers need more training in policy making so they can appreciate the nonlinearities of the policy process and understand what politics is, rather than something to be wary of. This is consistent with the US National Academies report, which urged for training among policy analysts and evaluators to incorporate social science findings on the conditions under which research is and is not used in policy making, rather than a more traditional education which privileges the value of scientific evidence over other decision-making modalities. ³ A deeper understanding of politics and policy making can illuminate more potential points of engagement and intersection, so researchers move beyond an inaccurate (and self-defeating) linear model that proposes research production on one end and decision making on the other, as implied by a problem framed as “two communities.” ²⁵ In fact, there are numerous other points of interaction and intermediaries, from media to advocates to interest groups, all of whom can establish relationships with researchers and with policy makers and interact at various points throughout the nonlinear policy process. ²³

Second, research institutions should take on more active roles in translation and reducing barriers for researchers. ²⁶ Universities and their dedicated policy-related centers can be engaged in more active outreach, relationship building with policy makers (and those trusted by policy makers, such as important advocates), learning who the appropriate committee staff are in state or local policy bodies, and other opportunities to forge personal relationships. ¹⁹ These investments are laborious, and as our findings make clear, individual researchers rarely can invest the type of time and energy needed. At one of our institutions, a health policy institute (Leonard Davis Institute of Health Economics) has developed a robust translation and dissemination program that relies on joint efforts of staff and faculty. Staff prepare briefs summarizing individual research publications and synthesize research related to policy areas—sometimes even engaging faculty outside of the institution. Twitter is used to build social media dissemination networks. A blog serves as a vehicle to deliver expert commentary. Conferences and forums focused on specific topics (eg, health insurance exchanges) help build personal relationships with policy makers. In addition, policy institutes can play another important role. In our prior research ²⁰ we found that researchers value having third parties promote their research to avoid perceptions of “self-promotion”— a concern that we found to be common among academic researchers. All of these efforts require substantial resources and the traditional model of grant-based funding for research is often not aligned with these efforts. To effectively serve the needs of the public, universities and funders will need to identify strategies to fund these efforts—possibly through more dedicated funding written into individual grants, philanthropy, or other mechanisms through institutional budgets.

Third, research institutions should evaluate whether tenure and promotion standards sufficiently prioritize and recognize public communication and policy impact. New measures of research visibility and impact such as Altmetric scores are being used more frequently by journals. However, we are not aware of many academic institutions using measures of public visibility when evaluating faculty for tenure. The American Sociological Association recently recommended that public impact be explicitly included as tenure criteria, to be measured by, for instance, soliciting letters from entities outside of academia to provide testimony as to the policy impact of scholars’ work. ²⁷ If evaluation standards change, faculty would have more direct incentive to engage policy makers and these activities would become normative in the research community.

Fourth, individual researchers who are especially committed to having impact (and who have institutional support, as described above) should adopt research approaches that connect policy makers, researchers, and intermediaries. Commentators have suggested various research approaches that apply knowledge transfer models directly into their design, such as models of exchange or “strategic science,” but with maintaining scientific rigor and integrity.^28,29 As an example, the new US-based Robert Wood Johnson Foundation program Interdisciplinary Research Leaders connects researchers and community leaders from the beginning with the goal of producing research that, by design, will have greater impact. Funders that care about their return on research investment should be encouraged to issue similar calls for proposals that prioritize action and dissemination and thus help to transform researchers’ traditional designs from investigator-initiated to research designs that have early and substantial policy maker and practitioner input.

Fifth, more research is needed to identify effective processes for promoting stronger linkages between researchers and policy makers and to evaluate both translation mechanisms and relationships. The knowledge exchange field should move beyond describing the problem (such as in the current study) to implementing and evaluating solutions. Such interventions, and their evaluations, should be developed in partnerships among researchers, intermediaries, and policy maker end users. As a simple intervention, researchers can experiment with different approaches to presenting research on a particular topic. ³⁰ More complicated, multiphase interventions based on behavioral change theory might be designed and tested with both researcher communities and policy maker communities (government agencies, elected officials, or staff) to determine how best to shift behaviors of all actors involved toward greater communication and incorporation of research. A notable example of this is the ongoing Supporting Policy In health with Research: an Intervention Trial (SPIRIT). ³¹ This study is a randomized trial involving 6 health policy agencies in Australia, and employs multiple interventions (audit feedback, training, supportive assistance, information exchange with researchers) with the goal of increasing agency capacity to use research evidence in their programs and ultimately more seeking out and use of scientific evidence. ³¹ Of course, other evaluation designs beyond randomized trials are also needed to assess the output of varying research-policy partnerships. Moreover, preliminary descriptive research such as developing and testing outcome measures like metrics of research use or research impact on policy making will also help push the knowledge transfer field forward.

Sixth, technologies such as social media have some appealing qualities: a low-resource-intensive, and potentially quick fix for communicating research evidence and connecting—albeit via virtual networks—with policy makers.^20,32 However, our study revealed ambivalence on both sides about the value (in 2013) of social media. State policy makers voiced their preference for face-to-face communication, and while some users of social networking sites may indeed interact in relationship-building ways, it remains an empirical question whether and how virtual social networks can substitute for “real” interactions in the policy process as well as whether researchers and their institutions are meaningfully connected with the appropriate end users of evidence. In addition, while tweeting and posting may seem to be efficient in terms of time (they are short communications, after all), crafting the right message at the right time and curating the research information takes strategic attention.

While this research has illuminated some concerns, our findings should not be interpreted to signify that change is required for all researchers or all policy makers; nor are researchers or policy makers the only appropriate targets for change. Some researchers conducting more basic health services research do not have interest in or capacity to translate their work themselves and may not need to change their approach, just as policy makers do not necessarily need to change their behaviors or relationships to make better policy. This study confirms the importance of other intermediaries or knowledge brokers as the focus for intervention, ³³ especially those who are already trusted by policy makers. Moving high-quality research syntheses into the hands of advocates, lobbyists, journalists, or other similar knowledge brokers might be a more scientifically valid and politically successful approach than trying to make the individual researcher the messenger of discrete study findings. As mentioned above, future research in knowledge transfer in health policy should consider more sophisticated models of the policy process and multiple “communities” of study (see, eg, critiques in Smith ²³ ).

Limitations

Our research had several limitations. First, policy maker study participants were recruited using a convenience sample of legislators and legislative staff attending the National Conference of State Legislatures’ annual Legislative Summit, and researcher participants were recruited from a random sample of university health services and health policy researchers attending the AcademyHealth’s Annual Research Meetings. Researchers and policy makers attending these meetings may not be representative of researchers and policy makers overall. Researchers attending the meeting have an applied focus which would lead them to have more, not less, interest in translation to policy settings. Similarly, policy makers in the study were all attending the Summit and thus have at least some interest in using evidence to inform policy making; as state-level policy makers, they are unlikely to reflect the views of policy makers at the local or federal level. In contrast, the researcher participants were drawn at random from a sampling frame, and so we are relatively confident that the researcher sample is representative of this population. That said, health services research is only one type of research relevant to health policy decision making. It will be important to assess the behaviors and attitudes of researchers in nonhealth fields that have health policy relevance (eg, education, transportation, housing) as well.

Second, we relied on self-report to measure attitudes as well as information translation and information-seeking behaviors. This method could be susceptible to recall or social desirability bias. However, both groups discussed their distrust and hesitancy openly; if their responses were socially biased, we would have anticipated them to present warmer attitudes toward researchers (for policy makers) and more enthusiasm about evidence translation and policy impact (for researchers). Other methodologies—such as examining institutional guidelines for tenure and promotion, or examining hearing testimony for documentation of research evidence—together could triangulate the barriers and facilitators that both groups face.