How academics & policymakers can collaborate effectively: Lessons from using behavioral science to improve U.S. federal government policies

Solution 1: Seek to Understand Each Other’s Goals & Identify Common Ground

Academics interested in pursuing policy-based research can look up strategic learning agendas, which are five-year plans for identifying and addressing important questions relevant to an agency’s mission and operations. ⁷ Federal agencies publish these documents to comply with the Foundations for Evidence-Based Policymaking Act of 2018. ⁸ They can be a good starting point for identifying a match between policy-relevant questions and an academic area of interest, and they can point academics toward organizations within federal agencies that might have leaders or staff who are interested in research collaborations.

For their part, policymakers can look for lines of inquiry in academic research that dovetail with their programs and policies. For example, the American Political Science Association’s Public Scholarship Program creates public-facing summaries of political science research that make academic work more accessible to those with limited time or who lack access to journals that are behind paywalls. Other resources for policymakers who want to learn about relevant behavioral science include translational outlets, such as this journal and its partner blog, the Behavioral Scientist, and other journals that take stock of behavioral science relevant to policy, such as the Journal of Behavioral Public Administration.

During early conversations, potential collaborators should be prepared to clearly communicate how the proposed project might meet shared goals and priorities and to be curious about the other’s needs. Academics should focus on learning as much as possible about the problem policymakers want to solve, in part by asking questions about its importance and its specific context, such as the constraints the policymakers face. After that, academics can begin developing ideas for solutions that align with and test current theories in their fields.

For their part, policymakers can explore what might make a collaboration appealing to academics, such as the possibility of publishing generalizable knowledge. Many academics may be willing to devote time to a project for free if they see an opportunity to test a certain idea, gain access to relevant data, and publish the findings. In line with this motivation, policymakers should expect to commit up front to transparency and sharing their findings, regardless of the result. (See the Sharing Results section later in this article for more on this topic.)

At times, these exploratory conversations may indicate that the policy needs of a given project do not mesh with the research goals of the academics. In such cases, academics should be frank about the conflict and attempt to connect their government partners with researchers who may be a better fit for the project.

Solution 2: Collaborate With a Wide Variety of Stakeholders to Fully Understand the Context of the Problem

To gain a deep understanding of the problem a policy is targeting, academics will want to talk to those who confront it on a daily basis. Grounding research in the experience of those affected by the problem can make studies and the programmatic decisions they inform more effective.

For example, the goal of one OES study was to develop an intervention to increase flu vaccination rates among veterans. ⁹ In the intervention, a health promotion staff member at a Veterans Affairs (VA) hospital provided feedback via emails to VA health care staff about their and their peers’ monthly vaccination performance metrics. This information was designed to motivate staff to boost their vaccination numbers, because many of them were not meeting goals set by the VA. Leaflets attached to the emails spelled out strategies for persuading the public to get vaccinated.

But the intervention produced no statistically significant differences in flu vaccination rates compared with a control condition in which no feedback or advice was given. In retrospect, providing performance metrics may have been insufficient to motivate health care staff who were working in an organizational culture that did not emphasize learning from performance metrics. Deeper knowledge of organizational practices and behavior through more extensive up-front engagement with people directly affected by the program could have helped the research team to shape the intervention in a more productive way.

By contrast, another OES study that involved a collaboration with several state governments produced promising results. In 2017, OES worked with the Department of Education (DOE) on a project designed to identify students who were eligible for a federal program that provided families experiencing housing insecurity with free meals, transportation, and a variety of additional educational supports. At the outset of the project, the research team, which included DOE staff, joined a call with state representatives in which the team explained the goals and rationale of the potential study. Later, the team met with representatives from states that were interested in participating in the study. During these conversations, the team learned that specific details about the homeless student program had been embedded in a long document containing lots of other information. This format led to information overload among the staff responsible for screening students for eligibility. These staff, called homeless student liaisons, therefore understandably overlooked the material relevant to them and their mission.

To fix this problem, the team created separate, shorter emails just about the homeless student program to send to the homeless student liaisons. ¹⁰ The emails contained action items related to identifying homeless students and were sent every two weeks to remind the liaisons of the goal. The refashioned communications led the liaisons to identify more than 3,000 additional homeless students eligible for support through federal programs, an average of 12% more than the control group did.

One approach to improving a team’s overall understanding of a process or system is to create a journey map ¹¹ or a behavioral map. The former outlines the steps of the process or system and helps partners understand people’s experiences when interacting with services. A behavioral map has a similar goal but emphasizes behavioral barriers and intervention opportunities—for instance, pain points that might prevent someone from applying for a benefit. Creating a journey map or behavioral map helps to clarify academic collaborators’ understanding of a process or system, highlights opportunities for improvement, and allows academic and policymaker collaborators to align on proposed interventions that address relevant barriers. Once all collaborators have jointly developed and agreed on a map, they can share it with stakeholders, such as program recipients, for feedback, which can help them refine the map and improve its accuracy. One example of such a map, seen in reference 12, outlines the steps for booking travel within the federal government and identifies questions and sources of confusion that discourage people from completing the process. It suggests that an intervention that reduces these psychological barriers could encourage people to book earlier and thereby reduce the cost of government travel.

Solution 1: Researchers Should Explain the Merits of Randomized Controlled Trials

Academics can play a key role in explaining that randomized evaluations are aimed at building evidence around what works, which is necessary for helping the most people over the long run. They should also point out that if the treatment’s effectiveness in a particular context is still being evaluated, the treatment may be just as likely to harm as to help recipients, or it may be no more effective—yet more expensive—than the status quo. So before a program is adopted and scaled up, it can be important to test its effects in the most rigorous way.

If limited resources constrain the number of people who can receive a treatment, randomization may also be the fairest way to determine who gets it. For example, an OES study with the Department of Defense tested whether email alerts from pharmacists to providers could reduce concurrent prescribing of opioids and benzodiazepines for patients in the military health system. ¹³ Pharmacists did not have time to send alerts to every provider whose patients were being prescribed both medications. Randomly selecting which patients’ providers received the emails each week was arguably the fairest way to administer the program, in addition to being the most robust way to determine whether the strategy worked.

Solution 2: Explore Approaches That Meet Partners Halfway

In some situations, research partners might consider a wait-list control design—in which control participants get the treatment after the trial is complete. In this way, everyone ultimately receives the potential benefits of the intervention, and the researchers are still able to conduct a randomized evaluation. Another option is a “holdout” design in which half of the participants are randomly assigned to receive the intervention first, with the other half receiving it at a later time.

OES used a wait-list control design in a collaboration with the Idaho Housing and Finance Association (IHFA) to boost application rates to their Homeowner Assistance Fund. ¹⁴ In this program, homeowners facing mortgage delinquency can receive up to $50,000 in aid. IHFA wanted to see if direct mailers could induce more homeowners to apply. However, they also wanted to ensure that all Idaho homeowners with properties in preforeclosure or foreclosure received mailers. So the team sent half of these homeowners a mailer early in the study, and the other half received a mailer one month later. This enabled the team to compare outcomes for the treatment and control groups during the month between the two mailings, while ensuring that all homeowners received mailers in a timely manner.

Solution 3: Pivot to Other Approaches

Randomization may not be feasible for other reasons. For instance, the design may violate contractual obligations or introduce complications that cannot be accommodated within a project’s time constraints. In these cases, academic collaborators can consider using nonrandomized, or quasi-experimental, designs. Quasi-experimental designs are useful when researchers or program administrators cannot randomly assign treatment and control groups. Instead, researchers assign people to these groups on the basis of some other aspect of the program’s structure or rollout. For example, the groups could be composed of people who fall just below or above an income threshold for a particular aid program or who live in adjacent counties or school districts. This approach has been taken in a number of OES studies (see reference 15 for an example of a recent OES project with a quasi-experimental design).

In some cases, the team might want to consider a descriptive study. These projects generally focus on uncovering key aspects of a certain process or element of a program and pointing out possible barriers or pain points. For example, in one OES descriptive study, researchers used Department of the Interior data to quantify the use of three methods of evaluating job candidates and show how each method affected the length of the hiring process and the type of person who was recruited or selected at each stage. ¹⁶

In another descriptive study, OES worked with the Small Business Administration (SBA) to examine how local emergency funds were distributed to businesses in the initial stages of the COVID-19 pandemic. ¹⁷ SBA was interested in the extent to which these supports were allocated equitably to historically underserved business owners, such as racial and ethnic minorities and women. By examining publicly available information about these programs and conducting interviews with local government officials charged with implementing them, the research team confirmed that there was a significant need for relief funding and also revealed that evidence was lacking concerning how to best address potential barriers to distributing aid equitably. This research brought important issues to light and gave SBA some specific questions to add to its future research agenda.

Descriptive studies like these generate helpful insights and then lay the groundwork for future impact evaluations by generating hypotheses and helping to build relationships between collaborators. By boosting trust, goodwill, and an understanding of the collaboration process among members of the project team, a descriptive study can also set the stage for fruitful future collaborations.

Solution 1: Communicate Regularly

We recommend scheduling regular check-ins with partners. Regular meetings and ad hoc calls can be helpful tools for addressing issues as they arise, particularly during stages that involve a lot of work or tricky barriers to navigate. For example, implementing random assignment to an intervention or control group can be difficult, especially for those attempting it for the first time. Academics can help by meeting with partners to clearly explain how randomization works in practice, conducting the randomization on the partner’s behalf where possible, and providing written instructions to help address potential difficulties.

Solution 2: Loop in Other Officials & Staff in the Policymakers’ Organization to Get Their Buy-In

Policy-related collaborations often involve numerous interested parties beyond those directly involved in the study. For example, a team that administers a program and a team that handles data from a study may be separate from teams making policy or funding decisions. It is important to communicate with all of these parties to ensure that they fully understand and support the study.

Policymakers are best positioned to know who the organizational stakeholders are and what type of communication will be most effective with them, particularly when it comes to navigating complex organizational dynamics. As such, policymakers can stress the importance of considering the needs of a range of interested parties at each stage of a study. For their part, academics can create tools such as slide decks, summary memos, or email drafts to help their partners share details about the collaboration with others in their agency. These conversations can build enthusiasm for ongoing research collaborations and generate new ideas for potential partnerships and learning opportunities.

It is also important to make sure that the agency employees charged with administering the program are given adequate support and information to carry out the experiment, which may have requirements that go beyond their standard operating procedures. For instance, if agency employees are to deliver a randomized treatment involving text messaging, they may by default record the responses to the texts in a binary way, such as by categorizing them as either “answered” or “declined.” However, for text messages that went unanswered, academics may want to distinguish between cases in which the recipient actively declined the text (say, by replying “STOP”), those in which a message was received but apparently ignored, and those in which a message was not received because of an error in delivery. Academics should be sure to communicate requests like these to their collaborators.

Solution 1: Discuss Data Handling Early & Formalize Data-Sharing Agreements

Early on, academics and policymakers should create data use agreements and memorandums of understanding that outline the types of data needed for the project, what format the data will be in, how the data will be transferred to academic partners and other agencies, and who will be responsible for data transfer and security. Drafting this agreement and sharing it among the project partners as early as possible can help collaborators understand the process and figure out how to ensure data access for members of the academic team. Academic partners should also find out who is responsible for handling the federal data relevant to their project, because this team may be separate from the primary collaborators. The academic partners should then meet with the data team to understand how they work and how to interpret and use the data they plan to share.

Solution 2: Discuss Ways to Encrypt or Analyze Data to Comply With Federal Rules While Enabling Scientific Discovery

More often than not, federal agencies are required to comply with rigorous data security standards, particularly when it comes to sharing information that identifies individuals. These restrictions can make data sharing between government and academic institutions challenging. For example, it is sometimes important to match a given individual’s data in one data set with their data in another data set, but it can be difficult to do that without relying on personally identifiable information, such as a name or social security number. Academics can show their government partners how to strip data of personal information and replace it with a common set of codes that allow individuals to be uniquely—but not personally—identified and tracked across data sets. Being open and careful about these processes can build goodwill while also protecting people’s data privacy. These conversations are important to have early in a collaboration to avoid any misunderstandings later on.

In some circumstances, data transfer from a federal agency to an academic partner may not be easy or even possible. To overcome this difficulty, academics can supply software that enables the government researchers to analyze the raw data themselves and provide the academics with the results of that analysis. For instance, perhaps what matters to the academics is the difference in an average between two groups, and that result can be shared even when the individual data points cannot be.

Solution 1: Commit to Transparency in Advance

It can be helpful to discuss at the beginning of the project how and when both academics and policymakers aim to use the eventual findings. Will any publication deadlines or scheduling constraints influence project timelines?

It is also helpful to agree early on to share all results in some format, even if results are null or unexpected. Null results can lead policymakers to change an ineffective program or help researchers see flaws in a study’s design.

For example, the OES project with IHFA that studied their Homeowner Assistance Fund found that direct mailers had no effect on application rates among at-risk homeowners (those with homes in preforeclosure or foreclosure). ¹⁴ Because of this null result, IHFA canceled plans to roll out the mailers more widely and reallocated the funding to other forms of outreach that targeted rural homeowners. This example illustrates one way in which null findings can inform program decision-making. In other cases, null results have motivated research to estimate the effects of a program or program change more precisely or to better understand how to boost a program’s efficacy. Because null results can inform policy decisions and future studies, OES commits to sharing all findings publicly, regardless of the results. ³

Solution 2: Communicate Results in Multiple Formats

For policymakers, generating brief, accessible documents to share with other interested parties can be useful. Policymakers can emphasize the value of creating and sharing plain-language descriptions of evidence that supports better program delivery. These documents can often be produced alongside academic articles aimed at specialized audiences with marginal additional effort. Scientists can also distribute preprint versions of the academic articles, enabling the release of results in advance of the final version’s publication in a peer-reviewed journal. OES project findings are disseminated in digestible and accessible ways, such as project abstracts, presentations to relevant agencies, and intervention packs, which lay out an intervention’s components and methods in an easy-to-use format so that others can reproduce it.

Academic collaborators should also discuss coauthorship on publications with policymakers. Coauthorship can be a way to formally recognize policymakers’ expertise and contributions to the scientific process and to support a culture of evidence building that is inclusive of valuable, rigorous work outside of university contexts. Coauthorship may, however, require policy partners to check with their agencies about whether any agency disclaimers or clearances are needed for publication. Presentations—such as panel discussions at conferences involving both academics and their policy partners—can also be an exciting way to share findings and encourage future collaborations.