Overcoming the Challenges of Collaboratively Adopting Artificial Intelligence in the Public Sector

Abstract

Despite the current popularity of artificial intelligence (AI) and a steady increase in publications over time, few studies have investigated AI in public contexts. As a result, assumptions about the drivers, challenges, and impacts of AI in government are far from conclusive. By using a case study that involves a large research university in England and two different county councils in a multiyear collaborative project around AI, we study the challenges that interorganizational collaborations face in adopting AI tools and implementing organizational routines to address them. Our findings reveal the most important challenges facing such collaborations: a resistance to sharing data due to privacy and security concerns, insufficient understanding of the required and available data, a lack of alignment between project interests and expectations around data sharing, and a lack of engagement across organizational hierarchy. Organizational routines capable of overcoming such challenges include working on-site, presenting the benefits of data sharing, reframing problems, designating joint appointments and boundary spanners, and connecting participants in the collaboration at all levels around project design and purpose.

Keywords

adoption of AI challenges of AI organizational routines interorganizational collaboration public sector

The European Commission (2019, p. 1) defines artificial intelligence (AI) as “systems that display intelligent behavior by analyzing their environment and taking actions—with some degree of autonomy—to achieve specific goals.” As a scientific discipline, AI includes a range of techniques, including machine learning (ML; e.g., deep learning and reinforcement learning), machine reasoning, and robotics (European Commission, 2019). However, the aspect of AI generally discussed in the public sphere is ML, an “algorithmic field that blends ideas from statistics, computer science and many other disciplines […] to design algorithms that process data, make predictions, and help make decisions” (Jordan, 2019).

For several decades, ML has been deployed successfully in both industry and academia. More recently, deep learning has made great progress in applications such as speech and language understanding, computer vision, and event and behavior prediction. These rapid technological advances hold the promise of automation and “human-intelligence augmentation” for organizations (Jordan, 2019).

AI technologies are currently experiencing a surge in diffusion and adoption by public organizations. Despite the current popularity of AI and a steady increase in publications over time, few studies have investigated AI in public contexts. As a result, assumptions about the drivers, challenges, and impacts of AI in government are far from conclusive (Gomes de Sousa et al., 2019).

In general, scholarly work on AI in the public sector has predominantly addressed public-sector use of AI in government functions related to general public services, economic affairs, and environmental protection in which AI techniques, such as ML, are used most frequently (Gomes de Sousa et al., 2019). Research is just beginning to identify the manifold challenges associated with public-sector adoption of AI. The few existing studies on this topic have highlighted the perils of AI, including its negative impact on the workforce, the need to balance privacy with data acquisition, the regulatory environment, and associated political and ethical implications (Wirtz et al., 2019). More research is needed to specifically address the challenges that organizations face when adopting AI, including those driven by resource scarcity, technical capacity and capability, and organizational path dependency (Mikhaylov et al., 2018). In addition, interorganizational collaboration can help organizations reap the benefits of AI adoption effectively and fully. Such collaborations enable partner organizations to combine their competencies and capacities, thereby strengthening public-service delivery or program and policy evaluation. Although many studies have investigated interorganizational collaboration, few have examined AI-based initiatives.

This article helps to bridge the gap by studying the challenges that interorganizational collaborations face in adopting AI tools and implementing organizational routines to address them. The research questions are as follows: (1) What challenges do interorganizational collaborations face in adopting AI? and (2) What organizational routines do managers use to overcome those challenges? For the purposes of this study, we build on the European Commission (2019) and Jordan (2019) and adopt an operational definition of AI in the public sector as a set of technologies, solutions, and processes designed to augment policy makers’ decision making by utilizing ML and big administrative data. We understand adoption as a process in two stages: development (conceiving, scoping, and building) and adoption (testing; Kim & Crowston, 2012). Finally, we define organizational routines as “patterns that participants use to guide, account for, and refer to specific performances of a routine” (Pentland & Feldman, 2005, p. 795) and that aim to produce “repetitive, recognizable patterns of interdependent actions, carried out by multiple actors” (Feldman & Pentland, 2003, p. 95).

The Challenges of AI in the Public Sector

Although AI in the public sector has been extremely underresearched, a small group of scholarly studies and practitioner-oriented reports suggest that interest in this topic is growing. These works have tended to present the benefits of AI applications for public services across many domains of the government, while recognizing that the potential of AI is undermined by technical, organizational, and policy challenges (Kankanhalli et al., 2019).

Dwivedi et al. (2019) and Sun and Medaglia (2019) have grouped these challenges into the following seven categories: (1) social challenges, (2) economic challenges, (3) technological challenges, (4) data challenges, (5) organizational and managerial challenges, (6) ethical challenges, and (7) political, legal, and policy challenges. Social challenges include issues related to societal norms and attitudes toward adopting AI in government. They include unrealistic expectations of AI technology, societal misunderstandings of AI capabilities, potential job losses, and increased inequality (Dwivedi et al., 2019; Korinek & Stiglitz, 2017; Risse, 2019; Sun & Medaglia, 2019). Economic challenges include obstacles to profitability and economic sustainability; they involve the costs and investments associated with the adoption of AI by public organizations. Wirtz et al. (2019) have argued that financial feasibility is one of the main challenges that organizations face when initiating AI programs. More specifically, two cost drivers often prevent organizations from implementing AI: the required investment in technology and the high demand for scarce AI experts, which increase education and salary costs.

Other challenges are technological, involving the nature and characteristics of AI technologies. In their study of AI in health care, Sun and Medaglia (2019) identify two technological challenges: (1) the lack of transparency of AI algorithms, which matters, given that algorithms are responsible for transforming data inputs into concrete decisions and (2) the difficulties of AI systems in processing unstructured data. Dwivedi et al. (2019) discuss the fact that AI tools cannot fully understand human situations or derive the right meaning from such situations. Kankanhalli et al. (2019) note that the AI tool infrastructure consists of several key elements that involve different technologies, making it a key issue how these technologies work together. The interoperability of AI systems is another important technological challenge: the lack of technical standards in the AI industry leads to hardware and software variations that create “an inconsistent technology ecosystem, which causes interoperability issues. Further, the system may not be interoperable with other government applications” (p. 305).

Since AI applications are heavily data-dependent, several studies have addressed data-related challenges. Sun and Medaglia (2019) discuss the insufficient size of the available database, the absence of data standards to control what data are collected, how they are collected, what format they are stored in, and the difficulties of data integration. Many studies have particularly examined the latter issue. As AI system data derive from multiple channels and different sources, sharing and integrating data among government agencies and departments and external stakeholders remains a perpetual challenge (Christodoulou et al., 2018). Technological challenges also impact the degree of data integration. As Desouza (2018) explains, the siloed nature of information technology (IT) systems in the public sector and the misalignment of agency-specific IT and data governance protocols make it difficult to integrate data on key thematic issues across systems. In sum, the lack of interoperability “limits how agencies can integrate multiple databases that ML algorithms can then analyze and use to provide richer insights” (p. 22). Data privacy and security present additional, interrelated data challenges since data-security problems can threaten data privacy (Kankanhalli et al., 2019). The use of AI in the public sector may entail the risk of massive surveillance and loss of privacy, while also encouraging malicious attacks to access and exploit data (Agarwal, 2018; Dwivedi et al., 2019; Krishnamurthy & Desouza, 2014; Mehr, 2017; Wirtz & Müller, 2019; Wirtz et al., 2019).

Sun and Medaglia (2019) define organizational and managerial challenges as challenges that relate to the strategy, human resources, and management practices of an organization attempting to adopt AI. Their research on AI adoption in public health care finds three important challenges. The first challenge, organizational resistance to sharing data, is particularly important because it highlights additional dilemmas including the question of who owns data and the tension between the need for data integration and the interest of individual organizations. It also shows the importance of trust among stakeholders (Dwivedi et al., 2019; Kankanhalli et al., 2019; Mikhaylov et al., 2018). The lack of in-house AI talent also represents major challenge, which is not easy to address, given that there are not enough specialists and experts in the job market with the skills needed to support and promote AI development (Wirtz et al., 2019). Finally, the threat of AI replacing an organization’s workforce can present a real challenge, even when fears are “nuanced by the fact that AI is framed as not capable of replacing specialized, skilled work” (p. 376). Additional challenges arise when an organization lacks clear leadership (Andrews, 2019), a strategy for adopting AI (Dwivedi et al., 2019; Sun & Medaglia, 2019), AI deployment guidelines that include criteria for standardizing data collection and sharing (Chen & Lee, 2017), and easy-to-use AI applications (Dwivedi et al., 2019). Krishnamurthy and Desouza (2014) note the importance of cultivating a culture of cross-agency collaboration, adopting crowd-centric approaches, and developing collaborative leadership and management support.

Given that AI gives rise to ethical considerations unlike those of traditional technologies (Bullock, 2019; Sun & Medaglia, 2019), studies have also investigated the impact of AI on moral principles. The existing literature discusses public distrust of AI-based decisions, the unethical use of shared data, the (in)compatibility of machine-versus-human value judgments, and the lack of transparency in decision-making processes, which may reflect bias and unfairness (Desouza, 2018; Dwivedi et al., 2019; Freeman et al., 2020; Gomez de Sousa et al., 2019; Janssen & Kuk, 2016; Kankanhalli et al., 2019; Mikhaylov et al., 2018; Sun & Medaglia, 2019; Veale et al., 2018; Wirtz et al., 2019). Privacy concerns about the public disclosure of sensitive information are also considered an ethical challenge (Krishnamurthy & Desouza, 2014).

Finally, AI adoption can be impeded by political, legal, and policy challenges. These can involve organizational responsibility for mistakes made using AI systems, the use of copyrights, and the lack of market-wide policy regulations (Dwivedi et al., 2019; Sun & Medaglia, 2019), among other issues. Despite some existing legislation, such as the General Data Protection Regulation (GDPR), which regulates personal data protection and privacy in the European Union, existing research shows that current legal frameworks must change significantly to effectively protect and incentivize AI adoption (Dwivedi et al., 2019; Wirtz et al., 2019). For this reason, policy makers are encouraged to establish AI policies that clearly address legal and ethical challenges, while also approving specific regulations on key issues, such as data privacy and security and technical standards (Alberti, 2018; Duan et al., 2019; Ishii, 2017; Kankanhalli et al., 2019).

Three important conclusions can be drawn from the literature review above. First, despite the rhetoric that AI adoption in the public sector faces major challenges, further research is needed to move from speculation to evidence gathering. Second, technological, data-related, and organizational challenges are not new in the literature. Digital government, knowledge transfer, and collaboration studies have acknowledged obstacles to collaboratively adopting technology in the public sector (e.g., Agranoff, 2006; Cristofoli et al., 2015; Gil-Garcia & Sayogo, 2016; Pardo et al., 2012; Raub & Von Wittich, 2004; Vangen, 2017; Vangen & Huxham, 2003; Willem & Buelens, 2007; Yang & Maxwell, 2011). Yet, both the scale and type of data required for AI projects pose unique challenges that must be explored (Desouza & Jacob, 2017). Third, most prior studies have emphasized challenges related to AI adoption in the public sector, while ignoring the need to implement organizational routines to address those challenges.

Research Design

To answer our research questions, we used a case study between a large research university in England and two different regional governments (county councils) involved in a multiyear collaborative project around AI. Case studies show how particular practices are developed in particular organizations and, therefore, help refine theory (Ospina et al., 2018). Although qualitative case studies are not representative and their results cannot be generalized, they allow us to study research questions in depth, while leaving room for unexpected, interesting findings that can form the basis for concrete hypotheses to be tested in future research (Marshall & Rossman, 2011; Yin, 2013). This approach is particularly useful when little existing research exists on a topic (Yin, 2013) as is the case here.

The case selection was guided by a combination of convenience and purposeful sampling, a technique often used to identify and select information-rich cases, while making the most effective use of limited resources (Patton, 2002). Convenience played an important role, as one of the authors of this article was appointed Chief Scientific Advisor of the collaboration, overseeing various stages of the project across both councils. Purposeful sampling involves identifying and selecting individuals or groups of individuals who are especially knowledgeable about or experienced with the phenomenon of interest (Creswell & Plano Clark, 2017). In this case, we could guarantee available participants with an interest in participating in our study, two important determinants of successful purposeful sampling (Bernard, 2017).

The Case Study

In 2015, a major research university in England partnered with two county councils to build, test, and deploy a suite of AI tools, aimed at improving council policy decisions to ensure better prevention and early intervention in education, social care, crime, and security. The collaboration, which was funded by the Higher Education Funding Council for England, came about for several reasons. Although both councils were already using standard data sources to inform their strategic decisions, they recognized the potential of innovative AI tools. Neither council had the organizational capacity to develop AI tools internally. By using the expertise and research that existed within the university, the councils were able to compensate for their own lack of AI specialist staff members and knowledge. Involving the university also strengthened their external legitimacy, increasing public acceptance of the project. Finally, the collaboration also strengthened the university’s relationship with local communities through its work to increase volunteering opportunities for students.

The partnership included an impact evaluation of public-service metrics and tools and risk stratification modeling tools based on ML, which resulted in several different projects, ranging from risk stratification (predictive modeling) platforms and evaluation frameworks for school readiness to domestic abuse, homeless youth, reoffending youth, and school sustainability. Since the legal framework (a combination of the GDPR and 2018 UK Data Protection Act [DPA]) held various departments responsible for specific functions needed to collect and curate the corresponding data (designated “data controllers” under GDPR and DPA), the development and testing of tools meant that data processors had to share data across collaborating organizations (under GDPR and DPA designation).

Sharing data between the controllers and processors to enable testing and implementation of AI solutions proved a complicated process, even under the GDPR and DPA legal frameworks. The councils also differed in their approaches. While Council 1 developed in-house tools and used the university in an advisory role, Council 2 codeveloped tools with the university, leading to intensive data sharing and the need for data sharing agreements.

Data and Methods

In-depth semistructured interviews were conducted to collect data and information during May and June 2018, once the AI tools had been built and tested by both councils and were ready to deploy. We interviewed 24 individuals from the three organizations (12 from the university, eight from Council 1, and four from Council 2), including top-level leadership, middle managers, bottom-level analysts, academics, and PhD students. Table 1 provides detailed profiles of the interviewees.

Table 1.

Interviewee Profiles.

Organization	Position	Role
University	Academic Directors of Program Evaluation	Strategic—Top management
	Deputy Director of Strategic Change	Strategic—Top management
	Academic Project Manager	Strategic—Top management
	Designer of Collaborative Project	Strategic—Top management
	Project Instigator	Strategic—Top level
	Knowledge Exchange Manager	Middle management (decision-making role for specific issues)
	Senior Data Development Manager	Middle management (decision-making role for specific issues)
	Initial Project Developer	Middle management (decision-making role for specific issues)
	Research Analyst	Operational—Bottom level (implementer)
	PhD Student Analyst	Operational—Bottom level (implementer)
	PhD Student for Program Evaluation	Operational—Bottom level (implementer)
	Volunteer Hub Coordinator	Operational—Bottom level (implementer)
Council 1	Director of Strategic Commission and Policy	Strategic—Top management
	Head of Corporate Strategy	Strategic—Top management
	Head of Strategy of the Public Sector Reform Unit	Strategic—Top management
	Head of Profession, Data and Analytics	Strategic—Top management
	Program Leader	Middle management (decision-making role for specific issues)
	Senior Project Officer	Middle management (decision-making role for specific issues)
	Intelligence Manager of the Adult Social Care Office	Operational—Bottom level (implementer)
	Data Science Fellow of the Senior Organizational Intelligence Office	Operational—Bottom level (implementer)
Council 2	Head of Knowledge and Intelligence	Strategic—Top management
	Head of Service of the Multiagency Safeguarding Hub	Middle management (decision-making role for specific issues)
	Assistant Director of the Children’s Commissioning	Middle management (decision-making role for specific issues)
	Main Project Link	Operational—Bottom level (implementer)

The interviews covered, but were not limited to, the following themes: the purpose of the project, various challenges, organizational routines implemented to overcome challenges, and the results and benefits of the collaborative process of adopting AI tools, which took place between 2015 and 2018. The interviews lasted 30–45 min and were recorded and transcribed verbatim. Given the coding strategy used (see below), a verbatim record of interviews facilitated data analysis by bringing us closer to the data (Halcomb & Davidson, 2006).

An inductive–deductive strategy was used to code the interview data (Charmaz, 2014). This approach drew on emerging AI literature, as well as on more consolidated research on digital government, collaboration, and knowledge transfer, to code data that matched existing concepts related to challenges and routines (e.g., goal clarity, leadership, training, resistance to sharing data) while remaining open to new codes emerging from the data. After the initial primary coding, a secondary coding placed related codes into larger categories. Finally, we identified themes that captured the analytical reflections from the categories.

Findings

Based on the interviews, our findings revealed the predominance of one key challenge: resistance to sharing data and transferring knowledge between organizations. Our results showed that the resistance to sharing data was simultaneously caused by (1) privacy and security concerns (reflecting institutional laws and regulations, the ways in which specific organizational cultures cope with privacy and security, and real threats to security and privacy, given the type of data AI is used for), (2) a lack of understanding of the available and required data, (3) a lack of interorganizational alignment between project interests and expectations around data sharing, and (4) a lack of engagement within the organizational hierarchy, leading to diverging expectations at the top and bottom levels of the organization. Our findings further suggested that, despite these challenges, organizational routines emerged to cope and iterate approaches that eventually caused the project to succeed. These organizational routines included working on-site, presenting the benefits of data sharing, reframing problems, designating joint appointments and boundary spanners, and connecting participants in the collaboration at all levels around the project design and purpose.

Challenge 1: Privacy and Security Concerns

Although data governance (privacy) and cybersecurity (security) concerns were present in both councils, they manifested differently, reflecting each council’s organizational culture and the way its legal unit interpreted the DPA/GDPR requirements. This, in turn, caused resistance to data sharing in both councils. However, this resistance was rooted in different views of data governance.

Council 1 had concerns about how personal information was used in projects and how data were released for ML algorithms in terms of definition and scope. These concerns caused some resistance to sharing data and a resulting lack of data availability during the first stages of the project. As “data controllers” within Council 1 were resistant to sharing large amounts of training data, they often released data sets that consisted of narrow lists of variables, with nonindividual-level identifiable aggregates, in order to comply with data governance privacy guidelines enforced by the data governance team. These privacy concerns were ultimately related to transparency, bias, and ethics. As one academic recalled, “they kept coming back and saying ‘we need to get approval for this from [the data governance team]', [which has] a duty to basically ensure data is shared safely with third-party organizations.” Unfortunately, “the council used that as an excuse not to share data.” As one Council 1 respondent explained, “first of all, people tend to be default nervous about sharing data. That may stem from just not having a full understanding of data protection and governance.” For instance,

at [Council 1] they are very wary of anything that would allow anybody not at their premises and not part of their institution to access any sensitive data, even it is anonymized; even if it is structured in ways which would mean it’s less sensitive.

Council 2 was concerned about how personal information was protected when shared with the university. The Council 2 leadership expressed concerns about cybersecurity in relation to current events, “even so this is a contentious issue; we don’t want something blowing up in the news like Cambridge Analytica and everyone suddenly saying, ‘oh my goodness what are we doing?!’” Several interviewees from units within Council 2 emphasized that much of the data needed for ML was “acutely sensitive, describing in detail some fairly upsetting situations for children we look after.” Ensuring that data left organizations in a secure manner was a crucial aspect of the data sharing process for this council.

Routine for Addressing Data Privacy and Security Concerns: On-Site Work Days and Presenting the Benefits of Data Sharing

At Council 2, data availability challenges related to cybersecurity were overcome by building trust at the individual level through the creation of a colocation routine, in which academics worked with participants “on-site” at Council 2 at least once a week to access organizational data. Describing this routine, one academic respondent said, “by having staff co-located there—because we are available at their offices, they can come to us and ask questions…. Our staff are there every week.” Another academic said, “on Wednesdays I work at [Council 2] and interact with everyone I’ve done projects with, which was helpful because I could go directly to them if I wanted any clarification.” From the perspective of Council 2, colocation was a beneficial routine:

The fact that the researchers come here to our building and sit with us for a whole day…. they are there to have some of those conversations and that cross-over of skills and ideas. They become part of the team. And that makes people feel engaged with it all.

Overall, trust built through face-to-face interactions at Council 2 enabled data sharing. In addition, work carried out on-site allowed academics to obtain “access to [Council 2] hardware; to laptops; so that we [could] access data, work on projects, and deliver something useful.” Once trust had developed over time, during on-site work days, the knowledge transfer began. Academics accessed what they needed to enable ML techniques, enhance organizational capacity, and train government staff. Interestingly, the routine of using “on-site” work days was not implemented in Council 1, where organizations took longer to resolve data resistance and availability issues.

While no single routine emerged to alleviate data governance concerns in Council 1, communicating the benefits that sharing data would bring to each organization seemed to limit organizational resistance. As one interviewee noted, “the more evidence you can provide, the happier people are. It is something we should be doing all the time.” In particular, “organizations are not prepared to give up their data unless they are really clear about what they are going to give up and why.” The same participant elaborated further on this idea: “information-sharing projects will always be set up in such a way that you may only use their kind of data for the specific reasons.” Eventually, project leaders realized that showing each organization-specific areas in which data sharing would increase the benefits of AI methods made the knowledge transfer process easier.

Challenge 2: Lack of Understanding of What Data Were Available or Needed

A lack of understanding of the data needed for AI projects also hindered data sharing. For data sharing to happen, organizations had to first understand the nature of the data required. As one interviewee shared:

The university needed to understand what was happening and then say “this is what we could offer to enhance this/to help you.” It was often up to us, [Council 1], to say what we wanted them to do. And I think that was the wrong way around.

A similar sentiment was expressed by an individual in Council 2: “we had started off thinking, ‘ah, so here’s some problems which would be well addressed by predictive modeling,’ rather than saying, ‘what problems is the organization facing? And how can we apply more advanced analytical thinking to that?’” At the start of the project development, leadership generally focused on ways of using AI to solve problems rather than thinking about the problems themselves or which AI solutions were most relevant. In other words, they overlooked the problem–action–analysis sequencing.

Routine to Address a Lack of Understanding of Data Needs: Reframe Problems

Both councils were initially confused about the nature of the problems driving data sharing. The collaborators overcame this challenge by reassessing the actions they hoped to take using AI solutions; these actions determined the analysis to be executed and thus the way data had to be shared. In one routine, discussion and brainstorming sessions were used to frame the organizational problems that AI needed to address, instead of starting with what method might be most interesting to use. As one interviewee said, by initially centering on the problems themselves, “we aren’t bound by solutions that then have to work.” One of the academics shared insights about returning to the sequencing needed for data sharing:

You’ve just got to start with: “what’s your problem?” and “how can I solve it?” and get to a common set of problems. If people feel like they are handing over the data and not getting problems solved they are going to be slower to do it.

Focusing on the sequence of defining organizational problems and then searching for appropriate AI solutions made it possible to achieve a better integration of data sets. According to one academic:

I think our discourse with [Council 2]—we have gotten them to explain the issues. And then we sort of have a process where we get them to define the scope of what it is they want to achieve. We can align our academic outputs from that because we make an assessment: “Well, is this something that just helps them to do the analysis work?”

By routinizing opportunities that allowed collaborators to constructively frame their organizational problems, the academics were better able to fill the skills and capacities gap for knowledge transfer, resulting in more willingness to share data.

Challenge 3: Alignment of Project Interests and Expectations on Data Sharing

In addition, several organizational factors made it difficult to form expectations or align project interests on sharing data between the university and public organizations. In the beginning, the academics had trouble coordinating project expectations because Council 1 was very large, making it very time-consuming to navigate the organization, communicate with stakeholders, and assign responsibility for collaborative projects. Second, both councils initially found it difficult to share information effectively with partners who had different interests. Council 2 was “smaller [and] easier to prove who has done what…to attribute responsibility, blame, praise, and credit.” By contrast, Council 1 was “monolithic” and “enormous.”

You’ve got a massive circle with 4,000 staff in it which is run by an operational director who is driven by a set of legislative and operational constraints which are huge. Whereby the fact that the CF program might be engaging with funky circles who think it’s wonderful and everything else…does not mean they really relate to each other very much at all.

Other interviewees commented: “you can’t really make organizational differences match; what happened was [that] over a period of time, we found academics that already had an interest with something that matched to a degree things that councils wanted to do”; and “you had to find a match between the right interests, and for the councils to have the right focus in the particular area” because it was “very much about matching rather than making them come together in any artificial way.” At the organizational level, the disconnect between project interests and expectations of data sharing efforts began to resolve when formal navigation approaches and ways of creating specific collaborative tasks were applied to Council 1.

Routines for Aligning Project Interests and Expectations on Data Sharing (Council 1): Designate Champions and Boundary Spanners

As soon as project leaders designated formal mechanisms to ensure role clarity and navigation, the academics found it easier to engage effectively with appropriate government staff. It then became easier to align interests, stabilize the data sharing process, and ensure knowledge transfer. As one respondent put it, “some of these risk-modeling projects we’ve been doing with collaborative data have needed really strong project management. They have been critical with setting up those discussions to sell the concepts—that we want to use the data in doing this type of thing.” One routine that emerged to help Council 1 with this challenge was the creation of a jointly appointed chief scientific advisor, a formal position requiring technical academic knowledge of AI as well as a grounding in public organizational realities. As one participant noted, the joint appointment created more openness between the university and councils: “the fact that the council has been open to [the joint appointment] role and they have been working together to develop this program speaks volumes.” A university respondent commented: “I think we should’ve gotten someone on site [at Council 1] sooner than we did with a tech background.” Another routine that helped project leaders better understand organizational needs and differences in a very large council was the creation of additional boundary spanners to coordinate information and tasks between partners.

I think once we got people in new roles with the clarity as to what their responsibilities and involvement in the partnership were, then it has been very productive. We have two people at [Council 1] now who are coming to regular meetings with the chief data scientist.

Challenge 4: Differences in Expectations Between the Top and Bottom of Each Organization

At the top level of leadership in Council 1, “there was the belief that [Council 1] had signed up to something as had the university; but fundamentally, the people who actually did the stuff with the data probably hadn’t signed up to it in the beginning.” Elaborating further on this question of cohesion and fragmentation, one participant said:

I think that the resistance may have come because maybe it was decided by someone at the senior management level who had agreed to something. But then when it comes down to the people who are actually doing the analysis in there and the analytical team…they probably haven’t had a say in it.

There were also issues about the sense of mutual dependency between the university and councils. “From our side, we were never really clear about what capability the university had.” Because there was no consensus on joint dependencies in Council 1, for example, initiating collaborative work toward AI projects (which required cooperation with bottom-level “data owners” and analysts) led to inertia. One analyst from Council 1 described the situation as follows: “I became involved in the projects mostly after the priorities had already been set;” another analyst said, “maybe the core experts of people doing the analysis aren’t in the ring, so maybe there is a discrepancy between what will be useful and possible.” Even in the beginning, the interdependency between the university and Council 2 was not always clear to all participants. At first,

it was more of what can we do with it? sort of thinking rather than going to senior management and senior service managers and asking them if we had this ability, what are your priorities for doing things with it etc.

The Council 2 leadership acknowledged the importance of inclusion in setting AI project priorities because “if you don’t have all those people there wanting it, it is very difficult to get anywhere at all.”

Routines for Achieving Cohesion: Connect With Participants in the Collaboration at all Levels for AI Project Design and Purpose

Breakthroughs occurred in both councils when project participants at all levels began to feel connected to the collaborative projects and understood better what the university could contribute. For both councils, routinizing ways to showcase potential projects and the skills that academics could contribute helped to build connections between members of the collaboration. The first example of an effective routine for showcasing work at Council 1 occurred when the leaders stepped back, creating dialogue and a context for engaging with the analysts. This enabled the leadership to see clearly what contributions the university could make:

Once they saw some of the work the university [did], it was helpful in shifting cooperation. Some of the modeling [academics] did for the data analytics was good, and actually, we thought, “That’s a model we can use!”

Project leaders had to focus on routinizing situations in which confidence in the analysts could be built incrementally: “doing test cases; showing what you can do—taking it one step at a time.” At Council 2, similar routines focused less on showcasing the skills of academics and more on showcasing potential AI projects to participants. It was helpful to create reports and allow members to see how AI could resolve the council’s problems. As a Council 2 manager recalled:

We gave the project team a data set that had some real value and allowed them to test out some theories too. [The academics] provided data analysis to the questions we’ve raised. The whole thing was very compelling in the end.

Another helpful approach involved showcasing examples that could “help the service deal with things that are relevant to them.” According to one Council 2 participant, it was particularly beneficial when project members met regularly to

underst[and] more of the process; question it; ha[ve] those open conversations that get you to a better end result—without saying why you’re doing it, what’s your starting point, what’s your end point, and working through the process…I could be asking completely the wrong thing.

Discussion

The first research question aimed to understand the challenges that influence interorganizational projects aimed at AI adoption. Our findings make two important contributions. First, resistance to sharing data, particularly during the stage when AI tools are developed and adopted, is the main barrier to adopting AI. This finding is consistent with the existing literature on organizational AI challenges (Christodoulou et al., 2018; Sun & Medaglia, 2019). Interestingly, other types of challenges appear to influence resistance to sharing data. On the one hand, both councils considered the ethical challenges involved in data sharing. Council 1 was particularly concerned about the use of shared data, while Council 2 focused more on ways to avoid disclosing sensitive information to the public. Although both councils considered data challenges, particularly privacy and security issues, they did so in different ways, based on different interpretations of the DPA/GDPR requirements. Finally, other organizational challenges, such as the need to reconcile data sharing with the interests and expectations of individual organizations and the lack of guidelines for those responsible for implementing the decisions of top managers, also caused resistance to sharing data. Although the literature refers to such challenges hindering the adoption of AI (e.g., Dwivedi et al., 2019; Janssen & Kuk, 2016; Krishnamurthy & Desouza, 2014; Mehr, 2017; Sun & Medaglia, 2019; Wirtz & Müller, 2019; Wirtz et al., 2019), our findings further indicate that the relationship between such challenges and AI adoption is not unilateral. Instead, various types of challenges combine and amplify each other (Gil-Garcia & Sayogo, 2016), creating additional challenges (in this case, the organizational challenge of resistance to sharing data) that influence adoption.

Second, our results place more emphasis on data challenges than previous studies of digital government have done (e.g., Gasco & Jimenez, 2011), suggesting that the nature of AI projects makes such factors particularly relevant (Desouza & Jacob, 2017; Kankanhalli et al., 2019; Krishnamurthy & Desouza, 2014; Sun & Medaglia, 2019).

The second research question aimed to identify the organizational routines that managers performed to overcome challenges. These organizational routines varied across councils and included working on-site, presenting the benefits of data sharing, reframing problems, designating joint appointments and boundary spanners, and connecting participants in the collaboration at all levels around project design and purpose. Generally speaking, our results show that public-sector organizations adopting AI need customizable, multistrategy approaches to shaping effective data sharing. Actors must adapt their organizational routines in response to feedback from various learning processes (Feldman & Pentland, 2003; Raub & Von Wittich, 2004). There is no one-size-fits-all approach to successful adoption. Further, our findings indicate that, although various different types of challenges exist, improving organizational processes is the key to overcoming them. In other words, organizations do not address privacy and security concerns by strengthening data privacy and security practices. Instead, they implement routines to increase trust and show the benefits of data sharing.

Some additional observations can be made regarding specific organizational routines. First, our findings indicate that building trust is essential for any practitioner–scholar collaboration to succeed (Kankanhalli et al., 2019; Mikhaylov et al., 2018; Yang & Maxwell, 2011). Our study is in line with previous findings in the literature on collaboration, which show that face-to-face interactions build trust, while the management activity of creating learning spaces can overcome institutional differences between participants in interorganizational projects (e.g., Vangen, 2017). Indeed, the social interactions and informal relationships that arose in these spaces between staff members from different organizations helped to align expectations and build trust, resulting in data sharing. As Willem and Buelens (2007) have shown, this routine was well suited to the bureaucratic nature of the two councils. If more formal mechanisms had been used, the process could have been much slower (Amayah, 2013). Interestingly, our findings do not resonate with recent studies that claim that technology can strengthen interorganizational collaboration (e.g., Livermore & Verbovaya, 2016). This may indicate that, in complex collaborations, such as AI projects, the face-to-face component becomes the key.

Second, various organizational routines such as clarifying roles and responsibilities within the collaboration (e.g., by appointing champions) and presenting the benefits of the project (based on data sharing) can succeed in overcoming distrust. Our results therefore confirm the success of several strategies addressed by previous studies on digital government, knowledge transfer, and collaboration (e.g., Amayah, 2013; Ansell & Gash, 2008; Cristofoli et al., 2015; Pardo et al., 2012; Vangen & Huxham, 2003; Willem & Buelens, 2007).

Finally, these findings indicate that engagement and participation across organizations are important for project success. In line with Saz-Carranza and Ospina (2010), our results also show that engagement should occur from the very beginning, as happened in Council 2, so that tensions can be addressed and basic agreements and procedures for project success agreed early on in the process (Ansell & Gash, 2008). We argue that the organizational routine of connecting members across an organization is even more important in AI projects, given the range of people involved in the collaboration (Sun & Medaglia, 2019). Although council leaders understood the strategic direction of the project, it was important for data analysts and academics to showcase their work using AI methods to fully communicate the potential of the project.

Conclusions

The goal of this study was to explore the challenges of interorganizational collaboration in the various stages of AI adoption as well as the use of organizational routines to address those challenges. Several findings are in line with current studies of AI in the public sector; they also confirm previous research on digital government, cross-sector collaboration, and knowledge transfer. The latter makes particular sense, given that AI projects are technological, collaborative, and based on heavy data sharing. There is no need to reinvent the wheel when assessing challenges and evaluating successful organizational routines.

However, we have also identified certain elements specific to these types of projects that should be addressed in future studies to gather more empirical evidence. First, although data challenges are not new in the literature, they seem particularly relevant in AI projects, given the risks associated with privacy and security and the volume of data sharing needed to create effective algorithms. Second, the combination of different types of challenges reinforces each other, causing resistance to data sharing and compromising the adoption of AI projects. More quantitative and qualitative research is needed to examine the specific interactions among challenges and their direct and indirect effect on the adoption of AI.

Third, despite various entangled challenges, successful organizational routines were mainly about improving organizational practices. We argue that, given the complexity of AI projects, future research should focus on ways of using specific organizational routines, which have received minimal attention in the AI adoption debate, to address organizational and other types of challenges. In particular, future studies could address discrepancies between top managers (the decision makers, who rarely understand AI specificities) and the implementers (AI experts and specialists).

Finally, this study is not without limitations. In particular, the research context (collaborative AI projects in England carried out by two councils and a university) and the qualitative approach used may limit the generalizability of our results. Nonetheless, we believe that this research has generated interesting results, adding stimulating ideas to the conversation about AI adoption in the public sector.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Author Biographies

Averill Campion is a PhD candidate at ESADE – Ramon Llull University (Barcelona, Spain). Her research interests include interorganizational collaborations, public private partnerships, and artificial intelligence implementation.

Mila Gasco-Hernandez is a Research Associate Professor at the Department of Public Administration and Policy and the Associate Research Director of the Center for Technology in Government, both at the University at Albany (Albany, NY, United States). She holds a MBA and a PhD in public policy. Her interests include digital government, open government, smart cities and communities, telework, artificial intelligence, and public innovation (particularly open innovation in the public sector).

Slava Jankin Mikhaylov is Professor of Data Science and Public Policy at the Hertie School. He is the Director of the Hertie School Data Science Lab. His research and teaching is primarily in the field of natural language processing and machine learning.

Marc Esteve is an Associate Professor of International Public Management at the School of Public Policy at the University College London. He is also the director of ESADEGov, at ESADE Business School. He received his PhD in Management Sciences from ESADE Business School-Ramon Llull University in 2012.

References

Agarwal

P. K.

(2018). Public administration challenges in the world of AI and bots. Public Administration Review, 78(6), 917–921.

Agranoff

(2006). Inside collaborative networks: Ten lessons for public managers [Special issue]. Public Administration Review, 66, 56–65.

Alberti

(2018, October 15–17). Artificial intelligence in the public sector: Opportunities and challenges. First Edition of the Seminars of the PhD School in Public, International and European Union Law of Università degli Studi di Milano, Gargnano, Italy.

Amayah

(2013). Determinants of knowledge sharing in a public-sector organization. Journal of Knowledge Management, 17(3), 454–471.

Andrews

(2019). Public administration, public leadership, and the construction of public value in the age of the algorithm and “big data.” Public Administration, 97(2), 296–310.

Ansell

Gash

(2008). Collaborative governance in theory and practice. Journal of Public Administration Research and Theory, 18(4), 543–571.

Bernard

H. R.

(2017). Research methods in anthropology: Qualitative and quantitative approaches. Rowman & Littlefield.

Bullock

(2019). Artificial intelligence, discretion, and bureaucracy. American Review of Public Administration, 49(7), 751–761.

Charmaz

(2014). Constructing grounded theory. Sage.

10.

Chen

Lee

(2017). Collaborative data networks for public service: Governance, management, and performance. Public Management Review, 20(5), 672–690.

11.

Christodoulou

Decker

Douka

A.-V.

Komopoulou

Peristeras

Sgagia

Tsarapatsanis

Vardouniotis

(2018, September 3–5). Da makes the public sector go round [Conference session]. International Conference on Electronic Government (EGOV) Krems, Austria.

12.

Creswell

J. W.

Vicki Plano

C. L.

(2017). Designing and conducting mixed methods research. SAGE Publications.

13.

Cristofoli

Maccio

Pedrazzi

(2015). Structure, mechanisms, and managers in successful networks. Public Management Review, 17(4), 489–516.

14.

Desouza

(2018). Delivering artificial intelligence in government: Challenges and opportunities. IBM Center for the Business of Government.

15.

Desouza

Jacob

(2017). Big data in the public sector: Lessons for practitioners and scholars. Administration & Society, 49(7), 1043–1064.

16.

Duan

Edwards

Dwivedi

(2019). Artificial intelligence for decision making in the era of big data—Evolution, challenges and research agenda. International Journal of Information Management, 48(October), 63–71.

17.

Dwivedi

Y. K.

Hughes

Ismagilova

Aarts

Coombs

Crick

Tom

Duan

Dwivedi

Rohita

Edwards

Eirug

Aled

Galanos

Vassilis

Ilavarasan

P. V.

Janssen

Jones

Kar

Kizgin

Hatice

Kronemann

Bianca

Lal

Lucini

… Williams

(2019). Artificial Intelligence (AI): Multidisciplinary perspectives on emerging challenges, opportunities, and agenda for research, practice and policy. International Journal of Information Management. Advance online publication. https://doi.org/10.1016/j.ijinfomgt.2019.08.002

18.

European Commission. (2019). A definition of AI: Main capabilities and scientific disciplines. High-Level Expert Group on Artificial Intelligence.

19.

Feldman

M. S.

Pentland

B. T.

(2003). Reconceptualizing organizational routines as a source of flexibility and change. Administrative Science Quarterly, 48, 94–118.

20.

Freeman

Sharkey

Cuella

M.-F.

(2020). Government by algorithm: Artificial intelligence in federal administrative agencies. Administrative Conference of the United States.

21.

Gasco

Jimenez

(2011, June 16–17). Interoperability in the justice field: Variables that affect implementation [Conference session]. Eleventh European Conference on E-Government—ECEG, University of Ljubljana, Ljubljana, Slovenia.

22.

Gil-Garcia

J. R.

& Sayogo

D. S.

(2016). Government inter-organizational information sharing initiatives: Understanding the main determinants of success. Government Information Quarterly, 33(3), 572–582.

23.

Gomes de Sousa

Pereira de Melo

E. R.

De Souza Bermejo

P. H.

Souza Farias

R. A.

Oliveira Gomes

(2019). How and where is artificial intelligence in the public sector going? A literature review and research agenda. Government Information Quarterly, 36(4), 101392.

24.

Halcomb

Davidson

(2006). Is verbatim transcription of interview data always necessary? Applied Nursing Research, 19(1), 38–42.

25.

Ishii

(2017). Comparative legal study on privacy and personal data protection for robots equipped with artificial intelligence: Looking at functional and technological aspects. AI & Society, 34, 509–533.

26.

Janssen

Kuk

(2016). The challenges and limits of big data algorithms in technocratic governance. Government Information Quarterly, 33(4), 371–377.

27.

Jordan

M. I.

(2019). Artificial intelligence—The revolution hasn’t happened yet. Harvard Data Science Review. Advance online publication. https://hdsr.mitpress.mit.edu/pub/wot7mkc1/release/9

28.

Kankanhalli

Charalabidis

Mellouli

(2019). IoT and AI for smart government: A research agenda. Government Information Quarterly, 36(2), 304–309.

29.

Kim

Crowston

(2012). Technology adoption and use theory review for studying scientists’ continued use of cyber-infrastructure. Proceedings of the American Society for Information Science and Technology, 48(1), 1–10.

30.

Korinek

Stiglitz

(2017). Artificial intelligence and its implications for income distribution and inequality (Working Paper 24174). National Bureau of Economic Research.

31.

Krishnamurthy

Desouza

(2014). Big data analytics: The case of the social security administration. Information Polity, 19(3), 165–178.

32.

Livermore

Verbovaya

(2016). Doing collaboration: How organizations use Facebook to foster collaboration. Human Service Organizations: Management, Leadership & Governance, 40(5), 553–571.

33.

Marshall

Rossman

G. B.

(2011). Designing qualitative research. Sage.

34.

Mehr

(2017). Artificial intelligence for citizen services and government. Harvard Kennedy School—Ash Center for Democratic Governance and Innovation.

35.

Mikhaylov

S. J.

Esteve

Campion

(2018). Artificial intelligence for the public sector: Opportunities and challenges of cross-sector collaboration. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 376(2128), 20170357.

36.

Ospina

Esteve

Seulki

(2018). Assessing qualitative studies in public management research. Public Administration Review, 78(4), 593–605.

37.

Pardo

Nam

Burke

(2012). E-government interoperability: Interaction of policy, management, and technology dimensions. Social Science Computer Review, 30(1), 7–23.

38.

Patton

(2002). Two decades of developments in qualitative inquiry: A personal, experiential perspective. Qualitative Social Work, 1(3), 261–283.

39.

Pentland

Feldman

M. S.

(2005). Organizational routines as a unit of analysis. Industrial and Corporate Change, 14(5), 793–815.

40.

Raub

Von Wittich

(2004). Implementing knowledge management: Three strategies for effective CKOs. European Management Journal, 22(6), 714–724.

41.

Risse

(2019). Human rights and artificial intelligence: An urgently needed agenda. Human Rights Quarterly, 41(1), 1–16.

42.

Saz-Carranza

Ospina

(2010). The behavioral dimension of governing interorganizational goal-directed networks: Managing the unity-diversity tension. Journal of Public Administration and Theory, 21(2), 327–365.

43.

Sun

T. Q.

& Medaglia

(2019). Mapping the challenges of artificial intelligence in the public sector: Evidence from public healthcare. Government Information Quarterly, 36(2), 368–383.

44.

Vangen

(2017). Culturally diverse collaborations: A focus on communication and shared understanding. Public Management Review, 19(3), 305–325.

45.

Vangen

Huxham

(2003). Nurturing collaborative relations: Building trust in interorganizational collaboration. The Journal of Applied Behavioral Science, 39(1), 5–31.

46.

Veale

Van Kleek

Binns

(2018, April 21−26). Fairness and accountability design needs for algorithmic support in high-stakes public sector decision-making [Conference session]. 2018 CHI Conference on Human Factors in Computing Systems (CHI’18), Montreal, Canada.

47.

Willem

Buelens

(2007). Knowledge sharing in public-sector organizations: The effect of organizational characteristics on interdepartmental knowledge sharing. Journal of Public Administration Research and Theory, 17(4), 581–606.

48.

Wirtz

Müller

(2019). An integrated artificial intelligence framework for public management. Public Management Review, 21(7), 1076–1100.

49.

Wirtz

Weyerer

Geyer

(2019). Artificial intelligence and the public sector—Applications and challenges. International Journal of Public Administration, 42(7), 596–615.

50.

Yang

T.-M.

Maxwell

(2011). Information-sharing in public organizations: A literature review of interpersonal, intra-organizational and inter-organizational success factors. Government Information Quarterly, 28(2), 164–175.

51.

Yin

R. K.

(2013). Case study research: Design and methods. Sage.